Rohan Soni

Case study · Personal LLM product · In progress

Redline: an LLM that has to show its source

Freelancers sign client agreements they can't easily read. Redline reads the agreement, explains it in plain English, and flags the risky clauses. The product only works if people can trust a flag, so every flag has to prove where it came from.

My role
Product owner: scope, trade-offs, success metric
Built with
Claude Code agents, working from my decisions
Stack
Next.js, TypeScript, Supabase, an LLM through OpenRouter
Status
Live prototype, no users yet. Name may change.

The problem

A freelancer gets a client agreement and a deadline. A lawyer costs more than the job is worth, so most people skim and sign. The clauses that hurt them, such as unlimited liability, late payment terms or broad IP assignment, are written to be skimmed past.

An LLM can read the agreement in seconds. The trouble is that an LLM can also invent a clause that isn't there, or quote one slightly wrong, and a freelancer has no way to tell. For this product, a confident wrong answer is worse than no answer at all.

Narrowing who it's for

The idea started as "contracts, leases and terms of service". Four research passes asked who has the pain, what goes wrong, what already exists, and who would pay. From those I narrowed it to freelancers reading a client's agreement, and dropped consumer terms of service because there's no counter-offer to make and existing tools already summarise the big services.

That research was desk research. Talking to freelancers directly is the next step, and it's the one I'd most want to do before calling the scope settled.

Decisions on paper before code

Before any application code existed, there was a PRD, a glossary and a written record (an ADR) for each product decision. I made the decisions; Claude Code agents drafted the documents from them, and I reviewed and edited them. There were 19 decision records before the first line of code, and 23 now.Redline repository, October 2026 The ones that shaped the product:

Grounding is enforced in code, not asked for in a prompt

Every flag carries the exact sentence it came from. Before a flag is displayed, code checks that sentence word for word against the reader's own document. If it doesn't match, the flag is never shown. A prompt instruction asks the model to behave. This check doesn't depend on the model behaving.

The model never writes its own hedge

The model returns how ambiguous a clause is and how harmful it could be as separate fields. The code decides whether to show "this could mean…" wording, and only when there really are two readings. Hedging stays consistent instead of depending on the model's mood.

The judge logs, it doesn't gate

A second LLM checks whether each flag matches the reader's own red lines. On the first live run it returned "doesn't fit" while its own reasoning argued the match held. If its verdict had been a gate, two correct flags would have disappeared without anyone noticing. So the judge's disagreements are logged for review, not used to block output.

Recall over precision, with a metric for silence

Missing a harmful clause costs a freelancer more than reading one extra flag, so flagging leans toward recall. A clean agreement is still a valid result. To catch a model that quietly stops flagging, I decided on a zero-flag-rate health metric, measured against a fixed baseline first and a rolling one later.

What I cut

Payments, sharing and OCR. OCR went because a citation is worthless when the text it points at was misread, and citations are the whole product.

The success metric is counter-offers sent, measured by copies of the drafted reply. Flags shown and documents read don't count. A freelancer who pushes back on a clause got value; one who only read a summary may not have.

Positioning

Redline works alongside a lawyer, never instead of one. That isn't only tone. The FTC's 2025 order against DoNotPay made "AI lawyer" claims a compliance line, and the product copy is written to stay on the right side of it.

Testing it like a sceptic

After the build I ran two sceptical-reader passes on the live app, testing it the way a hostile reviewer would. They found real problems. The worst: the app called a deliberately one-sided test agreement a clean read, because the model was scoring every planted clause below the flag threshold. It also once treated a recipe as a normal agreement.

A labelled set of agreements is what should catch that failure before release; the zero-flag rate is what watches for it after. For the second, Redline now refuses documents that aren't agreements. Severity calibration is the open problem I'm working on now, which is why this page says "in progress" and makes no accuracy claim.

How it was built

I made the product decisions and reviewed and edited the specs. Claude Code agents drafted the specs and wrote the code, working through the build tickets in order. After that I tested the live app as a sceptical reader, and a security review and fix branches followed. It's the way I work at my day job too: the judgment is mine and the agents do the drafting.

What I'd do next

  • Interview freelancers before adding anything else. The scope rests on desk research.
  • Fix severity calibration against a labelled set of agreements, then publish the eval results.
  • Measure the success metric with real users: how many counter-offers go out, split by soft and firm stance.