ConceptIntermediateShipping & Model Lifecycle / Prototyping with LLMs and rapid POCs / #7
What should you test with a prototype before writing the PRD?
The direct answer
Test whether the model can do the job at all, on real documents, not a clean sample. Then learn exactly how it gets things wrong, because that decides what the PRD is allowed to promise. Save the wait time and the nice-to-have polish for after, since neither one changes whether the tool actually works.
The ranking, by what breaks first if skipped
Prove the model can find what it's supposed to find at all, on real case files.Why: dependency. Every later test, and every section of the PRD, assumes the core task already works.
Learn exactly how it gets things wrong, not just how often.Why: reversibility. A privileged name missed inside a table costs a real, unrecoverable mistake once a document ships, and that decides whether the PRD can allow the model to act on its own anywhere near a table.
Pick the interaction pattern before it's a line of code.Why: this is the one decision hardest to walk back once engineering has built a pipeline around it.
Test how long someone actually waits, and what they do while waiting.Why: cheap to learn now, expensive to find out after launch that a six minute wait sends someone back to their inbox mid task.
Push the nice-to-have polish to the bottom of the list.Why: none of it changes whether the PRD's core scope is right, and it's the easiest thing to spend a week testing by mistake.
Re-run the same real documents whenever the model underneath changes.Why: the failure shape found under one version of the model won't necessarily hold under the next one.
How to answer this, stage by stage
Seven moves. The trap in this question is answering it with a list of things to click and test, when it's really asking whether you know which open question gets expensive the moment the PRD locks in an answer.
1
Ground it in one real product
Say it like this
"Let me make this real. Say a law firm called Caldwell and Rue is building Veilmark, a tool that reads a document and marks what has to be redacted before it goes to the other side of a case. Osian Vantrease is the product manager building it, and Verena Palfrey is the senior paralegal who'd actually use it. I'll answer against that."
Why this works
Grounds an abstract question about prototypes in one real product, so the ranking that follows isn't hypothetical.
2
Name your method before you use it
Say it like this
"I'd use ORDER here. Rank the open questions the prototype has to answer by what breaks first if you skip them, not just list everything you could possibly test before you write."
Why this works
Signals a method up front, so the answer doesn't sound like a checklist made up on the spot.
3
Say what the question is actually testing
Say it like this
"This isn't really asking me to list test cases. It's asking whether I know which unanswered question is cheap to get wrong right now, and which one gets expensive the moment the PRD writes it down as settled."
Why this works
Separates the real judgment call from a surface reading that treats the answer as a test plan.
4
Give the ranked answer straight
Say it like this
"Test whether the model can do the job at all, first, on real documents. Then learn exactly how it's wrong, because that decides what the PRD is allowed to promise. Then test how the wait feels. The nice-to-have polish questions, formatting, exports, that kind of thing, go dead last."
Why this works
This is deliverable 0, said out loud, in the order that actually matters.
5
Show what has to be true before anything else
Say it like this
"None of the rest of this matters until you know Veilmark can find a privileged name at all. Every section of the PRD, the workflow, the review step, the launch date, quietly assumes the core task works. Skip that test and you're writing a plan for a feature that might not exist yet."
Why this works
This is the dependency step. It shows the order isn't arbitrary, one thing genuinely has to be true before the next one is worth testing.
6
Back it with the real test, not a guess
Say it like this
"Here's what thirty real documents showed. Veilmark caught 48 of 50 privileged mentions sitting in plain paragraphs. It caught 18 of 30 sitting inside tables, the same kind of table a witness list always uses. That gap is what decides whether the PRD can say 'apply this on its own' anywhere, or has to say 'flag it and let a person confirm.'"
Why this works
A cheap test plus a real number is what makes the order defensible instead of just tidy.
7
Close on the rule, not the checklist
Say it like this
"So: can it do the job at all, first, because everything else assumes it can. Then how it's wrong, because that's the part you can't take back once a document has already been mailed. Then how long the wait feels. The nice-to-have questions go last, because none of them change whether the PRD's core scope is even right."
Why this works
Ends on the literal ranking the question asked for, defended instead of just listed.
Let's learn
Veilmark is a tool that reads a legal document and marks the parts that have to be blacked out before anyone outside the firm sees it: names, account numbers, anything a court would call privileged.
Every week, before Veilmark existed, the three paralegals at Caldwell and Rue spent about 14 hours between them doing this by hand. Page by page, circling what had to go, then covering it with a marker or a slow desktop tool nobody fully trusted alone.
Osian Vantrease, the product manager building Veilmark, could have written the PRD the ordinary way: describe the feature, say the model finds privileged content, hand the spec to engineering, and let the first real test happen in front of a client. Instead, before writing a single section of the PRD, he spent four days running the roughest version of Veilmark he could put together. One script. Thirty real documents from a closed, cleared-for-reuse matter. A spreadsheet to mark what it got right.
Knowledge spark: what makes something privileged?
A note counts as privileged when it was written between a lawyer and a client, in confidence, about the case. It has to come out before a document leaves the firm. Miss one, and the other side gets to read something they were never supposed to see, and there's no taking that back.
The extra mistakes were not the problem. Say that plainly, because it's the part most people skip past. Nobody expects a first prototype to catch everything. What decided the whole shape of the PRD wasn't how often Veilmark missed something. It was where.
Testing whether Veilmark can find anything at all comes before testing how it fails. Neither one comes before the PRD gets written.
In plain paragraphs, prose text, Veilmark found 48 of the 50 privileged mentions the paralegals had already caught by hand. Inside tables, the kind a witness list or an exhibit index always uses, it found 18 of 30.
Privileged mentions Veilmark caught in the first real-document test, prose vs. table
Prose textTable or exhibit list
Eighteen of thirty caught is not a rare miss. It's the shape of the mistake the PRD has to plan for, not the one the demo document ever showed.
We were not choosing whether the model makes mistakes. We were choosing whether the PRD would tell engineering to check.
Here's what that costs at its worst. A PRD that assumed one accuracy number everywhere would have let Veilmark redact a full production and send it, table names included, with nobody's eyes on the file. A privileged name buried in a witness list, missed, mailed to opposing counsel, isn't a bug report. It's a real, formal privilege waiver, the kind a firm has to disclose to a client and sometimes to a court.
The choice I would take back
Early on, the plan was to write the PRD off the demo Osian had already built: a clean, three-page NDA that ran perfectly every single time. It looked done. I would go back and swap that document for the messiest real production in the building before writing a single requirement, because the demo only ever proves the easy 96 percent.
What I would leave alone. Whether the redaction marks show up as solid black bars or a grey highlight in Veilmark's export is not worth a minute of testing time yet. A paralegal can live with either. Spend the four days on the part that decides whether a document can leave the building safely, not the part that decides how it looks doing it.
The lesson. A model that works on the document you're proud of only tells you that the document you're proud of works. The PRD has to plan around the part you haven't shown anyone yet, and you only find that part by testing something ugly on purpose.
Now here is the same thing as a story
The short version is above. Keep reading if you want to feel why the demo document was the wrong one to build a PRD on.
Osian Vantrease had built product features for six years, none of them anywhere near a courtroom. Legal was new. What he was good at, from the very first sprint, was picking one thing and proving it worked before he asked anyone to believe him.
Veilmark's first real version could do exactly one trick well: read a three-page mutual NDA and mark every name that had to come out. Osian ran it a hundred times. It never missed the client's name, never missed the date, never missed the signature block. He carried that document to every meeting he could find. Leadership liked it. Verena Palfrey, the senior paralegal at Caldwell and Rue, watched it run twice and said, quietly, that it looked better than anything she'd been handed to test in eleven years.
For a while, that one document was enough. Every demo used it. Every question got answered against it. Osian stopped pulling new documents to test with, because the one he had kept working.
Then, standing in the hallway after a demo, Verena asked a question that wasn't really a question at all. "Does it catch the names in the exhibit index too, or just in the letters?" She wasn't testing Osian. She genuinely wanted to know, because Caldwell and Rue's real productions were never three clean pages. They were three hundred, and most of that was tables: witness lists, exhibit indexes, signature logs.
Osian didn't know. Nobody had run it on a table.
One of these you can move by a week and nothing changes. The other one, once it ships, has already been read by the wrong person.
So instead of writing the PRD that week, the way the plan had said, Osian pulled thirty real documents from a closed matter and ran Veilmark against all of it, prose and tables both. It took an afternoon.
The prose held. Forty-eight of fifty. The tables didn't. Eighteen of thirty, which meant twelve names that should have come out would have gone through, invisible, inside a format nobody had thought to test, because the demo document never had one.
It was never really about the two missed pages. It was about which twelve names would have gone out the door before anyone knew to look.
A year earlier, in the meeting where the team picked which document to build the first demo around, someone had asked whether they should test on something messier, closer to a real production. The clean NDA was already working, and a messy file felt like borrowing trouble before they needed to. Nobody wrote down that this choice would still be the only test a PRD ever got written against.
I would go back and pick the ugly file. Not instead of the clean one, first. The NDA proves the model can do the job at all. The ugly one proves you know how it fails.
With that change, here's the replay. Same thirty documents, same afternoon, but now it happens before the PRD, not after engineering has already built a launch around "redact on its own, no review." Osian writes the PRD with a rule baked in: prose matches the model is sure about apply on their own, table matches always get flagged for a one-tap confirmation. Nine months later, when Caldwell and Rue runs an actual three-hundred-page production through it, Verena opens forty flagged items in about twelve minutes, and everything else has already gone.
One PRD assumed the demo was the whole product. The other assumed the demo was only the easy 96 percent, and built the harder part a place to get caught before a document ever left the building.
What I'd tell myself, back in that meeting: pick the file you're least proud of. The PRD only has to be honest about that one.
ORDER, spelled out before the PRD exists
BOUND would fit if the question were about sizing how many hours engineering needs for Veilmark. This sits earlier than that: which open question the prototype has to answer before the PRD can even be scoped. That's ORDER's job.
O, outcome. Every test in this list protects one thing: that the PRD gets written on real answers, not on assumptions that only turn out wrong once someone else has already built against them.
R, reversibility. The hardest thing to walk back is a core interaction pattern the PRD commits to before anyone's tested it: whether Veilmark acts on its own or waits for a person. Get that wrong, and it's not a bug fix once engineering has built the whole pipeline around "no review needed."
D, dependency. Nothing else is worth testing until you know Veilmark can do the core task at all. Every other section of the PRD, the workflow, the review step, the rollout plan, quietly assumes the answer is yes.
E, evidence. Cheap to check first: a handful of real documents run through the roughest version of the interaction you can build. Osian's version was one script and thirty real files. No interface, no launch plan, just: does it find the right things?
R, rank. Can it do the task at all, first, since everything else assumes it can. Then the shape of the common failure, prose versus table, since that decides what the PRD is allowed to promise. Then how it feels to wait for an answer. On Osian's rough build, a 20 page file came back in about 15 seconds. A real 200 page production took about 3 minutes, and by that point, watching Verena test it live, she had already checked her email once. Nice-to-have polish, export formatting, a shareable report, goes dead last, because none of it changes whether the first three questions come back right.
How long Veilmark's rough prototype took, by document length
The wait wasn't the failure. It just told Osian where a paralegal's attention leaves the task, which is the kind of thing you'd rather learn on a Tuesday afternoon than in a client meeting.
The check that keeps this ranking honest
Swap the outcome and the order should move. If a missed name in a table only ever cost a paralegal a second look before mailing, the failure-shape test could rank behind the wait-time test. It ranks second here because Caldwell and Rue would be carrying a real privilege waiver they'd have to report to a client, not a typo, if that table match had shipped.
Rank it again, where the miss reaches a field, not a filing
Elmsgate Growers Cooperative built Fieldscan: a tool that reads a photo an agronomist snaps in the field and names what's wrong with the crop, before anyone sprays anything.
O. Every version of Fieldscan's test order protects one thing: that a "spray this" call reflects what's actually wrong with the plant, not a guess from a photo that merely looks similar.
R. A wrong "it's just leaf curl" call, when it's actually blight, costs a field once it's acted on or ignored. There's no getting a growing season back once it's gone.
D. None of it matters until you know whether Fieldscan can name a disease at all from a real phone photo, taken in bad light, in an actual field, not a clean stock photo in a lab.
E. Cheap to check: run it against 20 real photos the co-op's agronomists already diagnosed last season, and count how many it gets right.
R. Same order: prove it can name a disease at all, first. Then learn its failure shape, whether it confuses two diseases that look alike but need opposite treatment, which is the costly kind of miss. Then test how long a photo takes to process out where the signal is weak. The shareable report PDF goes last.
Swap the trigger and it still runs
Caldwell and Rue doubles its caseload overnight. The order doesn't move. The failure-shape test matters more, not less, since more documents mean more chances for a table to slip past review.
Veilmark's underlying model gets noticeably better at reading tables. Doesn't reorder either. A better model still needs someone to confirm the one time it's wrong, or nobody can tell that time from the other twenty-nine.
Caldwell and Rue starts licensing Veilmark to a second firm. Doesn't reorder. The rule protects the same thing regardless of who's running it.
Where people run it wrong
Treating a clean demo as proof the whole product works, when a demo only ever proves the part you were brave enough to test.
Running the wait-time and polish tests first, because they're easier to set up than a real failure-shape test.
Testing once before launch and never re-running the same real documents after the model underneath quietly changes.
How to use it live
Say the outcome out loud before naming a single test. "Every test I run before this PRD protects one thing, that a person can trust what the model tells them it found." Then rank from there. Naming the outcome first turns a list of test ideas into an argument you can defend.
Flashcards (click a card to flip it)
1 · THE FRAMEWORK
Which framework fits "what to test before the PRD," and why not SPARK?
Tap to flip
ANSWER
ORDER, for ranking which open question the prototype has to answer first. SPARK is for designing the feature itself, not for ranking which test to run before you've even scoped it.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Osian Vantrease, the product manager building Veilmark, an AI redaction tool for Caldwell and Rue's paralegal team.
3 · THE HABIT
What did Osian stop doing because the demo kept working?
Tap to flip
ANSWER
He stopped pulling new documents to test with. Every demo, and every question about the tool, ran against the same clean three-page NDA that always worked.
4 · THE DEPENDENCY
What has to be tested before anything else about Veilmark?
Tap to flip
ANSWER
Whether it can find a privileged name at all, on a real document. Every other section of the PRD quietly assumes the core task already works.
5 · THE OLD DECISION
What decision would you take back, and why did it make sense at the time?
Tap to flip
ANSWER
Building the first demo around a clean, easy document instead of a messy real one. It made sense because the clean document proved the model could do the job at all. Nobody planned to test anything else before the PRD got written.
6 · THE NUMBER
Veilmark caught ___ of 50 privileged mentions in plain paragraphs, and ___ of 30 inside tables.
Tap to flip
ANSWER
48 of 50 in prose, 18 of 30 in tables. That gap is what decided whether the PRD could say "acts on its own" anywhere near a table.
7 · THE REPLAY
Same thirty documents, tested before the PRD instead of after. What changes?
Tap to flip
ANSWER
The PRD ships with a rule: prose matches the model is sure about apply on their own, table matches get flagged. Nine months later, on a real 300 page production, Verena opens 40 flagged items in about 12 minutes instead of an unnoticed leak.
8 · THE TRANSFER
Section 4 runs ORDER again on a different product. Which one, and what plays the role of the prose-versus-table gap there?
Tap to flip
ANSWER
Elmsgate Growers Cooperative's Fieldscan. The equivalent gap is confusing two diseases that look alike in a photo but need opposite treatment, not just failing to spot a disease at all.
Check yourself Score: 0 / 0
Fill in the blank
1. Veilmark's rough prototype caught ______ of 50 privileged mentions in plain paragraphs, and ______ of 30 inside tables.
Show hint
It's the number the whole ranking argument turns on.
Show answer
48 of 50, and 18 of 30. That gap is why the PRD couldn't treat every match the same way.
Multiple choice
2. Which question does this answer say has to be tested first, before anything else about Veilmark?
A. Whether the export format looks clean
B. Whether the model can find a privileged name at all, on a real document
C. How long a big production takes to process
D. Whether paralegals like the interface
Show hint
Ask what every other section of the PRD quietly assumes is already true.
Show answer
B. Every later test, and every PRD section, assumes the core task already works. Test that first, or the rest is a plan for a feature that might not exist.
True or false
3. True or false: since Veilmark rarely misses a table entry, the gap between prose and table accuracy didn't need its own line in the PRD. Say why.
True
False
Show hint
Think about what happens when something rare also can't be undone. Now check how rare 18 of 30 actually is.
Show answer
False. Eighteen of thirty caught in tables isn't rare at all, and even a rare miss that ships to opposing counsel is a real privilege waiver. Common and unrecoverable together is exactly what needs its own written rule.
Multiple choice
4. What does the dependency step (D) argue in this answer's ORDER?
A. Engineering should decide the interaction pattern, since they'll build it anyway
B. The wait-time test should happen before the accuracy test
C. Nothing else is worth testing until you know the model can do the core task at all
D. The PRD should be written before any prototype exists
Show hint
Ask what has to be true before anything else is even worth testing.
Show answer
C. Without a working core task, there's nothing for the other tests, or the PRD, to be built around.
Short answer, apply it yourself
5. Pick an AI feature you use or are building. What's one thing you'd test on the ugliest real input you have, before writing anything that assumes it works?
Show hint
Look for the input closest to your actual mess, not your best demo.
Show answer
Model answer: "A voice memo transcription app, tested against a recording with background noise and two people talking over each other, not the quiet solo recording used in every product demo."
Short answer, the number question
6. If Veilmark had caught 28 of 30 table mentions instead of 18, would the interaction pattern still need to flag table matches for confirmation? Say what changes and what doesn't.
Show hint
Reversibility is about whether a miss can be undone, not about how rare it is.
Show answer
Model answer: "What changes: how often a paralegal has to stop and confirm something, so the wait gets shorter. What doesn't change: one missed name in a table is still a real privilege waiver once it ships, so a confirmation step still belongs in the PRD, just a lighter one."
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
