ConceptAdvancedQuality, Cost & Token Economics / Eval design for product teams / #23
How do you know when your eval suite is good enough?
An eval suite is good enough the moment it stops finding new kinds of trouble, not the moment it finds zero trouble on a set you built to be easy.
The direct answer
An eval suite is good enough once it stops turning up new severity one failure categories for two review cycles running, not once it hits a fixed test count or a hundred percent pass rate. Past that point, keep the one guardrail that never turns off, in this case a check that verifies every citation against real case law before a brief leaves the building, and ship. That guardrail catches whatever the suite never thought to test for, so there is no real safety left to buy by testing longer.
Do this, in order
Ship once new severity one failures stop showing up for two review cycles running.Why: that is the flip from a suite still teaching you something to a suite finding the same absorbed mistakes dressed differently, and that is the moment more testing stops earning its keep.
Never trade away the one guardrail that runs forever: verify every citation against real case law before it leaves the building.Why: this is what catches the failure category testing did not think to write down yet, since a lab can prove a plateau, never a zero.
Track new failure categories per cycle, not pass rate or test count.Why: a pass rate can sit at a hundred percent on a set the team already knows how to pass, and a test count just rewards writing more of the same prompt.
Reset the plateau clock on every model swap or prompt change.Why: a new model version can reopen a category the old one had already closed, so last month's good enough does not carry forward on its own.
Watch the cost of delay in the same unit as the cost of the risk.Why: hours lost hand drafting past the plateau are just as real as hours burned cleaning up one incident, and counting only one side is how testing runs long for no extra safety.
Leave the two clean cycle rule alone, even under pressure to ship faster.Why: one quiet cycle can be luck. Two in a row is close enough to a pattern to trust, and this is the one place worth resisting a shortcut.
How to answer this, stage by stage
Nobody is grading whether you can describe an eval suite. They are grading whether you can tell the difference between a tool that has stopped teaching you anything new and a tool you are just tired of testing.
1
Scope it to one real product before answering in the abstract
Say it like this
"Let's ground this in one product. Brieftrace is a tool inside Roux and Sarsfield, a litigation firm. It reads a case file and drafts a first pass legal brief, citations and structure already in, for an associate to edit before it goes near a judge. Farrin Delaney runs eval on it."
Why this works
An abstract "how do you know a suite is good" answer turns into a definition of testing fast. One product makes the bar a real decision instead of a general rule.
2
Say your structure out loud before diving in
Say it like this
"I'll answer this in two parts. First, the actual signal that says the suite is ready, since it isn't a pass rate or a test count. Then what protects the product after that point, because no suite can honestly promise zero forever."
Why this works
Tells the interviewer you have a plan, not just an opinion about how much testing feels like enough.
3
Reframe the question before answering it
Say it like this
"This isn't really asking me to describe an eval suite. It's asking how you tell the difference between a tool that has stopped teaching you anything new, and a tool you're just tired of testing."
Why this works
Stops you from listing eval best practices and never actually answering what "good enough" is measured against.
4
Give the one decision, plainly
Say it like this
"An eval suite is good enough once it stops turning up new severity one failure categories for two review cycles running, not once it hits a pass rate or a fixed test count. Past that point, keep one permanent guardrail running on every output forever, and ship, because that guardrail catches what the suite never thought to test."
Why this works
This is the actual answer to the question, stated once, cleanly, before any story.
5
Prove it with the failure, cut to four sentences
Say it like this
"Here's what happens without that. Roux and Sarsfield set Brieftrace's original bar at zero failures found, full stop, so the team kept red teaming it for sixteen extra weeks after the failure count had already sat at zero for two cycles running. Associates kept hand drafting the whole time, since nobody trusted a tool still 'in testing.' That's about twenty four hundred hours of manual drafting the firm never got back, for a plateau that had already proven itself flat eight weeks earlier."
Why this works
Shows the real cost of the wrong bar, not just the mechanics of the right one.
6
Say what you'd measure going forward
Say it like this
"I'd keep tracking new severity one categories per cycle after launch too, not just before, since a model swap can reopen a category the old version had already closed. And I'd watch how often the citation check actually catches something live, because that number tells you whether the guardrail is doing real work or just sitting there."
Why this works
Shows you're thinking past launch day, and that the plateau isn't a badge you earn once.
7
Close on the decision, not the story
Say it like this
"So: ship at the plateau, not at zero. And keep the one check that never turns off, because that's what catches the thing the plateau couldn't see coming."
Why this works
Ending on the rule, not the anecdote, is what makes this sound like a method you'd actually reuse.
Let's learn
Picture a tool that reads a case file, the pleadings, the exhibits, the law that applies, and writes a first version of a legal brief: the argument already structured, the citations already dropped in. An associate at a litigation firm still edits it before it goes anywhere near a judge, but the blank page is gone.
Before a tool like this, a first pass brief took an associate about six hours: reading the file, finding the right cases, building the argument from nothing.
With the tool, a first draft is ready in about twenty five minutes. So the team does the responsible thing and tests it hard before trusting it with a real filing.
We were not testing whether the tool could write a good brief. We were testing whether we would ever feel safe enough to say we could stop.
Here is the turn. The extra testing is not the problem. What the team does next is the problem: they keep testing long after the tests stop finding anything new, because stopping still feels like the riskier move than continuing.
At its worst, a suite like this never ships. Every week it sits in testing is a week every associate keeps drafting by hand, for exactly the hours the tool was built to give back.
One of these costs is cheap and everyone watches it happen. The other one is quiet, and nobody is watching for it on purpose.
New severity one failure categories found, per weekly eval cycle
New severity one categories found that cycle
The count drops fast at first, seven new categories the first week, then five, three, two, one. By cycle 7 it hits zero, and cycle 8 confirms it: two cycles running with nothing new. That is the ship signal. The line stayed flat at zero for sixteen more weeks after that before the firm actually let Brieftrace ship, chasing a guarantee the flat line had already given them in week eight.
The decision that mattered
The firm set its original launch bar at zero failures found in testing, full stop. That made sense the week Brieftrace first shipped a draft, when nobody had any track record to lean on. It stopped making sense the week the failure count had already sat at zero for two cycles running, and testing kept going anyway because nobody had agreed on what zero, held steady, was supposed to mean.
What I'd leave alone: the two consecutive zero cycle rule itself. Never ship on one clean cycle alone, since one quiet week can be luck and not a real plateau. That one piece of caution is worth keeping even when everything else about the process gets faster.
The lesson: a suite that keeps finding new kinds of trouble is not done. A suite that has stopped finding anything new, for more than one cycle running, is not lucky. It is finished, and the only thing more testing buys you after that is a smaller number on a graph nobody outside the team will ever see.
Now here is the same thing as a story
Read the short version above for the two minute answer. Read this one for the meeting where "zero failures, full stop" sounded like the only responsible bar in the room.
Farrin Delaney spent four years as a litigation associate before she moved into product: six hours a brief, chasing citations by hand, building an argument from a blank page at eleven at night more times than she liked to count.
Roux and Sarsfield brought her on to run eval when they built Brieftrace, the tool that would draft that first pass in her place. The good months were good. Every Monday, her team ran fifty adversarial prompts against it, obscure jurisdictions, contradictory precedent, cases picked to trip up a model that is only as careful as its last correction. Early on, Brieftrace found new ways to fail almost every week: a citation to a case that did not exist, a real case cited for the opposite of what it actually held, a brief formatted for the wrong court entirely.
Then the habit changed, in three small steps. First, the weekly failure count started dropping, five new problems, then three, then one. Second, the team stopped being surprised by what it found, since the misses were mostly smaller versions of the same few mistakes. Third, when two cycles running turned up nothing new at all, nobody in the room actually said the word "ship."
The trigger was small. A new associate, sitting in on his first review meeting, asked why the team was still running the same fifty prompts against a tool that had not failed in eight weeks.
We did not spend sixteen weeks protecting the firm from a bad citation. We spent it protecting ourselves from having to say the word ship.
Farrin didn't have a clean answer for him. The honest one was that the original bar, set the day Brieftrace first drafted anything, had been zero failures found, full stop, and nobody had ever agreed on what would replace it once zero actually happened. So testing kept running. Sixteen more weeks of it, same fifty prompts, same zero result, while every associate in the building kept drafting briefs by hand at six hours a file.
It was never really about the failure count. Farrin's team had a number, zero, and they knew exactly what it meant the day they wrote it down. What they didn't have was a rule for when zero, held steady, counted as an answer instead of a habit.
The decision that opened the door went back to Brieftrace's first planning meeting, months earlier. Someone asked what "done testing" would look like, and the room agreed on zero failures found, because it was the only number that felt safe to say out loud to a room full of litigators. Nobody asked what zero would mean once it had already been true for two months running.
Run the same sixteen weeks again with one change: a rule, written down before Brieftrace ever drafted a word, that two cycles running with zero new severity one categories means ship, backed by a citation check that runs on every brief forever. Under that rule, Brieftrace ships in week eight instead of week twenty four. The firm banks about twenty four hundred hours of associate time across those sixteen weeks, hours that would otherwise have gone into hand drafting a tool that was already sitting there, tested and ready.
One design trusted a number nobody had written a stopping rule for. The other design trusted a rule that knew its own ending in advance.
What I'd tell Farrin's team, back in that first planning meeting: the day you write down a target with no stopping rule attached, you haven't set a bar. You've postponed a decision and called it caution. Nobody in that room meant to do that. That is exactly how it happens.
PICK, four moves for calling a suite done
Not a checklist for stalling. Four moves that build toward the one line that actually decides it: which kind of wrong you are willing to risk, and which you are not.
PPosition. Your pick, in one sentence, before any reasoning.
Ship Brieftrace once the eval suite stops finding new severity one failure categories for two cycles running, backed by a permanent citation check. Not once it hits a pass rate number or a fixed test count.
Say the pick first. "It depends" is not a position, and this question is testing whether you can commit to one.
IImpact. Who feels each kind of error, and in what unit.
Testing too long costs every associate real hours, six a brief, hand drafting what Brieftrace could already do in twenty five minutes. Testing too little risks a hallucinated citation reaching a judge, in a real filing, with the associate's name and the firm's name both on it.
Both sides get named in real units, hours on one side, a filed document on the other, not "risk" left as an abstract word.
CCost asymmetry. Which error is cheap and visible, which is hidden and expensive.
An extra week of testing is cheap, bounded, and everyone in the building can see it happening. A hallucinated citation that reaches a filed brief is hidden, since nobody flags it the moment it ships, and expensive in a currency hours can't fully measure: a court's trust, a bar complaint, a client walking. Optimize against the second one, with a guardrail that runs forever, not against padding the first one past the point it teaches you anything.
This is the step the whole pick turns on. Name which side is hidden, or the tradeoff isn't real yet.
Cost, by the numbers: hours spent, same unit
Hours lost to needless testing delayHours to clean up one filed bad citation
The delay actually costs more raw hours, twenty four hundred against about a thousand fifty a peer firm reported spending on cleanup, client notice, and a bar inquiry after one bad citation reached a judge. That is the point: past the plateau, more testing is not a free extra layer of safety. It is a cost of its own, and one that can outweigh the risk it claims to be guarding against.
KKill criteria. What evidence would flip the pick.
If a model swap or a prompt change makes new severity one categories start showing up again, the plateau is gone, and so is the ship decision, back into active testing until it flattens a second time.
This is what separates a confident pick from a stubborn one: a real condition that would change your mind, stated in advance.
Three things worth stating directly, since this is where the real judgment sits. The rejected alternative was a fixed target: either a hundred percent pass rate on the golden set, or a set count like five hundred test prompts. It lost because neither one tracks whether the team is still finding new kinds of trouble. A pass rate can sit at a hundred percent on a set the team already knows how to pass, and a fixed count just rewards writing more near duplicate prompts to hit the number. The AI specific failure worth naming by name is a hallucinated citation, the model inventing a case that does not exist, or describing a real case's holding as the opposite of what it actually says, confidently, in fluent legal language that reads exactly like a citation that checked out. The guardrail is an automatic check that verifies every citation against real case law before a brief leaves the building, and it runs on every single brief forever, no matter how clean the eval numbers get. And the trade being accepted on purpose is real: shipping at the plateau instead of chasing zero forever means trusting production monitoring, and one guardrail, to catch a category testing never got around to writing a prompt for. That is a real risk, traded against real hours saved every week the tool is live instead of sitting in testing.
The four, in one line each: P: ship at the two clean cycle plateau, not at zero. I: testing too long costs hours everyone can see, testing too little risks a filing nobody flags in time. C: the hidden and expensive error is the one that reaches a judge, optimize against that one. K: a new failure category reopens the plateau, no matter how good last month's numbers looked.
Same four moves, a loading dock instead of a courtroom
Not every hidden and expensive error is a legal one. Sometimes it is an environmental one, caught the same way: months late, by someone outside the building.
Emberrow Sanitation runs Manifestly, a tool that reads a load's waste stream and drafts the hazardous waste manifest a driver signs before hauling it off site. Nikoletta Barsov runs quality on it.
P, position: ship Manifestly once new severity one miscoding categories stop appearing for two review cycles running, the same rule as Brieftrace, backed by a permanent check: every code Manifestly drafts gets cross checked against the EPA's own hazardous waste code list before a driver ever signs.
I, impact: testing too long costs drivers about forty minutes a load, hand filling a manifest Manifestly could draft in under a minute. Testing too little risks a load miscoded as non hazardous, riding along in a truck with drivers, insurers, and a state agency all trusting a piece of paper that is quietly wrong.
C, cost asymmetry: an extra review cycle is cheap, and everyone on the quality team sees it happen that week. A miscoded manifest is hidden, since nobody double checks a routine load by hand anymore, and expensive in a way that shows up months later: an audit, a spill investigation, a permit review, none of it traceable back to the moment the code went wrong until someone goes looking for it.
There is no box between "code drafted" and "ships out." That gap is where a wrong code rides along unnoticed for months.
K, kill criteria: if Manifestly's model changes and even one new miscoding category shows up, the plateau resets, back into active testing until it flattens again. Same rule as the courtroom, just applied to a truck.
Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to the line: ship at the plateau, two clean cycles, backed by a check that never turns off.
Cost: there's no budget this quarter for weekly red team cycles. Run a smaller adversarial set, fifteen prompts instead of fifty, but keep the two clean cycle rule and the permanent guardrail. The size of the test can shrink. The discipline of the stopping rule can't.
The model got better, for real: say the new model is measurably more accurate on average. That is exactly when the plateau rule matters most, since a stronger model's mistakes get quieter and harder to catch by eye, and a team that trusts the average score instead of the discovery curve will miss exactly the failure a smarter model is good at hiding.
Where people run it wrong.
They treat a single clean cycle as proof, when one quiet week can be luck instead of a real plateau.
They let the same team that wrote the prompts also grade them, so the suite only tests for what its own builders already expected to fail at.
They keep testing past the plateau because stopping feels riskier than continuing, even once continuing stopped teaching anyone anything.
How to use it live. Say two numbers out loud before answering: how many cycles has it been since anything new turned up, and what backstop is running once you ship anyway. That buys a beat of thinking time, and it turns the rest of the answer into naming the one check that never turns off.
Flashcards (tap any card to flip it)
1 · THE FRAMEWORK
What framework is this, and what's its one job?
Tap to flip
ANSWER
PICK: commit to a position, then show the asymmetry. Built for tradeoff questions, when two kinds of error cost genuinely different things.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Farrin Delaney, product manager running eval on Brieftrace, the AI drafting assistant at Roux and Sarsfield. Drafted briefs by hand for four years before moving into product.
3 · THE HABIT
What did the team keep doing well past the point it was teaching them anything?
Tap to flip
ANSWER
Running the same weekly red team cycle for sixteen more weeks after the new failure count had already flattened at zero for two cycles running.
4 · THE SIGNAL
What's the two setting signal this answer turns on?
Tap to flip
ANSWER
Still finding new severity one failure categories each cycle, versus finding none for two cycles running. That second setting is the ship signal, not a pass rate or a test count.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Setting the launch bar at zero failures found in testing, full stop, instead of a flattened failure discovery rate plus a permanent citation verification guardrail.
6 · THE NUMBER
Fill in the blank: the firm burned about ___ hours drafting briefs by hand during the ___ extra weeks it delayed past the point new failures had stopped appearing.
Tap to flip
ANSWER
About 2,400 hours, over 16 extra weeks. That's more raw hours than the roughly 1,050 hours a peer firm reported spending cleaning up one filed bad citation.
7 · THE REPLAY
Same eval program, ship at the plateau instead of chasing zero forever, what changes?
Tap to flip
ANSWER
Brieftrace ships in week 8, the first point two clean cycles confirm the plateau, instead of week 24. The firm banks about 2,400 hours of associate time it would otherwise have burned drafting by hand.
8 · CROSS PRODUCT TRANSFER
Section 4 answers this same question again for a different product. Which product, and what's the hidden expensive error this time?
Tap to flip
ANSWER
Manifestly, a hazardous waste manifest drafting tool at Emberrow Sanitation. The hidden expensive error is a waste stream miscoded as non hazardous, caught only by an audit or a spill, months later.
Check yourself Score: 0 / 0
Fill in the blank
1. Fill in the blank: Brieftrace's eval suite counted as good enough once new severity one failure categories stayed at ___ for ___ cycles running.
Show hint
Look at the P step in the framework recap, and the line chart in Section 1.
Show answer
Zero, for two cycles. One clean cycle can be luck. Two in a row is the point the answer treats as a real plateau instead of a lucky week.
Multiple choice
2. Which signal actually told the team Brieftrace was ready to ship?
A. A hundred percent pass rate on the golden set.
B. Five hundred test prompts run against it.
C. Two review cycles running with zero new severity one failure categories.
D. One clean week plus a partner's sign off.
Show hint
Think about which of these actually tracks whether the team is still discovering new kinds of trouble.
Show answer
C. A pass rate or a test count can both be hit without ever proving the team stopped finding new problems. Two clean cycles running is the only signal that tracks discovery, not volume.
True or false
3. True or false: once the eval suite stopped finding new failures, the automatic citation verification check could be turned off to save time.
True
False
Show hint
Check the priority list's second item, and the "three things worth stating directly" paragraph in the framework recap.
Show answer
False. The guardrail never turns off, no matter how clean the eval numbers look, since it exists specifically to catch whatever the suite never thought to test for.
Short answer, name the reversal
4. What launch bar did Roux and Sarsfield originally set for Brieftrace, and why did it make sense at the time?
Show hint
Look at the block key box titled "The decision that mattered" in Section 1.
Show answer
Model answer: Zero failures found, full stop. It made sense the week Brieftrace first shipped a draft, when nobody had any track record to trust yet. It stopped making sense once the count had already sat at zero for two cycles running, because nobody had agreed what zero, held steady, was supposed to mean.
Multiple choice
5. The firm tested 16 extra weeks past cycle 8 with zero new failures the whole time, burning about 2,400 hours of manual drafting a shipped tool would have saved. What does that tell you about the bar they were using?
A. The extra weeks proved the product was safer than it would have been at cycle 8.
B. The bar was no longer about safety, since the discovery curve had already flattened for weeks.
C. They should have demanded an even higher pass rate before shipping.
D. Nothing. More testing time is always free.
Show hint
Look at the cost by the numbers chart, and what it says about testing past the plateau.
Show answer
B. Past the plateau, extra testing time is not a free safety margin. It is a real cost, and in this case it ran higher than the incident it was meant to guard against.
Short answer, apply it yourself
6. Pick an AI tool you use yourself that produces something risky if it's wrong. What would "the eval suite is good enough" look like for that tool, in your own words?
Show hint
Think about what a flattening discovery curve would look like for that tool, not a fixed score.
Show answer
Model answer: An AI assistant that drafts replies to work emails. Good enough once red teaming stops turning up new categories of wrong tone or wrong fact for two review rounds running, with a standing rule that it always shows the source it pulled a fact from, so a wrong one is still catchable after it ships.
Before you close the answer
Why this works
Tests whether you treat "good enough" as a fixed target, a pass rate, a test count, or as a discovery rate signal with a permanent backstop behind it. And whether you can name the cost of testing forever, not just the cost of shipping too soon. Most candidates only describe the second one.
Follow-up traps
"What if a whole new failure category shows up the week after you ship?" Response: that's exactly what the citation check is for. It doesn't depend on eval having thought of the category first, and it resets the plateau clock so eval goes and finds out why the model changed.
"Couldn't they have shipped at cycle 6, since the count was already down to one?" Response: no, one cycle at a low count can still be luck. Two clean cycles in a row is what tells you the curve actually flattened, not just dipped for a week.
If pressed
The citation check doesn't re-run the model at all. It's a separate, deterministic lookup against a real case law database that confirms a cited case exists and says what the brief claims it says, or flags it for a human before the brief goes out. A deterministic check bolted onto a probabilistic model is why it can run on every single brief forever without the cost scaling the way a full eval cycle does.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
