ConceptAdvancedEval-Driven Specification / Acceptance criteria for non-deterministic output / #17
How do you avoid acceptance criteria that are technically met but produce a bad product?
The direct answer
Don't let one downstream person's sign-off be the whole test. Split the bar in two: routine output can pass on a quick approval, but any case with real stakes has to clear an independent check against the actual source before it counts as met. Sign-off measures whether someone noticed a problem. It never measures whether the record is true.
Do this, in order
Split the bar: routine cases pass on sign-off, high-stakes cases need an independent check against the source.Why: one blended bar lets a badly-served slice hide inside a healthy-looking average.
Recut the sign-off numbers by the kind of case before you trust the aggregate.Why: the same approval rate can mean two very different accuracy rates underneath it.
Rule out that the sign-off metric itself is broken before you blame the model.Why: a fast approval time can be a real, honest number, not a tracking bug. Check which one it is first.
Name the specific way the criterion can be gamed, not a vague worry.Why: "approved" can mean the record is right, or that the reviewer had 47 seconds. Only a named mechanism tells you which.
Run the independent check on the highest-risk slice before general rollout, not after something goes wrong.Why: the gap is usually visible weeks earlier, in small signals nobody was watching yet.
Leave the fast approval path alone for the cases that are already accurate.Why: adding a heavier check everywhere slows down the majority of cases to fix a problem that lives in a minority of them.
How to answer this, stage by stage
Seven moves. This is a diagnosis wearing a definitions question's clothes, so most of the work is finding out what the criterion actually measures before you touch the fix.
1
Scope it to one criterion, one product
Say it like this
"Let me put a real shape on this. Say we build a tool that listens to a nonprofit board's meeting and writes the official minutes: who was there, what got voted on, who said yes and no. The launch bar the team wrote was, 'the chair approves the draft without asking for a correction, in at least 90 percent of meetings.' We hit 93 percent in month one."
Why this works
A vague question about "bad criteria" invites a vague answer. Naming the actual bar gives every later step something concrete to check against.
2
Say what the bar actually measures, out loud
Say it like this
"Before I trust that 93 percent, I want to be honest about what it counts. It doesn't count whether the minutes are right. It counts whether a tired chair, skimming an email at the end of a long day, happened to notice something wrong. Those are two different facts, and the bar only checks the second one."
Why this works
This is the whole answer in one breath. Separating "passed the check" from "is actually correct" is the move that finds the gap everyone else walks past.
3
Give the one decision up front
Say it like this
"So here's what I'd do. Keep the chair's quick approval for meetings where nothing was contested. But for any meeting with a recorded 'no,' an abstention, or a raised objection, add a second, independent check: someone compares the draft against the actual recording before it's allowed to count as passed. That part isn't optional, and it doesn't wait for the chair to notice."
Why this works
This matches deliverable zero. Saying the concrete rule before the story stops the interviewer from having to guess where the answer is going.
4
Rule out a broken measurement before you blame the model
Say it like this
"Before I go further, I'd check my own number. I pulled the timestamps: median time between the minutes email landing and the chair clicking approve is 47 seconds, across 220 real approvals, and it checks out against an independent log, so it's not a tracking bug. That number is real. It's just measuring how fast a tired person clicks a button, not whether the record is true."
Why this works
Skipping this is the most common mistake. Blame the model before ruling out your own instrumentation, and you chase the wrong fix for weeks.
5
Show the exact case where the bar passed and the product still failed
Say it like this
"Here's the case that made it real. A board votes 5 to 2 to move $40,000 out of reserves. The two 'no's never say the exact words 'I vote no,' they say 'I'm not comfortable with that,' so the tool logs it as unanimous. The chair approves it in under a minute. Three months later, a funder's compliance review asks the two dissenting members why they didn't object on the record. They did. The record just doesn't show it."
Why this works
One specific failure is worth more than a list of worries. It's the case the interviewer will remember, and it proves the abstract gap costs something real.
6
Name the check that would have caught it
Say it like this
"The check I'd run before shipping: pull every meeting with any recorded pushback and compare the draft, line by line, against the recording, whether or not the chair approved it. Run that on our own numbers and it finds a real, uncaught error in 14 of 36 contested meetings, 39 percent, against 2 of 104 routine ones. That's the number that should have gated the launch, not the chair's click."
Why this works
This is the strongest move in TRACE. It turns "I have a bad feeling about this" into a number nobody in the room can wave away.
7
Close on the line that ties the fix to what ships next
Say it like this
"So the 90 percent bar wasn't wrong, it was just pointed at the wrong question. It tells you whether the chair noticed a problem. It never told you whether the record was true. I'd keep it for the easy cases, and add a real check exactly where the stakes are highest."
Why this works
It ends on a decision, not a warning. It gives the interviewer something they can picture actually shipping.
If you remember one thing
Stage 4 and stage 6 are what's being graded. Rule out your own measurement first. Then run one check that compares the output straight against the source, on the cases most likely to be wrong. A bar that only checks whether someone downstream complained will always look healthy, right up until it isn't.
Let's learn
The tool listens in on a nonprofit board's meeting and writes up the official minutes afterward: who was there, what got voted on, who said yes and who said no, and what each person promised to do next.
Halyna Kovalenko has been the product manager for this tool, called BoardScribe, for ten months. Before it existed, a volunteer secretary took handwritten notes during the call, then spent two to three hours over the following week turning them into an official record, often finishing after the next meeting had already started. Now a draft lands in the chair's inbox about six minutes after the call ends. She reads it, clicks approve, and it becomes the file the nonprofit keeps for its own records and hands to its auditors.
Ten months ago, when BoardScribe was still a pilot running for three friendly boards, Halyna wrote the launch bar herself: the chair approves the draft without asking for a correction, in at least 90 percent of meetings. Those three boards mostly voted on routine things, budgets everyone had already agreed on, dates, reports. Nobody was fighting about anything, so a chair's quick nod was a fine stand-in for "this is right." The bar shipped, and month one landed at 93 percent.
What the bar actually measures: not the record, the chair's attention span
Here is the turn. The extra words the tool gets wrong are not the problem. Ninety-eight words out of a hundred come through exactly right. The problem is which two it gets wrong, and what the chair does about it. She doesn't reread the whole meeting against the draft. She skims it, and if nothing jumps out, she approves it.
Three months after launch, a board votes 5 to 2 to move $40,000 out of reserves to cover a shortfall. The two dissenting members never say the ritual words "I vote no." They say, "I'm not comfortable with that," and, "I'd rather we didn't." BoardScribe's rule for logging dissent only fires on the exact phrase, so it writes the motion up as unanimous. The chair reads the summary, sees "motion passed," and approves it in under a minute. That's the bar, technically met.
The bar didn't need a chair who reads more carefully. It needed a check that never depended on her noticing at all.
The cost showed up later, and it wasn't a typo. A set of minutes that quietly says "unanimous" when it wasn't isn't a small mistake. For a nonprofit, minutes are the legal record of what a board decided, the document a funder or an auditor pulls first. Three months on, during a routine compliance review, a program officer asks the two dissenting members why they didn't object on the record. They did, they say, right there in the meeting. The record just doesn't show it. Now the nonprofit is explaining a contradiction between two board members' word and its own official file, weeks before a grant renewal.
Knowledge spark: what board minutes are actually for
For a nonprofit, minutes aren't just notes. They're the official, legal record of what the board decided. Funders, auditors, and sometimes courts treat them as the truth of what happened in the room, whether or not that's exactly how the meeting went.
The choice I would take back. Writing one bar, "the chair approves without a correction," was the sensible call for a pilot where nothing was ever contested. I would take it back and write two bars from day one: a light one for routine meetings, and a strict one, an independent transcript check, for any meeting where a vote wasn't unanimous.
What I would write instead
Two bars, not one. Routine meetings: chair approval is enough, the stakes are low and the tool is already accurate there. Any meeting with a recorded "no," an abstention, or a raised objection: an independent check against the recording has to pass before the minutes count as done, whether or not the chair noticed anything wrong.
What I would leave alone. The 104 routine meetings in this quarter, reports read out, dates set, nothing contested, are already 98 percent accurate against the recording. Making every board wait on a human auditor to bless a routine set of minutes would slow down the 96 boards this tool is already serving well, to fix a problem that only shows up in the other kind of meeting.
The lesson. A bar that everyone downstream nods along to isn't the same as a bar that's true. I wrote a number that measured whether a tired person objected. I should have written one that measured whether the record matched the tape.
The weeks nobody was looking
The gap didn't start with the $40,000 motion. It started nine weeks earlier, in a signal nobody was reading as a signal.
Two lines, one dashboard, nine weeks
What the dashboard showed: chair-approval rate
Quiet signal: chairs replying same-day with a question about the minutes
wk 1wk 2wk 3wk 4wk 5wk 6wk 7wk 8wk 9
The approval rate stayed flat and green the whole time. The quiet signal, chairs replying with a clarifying question the same day, climbed from 1 a week to 9 in the week of the reserve-fund vote. Nobody connected the two, because only the first one had a dashboard.
Before blaming BoardScribe's transcription for those replies, Halyna's team checked whether the dashboard number itself might be broken, undercounting edits, or logging a reply as an approval by mistake. They pulled 220 real approval events and timed them against an independent audit log. The timestamps matched exactly. The 47-second median was real. That ruled out a tracking bug, and it pointed straight at the actual problem: the number was measuring attention, honestly, and attention was never the thing that needed measuring.
One number hiding two different truths
Once the team knew the bar was real but aimed wrong, they cut it the only way that mattered: by whether the meeting had anything contested in it.
Chair-approval rate vs. audited accuracy, by kind of meeting (140 meetings, one quarter)
94%
98%
89%
61%
Routine, 104 meetings no vote contested
Contested, 36 meetings at least one recorded "no"
Chair-approval rate
Audited accuracy, checked against the recording
On routine meetings, the two numbers nearly agree: approval is a fine stand-in for accuracy. On contested meetings, approval barely moves, 89 percent, while audited accuracy drops to 61 percent. The bar was healthy on exactly the meetings where being wrong costs the most.
Three ways the bar got hit for the wrong reason
Not because anyone cut a corner. A criterion built to be easy to check quietly stops checking the thing it was meant to guarantee.
Three separate, checkable ways to hit a bar without meeting its point
Way 1
Proxy swap. Approval stood in for accuracy.
The bar was written as "the chair approves," because that's easy to log automatically. But approval is a proxy, a stand-in for a truth nobody was directly checking. The stand-in stopped tracking the real thing the moment the meetings got harder than the pilot's.
How you'd check it: ask what the metric would look like if the reviewer never read a word. If the number would look the same, it's measuring the reviewer, not the record.
Way 2
A narrow trigger rule. Only the exact phrase counted.
The model's dissent rule fired on "I vote no" and little else. Real board members hedge, soften, and object in a dozen different sentences that never say the ritual words. A rule written once, against a clean pilot transcript, doesn't generalize to how people actually talk in a real meeting.
How you'd check it: pull transcripts where someone clearly pushed back, and check how many ways they phrased it. Count how many of those phrasings the rule would have missed.
Way 3
The calibration sample. The 90 percent bar was set on easy meetings.
The number 90 percent came from three pilot boards where almost nothing was ever contested. The bar was easy to hit by construction, because the sample used to choose it barely contained the kind of meeting that actually needed checking.
How you'd check it: look at what share of the calibration sample had any contested vote in it at all. If it's near zero, the bar was never tested against its own hardest case.
Running the independent check against every contested meeting in the quarter, not just the ones the chair flagged, is the one test that separates these from a guess:
Old gate: chair approval only
Any recorded pushback still counted as "passed" if the chair didn't object
14 of 36 contested meetings shipped with a real, uncaught error
New gate: approval + transcript check
Every meeting with a recorded "no," abstention, or objection gets compared to the tape
0 ship without a person checking the record against what was actually said
TRACE, the way the fix actually got found
This is a diagnosis question wearing a definitions question's coat, so TRACE is the framework, not a story about a habit changing. A "what if the error rate doubled" question would reach for FLIPS instead.
T, timeline. The gap opened nine weeks before the reserve-fund vote, in same-day replies from chairs asking clarifying questions. The approval-rate dashboard stayed flat and green the entire time, because it was never built to see this signal.
R, recut. Split by whether a meeting had a contested vote. One blended 93 percent hides a 98 percent segment sitting next to a 61 percent one.
A, assume nothing. Before blaming the model, Halyna's team checked whether the approval-rate logging itself was broken. It wasn't: 220 real timestamps, cross-checked against an independent log, confirmed the 47-second median was a real number, just the wrong one.
C, cause candidates. Three, named and separate: approval standing in for accuracy, a dissent rule too narrow to catch how people actually talk, and a launch bar calibrated on a sample with almost no contested meetings in it.
E, evidence test. Compare every contested meeting's draft against its recording, independent of whether the chair approved it. On this quarter's data, that finds 14 of 36 real errors that chair approval alone had let straight through.
Why E is the hard step
Anyone can say a bar is too soft. A test earns its place by checking the output against the actual source, not against another person's read of the output. If you check the model against a person's opinion of the model, you've just moved the proxy one step over, you haven't removed it.
Same gap, a claims desk after a hurricane
A property insurer builds an AI tool that drafts settlement letters after storm-damage claims. Its launch bar: the adjuster approves the drafted letter without changes, in at least 88 percent of claims. After a hurricane, claim volume triples, and the team watches the number to make sure quality holds up under the surge.
T. The approval rate stays near 88 percent the whole surge. The gap shows up first as a slow rise in calls to the state insurance hotline, weeks before anyone inside the company notices anything wrong on their own dashboard. R. Split by policyholder type. Renter-unit claims approve at 87 percent, homeowner claims at 89 percent, both near the bar. An independent contractor's on-site estimate finds renter claims underpriced by a median of $1,200 in 33 percent of cases, against 9 percent for homeowner claims. A. Before blaming the model's training data, the team checks adjuster review time during the surge. It drops to a median of 85 seconds a claim, from a normal 6 minutes, confirmed against real timestamps, not a logging bug. The number is real: adjusters are rubber-stamping under load, not reading closely. C. Three candidates: renter-unit repairs are underrepresented in the model's training data, the surge itself pushes adjusters toward faster, shallower review, and the 88 percent bar was calibrated on normal-volume claims, a condition that vanishes exactly when a hurricane hits. E. Run an independent contractor's estimate against a sample of renter-unit claims processed during the surge, before the tool is approved to run unsupervised in the next disaster response.
Swap the trigger and it still runs
Speed: instead of a model change, the trigger is a shrinking review window, a disaster surge, a end-of-quarter crunch. TRACE still starts with what the bar actually measures, not with how fast anything runs.
Cost: the team swaps in a cheaper model to cut spend. The by-segment check still has to show what the cheaper model quietly drops, not just what it saves.
The model really did get better: the case on this page. BoardScribe's transcription is 98 percent accurate word for word. The bar just never checked the one slice where being wrong is expensive.
Where people run it wrong
Treating "nobody complained" as proof the bar is fine, instead of checking whether anyone downstream had time to complain.
Writing one blended bar for every case, so a badly-served slice hides inside a healthy-looking average.
Setting the bar once during a friendly pilot and never rechecking it once the product starts touching real stakes.
If you are asked this cold
Buy yourself ten seconds by naming the gap out loud. "So there's what the criterion checks, and what it was supposed to guarantee, and those can drift apart without anyone noticing. Let me say which cases I'd go recheck first." That's not stalling. In nearly every version of this question, that's exactly where the real answer starts.
Flashcards (click a card to flip it)
This is a diagnosis question, not a flip story, so these eight test the TRACE moves and the real numbers instead of a habit changing.
1 · THE FRAMEWORK
Which framework fits "acceptance criteria that are technically met but the product's still bad," and why?
Tap to flip
ANSWER
TRACE. The real question is a diagnosis: why does a bar that's passing still let a bad product through. FLIPS is for "what if something changed," and here nothing in the model changed at all.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Halyna Kovalenko, product manager for BoardScribe, an AI tool that drafts nonprofit board minutes. She wrote the original 90 percent approval bar herself, ten months earlier, during the pilot.
3 · RULING IT OUT
What did Halyna's team check about the measurement itself before blaming the model?
Tap to flip
ANSWER
Whether the approval-logging pipeline was broken. It wasn't: 220 real approval timestamps matched an independent audit log exactly, so the 47-second median review time was real, not a tracking bug.
4 · THE THREE WAYS
Name the three ways the bar got hit for the wrong reason.
Tap to flip
ANSWER
Approval standing in for accuracy, a dissent rule too narrow to catch real speech, and a launch bar calibrated on a pilot sample with almost no contested meetings in it.
5 · THE NUMBER
Of the 36 contested meetings audited against the recording, how many had a real, uncaught error?
Tap to flip
ANSWER
14 of 36, 39 percent. Against just 2 of 104 on routine meetings, 2 percent, even though chair-approval rate barely differed between the two groups, 89 percent versus 94 percent.
6 · THE CHECK
Name the one test that turned a guess into a number, and what it holds still.
Tap to flip
ANSWER
Compare every contested meeting's draft against its actual recording, regardless of whether the chair approved it. It checks the output against the source directly, not against another person's opinion of the output.
7 · THE FIX
What does Halyna actually change about the acceptance bar?
Tap to flip
ANSWER
Splits it in two. Routine meetings keep the quick chair approval. Any meeting with a recorded "no," abstention, or objection must also pass an independent transcript check before it counts as done.
8 · CROSS-PRODUCT TRANSFER
Section 4 runs TRACE on a different product with the same shape of criterion problem. Which one, and what's different about the cause?
Tap to flip
ANSWER
An insurer's AI tool for drafting storm-damage settlement letters. The cause shifts from a narrow dissent-detection rule to training-data skew by policyholder type, worsened by adjusters rubber-stamping under a post-hurricane surge.
Check yourself Score: 0 / 0
Multiple choice
1. Why doesn't a 93 percent chair-approval rate prove BoardScribe's minutes are accurate?
A. It does prove it, since the chair reads every line before approving.
B. Approval only shows the chair didn't ask for a fix. It doesn't show the record matches what was actually said.
C. 93 percent is too low a number to mean anything either way.
D. Chairs aren't qualified to review AI-written minutes at all.
Show hint
Think about what the 47-second median review time tells you about how the chair is actually using that approve button.
Show answer
B. Approval is a proxy for accuracy, not a direct check of it. A busy chair skimming an email will only catch the errors that jump out, and a dropped dissenting vote often doesn't.
True or false
2. True or false: once the team confirmed the approval-logging pipeline wasn't broken, that meant the model's transcription was definitely the cause of the missing dissent.
True
False
Show hint
Ruling out a broken measurement only tells you the number is real. It doesn't yet tell you which of several possible causes produced it.
Show answer
False. Ruling out broken logging (the A step) just confirms the 47-second number is honest. It still took naming three separate cause candidates, and one evidence test, to find that the actual mechanism was a dissent rule too narrow to catch real speech.
Fill in the blank
3. Of the 36 contested board meetings audited against the recording, ______ had a real, uncaught error in the official minutes.
Show hint
Look at the grouped-bar chart's second bar in the "Contested" group, then compare it to the approval rate right next to it.
Show answer
14, or 39 percent. That's against just 2 of 104 routine meetings, 2 percent, even though the chair-approval rate for both groups looked almost the same, 89 percent and 94 percent.
Short answer
4. Name a segment of meetings where the fast chair-approval path should stay exactly as it is, and say why adding an audit there would be a mistake.
Show hint
Look at the by-segment chart. One bar pair is already almost identical.
Show answer
Model answer: "Routine meetings with no contested vote. They're already 98 percent accurate against the recording, so adding a human audit there slows down the 96 boards the tool is already serving well, to guard against a problem that doesn't live in that segment."
Short answer, apply it yourself
5. Think of a product you use where "nobody downstream complained" is basically the whole quality check. What's one case inside it where that check would probably miss something real?
Show hint
Think about a case where the person checking is rushed, or where being wrong wouldn't look obviously wrong.
Show answer
Model answer: "My bank's fraud alerts. I approve most of them in seconds without really reading the merchant name. A charge with a name close to one I recognize would probably slide through, the same way BoardScribe's minutes slid through when the words were close but not exact." Any honest answer works if it names a real case the downstream check is too rushed or too easy to fool to actually catch.
Multiple choice
6. A teammate says the real fix is simpler: just tell chairs to read the minutes more carefully. Why isn't that the same as the decision in this answer?
A. It is basically the same fix, both add more human review to the process.
B. Telling someone to try harder isn't a decision you can rely on twice. The independent check runs on the case's risk level, not on a person's willpower on a given night.
C. Chairs are volunteers, so asking them to read more carefully is unfair.
D. It would take longer to explain to chairs than to just build the check.
Show hint
One of these names a new dial being turned up. The other names an old decision being taken back.
Show answer
B. "Read more carefully" is a request, not a design. It depends on a tired person doing something different at 11pm with no new information. A check gated on risk level runs the same way every time, whether or not anyone remembers to try harder.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
