ConceptIntermediateEval-Driven Specification / Acceptance criteria for non-deterministic output / #21
What is the relationship between acceptance criteria and your eval set?
The direct answer
The eval set is the only thing that makes an acceptance criterion checkable at all. Write the criterion, then build a random, blind weekly sample pulled from the whole product, not the easy slice of it, graded by someone outside the team that wrote the criterion, with a real number that changes what ships. A criterion with no sample behind it, or a sample built by the same hands that wrote the sentence, isn't a requirement. It's a sentence waiting to be tested.
Do this, in order
Treat the eval set as the criterion's proof, not a task that happens after it.Why: a criterion nobody scores against a real sample never gets enforced, it just sits in the PRD looking responsible.
Sample randomly across the whole product, not the corner that already tests clean.Why: a flattering eval set can show 98 percent while the real defects sit in the slice nobody sampled.
Grade blind, by someone outside the team that wrote the thing being graded.Why: the person who built it reads its output more kindly than the person who has to live with a wrong claim.
Check the claim against the real, live thing, never against the spec that describes it.Why: a doc can match its own spec perfectly and still be wrong about what the product actually does.
Attach a real threshold with a real action to the score.Why: a percentage nobody acts on is a dashboard number, not a requirement.
Watch for the criterion getting written so loosely that no eval set could fail it.Why: a vague enough sentence passes any sample, and the next reader still has nothing real to trust.
Six moves, said the way you'd actually say them
Say the sample size, who grades it, and the two thresholds out loud, by name, or the rest just sounds like a nicer sentence.
1
Say the two documents aren't doing the same job
Say it like this
"People treat acceptance criteria and the eval set like two names for one thing. They're not. The criteria say what has to be true. The eval set is the only way anyone finds out whether it is. Write one without the other and you've written half a requirement."
Why this works
Naming the confusion first tells the interviewer you're not about to bolt a number onto a sentence and call it a day.
2
Put one real product under it
Say it like this
"Say this is Portway, a shipping API for online sellers, about 180 endpoints. An AI writes the reference doc and the code sample for every endpoint straight off the API spec. Get a field wrong in the docs and a developer's fulfillment code breaks in production, not in a code review."
Why this works
A requirement about "documentation" stays abstract until you can say who codes against a wrong sentence and what it costs them.
3
State the relationship in one line
Say it like this
"Here's the relationship. 'The docs must accurately describe the API' is a sentence nobody can fail a build on, on its own. The eval set is what turns it into a fact: pull a sample, check it against the real thing, get a number. No eval set, no criteria. Just a hope with good intentions."
Why this works
This is the reframe. Skip it and the number you name next sounds like a nice extra instead of the whole point.
4
Name the sample, the grader, and the bands
Say it like this
"Here's the number. Every week, pull 30 of the 180 endpoint doc pages at random, across every group, not just the mature ones. Run every code sample against real production responses, and have someone outside the docs team confirm every field, type, and required flag is actually true, blind to what the doc said. 96 percent or higher, it keeps publishing on every API change. Between 88 and 96, an engineer reviews the sample first. Under 88, that group goes back to a human writer."
Why this works
This is the answer to the question, spoken as a number. Everything before it was clearing ground for this one line.
5
Say out loud how the pairing gets gamed
Say it like this
"There are two ways this breaks even with an eval set in place. Write the criteria so loose, 'mostly matches the API,' that no sample could ever fail it. Or build the eval set to flatter the criteria already written, sample only the easy endpoints, grade against the spec text instead of the live API, and let the team that wrote the docs also score them. I'd flag both myself."
Why this works
Naming the gaming yourself tells the interviewer you understand what makes a spec real, not that you memorized a checklist.
6
Say what changes at each level of alignment
Say it like this
"With no eval set, the criteria mean nothing, whatever ships, ships. With a friendly, self-graded sample, the score looks great and the real gaps hide in the part nobody sampled. With a random, blind sample tied to a real threshold, the criteria become something you can actually fail a build on. That's the version worth shipping."
Why this works
This is the D step. It shows you're not grading the eval set as pass or fail, you're grading how honestly it's built.
7
Close on the one line
Say it like this
"So the relationship isn't that the eval set checks the criteria after the fact. It's that the eval set is the only thing that makes the criteria a criteria at all."
Why this works
This is the direct answer, restated as the last thing the interviewer hears. It's the sentence they'll remember.
If you only get through two stages
Stages 3 and 4 are the answer. Say what "accurately describe" was actually hiding, then say the sample size, who grades it, and the two thresholds. Everything else on this list is how you defend the pairing under follow-up.
Let's learn
Say a developer platform writes the reference documentation for every one of its API endpoints using AI instead of a person, and updates it the same day the API changes.
Before the generator, one technical writer kept the docs current by hand, working from engineering's change notes, about two weeks behind any real update, covering maybe 20 endpoint changes a month before the backlog grew again.
Now the generator rewrites the affected doc pages the moment the API spec changes: no lag, no backlog, all 180 endpoints kept current in one pass, every time.
Here's the part that matters. The pages didn't read worse. They read cleaner and more specific than the old ones ever did. What changed is that nobody had ever built a real way to check whether "accurately describe the API's behavior," the one line in the PRD, was actually true. An internal check existed, but it only ever sampled the 60 most mature endpoints and compared the text against the spec, not against what the live API actually returned.
The docs didn't get less accurate. The number everyone was watching just never measured the part that was wrong.
A clean 98 percent and an unchecked field, sitting on two different desks
What that costs at its worst: Emeric Dahlgren, a backend engineer at Loop Kitchens, an online meal-kit seller, built his fulfillment sync straight off Portway's tracking-webhook docs. The docs said the field estimated_delivery_date is always present in the payload. For a handful of regional carriers, the live API omits the field entirely instead of sending it empty. Emeric's sync assumed the key always existed, and for about three weeks, roughly 400 shipments from those carriers quietly dropped out of Loop Kitchens' order-tracking page with no delivery estimate shown, before a customer complaint led someone to check the logs.
Needs the tighter check
Claims a wrong word can break in production
Field presence and nullability: "always returned" versus "omitted when unavailable"
Carrier-specific behavior: which fields exist for which of the twelve carriers
Error codes and what actually triggers each one
Required versus optional parameters on write endpoints
These are the claims a developer codes against directly. Get one wrong and something breaks silently in someone else's production system.
Loose is fine here
Framing and style language
General "getting started" prose and tone
Suggested naming conventions in code samples
Which endpoints get grouped under which sidebar heading
Wording choices that don't claim a fact about behavior
A wrong guess here costs a developer a moment's confusion, not a broken integration. Leave it loose.
Knowledge spark: why matching the spec isn't the same as matching the API
The OpenAPI spec is a document a person wrote, describing what the API is supposed to do. The live API is the thing actually running. Nothing stops those two from drifting apart, a field gets made optional in the code but the spec never gets updated, and a generator that only checks its output against the spec will confidently repeat the spec's own mistake.
The leading edge: weekly blind match rate, if it had existed from week one
96 percent or higher, keep publishing live
88 to 96, an engineer reviews the sample first
under 88, that group goes back to a human writer
This is the rate a real, blind, all-endpoint sample would have shown, reconstructed after the fact. It crosses under 96 in week 2 and under 88 in week 5. Nobody was watching it. The dashboard the team actually watched sampled a different 60 endpoints and never touched the ones that were wrong.
The lagging outcome: support tickets tied to doc and API mismatches, per month
2 a month
Months 1 to 2, average
17 a month
Month Loop Kitchens' sync broke
Tickets lag the real drift by weeks, because a developer has to ship the code, watch it work in the sandbox, then notice it silently fails only for certain carriers in production. The blind rate above had already crossed both thresholds before a single one of these tickets reached a support queue.
The choice I'd take back
I wrote "the generated documentation must accurately describe the API's actual behavior" as the whole acceptance line, then let the team that built the generator also build and grade the sample that was supposed to check it, sampling only the group that was already solid and comparing it to the spec instead of the live API. I'd never separate those two decisions again: whoever owns the criteria has to own, or independently audit, the sample it's scored against.
What I'd leave alone. The framing and style language in the docs stays loose, and that's correct, not a shortcut. A wrong guess there costs a developer a moment's confusion, never a broken production sync. Rates are for the claims someone codes against directly. Sampling every sentence, including the harmless ones, just adds a checking job nobody needs.
The lesson. A criterion and the sample that scores it are one decision, not two. Write them apart, hand them to different owners, or let the same team build both, and somebody will end up with a sample that makes the sentence true no matter what the product actually does.
The Monday number Camille stopped opening
You don't need this to answer the question. It's here so "30 endpoints a week" stops being an abstraction and starts being a real Thursday.
Camille Duarte has been a product manager at Portway for three years. She wrote the PRD for the doc generator herself, and the line she put under "quality" was the one that sounded like the responsible, catch-all choice: the generated documentation must accurately describe the API's actual behavior. Nobody in the sign-off meeting argued with it. Nobody asked who would check it, or against what.
The engineer who built the generator, on his own initiative, added a small self-check: sample 20 doc pages from the rates-and-payments group, the 60 most mature endpoints, compare the text against the OpenAPI spec, log a percent match. For the first three months, Camille pulled that dashboard up every sprint review and cross-checked two or three of the sampled pages herself against a real API call in Postman, just to see with her own eyes.
The number always read 97 to 99 percent. Her own spot checks always matched it. So the habit thinned: two pages became one, then whichever page she had time for, then just the dashboard number itself, glanced at and trusted.
Nothing about the generator changed in that stretch. What changed was the platform underneath it: more carriers added, more optional fields, more behavior that varied case by case instead of following one clean rule. The generator kept writing the same confident, specific sentences it always had. On the mature endpoints, still sampled, still checked, that was fine. On the newer carrier-webhook and customs endpoints, forty of them, never once inside the sample, it increasingly wasn't.
We didn't lose accuracy. We lost the forty endpoints nobody had ever agreed to check.
There was no single bad release. It built the way most of these things build, a little at a time, unwatched. The first sign anyone outside the docs team saw was a message from Farida Khoury, a support engineer, in a shared channel: "third ticket this month about a webhook field being missing when the docs say it's always present, is this a known issue?" Nobody had an answer ready.
Two weeks after Farida's question, Emeric Dahlgren's fulfillment sync at Loop Kitchens hit the exact same gap, on a field the docs called permanent and the live API sometimes simply didn't send. His sync had been quietly dropping tracking updates for regional-carrier shipments for weeks. He filed the escalation that finally reached Farida's queue with a name attached to it.
Camille went back to the sign-off meeting in her head, the one from three years earlier where the quality line got approved with a single sentence under it, and everyone nodded because it sounded stricter than any number anyone could name on the spot. Nobody in that room asked who would build the sample, or whether the people building it would have any reason to make it easy on themselves.
Here's the replay. If a random, blind, all-180-endpoint sample had existed from week one, it would have shown 97, then 95 by week 2, already under the line where an engineer should be reading every sampled doc. By week 5 it would have shown 87, under the line where that endpoint group comes off automatic publishing. Both of those land before Emeric's sync ever ships, weeks before Farida had to ask her question in a channel at all.
With that sample running, the webhook group gets pulled into engineer review the moment the score first slips under 96. Someone reads the batch against real production traffic. The field that "is always present" never goes out described that way for a carrier that doesn't send it.
What Camille would tell herself, back in that sign-off meeting: she let one team write the requirement and build what checks it, in the same week, without asking whether that made the sample honest. It didn't. It made the sentence look tested. Those aren't the same thing, and the gap between them cost three weeks of somebody else's production logs before anyone noticed.
Two documents that only work as one: LEAD, applied here
This is a spec question about the relationship between two documents, not a person's habit snapping in two, so the framework is LEAD, not FLIPS or PICK. FLIPS needs a habit with exactly two settings already built in, and nothing here snaps, the sample was mis-scoped from the day it was built. PICK needs two options standing on equal footing; there's no real second option here, only degrees of how honestly the pairing gets built. The whole question is which number would make the sentence true or false.
L, link. The real outcome, not how the sentence reads or how fast the docs regenerate. Here, it's whether a developer can build working code from the docs without their integration breaking on something the docs got wrong.
E, early signal. The eval set itself, not a separate metric next to the criteria, is the thing that makes the criteria checkable at all. Without a random, blind, all-endpoint weekly sample behind it, "accurately describe the API" is a sentence with nothing standing behind it.
A, abuse. The pairing gets gamed two ways. Write the criteria so loose that any sample would pass it, no matter what the eval set finds. Or build the eval set to flatter the criteria already written, sample only the easy corner, grade against the spec instead of the live thing, let the same team do both.
D, decision. What changes at each level of alignment. No eval set: the criteria mean nothing, whatever ships, ships. A friendly, self-graded sample: the score looks great and the real gaps hide in the part nobody sampled. A random, blind sample tied to a real threshold: the criteria become something a build can actually fail.
The check that keeps the pairing honest
Ask who wrote the acceptance line and who built the sample that scores it. Same person, same team, same week, same sign-off meeting? That's the tell the pairing was never independent, and the score it produces is measuring agreement, not accuracy.
And if you want to be sure it really works, try it somewhere else
An HVAC platform's AI writes step-by-step repair procedures for field technicians, pulled from manufacturer service bulletins. "The repair steps must be safe and correct" is exactly as untestable as the shipping-docs version.
L. A technician follows the written steps and the unit gets fixed correctly, without a wrong torque value or a skipped safety step causing a callback or an injury.
E. Each week, pull 25 procedures at random, across every equipment model, not just the common ones. A technician who wasn't in the room checks every step, value, and safety warning against the real service bulletin and the actual unit.
A. Sampling only the three most common furnace models and skipping the older or regional units, where a wrong voltage or torque value actually hurts someone, so the score stays high while the real risk sits untouched.
D. Above the bar, procedures publish straight to the technician's tablet. In the middle band, a senior technician reviews before it ships. Below it, that model's procedures go back to a person writing them until the model's fixed.
Same rate, a different desk
Swap the trigger and it still runs
The endpoint count doubles. Doesn't matter. Sampling 30 out of 180, or 30 out of 400, costs the same afternoon.
The eval set gets slower to run. If checking against real production traffic instead of a cached spec triples the time per sample, the weekly count can shrink, but the criteria still only mean what the sample actually checks.
The generator gets better than the writer ever was. Rerun the same random, blind sample against the wider set it now covers. Only what's being sampled changes, never who checks it or how.
Where people run it wrong
Letting the team that owns the acceptance criteria also build and grade the eval set, so the sample quietly ends up testing what already works.
Treating the eval set as a one-time launch gate instead of a running weekly check, so criteria written once slowly stop meaning anything as the product grows past what was ever sampled.
Grading against the spec or the intended behavior instead of the real, live output, so the score can be perfect while the actual product is wrong.
How to say it if you're asked this cold
Buy yourself the time to build the pairing properly. "Before I give you a number, let me say what 'accurate' is actually being checked against, and by whom." That's not stalling. It's the L and E steps, said out loud, and it gives you somewhere honest to stand while the real sample takes shape in your head.
Flashcards (click a card to flip it)
1 · THE FRAMEWORK
Which framework fits this question, and why not FLIPS or PICK?
Tap to flip
ANSWER
LEAD, for a spec question about the relationship between two documents. FLIPS needs a habit with two settings already snapping; nothing snaps here, the sample was mis-scoped from day one. PICK needs two options on equal footing; there's only one honest way to build the pairing, and degrees of how far short of it a team falls.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Camille Duarte, a product manager at Portway, a shipping API platform, who wrote the original accuracy line and used to spot-check the generator's docs against real API calls herself.
3 · THE HABIT
What did Camille stop doing once the dashboard number kept looking fine?
Tap to flip
ANSWER
Personally cross-checking a few doc pages against real API calls each sprint. Two pages became one, then whichever she had time for, then just the dashboard number, glanced at and trusted.
4 · THE RELATIONSHIP
What's the actual relationship between the criteria and the eval set here?
Tap to flip
ANSWER
The criteria only became checkable once a random, blind, all-endpoint sample existed. The one that already existed had been quietly built, by the same team, to only ever sample the part that was already solid, so it made the sentence look tested without ever testing it.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Letting the team that wrote the acceptance line also build and grade the eval set, sampling only the 60 mature endpoints and checking against the spec text instead of real production traffic.
6 · THE NUMBER
The rewritten eval set samples ______ endpoints a week, out of about ______ total, and pulls a group back to a human writer under ______ percent.
Tap to flip
ANSWER
Thirty endpoints. About 180 total. Under 88 percent. Between 88 and 96, an engineer reviews the sample first; 96 or higher, it keeps publishing live.
7 · THE REPLAY
Same drift, real eval set from week one, what changes?
Tap to flip
ANSWER
The honest rate drops under 96 percent by week 2 and under 88 by week 5, both weeks before Emeric Dahlgren's fulfillment sync ever breaks. The webhook group gets pulled into engineer review the moment it first slips under 96, and the field that "is always present" never ships described that way for a carrier that doesn't send it.
8 · THE TRANSFER
Section four runs LEAD on a different product. Which one, and what's its early signal?
Tap to flip
ANSWER
An HVAC platform's AI-generated repair procedures for field technicians. Its early signal is the percent of a weekly random sample of 25 procedures, across every equipment model, a technician confirms matches the real service bulletin and unit.
Check yourself Score: 0 / 0
True or false
1. True or false: this gets fixed by rewriting the acceptance line to sound stricter, without changing what actually gets sampled to check it.
True
False
Show hint
Ask whether stricter wording changes who checks the claim, against what, or how wide the sample reaches.
Show answer
False. A stricter-sounding sentence checked by the same narrow, non-blind sample still passes the same way it always did. The words changed. What actually gets checked, and by whom, didn't.
Multiple choice
2. Which of these would count as a genuine pass under the rewritten criteria and eval set together?
A. The doc page uses precise, technical-sounding language.
B. The internal dashboard shows 98 percent this week.
C. Someone outside the docs team confirms, in a random weekly sample checked blind, that every documented field behaves that way in production.
D. No developer complained about that endpoint this month.
Show hint
Three of these describe how the docs looked or landed. Only one describes an independent check against the real API.
Show answer
C. Precise wording, a quiet dashboard, and no complaints are all facts about the surface, not proof the claims are true. The only real pass is an independent, blind check against the live system.
Fill in the blank
3. The honest eval set samples ______ endpoint doc pages a week, out of about ______ total, and pulls a group back to a human writer under ______ percent.
Show hint
The first two numbers are in the walkthrough's stage 4. The third is in the direct answer at the top.
Show answer
30. 180. 88. Between 88 and 96, an engineer reviews the sample before that group keeps publishing unsupervised.
Multiple choice
4. Which of these would make the weekly score look good without the criteria actually meaning anything?
A. Widening the sample from 30 endpoints to 60.
B. Writing the acceptance line so vaguely, "mostly matches the API," that no sample could ever fail it.
C. Having someone outside the docs team grade every sampled page.
D. Posting the weekly score where support and engineering both see it.
Show hint
The dangerous move removes the very thing the sample is supposed to be able to fail.
Show answer
B. A criterion loose enough that nothing could ever fail it stays "true" no matter how good or bad the eval set's real findings are. That's the abuse from the criteria side, not the sample side.
Short answer, apply it yourself
5. Pick a product you use yourself. Name one place it makes an "accurate" or "verified" promise you've never actually seen tested. What would the weekly blind-sample version of that check look like?
Show hint
Look for a claim with no number attached, like "in stock," "verified seller," or "matches your size."
Show answer
Model answer: "A secondhand clothing app that writes AI listings saying 'true to size, no flaws.' The weekly check: pull 50 items marked no flaws, at random, across every category, and have someone outside the listing team compare the text against the seller's real photos, blind to what the listing said. If the confirmed-accurate rate drifted, I'd know the model was gaming size and condition claims months before returns spiked."
Short answer
6. The flattering internal sample stayed at 97 to 99 percent for months while the honest rate had already dropped to 87. If the flattering sample had included even one of the 40 webhook and customs endpoints, would 98 percent likely have held? Why or why not?
Show hint
Ask where the real defects were concentrated, and whether the flattering sample ever went near that group.
Show answer
Model answer: Probably not. The real gaps were concentrated almost entirely in the 40 webhook and customs endpoints, the newer, less standardized ones the flattering sample never touched. Pulling even a handful from that group would likely have dragged the score down noticeably, which is exactly why a sample scoped to avoid a whole risky slice of the product can't be trusted just because its number looks clean.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
