ConceptFoundationalEval-Driven Specification / Writing an eval spec / #2
List the components of a complete eval spec.
The direct answer
Write the task-level success criteria first, one line per task the model handles, stating plainly what "got it right" means. Attach a hard pass/fail bar to those criteria, measured against a test set that's split by segment, not averaged across all of them. Everything else in the spec, the scoring method, the baseline, the release gate, only works because that criteria and that bar exist first.
The ranking, by what breaks first if skipped
Write task-level success criteria first, one per task, before anything else in the spec.Why: nothing downstream, not the rubric, not the bar, means anything until "got it right" has a plain definition.
Attach a pass/fail bar to those criteria that a new version has to clear before it ships.Why: without a bar, a regression doesn't get missed, it becomes unmeasurable. There's no number that was supposed to catch it.
Build the test set segmented by task type and by carrier or channel, not one pooled sample.Why: a segment that's 10 percent of volume can crater without moving a pooled average anywhere near a flag.
Pick a scoring method that actually checks the criteria, not a proxy for them.Why: a rubric that scores "sounded confident" instead of "gave the right port window" will pass the exact failure you built the spec to catch.
Set a baseline, the current live model or a human agent, so a score means something in isolation.Why: 91 percent is not information on its own. 91 percent against a live model already running at 94 is.
Decide who reviews a failed run and what happens next, before the first version fails the bar.Why: a gate nobody's on the hook to check is a suggestion, not a gate.
How to answer this, stage by stage
Eight moves. The trap in this question is answering it as a checklist recited from memory. An interviewer wants to see you rank the pieces by what actually breaks, not just name all of them in the order a template lists them.
1
Ground it in one real product
Say it like this
"Let me make this real. Say it's Driftline Mobile, a telecom carrier, and there's a chatbot in their app called Wayline that walks new customers through porting their number, activating their eSIM, and picking a plan. I'll list the eval spec's pieces against that, not in the abstract."
Why this works
A generic answer lists section headers. A grounded one shows you know what each piece is actually protecting.
2
Reframe from "list them" to "rank them"
Say it like this
"I can name the pieces, but the useful version of this question is which ones you can't afford to write last. Some of these you can bolt on next sprint. One of them, if it's missing, means you don't just find out late, you find out through a customer whose phone went dead for a day."
Why this works
Signals you're about to use a method, not recite a list you memorized before the interview.
3
Name the outcome every piece is protecting
Say it like this
"Every piece of this spec exists to protect one thing: that a broken version of Wayline gets caught before it reaches a real customer's activation, not after their number's already stuck between two carriers."
Why this works
Without stating the outcome first, any ranking that follows is just a preference dressed up as a method.
4
Find the piece nothing else can substitute for
Say it like this
"If there's no pass/fail bar, a bad version doesn't get caught late, it doesn't get caught at all, because there's no line anyone agreed it had to clear. That's the one piece that turns 'we think it's fine' into 'we checked, and it's fine.' Everything else can be patched in after the fact. That one can't undo what already reached a customer."
Why this works
This is the reversibility test, the strongest move in the whole ranking, and it's specific instead of "the eval matters most" hand-waving.
5
Show the dependency underneath the bar
Say it like this
"But you can't write a pass/fail bar out of nowhere. It has to be a bar on something, and that something is the task-level success criteria. So the real first line of the spec isn't the bar itself, it's deciding, in plain words, what 'Wayline got the port timing right' actually means for each task it does."
Why this works
Shows the pieces aren't a flat list, they're built on each other, which is what separates a ranking from an opinion.
6
Prove it with the failure it prevents
Say it like this
"Here's what happens without that bar. Driftline swapped in a cheaper model, it looked fine on the overall completion rate, and it quietly told everyone porting from one specific rival carrier that their number would be live within the hour. For that carrier it actually takes up to two days. People lost service for a day and nobody noticed for six weeks, because that carrier was only 11 percent of ports."
Why this works
A concrete, compressed failure is worth more than any amount of asserting that eval specs matter.
7
Say what's cheap to check first
Say it like this
"Before I'd commit real time to this, I'd check something cheap: pull the last few model or prompt changes shipped to Wayline and see if any of them were run against a segmented test set with a real pass line, or if the team just watched the dashboard afterward. That tells you in an afternoon whether this gap is real or hypothetical."
Why this works
A verifiable check beats a strong opinion, and it shows you'd confirm the problem before proposing the fix.
8
Close with the full list, in rank order
Say it like this
"So, in order: task-level success criteria, a pass/fail bar tied to them, a test set segmented by task and by carrier, a scoring method that actually checks the criteria, a baseline to compare against, and an owner for what happens when a run fails. The first two are the ones you can't recover from skipping. The rest you can add on the way."
Why this works
Ends on the literal list the question asked for, now with a defended order behind it instead of a memorized sequence.
Let's learn
An eval spec is the document that says how a team will know, before real customers see it, whether a new version of an AI feature is actually better or worse.
Say Driftline Mobile is a telecom carrier, and Wayline is the chatbot built into its app. A new customer opens it after buying a plan, and Wayline walks them through three things: porting their old number over, activating their eSIM, and picking any add-ons, like a second line for a kid or a family member.
Knowledge spark: what "segmented" means here
Instead of one big pooled score for "did the bot get it right," the test set gets split into smaller groups: porting from carrier A, porting from carrier B, eSIM on an iPhone, eSIM on Android. A model can look fine on the pooled number while one of those groups is badly broken.
Before Wayline shipped, about 35 percent of new customers had to call a live agent to finish setting up their plan, mostly to sort out a number port that didn't go the way the app described. After Wayline shipped, that number dropped to 9 percent. It was a real product, doing a real job.
Eighteen months in, Driftline's model team swapped Wayline's backbone model for a cheaper one, mostly to cut the cost of running it at scale. On the pooled numbers, nothing looked wrong. Overall activation completion barely moved, 91 percent before the swap, 89 percent after, well inside the range anyone would call normal noise.
Here is the part that matters. The cheaper model had never been tested against port requests from Quill Mobile, a smaller rival carrier that releases numbers on a 24 to 48 hour delay instead of the same-day release most carriers use. Wayline kept telling Quill-porting customers "your number will be active within the hour." For about a day, some of them had no working number on either carrier. Not a slow app. Not a wrong answer on screen. A dead phone.
Activations that needed a live agent call, before and after Wayline shipped
This is the number that made Wayline worth building. It is also the number the team kept watching, eighteen months later, when it stopped being the number that mattered.
The overall completion rate, 91 to 89, is the number Driftline's team actually had on a dashboard. Quill-porting customers specifically went from a 90 percent successful port to 52 percent, with the rest ending in a service gap. Quill was only about 11 percent of all ports. A slice that size can't drag a pooled average far enough for anyone to notice.
Successful port rate, Quill-porting customers vs. all customers, six weeks after the model swap
The amber line is Quill-porting customers, about 11 percent of volume. The navy line is everyone, the only number anyone was actually watching. It never left "normal week-to-week noise."
We didn't lose a phone call. We lost a family's connection for a day.
One customer had switched from Quill specifically so her elderly father, on the same account, would have coverage in a building with bad reception. For about eighteen hours during the port, neither of their phones worked. She called Driftline support four separate times before anyone could tell her what was happening.
The pass/fail bar can't exist until the first box does
The choice I would take back
When Wayline first shipped, Driftline's team decided that watching the live completion rate was enough of an eval. There was one model, one set of tasks, and completion rate moved cleanly with quality. That was a fair call at the time. Nobody rewrote it once the product added new carriers and new task types the average could quietly absorb a failure inside of. I'd take that back: write task-level success criteria per task, attach a segmented pass/fail bar to them, and gate every model change behind it before it reaches a customer.
What I would leave alone. Wayline's greeting message gets A/B tested for tone, warmer versus more formal, and that never needed a hard pass bar. Getting the wording wrong for a week costs nothing that can't be fixed with the next release.
The lesson. A missing eval component doesn't announce itself on the day it's skipped. The code ships, the dashboard looks fine, and the gap only shows up weeks later, in a slice of customers small enough to hide inside an average. The pass/fail bar isn't extra rigor bolted onto a working product. It's the only piece built to catch the failure before a person lives through it.
Now here is the same thing as a story
Skip this if the stage-by-stage version above is all you need. Read on if you want to feel why the bar and the criteria earn the top two spots, not just take it on faith.
Boluwatife Adisa has been the only product manager on Wayline since before it had a name, back when it was a Slack thread about how many customers were calling in confused during setup. She knows the port flow better than most of the engineers who built it.
For the first year, Wayline was the best thing on her roadmap. Every quarter, fewer people needed to call in. She used to sit with an engineer before every model change and watch a batch of real transcripts side by side, old version against new, one task at a time.
As Driftline added more carriers and more model updates started shipping, roughly one every few weeks, that habit thinned. First she stopped reading every transcript and just skimmed a sample. Then she stopped sampling by task and just watched the completion-rate line on the dashboard. It kept going up, or staying flat. She had other fires.
One of these you can fix on the next release. The other one, someone already lived through.
Nothing dropped. Nothing turned red. Six weeks after the cheaper model shipped, a new hire on the support team, three weeks into the job, caught Boluwatife by the coffee machine and asked, almost as small talk, why so many people who'd ported from Quill Mobile called back the next day sounding rattled.
Boluwatife pulled the Quill-specific numbers, something nobody had looked at on their own in months, and found it in under an hour. The new model had never been tested against Quill's 24 to 48 hour release delay. It was telling people their number would be live within the hour, the same line it gave every other carrier, because nobody had written down that Quill was a different task in the first place.
Nobody on the support floor had a chart in their head about it. They had a feeling about whether Wayline could be trusted with a port, and it only had two settings: fine, or not fine. A string of angry callbacks about dead phones flipped it, and the model getting fixed two days later didn't flip it back. For a month after, three of Driftline's most senior support reps quietly told Quill-porting callers to skip the app and call in directly.
The bug lasted two days. Not trusting the app with a Quill port lasted a month longer.
A year earlier, in the room where the first version of Wayline's eval got written, someone had asked whether they needed real per-task success criteria and a pass bar, or whether watching completion rate in production was enough. Back then there was one carrier's port flow, and completion rate tracked quality closely enough that "watch the number" wasn't a bad call. It just never got revisited once a second carrier, with a genuinely different task underneath the same button, joined the product.
I'd take that back, and here's what changes. With task-level criteria written per port type, and a pass bar tied to them, a segmented held-out set of Quill transcripts sits behind every model change, run automatically, taking maybe fifteen minutes. This version fails it the same afternoon it's built. It never reaches Wayline's live traffic. No new hire ever has to ask an uncomfortable question by the coffee machine.
The old way asked one question: did the average get better. The new way asks the harder one: did every task this bot does stay at least as good, one carrier at a time.
What I'd tell myself, back in that first eval review: skipping the segmented criteria felt like moving fast on paperwork. It was actually the one piece built to catch something before a customer's phone went dark for a day.
ORDER, ranking the pieces of the spec
TRACE would fit if a metric had already dropped and the job was finding out why. Nothing had dropped here, the pooled number stayed flat the whole time, so this is about ranking candidate gaps before one bites, which is ORDER's job, not TRACE's.
O, outcome. Every piece of the eval spec is competing to protect one thing: that a broken version of Wayline gets caught before it reaches a real customer's activation, not after their number is stuck between two carriers.
R, reversibility. A missing baseline is fixable next release, you just start comparing against whatever's live now. A missing scoring method can be patched by hand-grading a batch this week. A missing pass/fail bar is different: there's no line anyone agreed a new version had to clear, so a regression isn't caught late, it's simply never measured, and the first sign of it is a customer with no working phone.
D, dependency. The pass/fail bar can't exist on its own. It has to be a bar on something, and that something is task-level success criteria, written per task, before anyone writes a rubric or a threshold. Skip that, and a "pass/fail bar" is really just a vibe with a number attached.
E, evidence. Cheap to check: pull the last few model or prompt changes shipped to Wayline and search for a segmented test set with a documented pass line. At Driftline, none of the last four changes had one. All four were checked against the pooled completion-rate dashboard only.
R, rank. Task-level success criteria first, since nothing else means anything without it. The pass/fail bar tied to those criteria second, since its absence is what let this regression run for six weeks unmeasured. Then the segmented test set, the scoring method, the baseline, and the release-gate ownership, roughly in that order, each addable without a customer paying for the gap.
The check that keeps this ranking honest
Swap the outcome and the order should move. If a wrong port-timing message only ever cost a customer a shrug and a second, correct message five minutes later, the pass/fail bar could sit lower on this list. It doesn't get top rank because it's rigorous. It gets top rank because a real family's phones went dark for a day before anyone had a number that was supposed to catch it.
Run it where a mistake can't be reopened
A regional pharmacy chain runs an AI intake bot that reads a doctor's faxed prescription and files it into a patient's record, flagging anything it's unsure about for a pharmacist to check by hand.
O. Every piece of this eval spec protects one thing: a filed prescription matches what the doctor actually wrote, in the dose and the drug, before a patient ever picks it up.
R. A wrongly filed refill request is easy to reverse inside the pharmacy's same-day review queue, a pharmacist catches it and corrects the record. A wrongly filed dosage that clears the queue unflagged and gets dispensed is not reversible in the same way. The patient has already taken it.
D. The "flag if unsure" behavior quietly assumes task-level criteria already exist for what "unsure" should mean per drug class, dosage math versus drug-name lookup versus handwriting legibility. Without those criteria written down, the model's own confidence number is standing in for a definition nobody actually wrote.
E. A pharmacist pulled last month's flagged queue against a list of known-tricky handwriting cases. Fewer than half of the tricky cases had ever been run through the intake model before launch, only watched afterward as a "percent auto-filed" count.
R. Same order. Task-level criteria for what counts as a dosage error come first. The pass/fail bar, tied to those criteria and tested against known-hard handwriting, comes second. The flagging threshold, the pharmacist review queue, and the audit log all assume that foundation already exists.
One quarter's dosage errors from faxed prescriptions: caught by the review queue vs. dispensed
Those 4 are not a rounding error. No pharmacist's mistake, an intake model that was never run against a segmented set of known-hard handwriting before it started filing prescriptions unattended.
Swap the trigger and it still runs
Driftline moves Wayline's eval check from monthly to running on every model change. The order doesn't move. The pass bar gets checked more often, never skipped.
A new model vendor cuts inference cost 30 percent but performs worse on rare-carrier ports. Same order. The pass/fail bar matters more against a noisier model, not less, since it's the one piece built to catch the noise.
The new model scores higher on the vendor's own public benchmark. Doesn't reorder anything. A better public benchmark is a claim until it clears your own segmented pass bar.
Where people run it wrong
Treating a flat or improving pooled completion rate as proof nothing broke, when it only ever measured the average, never the worst segment.
Writing the scoring rubric or the release gate before task-level criteria exist, so a "pass/fail bar" is really a number nobody defined.
Spending the available time on the satisfying parts, like the chat interface, instead of the parts that are hardest to notice failing, like a rare carrier's port window.
If you're asked this cold
Say the outcome out loud before naming a single component. "Every piece of this spec is trying to protect one thing: a customer finds out their setup worked, not that it didn't." Then rank from there. Naming the outcome first is what turns a list into an argument.
Flashcards (click a card to flip it)
1 · THE FRAMEWORK
Which framework fits "list the components of a complete eval spec," and why not TRACE?
Tap to flip
ANSWER
ORDER, for ranking the spec's components by what's hardest to undo if it's missing. TRACE diagnoses a metric that already dropped. Here nothing had dropped, the pooled number stayed flat the whole time, so there's no drop to diagnose yet.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Boluwatife Adisa, the only product manager on Driftline Mobile's onboarding chatbot, Wayline, since before it had a name.
3 · THE HABIT
What habit let a regression reach real customers unnoticed for six weeks?
Tap to flip
ANSWER
Boluwatife used to review real transcripts task by task before every model change. As changes got more frequent, that thinned to watching only the pooled completion-rate line on the dashboard.
4 · THE DEPENDENCY
Which eval spec component has to exist before a pass/fail bar means anything?
Tap to flip
ANSWER
Task-level success criteria. A pass/fail bar is a line drawn on top of a definition of "got it right." Without that definition written per task, the bar is just a number with nothing underneath it.
5 · THE OLD DECISION
What decision would you take back, and why did it make sense at the time?
Tap to flip
ANSWER
Treating live completion rate as the whole eval, a year earlier, when Wayline had one carrier's port flow and the pooled number tracked quality closely enough. Nobody revisited it once a second carrier with a genuinely different task joined the product.
6 · THE NUMBER
Quill-porting customers' successful port rate fell from 90 percent to ___ percent, while the rate across all customers only moved from 91 to ___ percent.
Tap to flip
ANSWER
52 and 89. An 11-percent slice of volume falling that far barely dents an 89-percent-of-volume average sitting near 90, which is exactly why the dashboard never flagged it.
7 · THE REPLAY
Same model swap, spec with a segmented pass/fail bar this time. What changes?
Tap to flip
ANSWER
The new model gets run against a held-out set of Quill transcripts before it ships, about fifteen minutes of automated testing. It fails the bar the same afternoon it's built. It never reaches live traffic. No customer's phone goes dark.
8 · THE TRANSFER
Section 4 runs ORDER again on a different product, in a different industry. Which one, and what plays the role of the pass/fail bar there?
Tap to flip
ANSWER
A regional pharmacy's prescription intake bot. The equivalent is a pass bar tested against known-hard handwriting cases, dosage math and drug-name lookalikes, that a version has to clear before it's allowed to file prescriptions unattended.
Check yourself Score: 0 / 0
True or false
1. True or false: since Driftline's overall completion rate barely moved after the model swap, 91 to 89 percent, the eval spec didn't actually need a segmented pass/fail bar to catch this regression. Say why.
True
False
Show hint
Ask what an 11-percent segment falling from 90 to 52 percent does to an 89-percent-of-volume average.
Show answer
False. The overall number staying flat is exactly the problem, not proof nothing was wrong. Only a bar checked per segment, not the pooled average, could have caught Quill's collapse.
Multiple choice
2. Which two components does this answer rank above every other piece of the eval spec, and in what order?
A. The baseline, then the scoring method
B. Task-level success criteria, then the pass/fail bar
C. The release-gate owner, then the test set
D. The pass/fail bar, then the baseline
Show hint
One of these has to exist before the other means anything, per the dependency step.
Show answer
B. Success criteria come first because the bar can't be drawn on nothing. The bar comes second because its absence is what let a real regression run for six weeks unmeasured.
Fill in the blank
3. Before Wayline shipped, ______ percent of new activations needed a live agent call. After it shipped, that dropped to ______ percent.
Show hint
It's the number that made the product worth building in the first place, before the model swap eighteen months later.
Show answer
35 and 9. That's the headline win the eval spec exists to protect, not just the regression it later needed to catch.
Multiple choice
4. What does the dependency step (D) in this answer's ORDER argue?
A. A pass/fail bar can't mean anything until task-level success criteria define what it's a bar on
B. The baseline and the scoring method are the same component
C. Every carrier needs its own separate chatbot
D. The test set should never be segmented, since that fragments the sample size
Show hint
Ask what a pass/fail bar is a line drawn on top of.
Show answer
A. A bar needs a definition of "correct" underneath it. Without task-level criteria written first, a pass/fail bar is a number with nothing backing it.
Short answer, apply it yourself
5. Pick an AI feature you use or are building. Which component of its eval spec, if missing, would cause damage nobody could undo once a real person had already lived through it?
Show hint
Look for the component that decides whether a mistake gets caught before release, or lived through by a real person first.
Show answer
Model answer: "A ride-share app's AI fare estimator. The component hardest to add back after the fact is a pass/fail bar segmented by surge conditions: once it quietly under-quotes a fare during a surge and a driver cancels mid-ride over the mismatch, no rollback plan gets that rider their evening back."
Short answer, the number question
6. If Quill Mobile had made up 40 percent of Driftline's port volume instead of 11 percent, would the pooled completion-rate dashboard still have hidden the regression for six weeks? Say what changes and what doesn't.
Show hint
Reversibility is about what happens once a mistake is caught, not about how large the affected slice is.
Show answer
Model answer: "No, not for six weeks. At 40 percent of volume, a drop from 90 to 52 percent drags the pooled average far enough to look abnormal within a week or two. What doesn't change: only a segmented pass/fail bar, not a bigger dashboard drop, would have stopped the model from shipping in the first place. A worse dashboard complains sooner. It still doesn't say why, or which carrier."
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
