ConceptIntermediateQuality, Cost & Token Economics / Eval design for product teams / #9
Explain the difference between unit evals and end-to-end evals.
Here's what happens when a system passes its overall test every week and still ships a broken step underneath it: nothing looks wrong until someone goes looking, for the wrong reason, months later.
The direct answer
Build the end to end eval first, since it needs no split into steps and tells you fast whether the final answer is right for the person on the other end of it. Then break the pipeline into its real steps and give each one its own eval within days, because a regression, one step quietly getting worse, can hide for weeks inside an end to end score that barely moves while the other steps quietly cover for it. Neither eval alone is enough: the end to end score tells you something's wrong, the unit evals tell you where.
Do this, in order
Build the end to end eval first, since it needs no design work up front.Why: it's the cheapest way to know fast whether the whole chain gets the right answer for the person waiting on it.
Break the pipeline into its real, separately gradable steps as soon as that eval exists.Why: a unit eval needs a defined right answer at each step's own boundary, and you cannot write one on a step that doesn't have that boundary yet.
Give every step its own eval within days of that split, not months.Why: an end to end only setup lets a real regression hide inside a score that mostly still looks fine, the hardest kind of miss to trace back to its source.
Run the unit eval on the one step you just changed before waiting on a full end to end run.Why: it's the cheap check, seconds instead of a slower run that mixes in noise from steps you didn't touch.
Weight each step's eval bar by what a miss there actually costs.Why: a wrong payout number costs real money, a clunky sentence costs a minute's edit, and grading both the same wastes review time on the step that matters least.
How to answer this, stage by stage
Nobody's grading whether you know two definitions. They're grading whether you can rank a build order under a real dependency, and say why one eval alone would leave you blind.
1
Scope it to one concrete product before defining anything in the abstract
Say it like this
"Let's ground this in one product. ClaimForge is an agent at Greyholt Mutual. It reads a claim, checks the policy, works out the payout, and drafts the reply, four steps, one after another. Ophira Widmark owns eval design for it."
Why this works
A definition question turns into a lecture fast without one real pipeline to point at.
2
Define both terms in one line each, before ranking anything
Say it like this
"A unit eval checks one step by itself, feed it a claim and a policy, does the payout step get that one number right. An end to end eval checks the whole chain, does the letter the policyholder actually receives come out correct."
Why this works
The interviewer is listening for whether you know what each one actually measures, not just that two words exist.
3
Say your structure out loud
Say it like this
"I'm going to rank these two by what's hardest to undo if I only ever build one of them, because that's the real question hiding inside 'what's the difference.'"
Why this works
Signals a method instead of a glossary, and stops you drifting between two definitions with no landing point.
4
Name the outcome both evals are actually serving
Say it like this
"Neither eval is the goal. The goal is a policyholder getting the right payout, on time. Both evals only matter as far as they protect that."
Why this works
Without naming this first, ranking the two evals is just a coin flip dressed up as judgment.
5
Rank the two by what's hardest to diagnose if you only have the other
Say it like this
"If I only had the end to end score and the payout step started quietly rounding down, my weekly number might barely move, maybe 92 to 89, because the other three steps are still fine and the score has no way to say which one is wrong. That's the miss that hides for weeks and costs real money before anyone traces it."
Why this works
This is the actual judgment call the question is testing, not a definition.
6
Say the dependency, out loud
Say it like this
"Here's the catch. I can't write a real unit eval on the payout step until the pipeline is actually split into four separate, gradable steps with a defined right answer at each one. That split is a design decision, and it has to happen early, or there's nothing to check step by step."
Why this works
Shows you understand unit evals aren't free, they need a real split-into-steps decision first.
7
Give the cheap check that resolves it day to day
Say it like this
"Once the steps exist, I run the unit eval on the one step I just touched before I ever wait on a full four step run. It's seconds instead of minutes, and it doesn't mix in noise from three steps I didn't change."
Why this works
Turns the abstract ranking into something a candidate would actually do on a Tuesday.
8
Close on the ranked call, not the definitions
Say it like this
"So: build the end to end eval first, it needs no split into steps and tells me fast if ClaimForge works at all. Split the pipeline and give every step its own eval within days, because the end to end score alone can't tell me where a regression lives. Neither one on its own is enough."
Why this works
Closing on the rank is what makes this sound like a decision, not a glossary entry.
Let's learn
Here's what happens when a system passes its overall test every week and still ships a broken step underneath it: nothing looks wrong until someone goes looking, for the wrong reason, months later.
ClaimForge is an agent at Greyholt Mutual, an insurance company. It reads a claim, checks whether the policy covers it, works out the payout, and writes the reply. One step, then the next, then the next, then the last. Four steps chained into one pipeline.
Before ClaimForge, one senior adjuster did all four steps by hand. A full claim, read to reply, took close to half an hour. In an eight hour shift, one adjuster could fully process about 14 claims.
Now ClaimForge runs all four steps in under 90 seconds. Adjusters review a short summary instead of doing the work themselves, so the same team clears around 300 claims a day.
Knowledge spark: what's a golden set?
A stack of real claims where a human has already worked out the right answer, held back and never used to train the model. Every week, a reviewer checks what ClaimForge actually decided against that answer key. That's the ruler everything gets measured against.
Every week, a human reviewer checks 150 real claims against ClaimForge's own decisions. The blended pass rate across all four steps sat at 92 percent. Eight wrong out of 100 sounds like the whole story. It isn't. What matters is where those eight live, and whether one blended number can even tell you that.
A code change to the payout step shipped, a rounding bug in how it applied deductible carryovers from an earlier claim. Checked on its own, against a small daily set of 80 payout problems, the payout step's own accuracy fell from 98 percent to 71 percent, the same afternoon the code went out. But the blended weekly score, the one number the team actually watched, moved from 92 to 91, then drifted between 88 and 90 for the next month. That's inside the normal weekly wobble the team had always seen. Nobody looked twice.
The blended score never left its own noise band. The payout step, on its own, fell off a cliff the same day.
Blended end to end pass rate, week 0 to week 5
Blended end to end pass rate, vs the weekly golden set
Wk 0: 92 percent. Wk 1: 91 percent, the bug ships this week. Wk 2: 90. Wk 3: 89. Wk 4: 90. Wk 5: 88, the week finance's audit flags a pattern. Every point sits inside the 88 to 93 range the team had always treated as normal.
Payout step's own unit eval, before and after the code shipped
Passing its own daily setSame set, after the code shipped
Checked against its own 80 problem set, the payout step's accuracy dropped 27 points the same afternoon the rounding bug shipped. Nothing that sharp shows up anywhere in the blended weekly line above.
The bug only hit claims with an existing deductible carryover, about one in six. But one in six of 300 claims a day, for five weeks before finance's quarterly reconciliation flagged a pattern, is about 1,250 real claims that got a wrong payout number nobody caught in time.
The choice that mattered
For ClaimForge's first year, the team scored the whole pipeline as one pass or fail number, with no per step checkpoint. Splitting it into four separately graded steps felt like real engineering work for a product that was still small, and it made sense while claim volume was low enough that a human could skim most of the weekly sample by hand if something looked off. It stopped making sense the day the payout step's own accuracy could fall 27 points and the number the team actually watched didn't even flinch.
What I'd leave alone: the draft reply step doesn't need this kind of scrutiny. If it writes a slightly clunky sentence, the adjuster reviewing it fixes it in a minute before it goes out. Building a tight, daily graded unit eval for wording would spend review time on the one step that costs the least when it's wrong.
The lesson: a blended score can be completely honest and still hide the one step that's on fire. That's not a flaw in the number. That's what averages do. Three fine steps quietly cover for the one that isn't.
Now here is the same thing as a story
Read the long version below when you want to feel why the rank matters, not just be told the order.
Every Friday afternoon, Ophira Widmark pulls up the same report: ClaimForge's blended pass rate against the week's 150 golden claims.
She built the grading pipeline for ClaimForge from scratch, three years back, when it read maybe forty claims a day and one number was plenty to watch. She knows the shape of a normal week cold. A good Friday sits between 90 and 93. A rough one dips to 88 and climbs back the next week. She has never once needed to open the raw claims to know if a Friday is fine.
For most of that time, Friday afternoons were easy. Pull the number, note it, move on to whatever else needed her that week.
Then came a Friday in March that looked exactly like every other one. 91 percent. Two points off the week before, well inside the wobble she'd always seen. She noted it and closed the tab.
It kept looking like every other Friday for five weeks. 91. 90. 89. 90. 88. Each one, on its own, was nothing. Together, they were still nothing, because a slow drift inside a band she'd already decided was normal never once asked to be looked at twice.
It surfaced on a Tuesday, not a Friday, and not from her report at all. Finance ran their quarterly reconciliation and flagged a pattern: a cluster of claims with a deductible carryover, paid out slightly low, four weeks running. Someone forwarded it to her with one line: is this us or is this them.
It was never really about whether the blended number moved. It hadn't, not by more than a normal Friday ever moves. What mattered was that the blended number had no way to say which of the four steps a claim like this had even touched.
She spent the afternoon pulling every payout the model had calculated on a claim with a carryover, going back six weeks. Two hundred and eleven of them. Not a sample, all of them. Seventy of the two hundred eleven were wrong, low, from the same rounding error in how the model applied the carryover. Every other step, on the same claims, had done its job correctly. The coverage decision was right. The reply read fine. Only the number inside it was wrong.
The decision that let this hide went back to a meeting eighteen months earlier, when someone asked whether the four steps needed their own separate golden sets, or whether the blended weekly number was enough. The answer at the time was: the blended number's fine, splitting it four ways is a quarter of engineering time nobody's asking us to spend. Claim volume was still low enough that a human could skim most of the sample by hand if something looked off. Nobody came back to that question as volume climbed past 300 a day and skimming stopped being possible.
Run the same six weeks again with one change: a daily unit eval on the payout step, 80 claims with a carryover, graded against its own answer key. The rounding bug ships on a Tuesday afternoon. The payout step's own number falls from 98 to 71 percent that same afternoon, nowhere near any wobble it's ever had. Ophira sees it Wednesday morning, not five weeks and a finance audit later. The fix ships Thursday. Instead of 1,250 wrong payouts, there are about seventy, the ones that went out in the day and a half between the bug shipping and the alert firing.
What I'd tell myself, back in that meeting eighteen months earlier: the moment a number is standing in for four different jobs, ask whether it can still tell you which one broke, or whether it only ever could because the volume was too small to hide anything in yet. Nobody asked. That's on the room, not on the model.
ORDER, run on unit versus end to end
This isn't a coin flip between two useful evals. It's ORDER, run until one build order genuinely outranks the other.
OOutcome. What is everyone actually racing to protect?
Not either eval's own pass rate. Whether a policyholder gets the right payout, on time, on every claim, whichever of the four steps happens to touch it.
Name this first, or ranking the two evals is just gut feeling dressed up as method.
RReversibility. Which gap is harder to undo?
Having only the end to end score, and missing the unit evals, is the harder failure. A real per step miss can run for weeks, quietly costing real money, inside a blended number that looks like normal noise. Having only unit evals, and missing the end to end score, is a real gap too, but it tends to surface faster: the assembled output is what the policyholder actually reads and what a reviewer spot checks, so a bad handoff between two correct steps usually shows up in a complaint within days, not months.
This is the hard rank. Both gaps are real. Only one of them hides in plain sight.
Both gaps cost you something. Only one of them stays invisible.
DDependency. What needs what?
You cannot write a real unit eval on the payout step until the pipeline is broken into four real, separately gradable steps with a defined right answer at each boundary. That split is a design decision, made once, early, before any unit eval is possible.
This is why unit evals can't just be built on day one out of nowhere. The split has to come first.
Skip the first box and there's nothing left to grade step by step.
EEvidence. What's cheap to check right now?
Once the steps exist, run the unit eval on the one step you just changed, before waiting on a full four step run. That's seconds, not minutes, and it doesn't mix in noise from three steps you didn't touch.
Cheap enough to run after every change, on any step, before a blended number ever gets a chance to hide it.
RRank. Say the order, out loud.
Build the end to end eval first, it needs no split into steps and gives you a real floor number right away. Split the pipeline and add a unit eval per step within days, not months, because that's what makes a regression visible the day it ships instead of the week finance happens to notice. Neither one alone is enough on a chained system like this.
This is the line a strong candidate says out loud, not the one they leave implied.
Three things worth stating directly, since this is where the real judgment sits. The alternative Ophira's team considered right after finding the payout bug was widening the end to end sample, reviewing more than the usual 150 golden claims by hand each week. It lost because a bigger sample would only have caught the bug sooner by chance, still weeks later, and it still wouldn't say which of the four steps was wrong, someone would still have to read transcripts by hand to find that out. The AI specific failure mode worth naming by name is a silent regression inside a chained model pipeline, one step quietly getting worse while the others keep covering for it in the blended number, exactly the shape of failure that only shows up as noise until someone happens to slice the data the right way. The guardrail is two part: a daily unit eval on every step, graded against its own golden set, and a rule that a step's eval runs the same day any code touching that step ships, not on the weekly cadence the blended score runs on. That guardrail isn't free. Maintaining four separate golden sets instead of one, and running daily checks against them, costs real engineering time and a reviewer's hours every week, a cost accepted on purpose because a wrong payout number is worth catching in a day, not a quarter. And the bar was never zero misses, a system running 300 probabilistic claims a day can't promise that. It's an audited floor set by stakes: the payout and coverage steps clear 97 percent on their own daily golden set, checked every day, while the draft reply step gets a looser, weekly spot check, because a bad sentence there costs a minute's edit and a bad number costs someone their claim.
And if you want to be sure it really works, try it somewhere else
Same five ranks, a city permit office instead of an insurance claim, nothing about deductibles anywhere in sight.
Zoneworks is a permit review agent at Bramfield city hall. It reads a building permit application, checks it against the zoning code, calculates whether the setback and lot coverage numbers comply, and drafts the approval or rejection letter. Four steps, chained. Kalindi Zhirkov reviews the flagged and rejected applications.
Outcome: not either eval's own score. Whether an applicant gets a correct, citable permit decision, the first time, without an appeal.
Reversibility, the hard rank: grading only the final letter lets a wrong setback calculation slip through while the letter itself still reads fine, coherent, properly formatted, citing the right code section, just built on a bad number. That's the harder gap to undo. A resident builds an addition on a bad determination, and undoing that costs far more than catching it before the letter ever goes out.
A letter can read perfectly and still be built on the wrong number.
Dependency: Bramfield's team had to split "read the application" from "check zoning" from "calculate setback and coverage" from "draft the letter" as four real steps, with a defined right answer at each one, before a setback specific eval was even possible to write.
Same rank as before: build the end to end eval first, since it needs no split into steps and tells the office fast whether Zoneworks' letters are broadly correct. Add a unit eval on the setback calculation step within days, because a wrong setback number can hide inside a letter that still reads fine end to end, the same way a wrong payout hid inside a coherent claims reply.
Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to the rank, end to end first, unit evals per step within days, and name the one dependency that forces that order.
Cost: there's no budget this quarter for four separate golden sets and a nicer end to end dashboard. The step level evals win every time, on the steps where a miss costs real money. A prettier dashboard on a system nobody can localize a bug in is decoration.
The model got better, for real: say overall accuracy improved this quarter. That's not proof every step improved with it. One step can get quietly worse while three others improve enough to lift the blended number anyway.
Where people run it wrong.
They treat a blended pass rate sitting inside its historical range as proof nothing regressed, without ever checking the slice.
They build every step's unit eval to the same rigor, instead of weighting effort by what a miss on that specific step actually costs.
They wait for the end to end score to flag something before writing a single unit eval, instead of splitting the pipeline into steps on day one, whether or not anything looks wrong yet.
How to use it live. Say the real tension out loud before answering it: "is this asking me which eval to build, or which one to trust when they disagree." That buys a beat to think instead of reciting two definitions at the interviewer.
Flashcards (tap any card to flip it)
1 · THE FRAMEWORK
What framework is this, and what's its one job?
Tap to flip
ANSWER
ORDER: rank by what's hardest to undo. Built for prioritization questions where two things both want to be built first.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Ophira Widmark, eval engineer at Greyholt Mutual, owns eval design for ClaimForge, the four step claims agent. Built its grading pipeline from scratch three years back.
3 · THE OLD SETUP
What did the team build first, and what did it miss?
Tap to flip
ANSWER
One blended end to end score for the whole pipeline, no per step checkpoint. It could never say which of the four steps a real miss lived in.
4 · THE TWO LAYERS
What are the two eval layers being ranked, and why does the comparison matter?
Tap to flip
ANSWER
A unit eval on one step alone, versus an end to end eval on the whole chain. An end to end only setup lets a real regression in one step hide for weeks inside a pipeline that mostly still works.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Scoring the whole pipeline as one pass or fail number for the first year, with no per step golden set, because splitting it felt like extra engineering work while volume was still small.
6 · THE NUMBER
Fill in the blank: the payout step's own unit eval fell from 98 percent to ___ percent the day the rounding bug shipped, while the blended end to end score that same week only moved from 92 to ___ percent.
Tap to flip
ANSWER
71 percent, and 91 percent. The step level number told the real story. The blended number looked like a normal Friday.
7 · THE REPLAY
Same rounding bug, new design, what changes?
Tap to flip
ANSWER
A daily unit eval on the payout step flags the cliff the same afternoon the code ships. Ophira sees it Wednesday morning, not five weeks later through a finance audit. About seventy wrong payouts go out instead of 1,250.
8 · CROSS PRODUCT TRANSFER
Section 4 answers this same question again for a different product. Which product, and what's the same tension?
Tap to flip
ANSWER
Zoneworks, a municipal permit review agent at Bramfield city hall. Same tension: grading only the final approval letter lets a wrong setback calculation slip through, while a step level eval on the setback math catches it before the letter goes out.
Check yourself Score: 0 / 0
True or false
1. True or false: because ClaimForge's blended weekly pass rate never dropped below 88 percent while the payout bug was live, the bug's real impact was small.
True
False
Show hint
Check the payout step's own unit eval chart, not the blended line, and think about how many claims have a deductible carryover.
Show answer
False. The bug hit about one in six claims for five weeks, roughly 1,250 real claims, while the blended score stayed inside the 88 to 93 range the team had always treated as normal. The impact was real, the blended number just never showed it.
Multiple choice
2. Why couldn't Ophira have caught this bug just by watching the end to end score more closely?
A. Because she wasn't allowed to open the raw claims.
B. Because the end to end score blends four steps into one number, so a real miss in one step can sit inside a range that still looks like normal weekly noise.
C. Because the end to end eval only ran once a year.
D. Because the model needed retraining regardless of what any eval showed.
Show hint
Look at what a blended average is doing to three fine steps and one broken one.
Show answer
B. A blended number can only tell you that something, somewhere, changed. It has no way to say which of the four steps changed, or by how much, on its own.
Fill in the blank
3. The payout step's own unit eval accuracy fell from 98 percent to ___ percent the same afternoon the rounding bug shipped.
Show hint
It's the number marked on the bar chart in Section 1, right next to "same day, after."
Show answer
71 percent. A 27 point drop, on the same day, in a number the blended weekly score never came close to reflecting.
Short answer, name the rejected alternative
4. What alternative did Ophira's team consider once they found the payout bug, instead of building step level unit evals, and why did it lose?
Show hint
Look at what the team could have done to the weekly golden set itself, instead of splitting the pipeline.
Show answer
Model answer: Widening the end to end sample, reviewing more than the usual 150 golden claims by hand each week. It lost because a bigger sample would only have caught the bug sooner by chance, still weeks later, and it still wouldn't say which of the four steps was wrong, someone would still have to read transcripts by hand to find that out.
Short answer, apply it yourself
5. Pick a multi-step AI product you use yourself, one with more than one model call chained together. Name one step you'd want its own unit eval on, and why.
Show hint
Think of a step where a small mistake can't be fixed just by looking at the final output, it has to be right on its own.
Show answer
Model answer: A trip-planning app that pulls flight options, checks visa requirements, then drafts an itinerary. I'd want a unit eval on the visa-requirement step specifically, because a wrong visa answer can't be caught by rereading the finished itinerary, it has to be right on its own before anything downstream matters.
Multiple choice
6. Why does the fix give the payout and coverage steps a tighter, more frequently graded eval bar than the draft reply step?
A. Because the draft reply step runs on an older, cheaper model.
B. Because a miss on the payout or coverage step costs real money or a wrong coverage decision, while a miss on the draft reply step costs an adjuster a minute's edit.
C. Because the payout step takes longer to run, so it needs closer watching.
D. Because regulation requires a fixed schedule regardless of what each step actually does.
Show hint
Think about what actually happens after each kind of miss, not how often each step gets something wrong.
Show answer
B. The eval bar should match what being wrong costs on that step, not stay uniform for convenience. A probabilistic system can't promise zero misses anywhere, but it can promise a tighter, daily audited bar where a miss costs the most.
Before you close the answer
Why this works
Tests whether you understand these are two different questions, does it work at all, and where does it break, rather than a redundant pair, and whether you can rank a real build order under a real dependency instead of saying "build both" and stopping there.
Follow-up traps
"Isn't building both from day one just obviously right, so why does the order matter?" Response: the order matters because splitting the pipeline into steps is itself a real design decision with a cost. Naming which step's golden set to build first, on the step where a miss costs the most, is the actual judgment call, not "build both."
"What if you don't have engineering time to split the pipeline into gradable steps at all?" Response: then you're flying on a blended number alone, say so plainly, and put your remaining evidence budget entirely on a bigger, more frequent end to end sample on the single highest-stakes claim type, not spread thin evenly across all of them.
If pressed
The real production bar isn't zero misses, a system running 300 probabilistic claims a day can't promise that. It's an audited floor: the payout step's unit eval must clear 97 percent on its own daily golden set of claims with a deductible carryover, checked every day, not folded into the blended weekly number where a 27 point drop can hide behind three fine steps.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
