Artifact critiqueIntermediateEval-Driven Specification / Writing an eval spec / #18
Critique an eval spec that measures only accuracy on a single benchmark.
The direct answer
Don't let "score at least 80 on Recall-5" stand alone as the whole spec. One accuracy number on one benchmark can climb by getting very good at the easy, common job while staying blind to the harder one, and it counts a small miss the same as a bad one. Split the bar by the kind of visit it's actually testing, weight misses by how much they cost, and check that the benchmark's own mix still looks like what people use the feature for.
Do this, in order
Split the single accuracy bar into a bar per kind of visit, and never let one blended number stand for the whole feature.Why: a blended 84 let a whole segment sit at 65, quietly failing its own bar, for nine weeks straight.
Rule out a broken benchmark before you trust or blame the score.Why: if the scoring itself disagrees with a manual check, a gap by segment is noise, not proof.
Recut the score by session type before believing one number.Why: repeat orders scored 93. New-cuisine visits scored 65. The 84 average hid both of those.
Compare what the benchmark actually tests against what real traffic is.Why: a benchmark built mostly from repeat orders can't tell you anything about a first visit.
Weight misses by how much they cost, not just count them.Why: ignoring a stated preference outright is not the same mistake as picking a close substitute, but one score treats them the same.
Leave the repeat-order bar exactly as it is.Why: it already matches real behavior, ninety-three and fourteen percent. Adding review there only slows down what already works.
How to answer this, stage by stage
Eight moves. This one plays the critique straight for a moment, then has to prove the number itself isn't broken before anyone's allowed to rewrite it.
1
Pin the criterion to one real product and one number
Say it like this
"Let me put a number on this. Say there's a food app, Suppr, and when you open a restaurant on it, a strip shows five dishes it thinks you'll like. A year ago, Dewi Santoso, who owns the eval process for that strip, wrote the whole acceptance bar as one line: the model has to score at least 80 out of 100 on Recall-5, a benchmark built from fifty thousand real past orders."
Why this works
Naming the actual product, the actual number, and the actual benchmark gives every later step something to point at instead of a vague "the spec is weak."
2
Say what's actually missing from the line
Say it like this
"Here's what's wrong with it, plainly. It names a score. It never says which job that score is testing, and it never says what to do about the jobs it isn't testing at all. A single number standing in for a whole feature is a red flag before you even ask whether 80 is a good bar."
Why this works
Turns "this spec reads fine on paper" into a specific, defensible complaint an interviewer can't wave away.
3
Give the fix up front
Say it like this
"So here's what I'd do. I'd split the bar in two. Score at least 80 on repeat orders, and score at least 80 on new-restaurant and new-cuisine visits too, not whatever that slice happens to land on. And a miss that ignores something a person actually told the app, like a spice preference, should count for more than a miss that just picked a close substitute."
Why this works
Matches deliverable zero. Saying the actual rewrite, not just "test it more," shows you can fix a spec, not only poke holes in one.
4
Rule out a broken benchmark before you call it blind
Say it like this
"Before I say Recall-5 is testing the wrong thing, I'd check whether it's even scoring consistently. I'd have two people hand-check two hundred of the held-out orders and see if they land near Recall-5's own numbers. Dewi's team did exactly this: 197 out of 200 agreed. So the scoring itself wasn't the problem."
Why this works
Skipping this is the easy mistake. Calling a number wrong before checking it's even reliable means you might be fixing the wrong thing.
5
Recut the blended score by what kind of visit it actually was
Say it like this
"Split it by session type and the 84 falls apart. Repeat-order visits, sixty-eight percent of traffic, score 93, and only fourteen percent of those sessions end with someone opening search instead. First-time, new-cuisine visits, thirty-two percent of traffic, score 65, and sixty-eight percent of those sessions end in a manual search. One blended number hid a segment that was already failing its own bar."
Why this works
A thirty-point gap by segment is the whole argument, in two numbers, with no adjective doing the work.
6
Show where the gap actually cost something
Say it like this
"Here's the case that made it real. Suppr onboarded forty new Ethiopian and Eritrean restaurants and ran a push telling people to try something new. The strip kept recommending each restaurant's single best-selling combo to everyone, spice preference and all. One-star reviews mentioning the recommendation strip went from about two a week to twenty-six, in the first week of the push."
Why this works
One real spike with a number attached beats a general warning about "narrow benchmarks."
7
Name the three ways the number can lie, and the one check that nails it
Say it like this
"Three things are going on. One, Recall-5's fifty thousand examples are eighty-five percent repeat orders, so it barely tests the case that broke. Two, it counts every miss the same, whether it's a close substitute or a recommendation that ignored a stated preference outright. Three, it only ever tests one job: will the exact dish someone reorders show up in the top five. It has no idea whether the model can handle a first visit at all. The check that nails it: how often the top five for a session is just the restaurant's overall most-ordered dishes, no matter who's asking. Ninety-one percent of the time on new visits. Nineteen percent of the time on repeat visits, where the model actually had something personal to work with."
Why this works
Naming three checkable causes, then one test that separates them, is the strongest move TRACE has. A vague "the benchmark's too narrow" doesn't survive a follow-up question.
8
Say what you'd leave alone, then close
Say it like this
"I wouldn't touch the bar for repeat orders. It's already doing its job there, ninety-three and fourteen percent, and adding review weight to the part that already works just slows down two out of every three visits. So: keep the number, but make it answer for more than one job. A score with one benchmark behind it will always look finished, right up until the visit it was never built to handle."
Why this works
Ending on what stays, not just what changes, shows judgment instead of blanket distrust of every benchmark in the document.
If you remember one thing
Stage 4 and stage 7 are what's being graded. Rule out your own measurement first. Then turn the suspicion that a benchmark is too narrow into a number by comparing what it actually tests against what real traffic is. A single accuracy score will always look finished, right up until the job it was quietly never asked to do.
Let's learn
Say we build a strip on a food app, Suppr. Open any restaurant on it and five suggested dishes show up above the menu, before you've typed anything into search.
Before the model got any good at this, the strip was mostly noise. People ignored it and searched the menu themselves. A year ago, Dewi Santoso, who owns the eval process for Suppr's recommendations, wrote the whole acceptance bar as one line: the model has to score at least 80 out of 100 on Recall-5, a benchmark built from fifty thousand real past orders, before an update ships. For nine weeks the team tuned against that one number. It climbed from 61 to 84.
Two lines, nine weeks of tuning
What the dashboard showed: Recall-5 score
Quiet signal: share of menu visits that end in a manual search instead
wk 1wk 2wk 3wk 4wk 5wk 6wk 7wk 8wk 9
Recall-5 climbed twenty-three points in nine weeks. The share of menu visits that ended with someone searching instead of tapping a recommendation never left a four-point band. A score that measures the job should move that number. This one didn't.
Here is the turn. Eighty-four out of a hundred sounds like the strip got smart. It didn't get smart everywhere. It got very good at one job and stayed exactly as blind as before at another, and nobody had written down which job the score was actually for.
The model didn't get better at recommending. It got better at remembering what you'd already ordered.
Knowledge spark: what a benchmark is actually testing
A benchmark only tells you about the slice of real work it was built from. If most of a food app's order history is people going back to the same dish, a benchmark pulled from that history mostly teaches, and mostly rewards, remembering. It was never asked to test a stranger's first visit.
The cost showed up the week the strip shipped at 84. Suppr onboarded forty new Ethiopian and Eritrean restaurants across the city and ran a marketing push, Try Something New, sending thousands of people to a cuisine they'd never ordered before. The strip recommended the same five dishes, the restaurant's five best-sellers, to every single person who opened the menu, no matter what anyone had told the app about how mild they wanted their food. One-star reviews mentioning the strip went from about two a week to twenty-six.
The gap between the mark that looked like success and the mark that wasn't
The choice I would take back. A year ago, writing "score at least 80" as the whole bar was the sensible call. Recall-5 was new, the score was in the 60s, and there was plenty of obvious room to climb before anyone needed to ask what the number was actually testing. I would take it back and split the bar the day the criterion was written: one number for repeat orders, a separate number for new-restaurant and new-cuisine visits, checked and re-checked as the app grows into new cities.
What I would write instead
Score at least 80 on Recall-5's repeat-order slice, and at least 80 on the new-visit slice too, not 65. Weight a miss that ignores a stated preference outright at double a miss that just picked a close substitute. Recheck the benchmark's own repeat-to-discovery mix every time a new city or cuisine launches.
What I would leave alone. The repeat-order bar doesn't need touching. Sixty-eight percent of all menu visits, Recall-5 scores 93 there, and only twelve out of every hundred of those sessions end with a manual search. The benchmark and real life already agree on that slice. Adding a second layer of review there would just slow down the two-thirds of visits Suppr already gets right.
The lesson. A score that climbs for nine straight weeks is not proof of anything by itself. I wrote a bar with one number on it and no job attached. A number with no job attached will always look finished, right up until the visit it was never built to handle.
The week the new cuisines landed
Dewi Santoso has run eval for Suppr's recommendation team for three years. She can look at a dashboard and tell, inside a minute, whether a climbing score is a real gain or a lucky run of tuning.
She built Recall-5 herself: fifty thousand real orders pulled out of Suppr's history, with a simple rule behind it. Hide the dish a person ordered next. Show the model everything that came before it. Check whether the real dish lands anywhere in the model's top five guesses. Score at least 80, or the update doesn't ship.
For most of that year, watching the score climb felt like watching the right thing happen. It moved a point or two a sprint, and every point felt earned. Somewhere in there, without anyone deciding it on purpose, the team quietly stopped doing something it used to do every week: a growth associate would open fifteen newly onboarded restaurants by hand and check whether the five suggested dishes made sense for a total stranger walking in cold. Once Recall-5 was closing in on 80, that check started to feel like double work. Dewi signed off on dropping it the same week the score first crossed the bar.
Nine weeks after tuning began, the score hit 84 and the update shipped. The next morning, Suppr's marketing team launched Try Something New, timed to land alongside forty newly onboarded Ethiopian and Eritrean restaurants going live across the city. Thousands of people opened a cuisine they had never ordered from before, that same week.
The strip did what it always did. For every one of those new restaurants, it showed the same five dishes to everyone who opened it: the restaurant's five best-selling items, spice level and all, no matter what anyone had told the app about how mild they wanted their food.
By Thursday, one-star reviews mentioning the recommendation strip had gone from about two a week to twenty-six. Support tickets kept asking some version of the same question: why does this app keep suggesting the exact same thing to everyone?
We didn't lose people over one bad guess. We lost them over the same guess, repeated at every new restaurant, to every stranger who walked in.
I want to say the problem was that Recall-5 was too easy. It wasn't easy, exactly. It was answering a question nobody was asking that week. Most of Suppr's fifty thousand held-out orders were reorders, people going back to a place they already knew. So the benchmark got very good at rewarding a model for remembering. It never had a reason to reward a model for guessing well about someone it had never seen before.
So here is the decision I would take back.
A year earlier, when we wrote the bar, saying "score at least 80" as the whole rule was the sensible call. Recall-5 was brand new, the score sat in the 60s, and there was plenty of obvious room to climb before anyone needed to ask what the number was testing.
I would put back the split. Score the repeat-order slice and the new-visit slice separately, and never let one blended number cover for the other. Not because the split is interesting on its own, but because of what it would have shown in week seven: repeat orders at 93, new visits at 65, still nineteen points under the bar, still shipping, because the average looked fine.
That is the whole difference. One design hands you a single number. The other hands you the one number that was actually in trouble.
And the part I'd want to tell myself, if I could go back: we built a bar that only ever answered for the job that was already easy. Nobody decided that on purpose. We just never wrote down which job the score was for.
What the split actually showed
Before blaming Recall-5's coverage, Dewi's team checked whether it was even scoring consistently. Two people hand-checked two hundred of the held-out orders and landed within a few points of Recall-5's own numbers on 197 of them. The scoring itself wasn't broken. That left the coverage.
Score by session type, next to what people actually did about it
93%
14%
65%
68%
Repeat-order sessions 68% of menu visits
New-cuisine, first-visit sessions 32% of menu visits
Recall-5 score
Share of sessions that end in a manual search
Recall-5 barely resembles the same test on the two segments, 93 against 65. Real behavior tells them apart completely: a 14 percent search rate on repeat orders, a 68 percent search rate on a first visit. A blended average of 84 hides both numbers, and hides that the discovery slice never even cleared its own bar.
The result strip below is the check that turned the suspicion into a number: how often a session's top-five list was really personal, versus how often it was just whatever was already popular at that restaurant, handed to anyone who asked.
Shipping on the blended score alone
84 on Recall-5, nine weeks of tuning
91 percent of new-visit top-five lists were just the restaurant's raw popularity ranking, not a personal guess
Requiring the new-visit slice to clear 80 too
A bar that has to pass segment by segment
0 prompt updates have shipped since without that slice passing
Three ways an 84 can lie
Not because anyone cut a corner on purpose. A criterion that names one score and nothing else can be met honestly, sprint after sprint, and still stop meaning anything.
Three separate, checkable ways to hit a score without meeting its point
Way 1
Narrow. The benchmark barely tests the visit that broke.
Recall-5's fifty thousand orders were pulled from Suppr's order history, which skews toward people going back to a place they know. Eighty-five percent of the benchmark's own examples are repeat orders, against sixty-eight percent in real current traffic, and the gap gets wider the moment a new city or cuisine launches.
How you'd check it: pull the repeat-versus-discovery mix inside the benchmark's own example set and compare it to real traffic. Eighty-five against sixty-eight is a real gap.
Way 2
Blind to severity. A small swap scores the same as a broken preference.
Of the misses on new-visit sessions, thirty-nine percent were a close substitute a person probably wouldn't mind, twenty-six percent ignored something the person had actually told the app, like a spice preference, and nine percent recommended a dish that was out of stock at that hour. Recall-5 counts all three as an identical zero.
How you'd check it: for a sample of misses, sort them by what actually went wrong, not just that something did, and see if the "bad" ones and the "barely matters" ones are landing on the same score.
Way 3
Wrong task. It only ever tests reorders.
Recall-5 asks one question: will the exact dish a person orders next show up in the top five. That's a reorder-prediction task. A menu visit is really at least three different jobs: reordering a favorite, browsing an unfamiliar cuisine for the first time, and following a stated preference. The benchmark has no signal at all for the second and third.
How you'd check it: for a sample of sessions, check whether the top-five list actually used anything personal to that user, or whether it's identical to the restaurant's raw popularity ranking regardless of who's asking.
TRACE, run on a bar that already looked done
This is a diagnosis question wearing a critique's coat: the real question is why a spec that reads cleanly stopped predicting real quality. GUARD would fit if the harm here were about one group losing a fair shot at something; it's about a number nobody checked against the job it was supposed to stand in for.
T, timeline. The score climbed from 61 to 84 over nine weeks. The share of menu visits that ended in a manual search stayed inside a four-point band the entire time. The two lines should have moved together. Only one of them had a dashboard, and only that one got trusted enough to end a weekly manual check.
R, recut. Split by session type. Repeat orders: Recall-5 score 93, search rate 14 percent. New-cuisine, first-visit sessions: Recall-5 score 65, search rate 68 percent. A blended 84 hid a real result on the job that was already easy and a real result on the job that always mattered more.
A, assume nothing. Before blaming the benchmark's coverage, Dewi's team checked its own scoring: two people hand-checked two hundred held-out orders and landed close to Recall-5's own numbers on 197 of them. The scoring itself was fine. The gap was real.
C, cause candidates. Three, named and separate: the held-out set is eighty-five percent repeat orders, it scores every miss the same whether it's a close substitute or a broken preference, and it only ever tests one job, whether the exact reorder lands in the top five, with nothing to say about a first visit.
E, evidence test. Check how often a session's top-five list matched the restaurant's raw popularity ranking, no matter who was asking. Ninety-one percent of the time on new visits. Nineteen percent of the time on repeat visits, where the model had something personal to draw on. The benchmark rewards popularity because that's mostly what it was ever built from.
Why E is the hard step
Anyone can suspect a single benchmark is too narrow. A test earns its place by turning that suspicion into a number: how often the model's answer is just the laziest guess it could make without looking at who's actually asking. Do that comparison and you've checked something real. Call a benchmark "probably not representative" without it, and you've only said the same worry in a more confident voice.
Same shape, a clinic that only tested the easy visit
Alderbrook Veterinary Partners runs symptom triage across nine clinics. Its AI reads what a pet owner types into the check-in kiosk and ranks the three most likely reasons for the visit, so the front desk can route to the right vet. Kwame Boateng, the chain's clinical ops lead, wrote the launch bar two years ago: the model has to score at least 85 on CauseCheck, a benchmark built from past visit records, before an update ships.
T. CauseCheck's score climbed from 74 to 88 over eight weeks of tuning. The rate at which a vet tech overrides the suggested category before the vet even sees the patient never moved off roughly one in three, the whole time. R. Split by visit type. Routine and wellness visits, seventy-one percent of traffic: CauseCheck score 95, override rate 9 percent. Acute, unusual-symptom visits, twenty-nine percent: CauseCheck score 71, override rate 61 percent. A. Before blaming coverage, Kwame has two vet techs blind re-triage forty cases against CauseCheck's own numbers. They agree on all but one. The scoring itself checks out. C. Three candidates: CauseCheck's reference visits are eighty-nine percent routine, the score never weights a missed urgent cause any higher than a re-ordered ranking, and the rubric only really rewards a clear, textbook symptom pattern, which acute cases rarely are. E. Compare how often the top-three suggestions for an acute case were just the clinic's three most common visit reasons, regardless of the symptoms typed in. Eighty-four percent of the time on acute cases. Twenty-two percent of the time on routine ones. The benchmark barely tests the visit type a vet tech actually needs help with.
Swap the trigger and it still runs
Speed: instead of nine weeks of steady tuning, the trigger is a rushed model swap approved in three days before a big weekend promotion. TRACE still starts with what the number was ever built to predict, not with how fast the swap got signed off.
Cost: the team shrinks Recall-5's held-out set to cut nightly compute, on the idea a smaller sample is close enough. The recut still has to show which kind of visit got cut, not just how many orders.
The model really did get better: the case on this page. The ranking genuinely improved at guessing repeat orders. The benchmark just never had a task that could tell that real gain apart from a fake one on the visits it never tested.
Where people run it wrong
Treating "we cleared 80" as proof the feature works everywhere it runs, instead of asking which job the 80 was actually built from.
Reading nine weeks of climbing accuracy as the product getting better everywhere, when a fixed benchmark, tuned against long enough, can climb for reasons that have nothing to do with the harder job.
Writing the bar once at launch and never rechecking it as new cities, cuisines, or visit types get added to what the model actually has to handle.
If you are asked this cold
Buy yourself ten seconds by naming the split out loud. "So there's the one number the spec asks for, and there's however many different jobs the model's actually doing. A single number can hide one of those jobs failing completely. Let me say how I'd check which job that is." That's not stalling. That's where the real critique starts.
Flashcards (click a card to flip it)
This is a diagnosis question about a written spec, not a habit changing, so these eight test the TRACE moves and the real numbers behind them.
1 · THE FRAMEWORK
Which framework fits critiquing a spec that names one accuracy number and nothing else, and why?
Tap to flip
ANSWER
TRACE. The real task is a diagnosis wearing a critique's clothes: working out why a number that reads fine keeps passing while a whole kind of visit quietly fails. GUARD would fit a fairness harm; this is a coverage gap.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Dewi Santoso, eval and PM lead for Suppr's recommendation strip. She built the Recall-5 benchmark and wrote its 80-point bar herself, a year before the gap showed up.
3 · THE HABIT
What did the team stop doing once Recall-5 crossed 80?
Tap to flip
ANSWER
A weekly manual check: a growth associate hand-reviewing the five suggested dishes on newly onboarded restaurants. It started to feel like double work once the dashboard looked healthy.
4 · THE THREE WAYS
Name the three ways a single-benchmark accuracy number can be met honestly and still miss the point.
Tap to flip
ANSWER
Narrow (mostly repeat orders), blind to severity (a small swap scores the same as ignoring a stated preference), and the wrong task (only ever tests reorders, never a first visit).
5 · THE NUMBER
Recall-5 scored 93 on repeat-order sessions and only ______ on new-cuisine, first-visit sessions, even though the blended score of 84 cleared the bar.
Tap to flip
ANSWER
65. Nineteen points under the 80 bar, hidden inside an average that looked fine.
6 · THE CHECK
Name the one test that turned a suspicion into a number.
Tap to flip
ANSWER
Checking how often a session's top-five list matched the restaurant's raw popularity ranking, no matter who was asking. 91 percent of new-visit sessions got the identical generic list; only 19 percent of repeat sessions did.
7 · THE FIX
What does the rewritten criterion say that the old one-line bar didn't?
Tap to flip
ANSWER
It requires 80 on the new-visit slice too, not just the blended average, weights a miss that breaks a stated preference above a near-miss, and rechecks the benchmark's own mix whenever a new city or cuisine launches.
8 · CROSS-PRODUCT TRANSFER
Section 4 runs TRACE on a different product. Which one, and what's the number?
Tap to flip
ANSWER
Alderbrook Veterinary Partners' symptom-triage tool. CauseCheck scored 95 on routine visits but only 71 on acute ones, and 84 percent of its acute-case suggestions were just the clinic's three most common visit reasons, regardless of the symptoms typed in.
Check yourself Score: 0 / 0
Multiple choice
1. Dewi's criterion says the model must score at least 80 on Recall-5. What's the actual problem with the line, as written?
A. Eighty is too low a bar for a recommendation model to hit reliably.
B. It never says which job the score is testing, so a segment that's still failing can hide inside a passing average.
C. Recall-5 should be thrown out and replaced with a completely different benchmark.
D. The bar should be raised until new visitors stop searching manually.
Show hint
Look at what the line says, and what it never names, about which job the score covers.
Show answer
B. The line names a score with no stated job and no check on whether it covers the harder kind of visit. That's the actual gap, not the number itself.
Fill in the blank
2. Recall-5 scored ______ on new-cuisine, first-visit sessions, well under the bar, even though the blended score of 84 cleared it.
Show hint
Look at the second bar group in the segment chart.
Show answer
65. Nineteen points under the 80 bar, sitting quietly inside an average that looked fine.
True or false
3. True or false: once the two analysts confirmed Recall-5's automated score matched their manual check, that proved the 80-point bar was testing the right thing.
True
False
Show hint
Confirming the scoring agrees with itself only rules out one kind of problem.
Show answer
False. That check (the A step) only ruled out a broken measurement. It took the segment recut and the popularity-ranking check (R and E) to show the benchmark was testing the wrong mix of visits.
Short answer
4. Name a place in Suppr's process where you'd leave the current Recall-5 bar exactly as it is, and say why.
Show hint
Think about the session type where the benchmark and real behavior already agree.
Show answer
Model answer: "Keep it exactly as is for repeat-order sessions, sixty-eight percent of menu visits. There, Recall-5's 93 and the real 14 percent search rate already agree. A second review layer there would just slow down the majority of visits that already work."
Short answer, apply it yourself
5. Think of a pass or fail bar you rely on somewhere, a hiring rubric, a test suite, a certification score. What's one way that bar could hit its number while still missing the real thing it's supposed to stand in for?
Show hint
Look for a case where the measured thing is common and easy, while the thing that actually matters is rare and hard.
Show answer
Model answer: "A support team's average response time can look great while a canned, unhelpful reply gets sent fast to close the timer, and the customer's actual problem never gets solved." Any honest answer works if it names a real case where the measured thing and the real thing quietly came apart.
Multiple choice
6. A teammate says the real fix is simpler: just test Recall-5 on twice as many held-out orders. Why doesn't that fix what's wrong with the criterion?
A. Because doubling the set would take too long to compute every night.
B. Because more examples pulled from the same order history would still be mostly repeat orders, so the new-visit case stays just as untested.
C. Because Recall-5's scoring already disagreed with the manual check, so no size fixes it.
D. Because associates would stop trusting the dashboard entirely.
Show hint
One of these treats the count as the problem. The rest of the answer says the mix is the problem.
Show answer
B. More of the same skewed history doesn't add coverage. It just makes the narrow slice bigger while the new-visit case stays just as thin.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
