CaseAdvancedEval-Driven Specification / Acceptance criteria for non-deterministic output / #7
How would you write criteria for a feature where the worst case matters more than the average?
The direct answer
Write the release bar around the rare, dangerous mistake first, not the pooled average. Require near-perfect recall on the one class of miss that can actually hurt someone, measured on a set built to contain enough real examples of it, and gate release on that number before an overall accuracy figure gets reported at all. An average blends the common, cheap mistakes with the rare, expensive one, and the rare one always loses that blend.
How to write the criteria, in order
Set the release bar on the rare, dangerous class first, not the pooled average.Why: an average blends five thousand easy cases against one hard one, and the hard one always loses.
Build the test set by stacking it with real examples of the dangerous case, not sampling it naturally.Why: at a natural rate of 1 in 5,000, a random sample barely has enough of them to measure recall on at all.
Route every flagged item in the dangerous class to a person regardless of score, until the bar is actually met.Why: that is exactly where the hidden mistake costs $46,000 and keeps growing, not $6 and a shrug.
Say the cost asymmetry out loud in real numbers, not in feelings.Why: "$6 against $46,000" is a number someone can argue with. "We should be careful" is not.
Track the dangerous class's recall against the overall accuracy line every quarter, not just once at launch.Why: 71 percent hid inside a 97.4 percent average for months before anyone measured it on its own.
Leave the minor-defect criteria on the average metric, untouched.Why: getting corrosion and paint chips slightly wrong is cheap, and tightening that bar only slows the shop down for no safety gain.
Seven moves for walking the criteria out loud
This is a tradeoff wearing a spec's clothes, so PICK is the tool, not a checklist of rubric fields.
1
Scope it to one concrete criteria decision
Say it like this
"Let's say I'm writing the release bar for AeroSight, the tool that looks at a photo of an engine blade and flags a possible crack for an inspector to check. That's the feature. That's the criteria I'm going to write."
Why this works
Stops the answer floating at "AI evals in general" and gives the interviewer something concrete to push on.
2
Say your structure out loud
Say it like this
"I'm going to pick a position first, say who feels each kind of mistake and in what units, name why one kind is worse than the other, and then say what would change my mind. Four moves, in that order."
Why this works
Signals a method instead of a meandering answer, and tells the interviewer what's coming before you start.
3
Reframe what the question is really testing
Say it like this
"This isn't really 'write me a rubric.' It's 'do you know an average can hide the one mistake that actually matters.' So I'm not going to start with a number like 97 percent overall accuracy. I'm going to start with the one class of mistake that can hurt someone."
Why this works
Shows the interviewer you see past the surface ask to the real judgment being tested.
4
Give the position, with the actual mechanism in it
Say it like this
"My position: before AeroSight ships, it has to hit at least 99.5 percent recall on the safety-critical class, fan-blade root cracks and wing-spar cracks, tested on a set we deliberately stack with real examples of that class. Overall accuracy still gets reported, but it never gates the release on its own."
Why this works
PICK rewards committing to a real number and a real mechanism, not a vague promise to "be careful."
5
Name who feels each mistake, then prove the asymmetry with a real case
Say it like this
"Here's the split. A false alarm on a corroded rivet costs an inspector about four minutes to rule out, six dollars of shop time, and nobody outside the hangar ever hears about it. A missed root crack doesn't get caught until the next heavy check, eighteen months later, and by then you're looking at an unscheduled blade swap and a grounded aircraft, about forty-six thousand dollars, or it just keeps growing. We actually shipped a version that scored 97.4 percent overall and only caught 71 percent of real cracks, because cracks are one photo in five thousand, and an inspector caught the one that slipped through on a hunch, not because the tool told him to look."
Why this works
The real numbers and the real near miss make the asymmetry checkable, not just asserted.
6
Name what would change your mind
Say it like this
"I'd drop the special bar the day the crack-class recall, measured every quarter, catches up to the overall accuracy line. If the model stops having a blind spot on the rare class, there's nothing left for an average to hide, and one bar works fine for everything."
Why this works
Shows confidence without stubbornness. The interviewer sees you'd change course given real evidence, not just hold a position.
7
Close on the one line and what you'd leave alone
Say it like this
"So: gate release on crack recall, not on overall accuracy, and leave the corrosion and paint-chip criteria exactly where they are, because getting those slightly wrong costs six dollars, not an aircraft."
Why this works
Ends on the one line the interviewer should walk away remembering, and shows judgment by naming what does not need fixing.
One more thing before the walkthrough moves on: this bar covers one class of defect, not the whole spec. Most candidates hear "worst case matters more" and answer by tightening everything. Say which class gets the strict bar and which one stays on the average, and you've answered a harder, better question than the one that was technically asked.
Let's learn
AeroSight is a tool that looks at a photo of an aircraft part, an engine blade, a wing panel, a landing gear strut, and tells an inspector whether it might have a problem worth a closer look.
One photo, one score, one person who has the last word
Knowledge spark: what does recall actually mean here?
Of all the real cracks that were really there, how many did the tool actually catch. Not how many photos it looked at correctly overall, just the ones that were genuinely a crack. A tool can miss almost a third of real cracks and still look fine on paper if cracks are rare enough in the mix.
Before AeroSight, a technician reviewed every one of the roughly 1,200 inspection photos taken across the fleet each day, by hand, about 45 minutes of dedicated photo review on top of the physical inspection itself.
With AeroSight sorting the photos and only flagging the ones worth a second look, about 40 a day, that review time fell to about 12 minutes a shift.
Here is the turn. That 12-minute number was never the risk. What was hiding inside it was. The team had set the release bar as one number: overall defect-detection accuracy above 97 percent, pooled across every kind of defect, corrosion, paint chips, dents, and cracks, all averaged together. AeroSight cleared it at 97.4 percent. Nobody asked, separately, how good it was at the one defect that can actually bring a plane down.
We did not build a bar for how good it looks on average. We built one for the one photo it cannot afford to miss.
At its worst, that looks like this: a root crack in a fan blade reads clean in the photo, AeroSight scores it below the flag line, and nobody looks at it again until the next heavy check, eighteen months and roughly 2,200 flight cycles later. Or, in the version nobody wants to say out loud, it does not get caught then either, and the crack keeps growing.
The choice I would take back
Early on, the team picked one release number, 97 percent pooled accuracy, because it was simple to explain and simple to gate a launch on. I would take that back. The training and eval corpus behind AeroSight has one confirmed crack photo for every 5,000 corrosion and paint-chip photos. Any single number that averages across all of them is really just a corrosion score wearing a crack score's name.
Cost per event, in dollars
Caught the same shift, cheap
Not caught for 18 months, expensive
AeroSight wrongly flags a fine rivet or a paint chip
$6
AeroSight wrongly clears a fan-blade root crack
$46,000
The cheap bar is barely there on purpose. $6 covers four minutes of a technician's time to rule out a false alarm, folded straight into the shift. $46,000 covers an unscheduled blade swap plus the aircraft sitting grounded while it happens, once the crack is finally caught at the next heavy check. Roughly 7,600 times the cost, and it is the one that does not show up until months later.
What I would leave alone. The criteria for corrosion, paint chips, and minor dents within limits. Those are common, cheap to get slightly wrong, and a false alarm there costs a technician a few minutes, nothing more. Tightening that bar to 99.5 percent too would only slow the shop down without making anyone safer.
The lesson. We wrote the criteria to answer "is this tool accurate," and it answered that well. We never wrote a separate bar for "is it accurate on the one class that can hurt someone." Those are two different questions, and only the second one has a person's safety riding on it.
The blade Tenzin checked out of habit
You don't need this to answer the question. Read it if you want to feel why the bar has to sit on the rare class, not the average.
Tenzin Norbu can run a fingertip along a blade root and tell you, before he's finished the pass, whether it's worth a second look. He's been a certified airframe inspector at Whitcombe Airlines' maintenance hangar for eleven years, and every part that goes back on an aircraft has his sign-off on it.
AeroSight arrived on the hangar floor the spring before last, and for months it was just good. A photo scored in under a second, flagged or clear, and the flagged ones were almost always right. He'd still run his own borescope check on every root, out of habit, alongside the tool. It agreed with him. Every time.
So he started running the manual check on half the roots. Then on the ones AeroSight flagged only. By late summer, if the screen said clear, he moved on.
Same tool, two very different kinds of wrong
Then, one Wednesday, he was signing off a fan blade AeroSight had scored clear. Nothing about the screen looked wrong. But he paused with the borescope still in his hand, because the light had caught the root at an angle that felt, for half a second, off. He ran the manual check anyway, the way he used to on every blade. There was a hairline crack at the root, real, and past the point where you let it fly again.
He hadn't caught it because the tool told him to look. He'd caught it because eleven years of habit hadn't fully left him yet, and he happened to still have the tool in his hand that morning.
We did not just miss one crack. We built a bar that could not tell us we were missing them, because the number it reported was mostly a corrosion score.
Desta Alemu had owned the release criteria for AeroSight since before it shipped. When Tenzin's near miss reached her, the header number still looked fine. Pooled accuracy, checked that same week, read 97.4 percent. She almost logged it as one unlucky photo.
Then she pulled the corpus the launch decision had been made against: 600,000 inspection images, 550,000 with no defect at all, 49,880 with a minor defect, corrosion, paint chips, allowable dents, and 120, total, with a confirmed safety-critical crack. One in roughly 5,000. She reran the recall number on those 120 alone. AeroSight had caught 85 of them. Seventy-one percent.
Seventy-one percent had been sitting inside a 97.4 percent average the entire time, because 120 crack photos can barely move a number built from 600,000.
So Desta split the bar in two. Before any pooled accuracy number counts for anything, AeroSight has to clear 99.5 percent recall on a test set built to actually contain enough confirmed cracks to measure it, stacked well past the natural 1-in-5,000 rate. Corrosion and paint chips stay on the old pooled bar, since getting those slightly wrong has never cost anyone anything but four minutes.
The part she'd take back isn't the launch. It's picking one number, simple to explain, simple to gate a release on, and never asking whether that one number could actually see the mistake that mattered. A number that good on 600,000 photos hid exactly the class it needed to prove itself on.
Tenzin's flagged-photo review is back to about 15 minutes a shift, three minutes more than the old average-based bar, because the new one flags a few extra borderline corrosion spots on purpose rather than risk waving off a borderline crack. And the crack he caught on a hunch that Wednesday can't slip through silently the same way again, because nothing scoring below the new bar on a root photo reaches him marked as done.
PICK, boiled down to four lines
This is a tradeoff dressed as a spec, so PICK is the tool. A "how would you measure whether this feature is working" question would reach for LEAD instead.
P, position. Gate release on the safety-critical defect class first, recall at or above 99.5 percent on a set stacked with real examples. Overall pooled accuracy is reported alongside it, but it never gates the release by itself.
I, impact. A technician wrongly told to double-check a fine rivet loses about four minutes, six dollars, absorbed into the shift, nobody outside the hangar notices. A missed root crack reaches the aircraft's flight schedule for up to 18 months and 2,200 cycles before the next heavy check would even see it again, and the cost, roughly $46,000 plus a grounded aircraft, doesn't show up until then.
C, cost asymmetry. The corpus behind the model holds one confirmed crack photo for every 5,000 minor-defect photos. Any single, pooled number will always be dominated by how well the tool handles corrosion, never by how well it handles the crack that actually matters. Spend the caution on the class the average can hide, not on the one it already measures well.
K, kill criteria. Drop the special bar the day crack-class recall, tracked every quarter as more confirmed cases come in, catches up to the overall pooled accuracy line. At that point the model has no real blind spot left for an average to hide, and one bar measures the same thing either way.
Knowledge spark: why can't you just raise the overall bar instead?
Because the overall bar is a blend. Raising it from 97 to 99.5 percent barely touches the crack class, since cracks are 1 in 5,000 photos and can't move a pooled average much either way. You'd mostly be squeezing harder on corrosion, the part that was never the problem.
Crack-class recall, measured each quarter, against the kill line
Crack-class recall, on a growing confirmed-crack set
Kill line: matches overall pooled accuracy, 97%
Four straight quarters of retraining closed most of the gap, from 71 percent up to 96 percent, measured against a confirmed-crack set that grew from 120 to 480 images as more real cases came in. It still sits just under the 97 percent line where it would match overall pooled accuracy. Until that line is crossed, the separate bar for the crack class stays exactly where it is.
Run the four letters again, on a stuck elevator cable
Vantage Elevator Group runs a model, CablePulse, that reads vibration and load sensor data from a building's elevator cars and flags anything odd for a mechanic to inspect on the next visit. Same shape of question, a different shaft, a different kind of harm.
P. Gate any release of CablePulse on its recall for one class, an early-stage fatigue fracture in a suspension cable strand, at or above 99.5 percent, on a set stacked with confirmed real fractures. General sensor-anomaly accuracy is reported, but it never gates a release by itself. I. A false alarm on a healthy cable costs a mechanic about 20 minutes to inspect and clear, absorbed into a routine visit. A missed strand fracture isn't caught until the next scheduled service, and by then the building's facilities team is looking at an emergency cable swap, the car out of service for days, and a cost that runs into the tens of thousands. C. Confirmed cable fractures make up a tiny sliver of CablePulse's training data next to routine sensor noise, the same shape of imbalance as AeroSight's cracks. A pooled anomaly-detection number will always be dominated by how well it reads ordinary vibration, never by how well it catches the fracture that actually matters. K. Drop the special bar the day fracture-class recall, tracked quarterly, catches up to the pooled anomaly-detection accuracy. Until then, any car flagged near the fracture threshold gets a mechanic's eyes regardless of score.
What I would leave alone, in the elevator shaft
The routine sensor-noise criteria for things like a slightly uneven door-close time or a motor running a touch warm on a hot day. A false alarm there costs a mechanic a glance and nothing else. No car has ever been hurt by that kind of miss.
Swap the trigger and it still runs
Speed: AeroSight takes three seconds to score a photo instead of one. Doesn't move the bar, because the bar is about what gets missed, not how fast the tool decides.
Cost: running AeroSight gets five times more expensive per photo. Also doesn't move it. A missed root crack costs thousands of times a single scoring call, whatever that call happens to cost.
The model gets better: if crack-class recall genuinely closes the gap to overall accuracy, per the kill criteria, move the bar, don't erase it. The two numbers becoming the same thing is exactly the evidence that would let one bar do the job.
Where people run it wrong
Treating a strong pooled accuracy number as proof of safety, when the rare class was never measured on enough real examples to say anything about it at all.
Fixing it by making the whole model "stricter," which only changes which minor defects get flagged, not whether it catches the crack.
Writing "add a human review step" without saying which photos get it, so the review time ends up spent re-checking easy corrosion cases instead of the ones that actually needed a person.
If you are asked this cold
Say the reframe out loud before you list a single rubric field. "Give me a second, I want to separate what each kind of mistake actually costs before I pick a number." That's true, it's already stage one of the walkthrough, and it buys the time to find the real asymmetry instead of reciting a generic checklist.
Flashcards (click a card to flip it)
1 · THE FRAMEWORK
Which framework fits this question, and what's its hardest step?
Tap to flip
ANSWER
PICK, for a tradeoff. The hardest step is C, the cost asymmetry: naming why a pooled average will always favor the common, cheap mistake over the rare, dangerous one, and building the bar around the dangerous one instead.
2 · THE PERSON
Who owns the release criteria in this answer, and what did she find that the launch number missed?
Tap to flip
ANSWER
Desta Alemu, the product manager who owns AeroSight's release criteria. She found that a 97.4 percent pooled accuracy number was hiding a 71 percent recall on real cracks, because cracks were only 1 in 5,000 photos in the corpus behind the number.
3 · THE HABIT
What did Tenzin Norbu stop doing once AeroSight kept agreeing with him?
Tap to flip
ANSWER
Running his own manual borescope check on every blade root. He moved from checking all of them, to half, to only the ones AeroSight flagged, to trusting a clear screen outright.
4 · THE ASYMMETRY
Name the two kinds of mistake here and what each one costs.
Tap to flip
ANSWER
A false alarm on a fine rivet or paint chip: caught the same shift, about $6. A missed root crack: not caught for about 18 months, about $46,000 once you count the unscheduled blade swap and the grounded aircraft, and it keeps growing if it's missed again.
5 · THE POSITION
State the pick in one sentence, the way you'd say it out loud.
Tap to flip
ANSWER
Gate release on crack-class recall at 99.5 percent or higher, tested on a set stacked with real cracks, and only report overall pooled accuracy after that bar is met, never instead of it.
6 · THE NUMBER
When Tenzin's near miss happened, AeroSight's recall on real cracks alone was ______ percent, hidden inside a 97.4 percent pooled score.
Tap to flip
ANSWER
71. Eighty-five of 120 confirmed crack photos caught, missed by more than a quarter, and none of that showed up in the one number the launch decision was made against.
7 · THE KILL CRITERIA
What would make you drop the special bar and go back to one pooled number?
Tap to flip
ANSWER
The day crack-class recall, tracked every quarter, catches up to the overall pooled accuracy line. At that point the model has no blind spot left on the rare class for an average to hide, and one bar would measure the same thing either way.
8 · THE TRANSFER
Section 4 runs PICK again on a different product. Which one, and where's the strict bar there?
Tap to flip
ANSWER
Vantage Elevator Group's CablePulse, reading sensor data for elevator cars. The strict bar sits on recall for early-stage cable-strand fractures, not on general sensor-anomaly accuracy.
Check yourself Score: 0 / 0
Multiple choice
1. Which of these is the actual mechanism behind this answer's criteria bar?
A. Make AeroSight score every photo faster so inspectors see flags sooner.
B. Gate release on recall for the safety-critical defect class alone, measured on a set stacked with real examples, before any pooled accuracy number counts.
C. Require every inspector to double-check every photo by hand, regardless of what AeroSight says.
D. Raise the overall pooled accuracy bar from 97 to 99.5 percent.
Show hint
Three of these either don't touch the rare class at all, or throw away the time the tool was built to save.
Show answer
B. A wastes effort on the wrong problem, C undoes the whole point of the tool, and D barely moves the crack-class number since cracks are 1 in 5,000 photos in a pooled average. Only B actually targets the mistake that matters.
True or false
2. True or false: once AeroSight clears 97 percent pooled accuracy across all defect types, that alone is enough to ship it.
True
False
Show hint
Think about what 97.4 percent pooled accuracy was actually hiding at launch.
Show answer
False. That exact number, 97.4 percent, shipped alongside a 71 percent recall on real cracks. Pooled accuracy alone was never enough, and the answer's whole position is that the crack-class bar has to be checked and cleared separately.
Fill in the blank
3. Fill in the blank: the confirmed-crack corpus held about 1 crack photo for every ______ minor-defect and clean photos, which is why a pooled number couldn't see the crack-class problem.
Show hint
120 confirmed crack photos, out of a 600,000-image corpus.
Show answer
5,000. 120 crack photos out of 600,000 total images is roughly 1 in 5,000, small enough that it barely moves a pooled average no matter how badly the model handles it.
Multiple choice
4. Why couldn't the team just raise the overall accuracy bar to 99.5 percent instead of writing a separate crack-class bar?
A. Raising the bar would take too long to retrain for.
B. A pooled bar is dominated by the common defect classes, so raising it mostly squeezes harder on corrosion and barely touches crack-class recall.
C. Inspectors refused to work under a stricter number.
D. Regulators require exactly 97 percent, no more and no less.
Show hint
Ask what a "stricter" version of the same pooled number would actually be measuring more of.
Show answer
B. Turning the pooled dial up just makes the model better at the classes that already dominate the average, corrosion and paint chips. It never forces the model to prove anything about a class that's only 1 in 5,000 photos. The fix has to measure that class on its own, not tighten the blend it's hiding inside.
Short answer
5. If confirmed critical defects were 1 in 50,000 photos instead of 1 in 5,000, would splitting the criteria still make sense? Walk through it.
Show hint
Think about how much harder a pooled number gets to move as the rare class gets even rarer.
Show answer
Yes, even more so. At 1 in 5,000, cracks already barely moved a pooled average. At 1 in 50,000, a model could get every single crack wrong and the pooled number would hardly change. The rarer the dangerous case, the more completely an average metric hides it, which makes the case for a separate, class-specific bar stronger, never weaker.
Short answer, apply it yourself
6. Pick a place in your own work where one overall score covers several different kinds of mistake. Name the rare, expensive one hiding inside it, and what number you'd pull out to gate on separately.
Show hint
Look for the mistake type that's both the rarest in your data and the one you'd least want to explain to your manager after the fact.
Show answer
Model answer: "Our support ticket router scores 95 percent overall accuracy across ticket types. Buried in that average is billing disputes over $1,000, maybe 1 in 2,000 tickets, where a wrong routing means a refund deadline gets missed. I'd pull recall on that one category out as its own gate, tested on a set stacked with real high-value disputes, and never let the 95 percent overall number stand in for it." Any answer works if you can name the rare category the overall score is currently hiding.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
