Artifact critiqueAdvancedEval-Driven Specification / Writing an eval spec / #11
Write the pass criteria for a feature where partial credit is meaningful.
The direct answer
Score each item in the rubric on its own, not the total. Make the one item that shows real understanding a hard gate: any mismatch there fails the case outright, no matter what else lines up. Give every other item a small tolerance, because a one-point gap there is a judgment call, not a mistake.
Do this, in order
Score every rubric item on its own, and let the item that shows real understanding gate the whole case.Why: a flat score can hide one dangerous miss behind a pile of harmless near misses, and that's exactly what let a wrong grade sit unread for weeks.
Give every other item a one-point tolerance instead of demanding an exact match.Why: without it, honest judgment calls get treated the same as real mistakes, and the fail list grows too long for anyone to actually read.
Write a real low example and a real high example for the gate item, not just a label.Why: "shows understanding" with nothing under it drifts back to whatever a reviewer happens to feel that week.
Keep the gate to the one item that actually decides understanding, not every item.Why: gate everything and the fail list floods again, the same problem the tolerance was built to fix.
Leave every other partial-credit combination for later.Why: scoring every possible mix of right and wrong on day one delays shipping the one check that would have actually caught the miss.
Re-run the eval after any change and read the whole fail list, not a sample.Why: a fail list short enough to read in full is the entire point of building the gate.
How to answer this, stage by stage
Seven moves. Ground it in one flagged answer, before touching the criteria itself.
1
Scope it to one flagged answer, not "grading quality" in general
Say it like this
"Say I'm the PM for Inkline, the short-answer grader inside Cobalt Campus's quizzes. I'm not going to write pass criteria for grading quality in general. I'll build it against the personal-finance question about credit card minimum payments, since that's the one where a wrong-but-confident grade already slipped through."
Why this works
Criteria built for one real question is something an interviewer can picture. Criteria built for "quality" never gets specific enough to argue with.
2
Say your structure out loud
Say it like this
"Here's how I'll walk through it: what the old pass bar actually checked, what a pass bar for this kind of feature really has to settle, the criteria itself, what breaks if it's too loose or too strict, and what I'd leave out of it for now."
Why this works
Two seconds of structure tells the interviewer you have a plan, so they're following your answer instead of guessing where it's headed.
3
Reframe what the pass criteria actually have to settle
Say it like this
"Most people hear 'write pass criteria' and go looking for the right score threshold. The real gap is that an exact-match bar treats a defensible one-point gap and a five-point lie exactly the same. It can only tell you two scores don't match. It can't tell you which mismatch is dangerous."
Why this works
This is the actual insight being tested. Skip it and you've written a threshold, not criteria.
4
Give the anchor, and name the gate
Say it like this
"So the criteria check three things on this question. Did it say interest keeps adding to what's owed, that's the gate, worth two of five points, and it has to match the human's grade exactly. Did it give a real example, and did it say the total ends up higher than what was borrowed, both of those get a one-point tolerance, because a human grader's own call there is already a judgment call."
Why this works
Naming the gate and the tolerance separately is something an interviewer can picture as a real spec line, not a promise to "grade more carefully."
5
Name the risk, in both directions
Say it like this
"Loosen the gate to 'close enough on the total score' and the keyword-only answer slides through again, because it was five points high, well outside anyone's tolerance, but the gate never checked the one item that mattered. Tighten every item to an exact match and we're back where we started, buried under a hundred fails nobody reads."
Why this works
Naming both failure directions shows you understand the gate is a dial you can set wrong two different ways, not just one.
6
Prove it with the case that slipped through
Say it like this
"Here's what happened without it. A student wrote 'it costs more because you pay more over time and interest happens,' never once explaining why. A human grader gave it a one. Inkline gave it a five. That mismatch sat inside a pile of 117 unread mismatches for five weeks, across three model updates, until a TA doing an unrelated check noticed it out loud."
Why this works
A specific case, with a real count and a real number of weeks, does more work than "this could go wrong" ever will.
7
Close on the one line
Say it like this
"So: score each item on its own, gate the one that shows real understanding, give the rest a small tolerance, and leave every other partial-credit combination for later. The number I can point to: under the old bar, 117 unread mismatches. Under the new one, 11 real flags, and every one of them gets read."
Why this works
Interviewers remember the last line most, and this one hands them something they can check, not just a mood.
Let's learn
Here is what happens when an AI grader gets checked against one pass rate, and nobody ever writes down what a mismatch is actually supposed to mean.
Say we build Inkline, a tool inside Cobalt Campus that reads a student's written answer and gives it a score out of five, the same way a human grader would.
What checking Inkline looked like, before this pass bar existed
Knowledge spark: what's an eval?
A stack of already-graded examples, used to test an AI feature before it ships a change. Run it every time, not just once, and it tells you whether the new version is actually better than the old one.
Before Inkline shipped any update, the team ran it against three hundred answers a human had already graded, and checked whether Inkline's score matched the human's score exactly. On the first version, it matched a hundred and eighty-three times out of three hundred. Sixty-one out of a hundred. Good enough, the team figured, to keep shipping small updates and checking the number each time.
Then somebody improved Inkline's prompt to catch more partial credit, and the exact-match number barely moved. Still around sixty out of a hundred. Nobody thought that mattered.
Here is the turn. The hundred and seventeen mismatches were never really the problem. Most of them were a defensible one-point gap: the model gave a three where a human gave a four, both reasonable reads of the same shaky sentence. The real problem was that the exact-match bar couldn't tell those apart from something worse. So nobody read all hundred and seventeen. They read twenty, by feel, and moved on to the next launch.
The keyword-only answer, scored two ways
the old bar: one total scorethe interest item, the gatethe other two items
The old bar gave this exact answer a flat 5 out of 5. Scored item by item, the example and the total-cost claim both earn a point, which is fine. The interest item earns nothing, because the answer never says why interest matters, only that it does. A flat score averages that 0 away behind two fine points. A gate can't.
We didn't grade whether the answer understood interest. We graded whether its score happened to land near a human's.
At its worst, this let one specific kind of mistake through with nobody watching. A student wrote an answer that used every right word, interest, over time, cost, without once saying why any of it was true. A human graded it a one. The newer version of Inkline gave it a five. That exact case sat inside the pile of a hundred and seventeen unread mismatches for five weeks, across three separate model updates, before anyone opened it.
The decision that mattered
Score each rubric item on its own, not the total: the interest item, the example, and the total-cost claim. Make the interest item a gate. Score it wrong, and the answer fails outright, no matter what the other two say.
The choice I would take back. When the eval first shipped, the team picked exact match because it was the simplest thing to check, and back then Inkline only touched a handful of questions, so a quick spot check covered most of what mattered. That held up fine while sixty percent felt like a rough number worth keeping an eye on. It stopped holding up the day that same rough number was hiding a wrong grade that had already reached real students.
What I would leave alone. A short-answer question with one right number, like "what's the minimum payment shown on this statement," doesn't need any of this. The model's answer either matches the number sitting in the statement or it doesn't. There's no judgment call inside it for a gate to catch.
The lesson. Splitting a score into a gate and a tolerance doesn't fix Inkline's grading. It fixes what the pass rate was ever telling anyone. A number that treats a shaky-but-honest guess and a confident lie the same way was never really checking quality. It was checking whether two numbers happened to agree.
Now here is the same thing as a story
Read this one when you've got a few minutes, because a missed Saturday says more than any pass rate ever could.
Before Cobalt Campus hired her as a PM, Milena Sabo graded short answers herself, three years running, for the same personal-finance course Inkline now grades.
She built Inkline's eval herself: three hundred answers, already graded by hand, checked fresh every time an engineer wanted to ship a change to Inkline's prompt or its underlying model. For the first year, she read every mismatch by hand, right alongside the report, the same way she used to read a stack of paper quizzes.
As Inkline rolled out to more of the course catalog, the mismatch pile grew past what one person could read in an afternoon. She started skimming twenty, picked at random. Then she stopped picking at random and skimmed only the ones scored furthest from a five. Then, most weeks, she didn't open the report at all. Sixty percent matched exactly. Close enough, she told herself, and moved on to the next launch.
A fellow TA, checking answers for copied text ahead of finals, sent her one line: "this kid got a 5 for basically restating the question, is that right?"
She never watched a percentage. She watched whether the report was short enough to read.
Milena pulled that one answer, then pulled every mismatch from the five weeks since the last model update. Eleven of them looked exactly the same: an answer that used the right words without ever saying why they were true, scored a five by Inkline and a one by whichever human had graded it first. It took her most of a Saturday, four hours at her kitchen table, just to find the eleven inside a pile of a hundred and seventeen.
The eleven wrong grades were never the real damage. The real damage was that the report had been reading sixty percent for five weeks, and sixty percent had sounded like a number worth trusting.
The anchor: one item gated, two items given room
She remembered the meeting where the team first picked exact match. Someone had asked, half joking, "what else would passing even mean?" and nobody had a sharper answer that afternoon, because back then Inkline only graded one question a week and a slow spot check covered nearly all of it.
So here is what she wrote after that Saturday. Three items, not one score: does the answer say interest keeps adding to what's owed, does it give a real example, does it say the total ends up higher than what was borrowed. The interest item stands apart from the other two. Score it wrong, on its own, and the whole answer fails, no matter how the rest reads.
The same answer, scored under each bar
Run the same five weeks forward with the new criteria in place. The interest item has to match the human grader's call exactly, or the case fails outright. The other two items get a one-point tolerance. The same eleven answers don't hide inside a hundred and seventeen anymore. They show up as eleven, on their own, the first time Milena opens the report. Reading eleven takes twenty minutes, not a missed Saturday.
One design hands her a percentage. The other hands her a list she can actually finish reading.
And the thing I'd tell myself, back in that first meeting: I answered how do we know it's close. I never asked how do we know which kind of close.
SPARK, keyed to a pass bar
This question asks for pass criteria, not an interface, but it's still one concrete decision about the exact moment a score earns trust or breaks it, so SPARK still fits. A question asking how to measure Inkline's overall grading quality across the whole catalog would reach for LEAD instead.
S, situation. Before criteria like this exist, whoever runs the eval reads a report of matches and mismatches and either checks all of it by hand or, once the pile grows, checks a sample and trusts the rest.
P, payoff. Not "fewer wrong grades." The habit worth building: whoever opens the eval report next month can read every real flag in one sitting, and trust that a flag means something actually broke.
A, anchor. Score each rubric item on its own, not the total. One item, the one that shows real understanding, is a gate: any mismatch fails the case outright. The rest get a one-point tolerance, because a small gap there is a defensible judgment call.
R, risk. Set the tolerance too wide, or gate the total score instead of the specific item, and a confidently wrong answer slides through again, this time hiding inside "close enough." Gate every item instead of just one, and the fail list floods back to over a hundred cases nobody reads, the exact failure the tolerance was built to fix.
K, keep out. No attempt yet to score every possible mix of right and wrong across every rubric item on every question in the catalog. One gate item per question, chosen for whichever part of the answer actually proves understanding, is enough to start.
Why the anchor survives the risk
Check it against that Saturday. Does the gate still catch an answer that uses every right word without saying why? Yes, because the gate checks the one item built to test understanding, not how close the total score landed. A tolerance-only design, with no gate, would have let the same eleven through a second time.
And if you want to be sure it really works, try it somewhere else
A city permits office is a different business entirely, wearing a different kind of application, but the same gap between "it looks complete" and "it actually is" shows up there too.
Same framework, a different desk, a different kind of check
S. Dax Petrov reviews building-permit applications for the City of Bellcross, where an AI tool pre-screens each one for completeness before a human ever opens it. Today, without written criteria, whoever checks the pre-screener's flags just eyeballs whether they look about right against her own notes. P. The habit worth building: whoever reads the pre-screener's fail list can trust that a flagged application is actually missing something, not just worded differently than a human reviewer expected. A. Same shape, a different gate. Whether the site plan and the paid fee are actually present in the file fails the application outright if either is missing. Whether the stated reason for a variance "reads as thorough," a real judgment call, gets a tolerance instead. R. Loosen the gate to "mostly complete" and a missing site plan slides through disguised as close enough, the same failure as before, wearing a permit's clothes. Tighten every field to an exact match, including the judgment fields, and the fail list floods again during the one week permit volume spikes, right before a filing deadline. K. No attempt yet to score partial credit on every possible missing-field combination across every permit type the city handles. Two gate items, chosen for whatever a reviewer can never issue a permit without, is enough to start.
It took a contractor calling to ask why her complete application had sat untouched for a week, when the pre-screener had actually flagged it for a missing signature nobody had told her about, before anyone checked whether "mostly complete" had ever meant anything specific at all.
Swap the trigger and it still runs
Speed: even if the eval ran after every single change instead of once a week, a fast exact-match bar is still exact match. Speed doesn't fix what the bar was ever checking.
Cost: if grading a bigger eval set became free tomorrow, three thousand answers instead of three hundred, that still wouldn't tell anyone which mismatches were dangerous. A bigger pile of unread mismatches is still unread.
The model gets better: if Inkline's exact-match rate climbed to ninety-five in a hundred, the five left over would still need a gate to catch them. A rare miss dressed up in the right words is exactly the kind a total score keeps missing.
Where people run it wrong
Writing a gate item, then averaging its score back into the total anyway, so a failed gate still passes if the rest of the answer reads well.
Naming the gate but never writing what a real low answer looks like on it, so reviewers drift back to a shared gut feeling within a month.
Trying to score every rubric item down to the exact word choice in the first version, so the pass bar ships months late while ungated answers keep grading themselves the whole time.
How to use it live
If you're asked this cold, ask what the worst possible pass would look like, the one that would actually fool someone. Then ask whether today's pass bar could ever catch it. That question finds the missing gate faster than listing every rubric item from a blank page.
Flashcards (click a card to flip it)
1 · THE SITUATION
Before this pass bar existed, how did the team decide whether an Inkline update was ready to ship?
Tap to flip
ANSWER
They ran it against 300 already-graded answers and checked whether Inkline's score matched the human's exactly. Once the mismatch pile grew past what one person could read, they sampled 20 by feel instead.
2 · THE PAYOFF
What's the real habit these pass criteria are trying to build?
Tap to flip
ANSWER
Whoever opens the eval report can read every real flag in one sitting, and trust that a flag means something actually broke, not just that two numbers didn't match.
3 · THE ANCHOR
What does the pass bar actually check, and which part can fail a case by itself?
Tap to flip
ANSWER
Each rubric item, scored on its own. The interest item, the one that shows real understanding, is a gate: any mismatch there fails the case outright. The other two items get a one-point tolerance.
4 · THE RISK
What went wrong the one time this pass bar didn't exist yet?
Tap to flip
ANSWER
An answer that used every right word without ever explaining why got a 1 from a human grader and a 5 from Inkline. It sat inside a pile of 117 unread mismatches for 5 weeks, across 3 model updates, before anyone caught it.
5 · THE PROOF
How did Milena actually find out the wrong grade had gone out?
Tap to flip
ANSWER
Not from her own eval report, which had read close to 60 percent match for weeks. A fellow TA, checking answers for copied text, sent her one line asking if a 5 for basically restating the question was right.
6 · THE NUMBER
___ of the ___ mismatches Milena found that Saturday were the same keyword-only answer scored wrong.
Tap to flip
ANSWER
11 of 117. It took her four hours to find them inside a pile that size, because nothing separated a dangerous miss from a harmless one-point gap.
7 · THE REPLAY
Same five weeks, pass bar in place. What changes?
Tap to flip
ANSWER
The gate checks the interest item on its own and fails the same 11 cases outright, on the first read. The pile that used to hide them is gone. Reading the real flags takes 20 minutes instead of a missed Saturday.
8 · CROSS-PRODUCT
Section 4 runs SPARK again on a different product. Which one, and what does its anchor add or change?
Tap to flip
ANSWER
The City of Bellcross's building-permit pre-screener. Its anchor keeps the same shape, a gate plus a tolerance, but the gate checks whether a required document is actually present in the file, instead of whether an explanation shows real understanding.
Check yourself Score: 0 / 0
Fill in the blank
1. The pass bar that would have caught the wrong grade checks each rubric item on ___, and only the ___ item can fail a case outright, no matter what the rest scores.
Show hint
Look at what changed between the old bar and the new one: what got checked, and which single item can override everything else.
Show answer
Its own; interest. The other two items, example given and total-cost claim, get a one-point tolerance, but a mismatch on the interest item fails the case on its own, no averaging.
Multiple choice
2. Which pass bar design matches the anchor this answer argues for?
A. One exact-match score, no per-item breakdown.
B. Every rubric item scored with a one-point tolerance, none of them able to fail the case alone.
C. Each rubric item scored on its own, with one gate item that can fail the case outright and a tolerance on the rest.
D. Every rubric item required to match exactly, with no tolerance anywhere.
Show hint
The anchor needs both a named gate and a tolerance that isn't applied everywhere.
Show answer
C. A has no per-item breakdown at all. B has tolerance everywhere but no gate, so the exact same wrong grade could still pass. D is the too-strict version this answer warns against.
True or false
3. True or false: this pass bar should also score partial credit for every possible mix of right and wrong across every rubric item on every question in the catalog, starting with the first version.
True
False
Show hint
Think about what the keep-out step in this answer names on purpose.
Show answer
False. Scoring every partial-credit combination across every question is the keep-out. One gate item per question, chosen for whatever proves real understanding, is enough to start. Covering every combination on day one would have delayed the pass bar that actually caught the miss.
Short answer
4. What old decision does this answer take back, and why did it make sense when it was first made?
Show hint
Think about why exact match sounded like the obvious choice back when Inkline only graded a handful of questions.
Show answer
Model answer: Early on, the team picked exact match because it was the simplest thing to check, and Inkline only touched a handful of questions, so a slow spot check covered nearly all of it. That held up while a rough percentage felt like a number worth trusting. It stopped holding up the day that same percentage was hiding a wrong grade that had already reached real students.
Short answer, apply it yourself
5. Pick something you track yourself, at work or anywhere. If it looked fine on the surface, would you actually catch it if the substance underneath was wrong? What would need to change about how you check it?
Show hint
Look for whether your own check ever verifies the underlying fact, or only whether the total looks acceptable.
Show answer
Model answer: "I track how many of my invoices get paid on time. Right now an invoice counts as 'on time' the moment the payment clears, but nobody checks whether it cleared for the full amount. A real check would need to flag any invoice paid for less than what was billed, not just whichever ones cleared by the due date."
Multiple choice
6. Based on this answer's own numbers, if the gate had fired on every single case in the 300-answer eval set instead of only the ones that actually failed it, about how many would have needed Milena's own read?
A. 300, all of them.
B. 11, only the ones that actually failed the interest item.
C. 0, the gate never actually fires.
D. About 150, roughly half.
Show hint
The gate checks the interest item against the human's own grade. It doesn't slow down every case, only the ones that actually fail that one check.
Show answer
B. Only 11 of the 300 eval cases actually fail the interest item. The other 289 pass cleanly and never need a second look, which is what makes a targeted gate cheap instead of a blanket slowdown.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
