What does the eval spec say about statistical significance of a score difference?

How to answer this, stage by stage

Seven moves. The trap here is grading a coin flip as a win, so the whole answer is really about proving a gap is bigger than luck before anyone trusts it.

Let's learn

Here's what happens when a rule that worked for a year quietly stops working, one point at a time.

Say Kestrel Signal, a brand-monitoring company, builds a tool that reads social posts mentioning a client's brand and tags each one positive, negative, or neutral. Before a new model version ships, the team checks it against a golden set, 500 posts a person already tagged by hand, and compares the model's score to the old version's score.

In the tool's first year, model versions moved in big jumps, seven, ten, once fifteen points at a time, as the pipeline went from a simple keyword list to a real classifier. The rule was simple: if the new score beats the old score on those 500 posts, ship it. Nobody argued with a ten-point jump. Nobody needed to.

Now the pipeline is mature. Gains have shrunk to one, two, three points a quarter. The team's rule never changed. A new model scored 83 percent against the old model's 81, and by the old rule, that's a win. Ship it.

Here's the turn. The 2-point gap is not the real problem. The real problem is what the team does next: they tell clients the tool got more accurate this quarter, and clients start trusting a slightly different sentiment score on a brand's reputation than the one they trusted last month, based on a number that might be nothing but which 500 posts happened to get drawn.

Here's what that costs at its worst. A client reads "brand sentiment improved 2 points" the same week actual sentiment is quietly souring, and does nothing, because the dashboard said things were fine. The tool built to catch a real shift missed it, and told a story about a fake one instead.

The choice I would take back. Writing the ship rule as "beat the old score, any amount" instead of "beat it by more than chance could produce." That was fine when gains were seven-plus points, since no realistic coin flip explains seven points. It stopped being fine the day the gains shrank to two.

What I would leave alone. A model that actually breaks doesn't need a significance test. If a version's score fell from 81 to 40 because it stopped reading a whole language correctly, nobody needs statistics to know that's real. Save the noise-band check for the close calls, not the obvious ones.

The lesson. A ship rule that only works while the wins are big was never really a rule. It was a coin flip we got away with, for as long as luck and real progress happened to move in the same direction.

Now here is the same thing as a story

Skip this if a 2-point "win" already makes you reach for a calculator. Read on if it doesn't yet.

The dashboard on Rasheed Osman's second monitor has one line at the top, in bold: this release's accuracy score.

He's been Kestrel Signal's eval lead for four years, since before the sentiment classifier could tell a sarcastic complaint from a real compliment. He wrote the release checklist himself: score the new model against the 500-post golden set, compare it to the old model's score, ship if it's higher.

For the tool's first year, that checklist never had to work hard. The classifier went from guessing to genuinely reading, and each quarter's score jumped seven, ten, once fifteen points. Rasheed would glance at the two numbers, see daylight between them, and move on to the next release. Nobody ever asked whether the jump was real. A ten-point jump doesn't need asking.

Somewhere around the fifth release, the jumps started shrinking. Six points. Then three. Rasheed noticed, and mostly felt good about it, because a maturing model is supposed to slow down. He kept the checklist exactly as it was.

Then a Tuesday release scored 83 percent against the old model's 81. Two points. By the checklist, a win. Rasheed was about to move it into the release notes when Wendell Cho, a data scientist two desks over, leaned in and asked, not unkindly, "You'd ship on a coin flip?"

Rasheed laughed, then didn't.

He ran the numbers that afternoon. On a golden set of 500 posts, chance alone, no real improvement at all, can produce a swing of close to five points, just from which 500 posts happened to get pulled that week. The 2-point "win" sat well inside that swing. It wasn't proof of anything. It was noise wearing a green checkmark.

He didn't blame the model. The model might genuinely be a little better. It might also be a little worse. Five hundred posts couldn't tell him which, and the checklist had never once asked the question.

He thought back to the meeting where the checklist got written, over a year earlier, in the tool's first month. Someone had asked whether they needed anything fancier than "did the score go up." At the time, every release was moving by double digits, and adding a significance test felt like solving a problem they didn't have yet.

The spec Rasheed rewrote that week requires a two-proportion significance test at 95 percent confidence, and grows the golden set to 800 posts, enough to trust a real 5-point move without asking every release to clear the almost 12-times-bigger bar a 2-point move would need.

The next release scored 84 against 83, one point. By the old checklist, another win to announce. By the new spec, it doesn't clear the noise band, so it holds, unannounced, until the gap grows or the evidence does.

The part he'd go back and tell himself: I asked whether the score went up. I never asked whether 500 posts could actually tell me that.

Same two numbers, 81 and 83. The fix was never a better model. It was a second question before the stamp.

BOUND, run against Kestrel Signal's two-point "win"

This is a sizing question hiding inside a spec-writing question, so BOUND fits and FLIPS doesn't. Nobody's habit snapped here. A checklist that was honest at launch quietly stopped being honest as the real gains shrank.

The range against reality. Today's 500-example set can trust a big move and nothing smaller.

B, break it down. The question is whether a random draw of the same posts could have produced this size of gap between two models that are actually tied. That needs a sample size, a confidence bar, and the size of gap we've agreed actually matters.
O, own the numbers. Today's golden set, 500 posts, hand-tagged by Kestrel's own review team. Observed scores, 81 percent old, 83 percent new, from the last release. A proposed confidence bar of 95 percent, two-tailed, the standard the team already uses for client-facing claims.
U, use a range. To trust a 10-point move, about 185 examples per version. To trust a 5-point move, about 860. To trust the 2-point move the team actually saw, about 5,800. A single number can't answer this question; the honest answer is a range tied to the size of gap in play.
N, nail the sanity check. 800 examples against Kestrel's real traffic, about 45,000 mentions a day, is under 2 percent of a single day, less than half an hour of live posts. A small, fair check for a real regression, and exactly why a 2-point wobble inside it reads as noise, not news.
D, direction. The size of gap the team decides matters swings the required sample size hardest. Halve it, from 5 points to 2.5, and the sample size roughly quadruples. Tighten the confidence bar from 95 to 99 percent and it moves about half as much. Let human-rater agreement get noisier and it barely moves at all by comparison. If someone asks why not the other two levers, name both, but the gap size is the one worth fighting over.

And if you want to be sure it really works, try it somewhere else

Foxglove Veterinary Network runs an after-hours phone line where an AI assistant decides whether a caller's animal needs an emergency vet tonight or can wait for a morning appointment. A new triage model scores 90 percent agreement with a vet's own call on whether a case was truly an emergency, against the old model's 85.

B, break it down. Same shape of question: could a random draw of calls produce this gap between two models that are actually tied, on the specific claim that matters, catching a real emergency, not the overall agreement rate.
O, own the numbers. Old model's emergency-catch rate, 85 percent. New model's, 90 percent. Emergency calls make up about 8 percent of the line's roughly 300 calls a night, the rest are routine.
U, use a range. To trust a real 5-point move in the emergency-catch rate at 95 percent confidence, about 683 actual emergency examples are needed, one number Consolata Mwangi, Foxglove's eval lead, worked out the same way Kestrel Signal did.
N, nail the sanity check. Here's the catch a plain random pull misses. Emergency calls are only 8 percent of traffic, so a random sample big enough to hold 683 of them would need about 8,500 calls total, months of logging for a single check. Consolata's fix: deliberately pull about 700 real emergency-flagged calls from the archive, plus a matched routine sample, instead of waiting on an 8,500-call random draw.
D, direction. Here it isn't the size of gap the team cares about that swings the plan the most, the target stayed at 5 points both times. It's how rare the group being measured is. Emergency calls being 8 percent of traffic, not 50, is what turns a 683-example requirement into an 8,500-call logistics problem, the opposite lesson from Kestrel Signal, where the gap size was the lever and the underlying rate barely mattered.

Swap the trigger and it still runs.
Speed: an interviewer asks how fast you'd get a usable check running before a compliance deadline. Same equation, run with a looser target gap, 8 points instead of 5, so a smaller, faster-to-label 300-example set clears the bar, tightened once there's time to grow it.
Cost: finance caps how many posts get sent to human labelers each quarter. Same equation, solved backwards: at a budget of 400 labeled posts, the smallest gap Kestrel Signal can actually trust is about 6 to 7 points, not 5, so the ship bar has to move to match what the budget can prove.
The model got better: a cheaper labeling vendor cuts the golden set's cost per example in half. Same confidence and power targets, only the affordable sample size moves, and 5,800 examples for a 2-point gap stops being a hypothetical and starts being a line item.

Where people run it wrong.
They celebrate any score that went up, without asking whether their sample could tell a real move from noise.
They grow the golden set once, then treat every future gap as automatically trustworthy, instead of resizing it to whatever gap they're actually trying to catch that quarter.
They chase a tighter confidence bar, 95 to 99 percent, when the sample size is really being eaten by how small a difference they've decided matters, or by how rare the group they're measuring is.

How to use it live. Say the noise-band question out loud before touching a number: could a random draw of this same set produce a gap this size, if the two models were actually tied. That buys a few seconds to remember which numbers you can actually defend, and tells the interviewer the arithmetic is coming, not a guess dressed up as one.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0