Critique a golden set built entirely from examples the team wrote themselves.

How to answer this, stage by stage

Seven moves. This one plays the critique straight, then has to prove its own grader isn't broken before anyone's allowed to blame the examples.

Let's learn

Say we build a score. Larkmoor Crate is a subscription box company, and every week the score looks at each subscriber and rates how likely they are to cancel in the next thirty days, so the retention team knows who to email first with a discount or a bonus box.

Before anyone tuned anything, the score was new and rough. It only matched seventy-one out of every hundred of Estelle's own two hundred and forty hand-written cases. Nobody trusted it yet, and that felt right. Plenty of headroom, plenty of reason to keep working.

Fourteen months and six retrains later, that same score sits at ninety-two. Every retrain beat the one before it. Every retrain replaced what was already live.

Here is the turn. Those twenty-one extra points are not the real problem. The real problem is what never showed up anywhere on that climbing line: the model's actual ability to catch a subscriber who cancels without complaining first. That number, the real early-warning catch rate, sat at fifty-eight percent when the golden score was seventy-one. It sits at fifty-four percent now that the golden score is ninety-two.

At its worst, this costs Larkmoor Crate subscribers nobody ever tries to save. Cancellations that go quiet, no ticket, no complaint, get caught early only nineteen percent of the time. Almost half of everyone who leaves, 185 of last quarter's 420 cancellations, leaves exactly that way.

The gap between a climbing score and the subscriber it never once flagged

The choice I would take back. Fourteen months ago, writing two hundred and forty examples by hand, instead of pulling and labeling real historical cancellations, was the sensible call. There wasn't a clean history to pull from yet, and two hundred and forty hand-picked cases let a new retrain get checked in an afternoon instead of a week. I would take it back and put a real, refreshed sample of production outcomes on equal footing with it, checked every quarter, not just at launch.

What I would leave alone. The vocal-complainer segment doesn't need touching. The model already catches eighty-one percent of those cancellations early, because that's exactly the kind of case the golden set is full of. Adding more review there would just slow down the part that already works.

The lesson. A test a team writes for itself will always agree with what the team already believes about how people leave. It takes real, boring, unremarkable silence to prove it wrong, and nobody goes looking for silence on purpose.

A three-year subscriber who never called anyone

Read the short version above if you're short on time. This is the long version, for when you want to feel exactly where the fourteen months went.

Estelle Marchand has run eval for Larkmoor Crate's churn model for three years. She can look at a retrain's numbers and tell, inside a minute, whether a climbing score means something real or just a lucky week of tuning.

She built the golden set herself, with two teammates, in the week the model first launched. Two hundred and forty example subscribers, each one written out by hand: what they ordered, what they said to support, and when they finally canceled. It was fast to build and easy to check by eye, and at the time, pulling and cleaning real historical cancellations across three different support tools would have taken a month the team didn't have.

For most of that first year, watching the score climb felt like watching the model get smarter. Seventy-one. Seventy-nine. Eighty-four. Every point felt earned. Somewhere in there, without anyone deciding it on purpose, the team quietly stopped doing something it used to do every quarter: pull fifty real cancellations at random and check by hand whether the flagged list had actually caught them. Once the golden score was closing in on ninety, that check started to feel like double work. Estelle signed off on dropping it the same quarter the score first passed ninety.

Fourteen months in, the score hit ninety-two, and the sixth retrain replaced production without anyone blinking.

Then a new retention analyst, Isadora Blythe, three weeks into the job, was pulling the quarter's cancellations for a business review and cross-checking them against the flagged list, mostly to learn how the system worked. One name didn't line up. A subscriber on the eighty-nine-dollar Reserve box, three years with Larkmoor Crate, canceled in April. Zero flags. Not once, across six retrains, had the score ever marked her as at risk.

Isadora asked Estelle why. Estelle didn't have an answer, and that bothered her more than the miss itself.

I want to say the problem was that the golden set was too easy. It wasn't easy, exactly. It was answering a question nobody was asking anymore. Every one of Estelle's two hundred and forty examples had a support ticket in it somewhere, because a person writing a training case almost always gives it a reason. Real subscribers don't. Some of them just let the card decline and never say a word.

So here is the decision I would take back.

Fourteen months earlier, when the team wrote the golden set, building it from imagination instead of real records was the sensible call. There wasn't a clean history to pull from yet, and two hundred and forty hand-picked cases let a new retrain get checked in an afternoon instead of a week.

I would put a real, refreshed sample back alongside it. Pull real cancellations and real non-cancellations every quarter, score the model against both sets side by side, and never let the golden set's number stand alone again. Not because the golden set is worthless. Because of what it would have shown, months earlier: eighty-one percent caught among subscribers who complained first, nineteen percent among the ones who went quiet, both hiding inside one number that looked fine.

That is the whole difference. One process trusts a number the team wrote about itself. The other checks that number against what subscribers actually do.

And the part I'd want to tell myself, if I could go back: we built a bar that only ever answered for the kind of leaving we already knew how to picture. Nobody decided that on purpose. We just never wrote down the kind we couldn't.

What the recut actually showed

Before blaming the golden set's coverage, Estelle's team checked whether the score was even being read right. Two people hand-checked a sample of real subscriber outcomes against what the golden set said they should be, and agreed on nearly all of them. The scoring itself wasn't broken. That left the coverage.

Three ways a score can pass and still be lying

Not because anyone cut a corner on purpose. A set built entirely from what the team already believed can be met honestly, retrain after retrain, and still stop meaning anything.

Three separate, checkable ways to write a set that passes and still misses the point

TRACE, run on a set nobody had touched since launch

This looks like a critique of a document, but the actual job is diagnosis: work out why a set the team trusted for fourteen months quietly stopped matching reality. GUARD would fit if the harm were one group losing a fair shot at an appeal; here the harm is a whole kind of subscriber the set never learned existed.

Same shape, a fraud list built entirely from textbook claims

Palisade Assurance runs a fraud-flag model across its auto claims. Percival Odom, the special investigations unit's model owner, wrote the acceptance set two years ago: one hundred and fifty claim narratives, drafted by senior investigators from memory of the fraud patterns they knew best, scored on a benchmark called FraudMatch. Every retrain since has had to beat the last version's score on that set before it replaces the one flagging live claims.

T. FraudMatch's score climbed from sixty-eight to eighty-nine over eight quarterly retrains. The share of confirmed fraud caught among minimal-text claims, the ones filed through the app with barely a sentence attached, never moved off roughly one in four.
R. Split by claim length. Full-narrative claims, sixty-one percent of volume: FraudMatch catches confirmed fraud eighty-one percent of the time. Minimal-text claims, under fifteen words, thirty-nine percent of volume: twenty-four percent.
A. Before blaming coverage, two investigators blind-rescore forty flagged claims against FraudMatch's own output. They agree on thirty-nine of forty. The scoring checks out.
C. Three candidates: the acceptance set's narratives average forty-five words, none written by an investigator who'd ever seen a nine-word claim score high; the team never wrote a minimal-text example in at all, since those felt too thin to make a convincing teaching case; and the set's language is formal investigator prose, "claimant exhibited an inconsistent timeline," nothing like how a real app-submitted claim actually reads.
E. Compare the share of claims under fifteen words. Four of the acceptance set's one hundred and fifty, about three percent. Fifty-eight percent of real flagged claims. FraudMatch barely tests the claim type the SIU reviews most.

Flashcards (click a card to flip it)

This is a diagnosis question about a written test set, not a habit changing, so these eight test the TRACE moves and the real numbers behind them.

Check yourself Score: 0 / 0