How large should a golden set be, and what determines the answer?

How to answer this, stage by stage

Eight moves. This is arithmetic plus one ownership decision, so what's being graded is whether your structure holds up under a follow-up question, not whether your voice cracks at the right line.

Let's learn

How do you know a receipt-categorization model that claims 96 percent accuracy is actually right, and not just right on the ten receipts someone happened to check? Say we build a feature that reads a photo of a receipt and files it under one expense category out of a list of 22: groceries, dining, travel, fuel, software, office supplies, and so on. A golden set is the fixed pile of receipts you check every version of that model against, the same pile every time, so a change in the pass rate means the model actually changed, not that the test changed under it.

The equation is short. Golden set size equals the number of categories times a per-category floor, plus an edge-case layer added on top. The floor is what makes each category's own pass rate trustworthy on its own, not just the overall average. The edge layer is a separate budget, spent on the receipts that don't sit cleanly in one category.

Say the target set of 925 examples takes two labelers, cross-checking each other, about a week. Against a support queue that logs roughly 300 miscategorization complaints a month, 925 is close to a quarter's worth of real disputed receipts. Small enough to actually build. Large enough to have seen the weird ones.

But the size was never really the hard part. The hard part is who keeps the set true once the taxonomy stops holding still. Section 2 has the number that made that real for one data lead.

Now here is the same thing as a story

Skip this if you already believe a spreadsheet can quietly go stale under a team's feet. Read on if you don't.

Kagiso Molefe is the data lead at Cinder, a fintech app that scans a photo of a receipt and files it into one of an accountant's expense categories. She joined eight months after launch, which means she inherited the golden set instead of building it.

It had been built once, at launch, by a contractor the founding team brought in for three weeks. 600 receipts, 22 categories, roughly 27 examples each. Nobody's name was on it. It sat in a shared folder called "eval_data_final_v2" and nothing about it had a date attached to when it should be looked at again.

For most of a year, that was fine. The model's pass rate against the set held steady in the low nineties, quarter after quarter. Every release, someone ran the eval, saw the number, and shipped.

Then a new engineer, three weeks into the job, was reading through the taxonomy and asked a question nobody could answer. "Wait, why do we have zero examples of Buy Now, Pay Later receipts in here? Klarna and Afterpay must be a third of what people are paying with now."

Kagiso pulled the golden set folder up on the call. He was right. Zero. Not thin coverage, none at all. BNPL wasn't a merchant category that existed when the contractor built the set, so it had never been added, and nobody had owned making sure it got added later.

She ran a spot check that afternoon, pulling 200 real BNPL receipts from the last month of production traffic and hand-labeling them. The model was filing 31 percent of them under "software subscriptions," because the checkout page and the recurring-charge language looked close enough to a SaaS bill to the model, and the golden set had never once told it otherwise.

The pass rate the team had been watching for a year had never dropped. It couldn't have. It was measuring a version of Cinder's receipts that had stopped existing months earlier.

So here is the decision Kagiso took back. The set had been built as a one-time deliverable, priced and scoped like a contractor task with a clean end date, instead of a living asset with a name attached to it. She gave it one: hers. And a rule to go with it, a refresh review every quarter, plus a standing trigger, any merchant type that crosses one percent of monthly receipt volume gets golden examples added within two weeks of being flagged, not at the next scheduled review.

She rebuilt the set at 924 examples, 22 categories at a 30-example floor, plus a 40 percent edge-case layer loaded onto the categories people actually disputed: split-category receipts, business-versus-personal, refunds, and now BNPL. Not spread evenly. The evenly-spread version had looked complete on a slide for a year and had caught none of this.

What she'd leave entirely alone: categories like rent and utilities, which are almost always a single clean line item with no real ambiguity. Thirty examples there will hold for years. She didn't thicken them, and she didn't need to.

The thing I'd want Kagiso to say out loud, if she were in the interview: the equation told her how big the set should be. It never told her who was supposed to notice when the world underneath it changed. That was a role, not a number, and for a year nobody had it.

Same equation. The missing piece was never a number, it was a name on the folder.

The formula, recapped as BOUND

This is a sizing question, so the framework is BOUND. There's no behavior snapping here and no leading metric to chase, just arithmetic, said out loud, with the assumptions owned. FLIPS would force a person's habit onto a question that's really about a formula and a folder nobody claimed.

The range, with the number you can actually feel marked separately

B, break it down. Number of categories times a per-category floor, plus an edge-case layer added on top and loaded onto the disputed categories, not spread evenly.
O, own the numbers. 22 categories in Cinder's taxonomy. A 30-example floor, the point below which a category's own pass rate is noise. A 40 percent edge layer, sized off the real share of disputed receipts a support queue actually sees.
U, use a range. About 500 for a thin set to about 1,760 for a rigorous, audit-facing one. Target around 925 for a normal launch.
N, nail the sanity check. 925 examples is close to one quarter's worth of Cinder's real miscategorization complaints. Enough to have seen the weird ones. Small enough for two labelers to clear in a week.
D, direction. The per-category floor swings the total the most, not the edge layer and not the category count, because it's multiplied by every one of the 22 categories. Cut it to save budget and the whole set quietly shrinks by a third.

And if you want to be sure it really works, try it somewhere else

A veterinary triage app reads a pet owner's typed description of what's wrong and sorts it into one of four urgency tiers: emergency now, same-day visit, this-week appointment, monitor at home.

B, break it down. Number of tiers times a per-tier floor, plus a boundary layer sized around the pairs of adjacent tiers where a real symptom description could plausibly land in either.
O, own the numbers. Only 4 tiers, so I'd raise the floor well above the fintech case, 40 examples a tier, because a miss on the smallest tier, emergency, is the one that can't be absorbed by volume. Plus a boundary layer of about 100 examples split across the three adjacent-tier pairs.
U, use a range. Roughly 160 examples at the thin end to about 400 at the rigorous end, far smaller in raw count than the receipt case, because there are only 4 tiers instead of 22 categories.
N, nail the sanity check. A clinic answering the triage line might see 50 genuinely ambiguous descriptions a week. A 260-example golden set is about a month of those, enough to have actually argued about the hard ones.
D, direction. Here it isn't the floor that swings the number, it's the cost of missing an emergency. That single fact justifies a much bigger floor on one tier than the formula would suggest on its own, the opposite lesson from the fintech case, where every category costs roughly the same to miss.

Swap the trigger and it still runs.
Speed: an interviewer asks how fast the set needs to be ready before the next compliance audit. Same equation, solve for how many labelers running in parallel clear it by the deadline.
Cost: finance caps the labeling budget at $6,000. Same equation, run backwards, divide the budget by the cost per labeled example to find the real ceiling on set size.
The model got better: a new categorization model ships strong out of the box. Same equation, same target size, because the set's job is measuring the model, not training it. A stronger baseline doesn't lower the bar for trusting the measurement.

Where people run it wrong.
They build the set once at launch and never revisit it, so it quietly stops representing the merchants or symptoms actually showing up in production.
They spread coverage evenly across every category, which looks complete on a slide and never stresses the boundary calls where the model actually fails.
They let whoever's free build it, with no name on the file, so nobody's job is to notice when it's gone stale.

How to use it live. Say the equation before you touch a single number: "size equals categories times a per-category floor, plus an edge-case layer on top." That sentence buys you ten seconds to recall a real floor and a real percentage, and it shows the interviewer you have a method, not a guess.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0