What is a holdout set and when would you use one for an AI product?

How to answer this, stage by stage

Seven moves. The trap is naming a fraction, twenty percent, and stopping there, with nothing behind it.

Let's learn

Repline is the part of Ironloop, a home strength-training app, that watches a lifter's squat or deadlift video and scores their form.

Before anyone measured it carefully, the team judged Repline the same way every Friday: pull ten random clips, watch them together, and ship the update if none looked obviously wrong.

Now there's a real golden set. 400 clips, each one scored by a certified strength coach, covering the ways a squat or deadlift actually goes wrong: knees caving in, missed depth, a rounding lower back, the bar drifting forward, a bad camera angle, a rep too fast to count, even two people sharing one small home gym.

Here's the turn. The problem isn't whether Repline agrees with the coach on those 400 clips. Of course it mostly does. The engineers tuning the model's thresholds, how far the knees can drift, how deep counts as deep enough, have been staring at every one of those same 400 clips for weeks. A model allowed to see the answer key while it studies will always pass a test written from that same key.

What that costs at its worst: Repline reports 96 percent agreement with the coach, leadership signs off, and a real user's knee-valgus rep still gets scored "good form," because a small fix was made for that exact kind of clip, memorized rather than actually learned.

The choice I would take back. Letting every engineer tuning the model see the entire golden set. That made sense back when the set was brand new, only 40 clips, and holding any of it back felt wasteful. It stopped making sense the moment the set was big enough for someone to quietly tune around specific clips without meaning to.

What I would leave alone. The Friday review, ten random clips the whole team watches together, doesn't need to change. Nobody's reporting a number off that habit. It's a gut check, and a gut check doesn't need to be blind.

The lesson. A score computed from data the model already learned from isn't really a score. It's the model telling you what you already told it.

Now here is the same thing as a story

The short version is above. Read on if you want to feel why a sealed folder beat a number everyone already believed.

Every Thursday afternoon, Akosua opens the same shared folder and scrolls through all 400 scored clips, one more time, just to be sure nothing's drifted. She's the product manager on Repline, and she wrote the fault-category list herself, back when it was eight lines on a sticky note: knees, depth, back, bar path, camera, tempo, crowding, and clean.

She can spot a valgus knee from a thumbnail before the video even loads. Ask her which clip is the hardest one in the set and she'll tell you without checking, clip 214, a squat filmed from the worst possible angle in someone's garage.

When the golden set first came together, it felt like a small miracle, and for a few good weeks it was. Any engineer tuning a threshold could open the whole folder, watch exactly which clips their change helped or hurt, and ship with confidence. Nobody thought twice about it. The set was small, 40 clips at first, and holding any of it back had felt like throwing away scarce, hard-won data.

The habit crept in three small steps. First, engineers started naming clips by number in the team channel, "fixed the thing that broke 214." Then someone wrote a quick check that literally asserted clip 214 now passed, a fast way to confirm a fix landed. Then, without anyone deciding it on purpose, "did all 400 pass" quietly became the actual bar for shipping.

Nobody noticed, until a new hire on the ML team asked a small question in standup. "What's our holdout look like?" Silence. Akosua realized, mid-meeting, that there wasn't one. There never had been.

She went back and pulled the threshold code that weekend. Buried in a config file was a line that read, in effect, if this is one of clips 88, 214, or 305, allow two extra degrees of knee drift. Someone had tuned the model around three specific, hard clips instead of around the general rule those clips were supposed to represent.

Most of the set stays open for tuning. A fifth sits apart, sealed, until the very last step.

The old decision, from that first small meeting when the set was only 40 clips: let everyone see everything, because holding data back when there was so little of it felt like the wasteful move. What nobody said out loud in that meeting was that the golden set wouldn't stay small, and the day it grew past what one person could remember clip by clip was exactly the day it needed a slice nobody was allowed to touch.

So Akosua sealed 80 clips that Thursday night, picked to spread across all eight fault categories, and locked them out of every tuning script the team used. Then she reran the numbers against just that sealed 80.

Reported agreement dropped from a memorized 96 percent to an honest 89 percent. Still fine, on the whole. But knee-valgus accuracy on the sealed clips came back at 78 percent, against 95 percent on the clips the model had been tuned against. That 17-point gap was the whole story, in one number. The model hadn't learned to see a caving knee. It had learned to recognize the specific knees it had already been shown.

The part I'd go back and tell myself: I thought we needed more data. We needed a slice of the data nobody was allowed to look at.

BOUND, sized for a slice nobody's allowed to open

This is a sizing question wearing a definition's clothes, so BOUND fits and FLIPS doesn't. Nobody's habit snapped on a Tuesday here. A team just had to decide how much of its own test to hide from itself.

B, break it down. Holdout size equals the golden set's total, times the fraction sealed off from everyone tuning the model, added as one product, not guessed as a single number.
O, own the numbers. A fifth, 20 percent, held back. Eight fault categories add up to 400 clips: clean reps, knee valgus, missed depth, back rounding, bar drift, bad camera angle, fast tempo, and crowded frames. A fifth of 400 is 80.
U, use a range. At the point estimate, 80. If the set only reaches 250 by launch, 50. If it grows to 600 as more fault categories get written, 120. Honest range: 50 to 120.
N, nail the sanity check. With 80 sealed clips, a measured 90 percent agreement is honest only to within about 7 points, so it could really be anywhere from 83 to 97. That's enough to catch a model that's actually broken. It's not enough to catch a quiet two or three point slip.
D, direction. Growing the whole golden set from 250 to 600 moves the holdout by 70 examples. Wanting a tighter number instead, cutting the wobble from about 10 points down to about 5, moves it by about 105 examples on its own. How sure the team needs to be swings this more than how big the golden set gets.

The range against the number the Friday spot-check already runs on, for scale.

And if you want to be sure it really works, try it somewhere else

Ironseam Fabrication builds the tool trainee welders use to grade their own bead technique from shop-floor photos: porosity, undercut, lack of fusion, overlap, or a clean pass. Five categories, fixed by the welding standard the shop certifies against, not something a product team gets to invent.

B, break it down. Holdout size equals the total golden set, times the fraction sealed off, added up across the five fixed categories.
O, own the numbers. Oluwaseun Bankole, Ironseam's eval lead, sizes each category by how much a defect's look actually varies: clean pass at 30, porosity at 45, undercut at 45, overlap at 30, lack of fusion at 50, the highest, because it's the one that can fail a weld nobody can see failed just by looking. Total: 200. Fraction held back: a quarter, 25 percent, higher than Repline's, because a wrong call here can fail a real weld. Holdout: 50.
U, use a range. The five categories are fixed by the standard, so the swing comes from budget instead: a lean pass at 20 examples per category gives 100 total, 25 held out. A thorough pass at 60 each gives 300 total, 75 held out. Range: 25 to 75.
N, nail the sanity check. At 50 held-out photos, a measured 88 percent agreement with a certified inspector is honest only to within about 9 points, so the real number could sit anywhere from 79 to 97. Against an 85 percent pass bar that decides whether a trainee gets signed off, that's too wide to call.
D, direction. In principle, tightening the confidence bar would move the holdout more than growing the total would, same shape as Repline's answer. But the inspection standard fixes how sure the shop has to be. Oluwaseun can't loosen or tighten that bar to make his life easier. The lever he can actually pull is the per-category budget, and that's what he watches.

The old decision Ironseam would take back is a cousin of Repline's, not a copy of it. Ironloop had let everyone tuning the model see the whole set because the set was once too small to spare any of it. Ironseam's old habit was different: a single inspector graded every photo used for both training the eye and proving the eye was trained, because the shop only had one inspector to spare. Same mistake wearing a different coat: nothing was ever kept apart from the process that was supposed to be judged by it.

Swap the trigger and it still runs.
Speed: an interviewer wants a usable answer before the meeting ends. Round the fraction to a clean quarter, keep 200 as the working total, revisit the exact split once real shop photos start arriving.
Cost: the shop can only pay an inspector for 40 hours of grading, total. At about 12 minutes per graded photo, that's a hard ceiling of 200 examples, whatever the split ends up being. The holdout gets carved from what they can afford, not from a bigger number they wish they had.
The model got better: a new camera rig gives sharper, better-lit photos, so fewer shots get thrown out as unreadable. That doesn't shrink the holdout. A clearer photo doesn't make a 50-example sample any less wobbly, it just means fewer of the 50 go to waste.

Where people run it wrong.
They let the same photos train the model's eye and then grade that eye, so a threshold tuned around one bad weld gets "proven" against that exact weld.
They pick a fraction, a fifth, a quarter, without checking whether that many held-back examples can actually tell a real problem from noise.
They size the holdout once at launch and never revisit it as the stakes grow, more welders, more weld types, past what the original number was ever built to answer.

How to use it live. Say the equation before naming a single number: total set, times the fraction sealed off, and that fraction gets checked against how sure the answer needs to make you, not against a habit. That buys a few real seconds to do the arithmetic instead of guessing out loud.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0