How many examples belong in a first eval set and how do you choose them?

How to answer this, stage by stage

Nobody's grading you on landing exactly sixty. They're grading whether you built the number from named situations instead of reaching for a round one. Seven moves get you there.

Let's learn

Meridian Fit's AI drafts a full week of workouts from about five minutes of questions: the goal, the experience level, what equipment is on hand, any injuries, and how much time there is each day.

Before that feature goes anywhere near a wide launch, someone has to build a small stack of test intakes, run each one through the model, and grade the plan it drafts by hand. The question is always the same: how many examples, and which ones.

The easy way to answer the sizing question: pick a round number. A hundred, say, because that's the figure a rival app's engineering blog used when they launched something similar. Grab the hundred most recent sign-ups, grade the plan against each one, ship if most of them look fine.

Here's the turn. A hundred examples pulled at random from recent sign-ups isn't a hundred different tests. Most new users share the same shape: new to exercise, wanting to lose some weight, working out at home. Pull a hundred people at random and around ninety of them look almost the same. The other ten barely touch what actually breaks a plan, an old knee injury, a request for six days a week in a full gym, a goal that contradicts itself.

At its worst, that gap doesn't show up until real people are using the feature. The one plan that tells someone with a bad knee to do walking lunges for a week never gets caught in review, because nobody built an example that looked like them.

The choice I would take back. We'd have picked a total first, a hundred, because it was easy to defend in a roadmap doc, then filled it with whoever happened to sign up that week. I'd take that back. Name the situations that fail differently first. Let the count follow from that.

What I would leave alone. Not every part of this feature needs a stratified eval set. The wording on the plan's daily nudge message, "Great work today!", doesn't split into meaningfully different situations. A flat sample of twenty is plenty there, because a bad line costs a shrug, not a hurt knee.

The lesson. An eval set's size isn't a number borrowed from someone else's launch post. It's the number of situations that actually break differently, times enough examples per situation to trust the grade, plus room for the ones nobody thought of yet.

Now here is the same thing as a story

Skip this part if you already believe a hundred random examples can still cover almost nothing. Read on if you don't.

Every Monday before the office fills up, Farrukh Tashkentov pulls last week's new intake surveys and reads through a handful of them, one at a time, while his coffee is still too hot to drink.

He's been the product manager on Meridian Fit's plan-drafting feature for fourteen months. Hand him any intake and he can usually guess, before he finishes reading it, what kind of week the model is about to draft.

The team needed a first eval set before the feature opened past its closed beta, and a number already had momentum behind it. Nobody had pushed back on it across two planning meetings: a hundred examples, the figure a rival app, Ridgeback Fit, had put in an engineering blog post about their own AI trainer's launch. A hundred sounded thorough. It was also easy. Pull the hundred most recent completed intakes, grade the plan against each one, ship if ninety look right.

Then, in that same stand-up, the newest engineer on the team, three weeks in, asked the plain question nobody senior had bothered with. "A hundred of what, though?"

Farrukh didn't have a real answer beyond "recent sign-ups." So instead of defending the number, he went and actually looked at it.

Ninety of the hundred were the same shape: new to exercise, wanting to lose some weight, working out at home with whatever they had lying around. Six were an intermediate lifter with a full gym. Three were someone training for a race with no equipment at all. One mentioned a bad knee. Not one was an advanced lifter asking for six days a week.

A hundred pulled at random, weighed against sixty built on purpose

He didn't panic, and he didn't just bolt more examples onto the pile. He sat down with Meridian Fit's head trainer, the one person on the team who could say, for any two intakes, whether the right plan for one would actually be wrong for the other. Together they named six situations where a wrong plan fails in a genuinely different way, not six possible answers to a survey question, six real shapes.

Eight examples in each, built on purpose this time, not pulled at random. Forty-eight. Twelve more for the edge cases the random hundred had walked straight past: a request asking for muscle gain and ten kilos of weight loss in one month, a stated heart condition, someone with no equipment and eight minutes a day. Sixty in total.

Weeks earlier, in the meeting where the hundred got written into the roadmap doc, it had felt like the safe, defensible choice. A real company had used it. Nobody in the room could name a reason it was wrong. That was the actual problem. Nobody had tried.

He graded the new sixty over one afternoon, the same length of time the old hundred would have taken. Three of them failed. One was the returning-from-injury case: the model had put a stated bad knee into walking lunges, twice, in two different weeks of the same plan. The random hundred, with its single injury mention, never had the numbers to catch that even once.

The thing I'd want to tell myself, back in that planning meeting: the number was never the risk. What the number was made of was the risk. I wrote down a total before I'd named a single situation it had to cover.

The letters, spelled out against this exact sixty

This is an estimation question with a build-up hiding inside it, so BOUND fits, not FLIPS. Nobody's habit is fading here. It's an argument about what the total is made of.

B, break it down. The total isn't one number, it's a sum: the number of situations that fail differently, times examples per situation, plus a fixed slice for edge cases.
O, own the numbers. Six categories, eight examples each, forty-eight. Twelve edge cases on top. Sixty total.
U, use a range. Thirty-five if only the four most common situations get covered at six examples each. Ninety if all six get twelve examples plus twenty edge cases. Sixty sits between those on purpose.
N, nail the sanity check. Sixty examples at about eight minutes of real review each is roughly eight hours, one workday. The two-hundred-example version some engineers floated would have taken over three days before anyone started iterating.
D, direction. Whether you cross every survey field into its own category, or hand-pick the ones that actually fail differently, moves this number more than anything else on the list.

The range, with the borrowed number marked for scale, not as one of the bounds

And if you want to be sure it really works, try it somewhere else

A veterinary chain's AI drafts at-home care instructions after each visit, tailored to the species, the procedure, and what the owner said about their household.

B, break it down. Same shape: the number of situations where wrong instructions actually hurt the animal or the owner's follow-through, times examples per situation, plus edge cases.
O, own the numbers. Six categories: a routine spay or neuter, a dental cleaning, an orthopedic surgery needing strict activity limits, a new chronic diagnosis the owner has to learn a skill for, an exotic pet with different physiology warnings, and a multi-pet household where the recovering animal needs to be kept apart from the others. Six examples each, thirty-six. Ten edge cases, an owner who needs plainer language, a second unrelated condition, a household with young kids and a large recovering dog. Forty-six total.
U, use a range. About thirty if only the four most common situations get covered at five examples each plus eight edge cases. About sixty-five if all six get eight examples plus eighteen edge cases.
N, nail the sanity check. Forty-six examples at ten minutes of real review each, checking instructions against the actual procedure, is about seven and a half hours, close enough to one workday.
D, direction. Same lever as the workout-plan case: crossing every species-and-procedure combination literally would run into the hundreds. Hand-picking the six situations that hurt the animal differently is what keeps the number small enough to actually build.

Swap the trigger and it still runs.
Speed: an interviewer caps it at one afternoon before a review meeting. Same six categories, drop to five examples each instead of eight, keep the edge-case slice, because the category list is what protects the estimate, not the per-category depth.
Cost: a senior trainer's review time runs forty dollars an hour and the review budget is five hundred dollars. Work backward from that ceiling to how many examples the budget actually affords, same equation, solved in the other direction.
The model got better: after a few rounds the drafter stops making category-specific mistakes. Keep the same six categories, but drop to four examples each, since the job now is catching regressions, not discovering new failure shapes.

Where people run it wrong.
They pick a round number first, then back into a method that happens to produce it, instead of building the number from named categories.
They cross every survey field literally and end up with hundreds of near-empty categories, so the eval set becomes unreviewable and nobody finishes grading it.
They build the category list once at launch and never revisit it, so a new goal type or equipment option ships months later with zero coverage in the eval set.

How to use it live. Say the equation out loud before any number: "I'd size this by category, not as one flat total, how many genuinely different situations, times enough examples to trust each one, plus room for edge cases." That sentence buys you time to actually do the arithmetic, and it tells the interviewer a round number isn't coming.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0