Explain how you would use a pilot to build the eval set for production.

How to answer this, stage by stage

Eight moves. The trap is naming one confident-sounding percentage and never showing which buckets it's actually built from.

Let's learn

Fanout is the part of Coppervale's help desk software that reads a new support ticket the moment it lands and decides two things: which team should own it, and how urgent it is.

Before Fanout, a queue manager on each pilot account read every new ticket by hand, worked out the team and the urgency, and moved it into place. That took about ninety seconds a ticket, and on a busy account with two hundred tickets a day, sorting the queue alone ate most of a shift.

With Fanout running, most tickets get sorted the second they arrive, in under a second, with nobody touching them. Across the pilot's five weeks and six thousand tickets, eighty seven of every hundred got auto-routed clean, no human ever needed to look.

Here's the turn. The tickets Fanout gets wrong are not the real problem, there simply aren't many of them, about four in a hundred. The real problem is that Fanout is just as confident when it's wrong as when it's right. It doesn't hesitate, it doesn't flag itself, it just quietly puts the ticket in the wrong queue and moves on. Nobody finds out until an agent in that queue happens to read it, or a customer chases it.

Fanout never tells you which branch a ticket took. The confident-and-wrong branch looks identical to the confident-and-right one from the inside.

At its worst, that cost more than a wasted queue slot. In week four of the pilot, a ticket that read like an ordinary "I can't log in" question was actually an account takeover, someone else had gotten into the account. Fanout filed it under General, the same queue as every routine password question that week. It sat there for three days before an agent working through the backlog in order finally opened it. By then the customer had posted about it publicly.

The choice I would take back. The very first golden set for Fanout's launch was a flat ten percent random sample, pull every tenth ticket off the six thousand, six hundred examples. That felt thorough. Check it and you find only twenty four of those six hundred were tickets Fanout actually got wrong, and by chance, not one of them was a security ticket.

What I would leave alone. Refund and cancellation tickets need no extra depth. One clear intent, one obvious queue, the cleanest bucket in the whole pilot, barely any caught-wrong tickets across the entire five weeks. Giving that queue thirty more hand-labeled examples wouldn't teach the eval set anything it doesn't already know.

The lesson. A pilot doesn't hand you an eval set for free. It hands you six thousand tickets. The pilot's real job is to make the rare four percent easy to find. Finding them and labeling them is still yours to do.

Now here is the same thing as a story

The short version is above. Read on if you want to feel why one queue almost shipped without enough evidence behind it.

Kacper Zielinski can sort a support ticket in about four seconds, before he's even finished reading the subject line. He's the eval lead on Fanout at Coppervale, and he built its very first golden set the same week the pilot opened.

That first golden set was simple on purpose. Pull every tenth ticket off the pile, six hundred tickets, label them by hand with one other reviewer over four days. For the first three weeks of the pilot, that felt like plenty. Every morning Kacper ran Fanout's routing against those six hundred examples, and the number came back the same: ninety six percent. He'd glance at the dashboard around ten, see ninety six, and move on to the rest of his day.

The habit thinned the way habits do. First, nobody rebuilt the sample as the pilot's account mix grew from two customers to five. Then, when a new hire asked in standup whether the six hundred examples still matched what Fanout actually handled, the honest answer was "probably," and the meeting moved on. Then, quietest of all, Kacper stopped opening the six hundred examples himself. He just watched the number. Ninety six percent had started to feel like a fact instead of a measurement someone had actually checked lately.

Then, in week four, a ticket came in from one of the pilot accounts that read like an ordinary password question. Fanout filed it under General, the same queue as every other login question that week. It wasn't ordinary. Someone else had gotten into the account. The ticket sat in General for three days before an agent working the backlog in order finally opened it. By then the customer had posted about it publicly.

Kacper pulled the golden set that afternoon. Of the six hundred examples, twenty four were tickets Fanout had actually gotten wrong. None of them were security tickets. Security made up a small slice of pilot traffic to begin with, and a flat random sample hadn't caught enough of it to tell him anything.

The decision he'd take back was easy to remember, because it took about two minutes to make. In the kickoff planning call for the pilot, someone suggested pulling a flat ten percent sample, since that was the standard move for every other eval at Coppervale, and nobody in the room had a reason to argue. It made sense. It was fast, it felt proportionate, and at the time nobody knew which queues would turn out to be rare.

So he rebuilt it. Three buckets instead of one flat sample: a thin random slice of the clean auto-routes, a denser slice of the flagged-unsure tickets, and every single caught-wrong ticket, all two hundred forty of them, no sampling. Five hundred ninety four examples total. Run queue by queue, five of Fanout's six queues cleared a floor of thirty caught mistakes each, enough to trust that queue's own number. Security had twenty two. Below the floor. So instead of shipping on the strength of one comfortable ninety six percent, Coppervale ran two more pilot weeks pointed only at security tickets before Fanout went out past the pilot accounts.

One design tells you the average is fine. The other tells you exactly which queue you don't actually know anything about yet.

The part I'd tell my past self, from that two-minute planning call: a flat percentage feels fair because it treats every ticket the same. That's exactly the problem. The tickets that matter most were never going to show up in their fair share.

BOUND, run on Fanout's own pilot, letter by letter

This is a sizing question wearing a process question's clothes, so BOUND fits and FLIPS doesn't reach far enough on its own. Nobody's habit here needs a two-setting switch. A team had to decide how much evidence a five-week pilot actually owed its rarest queue.

B, break it down. Eval set size equals three additions: a thin random slice of the confident-and-right tickets, plus a denser slice of the flagged-as-unsure ones, plus every single caught-wrong ticket. Never one flat percentage split evenly.
O, own the numbers. Pilot volume: two hundred forty tickets a day, twenty five business days, six thousand total. Buckets: clean 5,220 (87%), flagged 540 (9%), caught-wrong 240 (4%). Sample: clean 1 in 30, 174. Flagged 1 in 3, 180. Caught-wrong, all of it, 240. Total: 594.
U, use a range. At the point estimate, six thousand tickets, caught-wrong lands at 240. At a slower pilot week, roughly 180 tickets a day, the total drops to 4,500 and caught-wrong to about 180, thirty per queue. At the busiest observed rate, about 300 a day, the total climbs to 7,500 and caught-wrong to about 300, fifty per queue. Honest range: thirty to fifty caught examples per queue, forty as the working number.
N, nail the sanity check. Five hundred ninety four examples, about six minutes each, comes to roughly fifty nine hours, a week and a half for one reviewer, comfortable inside a five-week pilot if labeling happens as tickets arrive. And the caught-wrong bucket alone, 240 tickets, is about four percent of the pilot's real traffic, but forty percent of the finished eval set, a ten-times deliberate over-representation of the tickets that matter most. That's the point, not an accident.
D, direction. Cutting the pilot from five weeks to three doesn't shrink the total evenly. It drops the caught-wrong bucket from 240 to about 144, roughly twenty four per queue, under the thirty-example floor for every queue, not just security. Pilot length, not sample size, is the assumption that decides whether the rare queues ever clear the floor.

Security's real count sat below the floor every other queue cleared. That's a finding, not a rounding error.

Two things worth saying here, since this is where the AI-specific judgment actually sits. First, the alternative I rejected: pulling one flat random sample across all six thousand pilot tickets, the same move Coppervale used for every other eval. At the same rough budget, six hundred examples, that gets you twenty four failure examples total, and by chance, none from security. Rejected, because a flat sample spends its budget proportionally to how often something happens, not to how much it costs when it's rare and wrong. Second, Fanout doesn't ship on a promise that it's never wrong. It ships when each queue clears an accuracy bar against this eval set, most of the time, on a genuinely hard sample, not a bar met by an easy one. The same eighty five percent confidence cutoff that sends a ticket to the flagged-unsure bucket instead of auto-routing it is the guardrail against Fanout's most expensive habit: being wrong with no hesitation at all.

And if you want to be sure it really works, try it somewhere else

Same three buckets, a completely different product and industry, so the method proves itself instead of repeating a story you happened to prepare.

Thistledown Veterinary Group runs Beacon across a dozen clinics, a tool that reads an incoming pet-owner message, a portal note or a text, and decides whether it needs a vet's eyes today or can wait for routine scheduling.

Priska Onwuka runs clinic operations across Thistledown's three pilot clinics, and she's the one who ends up sizing Beacon's golden set.

B, break it down. Golden set size equals the same three additions: a thin random slice of the messages Beacon triaged with confidence and got right, a denser slice of the ones it flagged as unsure, and every message a vet tech had to reclassify after the fact, urgent filed as routine, or the other way round.
O, own the numbers. Three pilot clinics, about ninety messages a day between them, over a four-week pilot, twenty business days: 1,800 messages total. Clean 1,560 (87%), flagged 180 (10%), reclassified 60 (just over 3%). Sample: clean 1 in 20, 78. Flagged 1 in 2, 90. Reclassified, all of it, 60. Total: 228.
U, use a range. At 1,800 messages for three clinics. Two clinics instead of three lands closer to 1,200, about forty reclassified messages. A fourth clinic added mid-pilot pushes it past 2,400, closer to eighty. Range: forty to eighty reclassified messages, sixty as the working number.
N, nail the sanity check. 228 examples, about five minutes each, a shorter message than a support ticket, comes to about nineteen hours, well under a week for one reviewer, comfortable inside a four-week pilot. But split those sixty reclassified messages across Beacon's four message types, urgent-symptom, appointment, refill, billing, and urgent-symptom, the one category where a miss actually risks a pet's life, had only nine. Under any reasonable floor.
D, direction. Priska can't easily change how many messages three busy clinics generate a day. What she can change is whether a fourth clinic joins the pilot, or whether vets deliberately submit a handful of known urgent-symptom scenarios as seeded test messages instead of waiting for enough real ones to turn up on their own. Seeding moves the number that Kacper's lever, pilot length, never could.

The old decision Thistledown would take back is a cousin of Coppervale's, not a copy. Fanout's team sized by one flat percentage of everything. Beacon's very first eval was sized per clinic instead, twenty examples per pilot clinic, chosen back when Beacon only sorted two message types and clinic was genuinely the whole story. Same mistake in a different coat: sizing an eval by the axis that was true once, and never checking whether it's still the right axis once the product's real shape changed.

Swap the trigger and it still runs.
Speed: an interviewer cuts you off after ninety seconds. Skip straight to: hand-label every caught mistake, thin-sample the easy stuff, and check whether each queue clears about thirty examples before trusting its number.
Cost: leadership says the pilot can only run three weeks, not five, to hit a launch date. Don't shrink every bucket evenly. Protect the caught-wrong bucket first, still take all of it, and say plainly which queues won't clear the floor yet, instead of quietly shipping a thinner set as if it were the plan all along.
The model got better, for real: suppose Fanout's own confidence calibration improves and almost nothing gets flagged as unsure anymore. That doesn't mean the eval set can shrink. Fewer flagged tickets just means fewer genuinely ambiguous cases, it says nothing about whether the model is still confidently wrong on a rare ticket type it's never been tested against.

Where people run it wrong.
They size the golden set as one flat percentage of total pilot volume, and never check whether that percentage actually reaches the rare, expensive queue.
They stop trusting the eval set the moment the overall accuracy number looks good, instead of checking it queue by queue.
They treat the pilot's calendar length as fixed and unrelated to the eval set's coverage, instead of the one lever that actually decides whether the rare queues ever get enough examples.

How to use it live. Say the equation before naming a single number: three buckets added together, never one flat percentage across everything. That buys real thinking time instead of a guess said with confidence.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0