How many examples do you need before a feasibility judgement is trustworthy?

Kestrion Precision Works machines turbine and pump parts, and RivetEye is the photo-based tool at the inspection bench that checks each part for six kinds of defect before it ships. Marisol Iyengar is the AI PM who had to decide how many example photos were enough to trust a feasibility call.

How to answer this, stage by stage

Nobody is scoring whether you can name a big enough number off the top of your head. They are scoring whether you can show your work for why that number, and not a smaller one, is the one worth trusting.

Let's learn

For six years, an inspector at Kestrion checked every machined part by hand under a loupe, about six minutes a part, roughly 140 parts a shift.

RivetEye photographs each part instead and sorts it in under four seconds, so an inspector reviews the flagged pile instead of checking every part from scratch.

The five letters, held up as one page. Own the numbers is the step a forty-photo sample skipped.

Here's the turn: Marisol judged RivetEye feasible off 40 example photos pulled from a vendor demo set, staged under good light, hitting 96 percent accuracy against that set. Six of Kestrion's defect types were in that set, but the rarest one, a casting void that shows up in about 3 percent of parts, appeared only twice. Two correct calls out of two tells you almost nothing. It felt like proof anyway.

Forty photos never even reached the first branch of this tree.

At its worst, a turbine housing with an undetected casting void ships to a customer, and the flaw surfaces during their own incoming inspection instead of Kestrion's, turning a private catch into a public one and a batch recall.

What I would leave alone: for hairline cracks, showing up in nearly a third of scrap parts, the original 40-photo sample was genuinely plenty. That accuracy held up in production without a single surprise.

The lesson: a sample is not trustworthy because the total looks big. It is trustworthy because the thing you are most worried about missing has actually shown up enough times to be tested, not just hoped for.

Now here is the same thing as a story

The short version above is what you'd say defending a sample size in a planning review. Read this one for what it felt like the week a routine audit turned a comfortable number into an uncomfortable one.

Marisol could tell a defect classifier that was actually ready from one that just looked ready, most of the time. She had shipped two smaller vision tools before RivetEye, both fine, both boring in the best way.

Same word, sample, describing two very different collections of photos.

RivetEye's early weeks were good ones. The demo set's 96 percent held up on the shop floor for the common defects, hairline cracks and chatter marks, week after week. Inspectors started trusting the flagged pile and skimming past the rest, since checking every single one kept confirming it was fine.

Four things the 40-photo demo set never had a chance to prove.

Then Kestrion's quality director ran a surprise audit, pulling ten recently shipped parts at random. One had a casting void RivetEye had scored as clean.

Feasible got decided in week 3. Nobody asked again until week 26.

The real question was never whether RivetEye was 96 percent accurate. It was whether that number had ever been tested on a defect rare enough that two lucky catches could pass for proof.

When the 40-photo sample was first approved, someone said, "it's well above our bar, let's ship it," and it sounded reasonable, since the bar itself had never been rewritten since the day it covered one defect, not six.

Rerun that same audit with a 600-photo sample built to cover all six defect types across every line: the casting void class alone had been tested against 40 real examples, not two, and its true catch rate, 71 percent, was known and flagged for a second check before RivetEye ever reached the shop floor.

What I'd tell myself, hearing about that one shipped part: 96 percent was never the number that mattered. The number that mattered was two, and nobody had asked what two examples could actually prove.

BOUND, the arithmetic behind a number that felt safe enoughNot a script against ever trusting a small sample. BOUND is what tells you exactly which class never got a fair test.

The recap, one line per letter: break it down is coverage plus rare-rate trust plus line spread, own the numbers is stating where the 3 percent and 300 came from, use a range is 500 to 650 instead of one figure, nail the sanity check is explaining the ten-times jump before anyone questions it, and direction is naming the rarest defect's own rate as the number worth watching.

And if you want to be sure it really works, try it somewhere elseSame five letters, a municipal recycling line instead of a machine shop. The rare class that never got tested changes. The missing arithmetic doesn't.

Solveig Marrapodi runs product at Thistlewood Waste Services, where BinScope photographs items on a sorting belt and flags what should be pulled before it reaches the baler. A pilot judged BinScope feasible off 60 photos, mostly plastic bottles and cardboard, hitting 94 percent. Mapped onto BOUND: break it down is coverage of every material type plus enough of the rarest contaminant, a coffee cup lined with foil that looks like paper but jams the baler, plus spread across the two collection routes that feed the belt. Own the numbers means pulling from Thistlewood's own six months of jam reports, where the foil-lined cup shows up in about 2 percent of loads. Use a range covers the 250 to 400 photos of that one item needed to trust its catch rate. Nail the sanity check means explaining why a 94 percent pilot number still let three jams through in the first month. Direction names the one thing that would flip the plan: if the foil-lined cup turned out to be rarer than 2 percent, fewer examples would already be enough.

Checking the math before the go, no-go call is the step a 60-photo pilot skipped the first time.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "size the sample off your rarest case, not your total count," and stop.
Cost: there is no time to gather hundreds of examples before a decision is due. Say so honestly, and name which rare class is still untested rather than pretending the small sample covers it.
The model actually gets better, for real: if a new version of RivetEye claims a higher accuracy, that claim still needs the same rare-class test before anyone trusts it, since a better model can still be tested on too few hard examples.

Where people run it wrong.
They treat a big total photo count as proof, without checking how many of those covered the rare case that actually matters.
They trust a percentage with no visible equation behind it, as if a number alone were the same thing as evidence.
They set a sample-size rule once, for one easy case, and never revisit it as the tool takes on harder, rarer cases.

How to use it live. The moment someone asks how many examples are enough, ask yourself: what is the rarest thing this tool has to catch, and how many times has it actually been seen. Name that number, and the rest of the sample size follows on its own.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0