How do you tell when the real problem is bad data rather than a missing model?

Speckhawk watches stamped steel brackets pass under a camera on Varnhollow Stamping's line and flags scratches, dents, and weld splash before they reach plating. Smaranda Prundeanu runs its roadmap at Ridgemark Vision. Fiifi Danquah builds the models under it. For four months, one defect kept refusing to get better no matter which model they tried. Then Fiifi stopped trying models.

How to answer this, stage by stage

Nobody is grading whether you can define "data quality." They are grading whether you would have caught yourself, three rebuilds deep, before spending a fourth.

Let's learn

Here's what happens when a camera that's supposed to catch defects keeps missing the exact same kind, no matter how many times you rebuild it.

Say a company builds a camera that watches stamped steel brackets go by on a factory line, checks each one for scratches, dents, and weld splash, and pulls the bad ones before they reach plating.

Before the camera, three inspectors per shift checked brackets by eye, at a glance, about two seconds each. They caught the big stuff: dents, cracks, obvious splash. They missed the small stuff nobody has time to catch that fast under factory light. About 46 brackets a week reached plating with a scratch nobody had flagged.

Speckhawk's actual job, four steps. The fourth box is the one that kept happening no matter which model ran the second box.

Now the camera runs. Every bracket gets checked. On paper, the launch numbers looked solid: 84 percent of all defects caught, dents and weld splash barely a hair behind a trained eye. Management called it a win.

Here's the turn. That 84 percent overall number was hiding a much smaller number underneath it. On hairline scratches, marks under two millimeters, Speckhawk's recall was 58 percent. So the team rebuilt it. A deeper network six weeks later: 61 percent. A three-model ensemble ten weeks after that: 63 percent. Four months, roughly $150,000 in engineering and compute, and the number that mattered had moved by five points.

What it costs at its worst: a team keeps rebuilding a model against data that was never reliable, spending real months and real budget chasing an accuracy ceiling that was never going to move, while the actual defect, the one a customer would find, keeps reaching plating at the same rate the whole time.

What I would leave alone: dents and weld splash never needed any of this. Inspectors agreed on those calls well over 90 percent of the time, because a dent either changes the bracket's shape or it doesn't. There is no hidden data problem to go chasing there, and adding an audit step to a class that already works would just slow the team down for nothing.

The lesson: a model can get rebuilt three times and still be innocent. If the same failure shows up the same way no matter what architecture is running it, the architecture was never on trial. The labels were, and nobody had thought to check them yet.

Now here is the same thing as a story

The short version is above, for saying out loud in an interview. Read this one for the actual week the audit and the arithmetic finally landed on the same desk.

Fiifi Danquah has a rule he picked up from a bad year at his last job: before he'll believe a model is broken, he wants to know whether the ground truth underneath it is even telling one consistent story. He'd built the first version of Speckhawk himself, training it on eleven months of Varnhollow's own labeled photos, back when the whole defect taxonomy fit on one page.

Smaranda Prundeanu runs Speckhawk's roadmap at Ridgemark Vision, and she'd promised Varnhollow's quality manager, Amaia Zubiri, that hairline scratches would be caught before the next customer audit. Varnhollow's biggest client, a seat-frame manufacturer, rejected any shipment with a visible fine scratch on the mounting face. It was the one defect Amaia's team actually got graded on.

Four gauges, one question. Which of these needles was ever going to move by trying a new engine.

The first rebuild, a deeper network, felt like the obvious move. Fiifi's team spent six weeks on it. Hairline recall crept from 58 to 61 percent. Barely worth the champagne. So they tried an ensemble next, stacking a vision transformer on top of the deeper net, ten more weeks, another round of GPU time. 63 percent.

Every other defect class was fine. Dents sat above 94 percent. Weld splash sat above 91. It was only hairline scratches, only that one class, stuck in the same narrow band no matter what ran underneath it.

Smaranda started drafting the pitch for a fourth rebuild, a bigger ensemble, more compute, another quarter. Fiifi asked her to hold off for one week.

He'd noticed something smaller than a metric drop. Looking back through three separate error logs, from three separate architectures, the same images kept showing up as false negatives. Not similar images. The same 400 or so specific photos, missed by the baseline, missed by the deeper net, missed by the ensemble. Three different models, three different ways of seeing, and they'd all failed on exactly the same pictures.

That's not what a hard-to-see pattern looks like. A genuinely hard pattern trips up different architectures in different, scattered ways. The same exact photos failing every single time meant something else was going on, something the model couldn't possibly be the cause of.

Fiifi pulled a random sample of 400 labeled hairline-scratch images and did something nobody had done since launch: he checked the metadata. Every label in Speckhawk's training tool is tagged with which reviewer made the call, and which shift they were on.

Four questions asked of a random sample, none of which anyone had thought to ask in eleven months of relying on the labels.

Day shift labelers, trained at launch by the original quality lead before she left Varnhollow, called any visible mark on the mounting face a reportable defect, 89 percent of the time. Night shift labelers, trained months later by a different supervisor who'd inherited the job with no written rubric, called the same kind of mark reportable only 31 percent of the time. Out of 6,700 hairline-scratch labels in the training set, 2,300 of them, about a third, were the exact kind of mark that got two flatly different answers depending on who was on the clock.

Fiifi brought the numbers to Smaranda before she finished the fourth-rebuild pitch. "We've been asking the model to learn a rule that contradicts itself," he told her. "Half of it says a faint mark is fine. The other half says the same mark fails an audit. No architecture can average those into one correct answer, because there isn't one."

Smaranda considered a faster option: quietly drop hairline scratches from what Speckhawk reports at all, since it was the one class dragging the launch metric down, and let inspectors keep catching those by eye the way they always had. She and Amaia's team rejected it fast. Hairline scratches were exactly what Varnhollow's biggest customer audited for. Hiding the class wouldn't fix Varnhollow's problem, it would just stop Speckhawk from admitting it had one.

So instead, Amaia's quality team sat down with Fiifi and wrote an actual rubric: a scratch counted as reportable past a measured length and a measured change in surface reflectivity, checked with the same handheld gauge every inspector already carried. Not a feeling. A number. The 2,300 disputed images got relabeled by two independent reviewers each, with disagreements settled by a third. Three weeks, about $9,000 in reviewer time.

Then Fiifi did the part that mattered most: he didn't touch the model. He retrained the original first version, the one from eleven months earlier, on the corrected labels. Hairline recall went from 58 to 89 percent. Same code. Different ground truth.

What Fiifi would tell his past self, back when the fourth rebuild pitch was already half-written: a model that fails the same way three times in a row isn't asking to be rebuilt again. It's asking someone to go check what it was actually being taught.

LEAD, for telling a data problem from a model problem

Not a way to prove a model is bad. LEAD forces the question of whether it's even fair to ask the model to fix this, before another dollar goes toward finding out.

Same starting complaint, two branches, opposite fixes. Guessing wrong here is what cost Varnhollow four months.

Three expensive habits that all leave the ceiling exactly where it was, and the one cheap check that actually moved it.

The recap, one line per letter: link "the model isn't working" to a real diagnostic check, not a feeling about the number. The early signal is a specific, repeated failure pattern across different architectures, the tell that the ceiling sits in the data. The abuse is reaching for a bigger model, more fine-tuning, or quietly dropping the stubborn class, all before checking a single label. And the decision is what makes any of it real: audit a random sample for label agreement first, fix the rubric, then retest the model you already have.

And if you want to be sure it really works, try it somewhere else

Same four letters, an insurance claims desk instead of a press line, and this time the two disagreeing labelers are two regional offices, not two shifts.

Cutlass Mutual runs ClaimWard, a model that flags auto claims worth a closer look before an adjuster signs off. Sipho Kunene leads analytics there, and he watched his team run the exact same three-rebuild pattern Fiifi had, on a completely different kind of data.

ClaimWard's recall on one specific claim shape, a claim following a prior repair on the same vehicle part, sat stuck between 48 and 55 percent across three model versions, while every other claim category performed well above 80. Two rebuilds, similar cost, similar five-to-seven-point gains. The team was drafting a case for a fourth.

Prior-repair claims were the one type sitting in the low-confidence, low-agreement corner. Every other claim shape lived somewhere the model could actually learn from.

Mapped onto LEAD, the shape holds exactly. The link is a real diagnostic check, not a feeling about the model, is it failing to learn a real fraud pattern, or does the label under a prior-repair claim contradict itself depending on which office touched it. The early signal was the same tell: three different model versions missing the same specific claims, not a scattered set of hard cases. The abuse was the same reflex too, two rebuilds funded before anyone cross-tabbed a label against which office wrote it. And the decision matched Fiifi's move exactly: audit a random sample, 350 prior-repair claims, find the regional split, write one shared rubric, relabel, then retest the original model. Recall on that claim type rose to 85 percent, no new architecture involved.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: when several models fail a class the same specific way, check the labels before the architecture.
Cost: no budget for a formal audit. Whoever owns the model pulls twenty random examples from the stalled class by hand and just reads them, checking whether they'd label each one the same way a colleague would.
The model got better, for real: say a new base model genuinely raises overall accuracy. Check the stalled class anyway. A real model gain can lift the average while the specific, mislabeled subclass stays exactly where it was, hidden under a better-looking headline number.

Where people run it wrong.
They read "recall is stuck" as proof the model has hit its ceiling, without ever checking whether the ceiling is even the model's to move.
They treat an overall accuracy number as clean, when it's really an average hiding one broken subclass underneath a lot of healthy ones.
They fix the labels once and never check again, when a new labeling team or a new shift can quietly reintroduce the exact same drift months later.

How to use it live. Before answering, ask yourself one plain question out loud: "if I retrained the exact same model on cleaner labels, would this number move?" If you genuinely don't know, say that's the first thing you'd go check, and let that be the answer instead of guessing at an architecture fix.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0