Examine how an AI product changed after a major model upgrade.

Vantpoint scans a radiology image and flags the ones likely to show something critical, so a radiologist reviews those first. Dr. Fennimore Adisa reads scans at a mid-size regional hospital, and her habit of spot-checking Vantpoint's "clear" pile is the whole reason this story has an ending worth telling.

How to answer this, stage by stage

Seven moves. The second one reframes the whole question before any numbers show up.

Let's learn

Vantpoint reads a radiology scan and flags the ones most likely to show something critical, so a radiologist reads those first, before working through the rest of the day's scans in order.

Before the v3 upgrade, Vantpoint caught about 82% of genuinely critical findings, so radiologists like Fennimore kept a habit of spot-checking a random 15% of the "cleared" pile too, just in case.

With v3, sensitivity jumped to 94%, a real, substantial improvement that showed up clearly in every monthly accuracy report the hospital ran.

The turn: the accuracy gain was never the risk. The risk was what radiologists quietly stopped doing the moment the headline number looked this good.

At its worst: the hospital reports a record accuracy quarter to its board, while the actual habit that used to catch Vantpoint's rare misses has quietly disappeared, months before anyone connects the two.

What I would leave alone: scans Vantpoint actively flags as likely-critical still get full radiologist attention, no change needed there. The entire risk lives in the pile Vantpoint marks clear.

The lesson: a genuinely better model is still a perturbation. The headline number that improves is rarely the number that predicts what happens next.

Now here is the same thing as a story

Use the short version above under time pressure. Read this one for how close the missed case actually came.

Fennimore has read scans for twelve years and built her own quiet habit years before Vantpoint ever existed: always glance at a handful of the "normal" pile, never just the flagged ones.

The spot-check step, third in line, is the one that quietly emptied out after the upgrade.

The weeks after v3 launched felt genuinely great. Vantpoint's flags were sharper, its false alarms dropped, and Fennimore's own caseload of confirmed criticals per week held steady even as her total review time shrank.

Sensitivity is the clock everyone was watching. It was never going to ring first.

The habit thinned quietly, in three small steps nobody wrote down. First, Fennimore spot-checked ten cleared scans a day instead of fifteen. Then five. Then, some days, none at all, since the flagged pile alone already felt thorough enough to fill her attention.

Eight weeks separate the moment the spot-check habit collapsed from the moment anyone noticed a real problem.

Week eleven, a rare, atypical presentation of a genuinely critical finding sat in Vantpoint's cleared pile. Nobody spot-checked that batch that day. It surfaced two weeks later, at a routine follow-up, not through the hospital's own review process.

The metric that would have warned everyone earliest is exactly the one that never made it onto a dashboard.

The review afterward found nothing wrong with Vantpoint's model. What it found was a hospital-wide spot-check rate that had fallen from 15% to 2% within three weeks of the upgrade, a chart nobody had been keeping.

Three of these four already existed somewhere. Only the alert on a falling spot-check rate had to be built new.

The old dashboard asked the hospital to trust that a rising sensitivity number meant everything downstream was fine. The new one tracks the spot-check rate directly and triggers a mandatory audit the moment it falls too far.

I picked sensitivity as the one number to report every quarter because it was the cleanest, most shareable win. It took one rare finding sitting unspotted for two extra weeks to see that the number I was watching had never been built to catch what radiologists quietly stopped doing.

LEAD, in one screenNot a dashboard of everything. LEAD is what finds the one number that would have rung first.

Four letters, and the early signal is the one a headline-metric answer always misses.

The recap, one line per letter: link is catching every real critical finding, early signal is spot-check rate collapsing weeks ahead of any missed case, abuse is a hollow, rushed spot-check logged just to hit a number, and decision is a mandatory audit triggered automatically below 8%.

And if you want to be sure it really works, try it somewhere elseSame four letters, a waste-management sorting line instead of a hospital.

Sableton runs an AI vision system that sorts recyclables on a conveyor line, flagging contaminated batches for a human sorter to pull before they reach the baler. Bellamy Grennock oversees quality control at a regional materials recovery facility.

Mapped onto LEAD after Sableton's own major model upgrade: link is keeping contamination out of baled recyclable loads, not the model's raw sort-accuracy number. Early signal is how often human sorters manually re-check a batch the model marked clean, which fell sharply the week accuracy jumped, well before any contaminated bale actually got rejected by a buyer downstream. Abuse: a sorter could tap "checked" on a batch without actually pulling it off the line to inspect. Decision: below a set manual re-check rate, the facility automatically routes a larger random sample of "clean" batches to a slow-lane manual inspection station for that shift.

Swap "spot-check rate" for "manual re-check rate" and the same dashboard gap holds for a sorting line.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "watch the human spot-check rate, not the accuracy number, it collapsed weeks before any missed case," and stop.
Cost: there's no budget this quarter to build a new dashboard metric. Say so honestly, and start with a simple weekly manual count of spot-checks logged, even without automated tracking yet.
The model gets worse, for contrast: if Vantpoint's next update genuinely regressed instead of improving, the risk flips to substitution or workaround behavior, radiologists rationing trust toward easy cases, a completely different leading signal to watch.

Where people run it wrong.
They treat a model upgrade as purely good news and stop looking for any new risk it might introduce.
They report the headline accuracy metric as the whole story, missing that it can't see a human habit disappearing underneath it.
They build a metric and never attach a real decision to it, so it just sits on a dashboard nobody acts on until it's too late.

How to use it live. When someone describes a product after a big model upgrade, ask yourself: what human behavior might this improvement quietly end. That behavior, not the upgrade's own scorecard, is usually the real story.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0