Describe the rollback plan for a migration that has already shipped.

How to answer this, stage by stage

Nobody is grading whether you can say "we'd roll back." They are grading whether you know a rollback has a real backlog behind it, and whether you can size that backlog instead of waving at it. Seven moves get you there.

Let's learn

Say we build a model that looks at a chest or abdominal X-ray and flags which ones need a radiologist's urgent attention versus routine review. Before this kind of triage flag existed, every scan sat in one queue, read in roughly the order it arrived, and a genuinely urgent case could wait behind a dozen routine ones if it happened to land at a busy hour.

Then Harborview Diagnostics deployed Halcyon, its first triage-flagging model. Urgent-looking scans jump the queue automatically. Average time-to-read on a true urgent case dropped from around ninety minutes to under twenty. Nobody had to guess which scans mattered most, the queue itself reordered around it.

Now Harborview has migrated from Halcyon to Northlight, a newer model that reads faster and, on the pre-launch evaluation, matched or beat Halcyon on most finding types. Nine days after full cutover, a radiologist doing an unrelated follow-up noticed a prior scan Northlight had scored low-urgency actually showed a pneumothorax it should have flagged. The patient turned out fine. The near miss is what matters.

Say plainly: the fact that Northlight is faster and better on average is not the problem. The problem is that "better on average" said nothing about whether it stayed as good as Halcyon on pneumothorax specifically, a finding serious enough that an eighteen-point recall gap means real patients sitting in a routine queue when they should be in an urgent one.

At its worst, Northlight stays live for weeks instead of days before the near miss happens, or before anyone notices at all, since nothing about a quietly lower recall trips an alarm on its own. Every extra day adds roughly 40 more true pneumothorax cases to a growing look-back list, and each one is a chance that the next near miss isn't a near miss.

The choice I would take back. When Northlight's migration was first planned, the go-live monitoring was set up to watch the aggregate flag rate, the percentage of all scans getting flagged urgent, staying roughly steady week over week. That felt like a reasonable proxy for "the model is behaving normally." It also meant a recall drop on one specific, rare-ish finding could sit completely invisible inside a flag rate that looked perfectly stable overall.

What I would leave alone. Harborview also uses a much lower-stakes model that flags scans for missing patient-position metadata before filing. If that model regressed for nine days, the fix is just re-tagging some files, there's no patient-facing urgency, and no look-back list is needed at all.

The lesson. A rollback plan that starts with "flip the routing back" and stops there hasn't actually planned anything. The routing flip is the easy part. The real plan is the list of names behind it, and how fast you can get eyes on every one of them.

Now here is the same thing as a story

The short version is above. Read on if you want to feel why day nine, not the regression itself, was the real problem.

Esther Villanueva has led Harborview's imaging-AI roadmap for three years. She ran the Halcyon launch herself and wrote the monitoring plan for it. She's the one people call at 11pm if something about a migration feels wrong, because she's usually already looked at it by the time they ask.

The Northlight migration looked clean on paper. Pre-launch evaluation showed it matching or beating Halcyon across every finding category the eval set covered, including a respectable 89 percent recall on pneumothorax specifically. The go-live dashboard, watching overall flag rate, sat calm for the first week. Esther checked it daily, saw nothing, and by day seven had scaled back to a quick morning glance.

On day nine, Dr. Amara Reyes, a radiologist doing a routine six-week follow-up on an unrelated patient, pulled up the prior scan from that patient's first visit. It had been scored low-urgency by Northlight. Looking at it now, with a trained eye and no queue pressure, she saw a small pneumothorax that should have triggered an urgent flag. She called Esther directly instead of filing a ticket, because she wasn't sure yet whether it was a fluke or a pattern.

The technical rollback itself took under fifteen hours end to end. Building and clearing the look-back list took most of that time, not the routing change.

It was not a fluke. Esther pulled every scan Northlight had scored as low-urgency over the full nine-day window and re-ran them through Halcyon overnight. Of 356 scans that Halcyon's read suggested were true pneumothorax cases, Northlight had agreed with Halcyon's urgency on 253 and disagreed, scoring them lower, on 103.

A hundred three sounds alarming stated cold. It's also, as Esther pointed out at the incident review the next morning, less than three days' worth of Harborview's normal weekly QA review load, if two radiologists were pulled onto it specifically for a day. By hour fourteen, every one of the 103 disagreement cases had a second human read. Four were confirmed as genuine misses requiring the patient be contacted for follow-up. The other 99 were confirmed as correctly non-urgent on closer review, disagreements that turned out not to matter clinically.

What stayed with Esther wasn't the four confirmed misses, upsetting as each one was, it was how close the whole thing came to running for weeks instead of nine days. If Dr. Reyes hadn't happened to pull that particular prior scan on that particular follow-up, nothing about Harborview's monitoring would have caught the regression on its own, because the aggregate flag rate never moved enough to notice.

The old monitoring watched one number: the percentage of all scans flagged urgent, expected to hold roughly steady. Pneumothorax made up such a small share of total volume that even an eighteen-point recall drop on it barely nudged the aggregate. At the original launch-readiness review, someone had actually proposed per-finding recall tracking as part of go-live monitoring. It was deprioritized as extra dashboard work for a migration already on a tight timeline.

The second version of the monitoring tracks recall by finding category against the golden set continuously, not just at the pre-launch gate, with an automatic alert if any category drops more than eight points from its pre-launch baseline over a rolling three-day window. Run the same Northlight migration through that design and the pneumothorax gap trips the alert around day three, not day nine, cutting the eventual look-back list from 103 cases to somewhere closer to 35.

The thing Esther would tell her past self, back at that launch-readiness review: a dashboard that only watches the average will only ever catch a problem big enough to move the average. The finding that matters most is rarely the one common enough to do that.

BOUND: the arithmetic behind the rollback

This is a sizing question, how big is the rollback's real workload and how fast can it be cleared, not a person's habit flipping between two settings, so BOUND fits and FLIPS doesn't.

B, break it down. Rollback cost has two parts. Technical cutover, routing traffic back to the old model, is close to fixed and fast. The real variable cost is the look-back list: scans scored during the live window, times the share that were true positives on the affected finding, times the share the new model likely under-flagged, times the review time per disagreement case.
O, own the numbers. Nine days live, 2,200 scans a day, 19,800 total scans scored by Northlight. Pneumothorax base rate about 1.8 percent, roughly 356 true cases in that window. Pre-launch recall was assumed 89 percent; the live re-score against Halcyon's read showed roughly 71 percent, an 18-point gap, meaning about 103 disagreement cases. At six minutes of radiologist time per case to pull and compare, that's about ten hours of review work.
U, use a range. If the real recall gap is smaller, 10 points instead of 18, the list is closer to 57 cases, about six hours. If it's worse, 25 points, it's closer to 142 cases, over fourteen hours. The gap itself is the least trustworthy number in the whole estimate this early, based on one near miss and an overnight re-score, not a full audit.
N, nail the sanity check. Harborview's radiology team already handles about 40 routine look-back QA reviews a week. A one-time 103-case surge is a heavy single day for two radiologists pulled specifically onto it, not a multi-week emergency. Worth escalating loudly, not worth panicking over.
D, direction. The lever that swings this number most isn't the recall gap, it's the detection window. Catching the regression on day two instead of day nine would have limited the look-back list to roughly 79 scans' worth of true-positive exposure instead of 356, an almost eighty percent reduction, larger than any realistic improvement in Northlight's actual recall would produce on its own.

One more thing the arithmetic alone doesn't show: not every disagreement case in the look-back list turns out to be a real miss. At Harborview, only 4 of the 103 disagreement cases were confirmed as genuine misses on second read; the other 99 were correctly non-urgent, a disagreement that didn't matter clinically. That doesn't shrink the review workload, every disagreement still needs a human look to know which bucket it's in, but it does mean the headline "103 cases" and the true patient-harm count are two different numbers, and a rollback plan has to report both.

And if you want to be sure it really works, try it somewhere else

Kelpwatch Fisheries Cooperative runs an AI tool that reviews trawler camera footage and flags likely bycatch, protected species caught unintentionally, for a human observer to confirm before the catch is logged. The cooperative just migrated that classifier to a newer version.

B, break it down. Same shape, different unit. Rollback cost is technical cutover, near-fixed and fast, plus the look-back list: footage segments reviewed during the live window, times the share with a true protected-species event, times the share the new model likely missed, times review time per disagreement.
O, own the numbers. The new classifier ran for eleven days across about 60 trawler-hours of footage a day, 660 hours total. Protected-species bycatch events happen in roughly 0.6 percent of footage-hours. That's about 4 true events in the window. Pre-launch recall was assumed 85 percent; a spot-check against the old classifier's read on the same footage suggested it may have missed one of the four, a 25-point-equivalent gap on a very small base count. At twenty minutes of reviewer time per flagged disagreement, that's a little over an hour of review work.
U, use a range. With such a small true-event count, the range is wide in relative terms: zero missed events would mean no review backlog at all, two missed would mean roughly two and a half hours. The small base rate makes this estimate genuinely fragile until more footage is reviewed.
N, nail the sanity check. Kelpwatch's compliance team already reviews flagged footage as routine practice, several hours a week. An extra hour or two from this rollback is well within normal capacity, not something that needs extra staffing.
D, direction. Same lever as Harborview, at a much smaller scale: the number of days the regressed model ran before anyone checked matters more than the size of the recall gap itself, because with so few true events in any given window, every additional day adds real, not proportional, exposure.

Swap the trigger and it still runs.
Speed: an interviewer caps the answer at ninety seconds. Skip straight to the shape: cut over immediately, build the full look-back list from the live window, flag every disagreement for a human, size review time against the disagreement count, not the whole list.
Cost: the radiology team is already short-staffed this week. Don't shrink the look-back list to fit available hours, borrow radiologists from a lower-urgency queue for one day instead, and say plainly that the tradeoff is a slower routine backlog elsewhere, not a smaller safety check here.
The model got better: Northlight actually turns out to beat Halcyon on every finding category once the full audit finishes. That doesn't make the look-back list unnecessary in the moment, the near miss still happened and still needs confirming, it just means the eventual finding is reassurance rather than a real gap.

Where people run it wrong.
They treat "roll back" as purely a technical action and stop the plan there, with no mention of what happens to the patients or cases already touched by the regressed model.
They build the look-back list only from complaints or flagged tickets, missing exactly the silent under-flags that never generated one.
They watch an aggregate metric for early warning and assume it will catch a regression on a rare-but-serious category, when a rare category can move sharply without denting an average built mostly from common cases.

How to use it live. Say the shape before naming a single number: "rolling back isn't the hard part, that's a routing change. The real plan is the look-back list, everyone the regressed model already touched, and how fast we can get a second, trusted look at every one of them." That buys the room to ask what's actually knowable this early, instead of promising a number nobody can defend yet.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0