How do you triage an incident where the code is fine and the model is the problem?

How to answer this, stage by stage

Nobody is grading whether you can say "check the logs." They're grading whether you can tell a real regression apart from a world that quietly moved, without guessing. Seven moves get you there.

Let's learn

What does it mean when engineering swears the code is clean and the numbers are still wrong? Say we build a model that scores every payment in real time and holds the risky ones back before money moves.

Before a model like this, a fraud team catches bad transactions the old way: rules a person writes by hand, a list of stolen card ranges, a few dozen "if this then hold" checks updated every week. It catches the obvious stuff and misses anything a fraud ring hasn't tried before. On a normal week, about 40 percent of real fraud gets through, and one analyst spends most of her week just keeping the rule list current.

Then a company like Torvane Pay ships Sentinel, a model that replaces most of that by hand work. It scores 2.3 million payments a day and catches, on a normal week, 91 out of every 100 real fraud attempts before they turn into a customer's chargeback. The team stops hand-writing rules and starts trusting the score.

Then one Monday, the chargeback report shows confirmed fraud loss at nearly double its usual share of revenue. Everyone's first question is whether engineering broke something in the last deploy. Say plainly: that's not the useful question. Nothing shipped that week. The code is fine. The real question is which of three things actually happened before anyone looked: did the retrain quietly make the model worse, did the world change and the model hasn't caught up yet, or is this just a small, loud number that would have settled on its own.

At its worst, the team spends a week rolling back a model version that never did anything wrong, loses the real gains that version made everywhere else, ships a rushed patch that overcorrects and blocks good customers, and the actual gap, a fraud ring using a pattern nobody has taught the model yet, stays wide open the entire time.

The choice I would take back. When Sentinel v4 went live, the team retired v3 completely. Nothing kept running in the background. That felt tidy, one model to monitor instead of two. It also meant that six weeks later, when the real question became "did v4 actually get worse at this," there was no old model left to check it against, only v4's own word for it.

What I would leave alone. Sentinel's step-up authentication screen, the one-time code a flagged customer sees, hasn't changed and doesn't need touching here. Whatever this turns out to be, the fix lives in the score, not in what happens after a hold fires.

The lesson. A model that's wrong and a measurement that's wrong look identical from the dashboard. The only way to tell them apart is to check the measurement first, on purpose, before anyone starts rewriting the model to fix a problem that might not even be there.

Now here is the same thing as a story

Read this version when you have time to feel why the shortcut was tempting in the first place.

Anouska Brannick has run fraud risk product at Torvane Pay for five years. Ask her what year a specific chargeback rule got written and she'll usually just know, the kind of memory that comes from having actually shipped every version of the thing herself.

When Sentinel v4 went live, the rollout was Torvane's cleanest yet. Fraud loss held steady through the whole quarter. The dashboard Anouska checked every Monday morning stayed green, week after week, and the team started trusting the retrain process itself, not just the model it produced.

Somewhere in there, the team's incident habit changed too, quietly. Early on, any fraud spike got the full workup: a timeline, a recut by segment, a replay against the old model. By month four, the workup had shrunk to one question: did anything deploy. If the answer was no, the model got the blame by default, because nine times out of ten that really was where the bug lived, and a shortcut that keeps working stops feeling like a shortcut.

Then, on a Monday, the chargeback dashboard showed confirmed fraud loss at nearly double its usual share of revenue. Nothing had deployed that week. Engineering confirmed it inside the hour: the pipeline was healthy, latency was normal, no code had touched anything near this metric.

Someone on the call said the four words that end these meetings fast. "So it's the model."

Anouska's first move wasn't to agree or disagree. She asked for a timeline. Sentinel v4 had shipped eight weeks earlier. Two weeks after that, a new card-testing pattern started showing up in raw transaction logs, small, no alerts, because nothing alerts on a pattern the model has never been told to watch for.

The retrain looked fine. The real break sat two weeks past it, invisible. Nobody saw either one, because the number that finally moved didn't move until six weeks after that.

The proposal on the table was to roll Sentinel back to v3 that afternoon, as a precaution, while someone looked into it properly later. It's a reasonable-sounding move. It's also exactly the kind of decision that gets made under pressure without anyone checking whether it would even help.

Anouska stopped the rollback. Not because she was sure it was wrong. Because nobody on the call could actually say what v3 would have done differently, and Torvane hadn't kept it running anywhere to check.

Eight weeks earlier, in the cutover meeting for v4, someone had asked whether to keep v3 running in the background for a while. The answer was no, cleaner infrastructure, one model to monitor, one thing to reason about. It felt like good hygiene. Nobody in that room was thinking about a Monday eight weeks out when they'd need v3 to answer a question it could no longer answer.

Standing up a shadow copy of v3 from a saved checkpoint and replaying six weeks of real transaction history through it took the infrastructure team about eleven hours, most of it waiting on a batch job. Not a rollback. A rebuild done from evidence instead of instinct. By Wednesday morning the numbers were in: v3 missed the new card-testing pattern 75 percent of the time, v4 missed it 78 percent. Close enough that the gap was noise, not a regression. On the established patterns, the ones the team actually retrained v4 to improve, v3 sat at 85 percent caught against v4's 93.

A rollback would have thrown away eight points of real improvement on the patterns v4 was built to fix, and still missed the new pattern just as badly, because the new pattern was never v4's mistake. It was reality's.

What I'd tell myself, back in that cutover meeting: the version you're retiring isn't done being useful the day you stop trusting it as your daily driver. Keep it alive a while longer, quietly, in the background, because the day you need to ask it a question is a day you can't predict.

TRACE, laid out plainly: five moves before you're allowed to blame the model

Something broke and the code checked out clean, so this is a diagnosis question, not a person's habit flipping between two settings. TRACE fits. FLIPS doesn't.

T, timeline. When did this actually start, not when did anyone notice. Include things that looked like wins near the real start date, not just the day the number moved.
→ Sentinel v4 shipped eight weeks before the incident opened. The new fraud pattern showed up two weeks after that, six full weeks before the first chargeback landed.
R, recut. Slice the number by segment before reacting to the blended average. A modest overall drop is often one slice cratering while the rest of the book looks fine.
→ 91 to 89 percent overall hid a real split: 93 percent caught on established patterns, 22 percent caught on the new one.
A, assume nothing. Rule out the boring explanations before blaming behavior: bad attribution between the label and the model version, a sample too thin to trust, a reporting lag that makes two different weeks look like one.
→ The chargeback-to-model mapping checked out clean, the new-pattern slice was over a thousand cases, not noise, and the six-week dispute window explained why the report and the real event landed on different weeks entirely.
C, cause candidates. Name three real suspects, not a shrug: a regression from the retrain, a genuine shift in the world the model has never seen, or a normal week misread as a crisis.
→ Regression from v4's retrain, a genuinely new fraud pattern neither model version has ever seen, or noise on a small slice.
E, evidence test. Find the one check that tells your top two suspects apart, so the answer doesn't come down to whoever argues loudest.
→ Replay the real transactions through v3 in shadow. It missed the new pattern almost identically to v4, which clears the retrain and confirms a real shift in the world instead.

One more thing the test alone doesn't show: 78 percent missed on v4 and 75 percent missed on v3 isn't treated as "basically the same number" by eyeballing it. With over a thousand cases in that slice, a three-point gap sits well inside the range you'd expect from chance alone. The bar for calling something a real regression isn't a nicer-looking chart. It's whether the gap survives a sample-size check, not whether it feels different.

And if you want to be sure it really works, try it somewhere else

Corvide Health runs Vireo, a model that flags CT chest scans for a specific urgent finding before a radiologist opens the file. One month, the urgent-flag rate for that finding drops across the network. Radiologists start catching things Vireo should have caught first. On-call engineering checks it the same way: no deploy, clean pipeline, code fine.

T, timeline. Vireo v3 shipped five weeks before anyone noticed. That same week, unrelated to the model launch, Corvide's downtown imaging center swapped in a new CT scanner brand with different default image settings, a hardware change nobody thought to connect to a software metric.
R, recut. By site, not by network average: downtown's catch rate for that finding fell from 88 percent to 41 percent. The other eleven sites, still on the old scanner brand, held around 89 percent.
A, assume nothing. Check that scan metadata tags the right site, not a mislabeled feed. Check the case count at that one site is big enough to trust, about 340 relevant scans in the window, plenty. Check the drop isn't just fewer real cases that month, confirmed against the site's usual case mix, no change there.
C, cause candidates. A regression from v3's retrain, a real shift from the new scanner's image calibration, or noise from one smaller site's numbers.
E, evidence test. Replay downtown's post-swap scans through v2, the retained previous version. It also missed them, catching only 39 percent against v3's 41. On the eleven sites still using the old scanner, v3 actually beat v2, 91 percent against 88. The new scanner's pixel calibration was the real cause, not the retrain.

Swap the trigger and it still runs.
Speed: an interviewer caps the answer at ninety seconds. Skip straight to the core move: recut the number by segment before blaming the model, then settle the top two suspects with one replay against the retired version.
Cost: engineering can't spare a standing shadow copy for every model. The fallback is a smaller, sampled offline replay from a saved checkpoint, slower to get an answer, but it keeps the same test alive without the ongoing compute bill.
The model got better: it turns out v4 catches the new pattern fine too, along with everything else. That doesn't make the diagnosis unnecessary. It's still the only way anyone would know that for certain instead of assuming it from a metric that could have improved for the wrong reason as well.

Where people run it wrong.
They see the code is clean and stop there, treating "not code" as the same thing as "definitely the model," when the measurement itself could be the actual problem.
They react to the blended average instead of recutting it, and fix the whole model to solve a problem that lives in one narrow slice.
They set a flat "must never miss it" bar for a model that is fundamentally probabilistic, instead of a calibrated pass rate checked against what the previous version would have done on the same cases.

How to use it live. Say the split before naming a single suspect: "this could be the model, the world, or the measurement, and I want to rule out the boring ones first." That buys the room patience while you actually check, instead of reacting to the first plausible story anyone tells.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0