What changes about your rollback plan when the thing you might roll back is a model, not code?

Marrowatch is Gravelock Systems' real-time transaction-scoring model, licensed to forty banks to catch fraud before a charge clears. Zvonko Dresner owns its rollback policy. Two weeks after the newest version shipped, Miroslava Petkanova, who runs fraud operations at Ironmere Bank, called wanting it pulled, fast. Her merchants were getting blocked mid-sale.

How to answer this, stage by stage

Nobody's grading whether you know the word "rollback." They're grading whether you'll ask what a model rollback actually restores, and what it can't.

Let's learn

Marrowatch reads every card transaction moving through a bank's system in real time and scores it for fraud risk before the charge clears.

Before Marrowatch 5, the old version had a known blind spot. Marrowatch 4 caught about 6 percent of card-testing sequences, because each charge, looked at alone, was too small to flag. Fraud teams across Gravelock's forty banks manually reviewed roughly 40 flagged sequences a day network wide, and most of the real ones slipped past anyway.

Marrowatch 5 was retrained specifically on months of confirmed card-testing sequences pulled from across all forty banks. Live, across the network, its catch rate on that exact pattern came in at 91 percent. Real money, saved before it moved: an estimated $410,000 in prevented card-testing losses across the platform in the first two weeks alone.

A model version isn't just a bigger file. It's carrying real things a plain code rollback was never built to notice, let alone restore.

Two weeks in, a different number started climbing. At Ironmere Bank, high-frequency micro-ticket merchants, parking apps, vending machines, laundromats, coffee kiosks, started getting legitimate charges declined. Their whole business is many small, fast transactions, which is structurally close to what card-testing looks like from the outside.

Here's the turn. Those extra declines were never really the problem. The problem was what happened next: the reflex was to reach for the biggest lever in the runbook, a full rollback to Marrowatch 4, before anyone checked whether that lever actually did what its name promised.

What it costs at its worst: a blind full rollback would have stripped card-testing protection from all forty banks for the roughly thirty hours it took to rebuild Marrowatch 4's retired pipeline, an estimated $48,000 in fresh card-testing losses network wide in that window alone, to fix a problem that a two-hour segment override could have solved just as fast, for six hundred accounts at one bank.

What I would leave alone: Marrowatch's sanctions and watchlist check, a fixed list of blocked account numbers, is ordinary code, not a trained model. Rolling that back is a normal code rollback, exact and clean, with nothing learned to leave behind.

The lesson: we wrote a rollback plan for code, then quietly kept using it once Marrowatch became a model with real learned behavior, and never once asked whether the two things fail the same way. They don't.

Now here is the same thing as a story

Read the short version above when you're in the room. Read this one when you want to feel why the fastest-looking fix was actually the slowest one, and the quietest one to go wrong.

Zvonko Dresner has shipped every Marrowatch release for four years. He is the person who reads a model card twice before anyone else has finished their coffee, and the one who insists on a canary rollout even when everyone else just wants it live.

Marrowatch 5 was the biggest release yet, retrained on months of confirmed card-testing sequences pulled from all forty banks. It went out on a Monday, and for two weeks it looked like the best release Zvonko had shipped. Card-testing losses across the network dropped hard. Nobody complained. Zvonko started drafting the release notes for the next one.

Then, on a Tuesday afternoon, Miroslava Petkanova called in from Ironmere Bank. Her team was fielding calls from small merchants whose card readers had started rejecting perfectly normal customers, a coffee kiosk that couldn't ring up a two-dollar latte, a parking app locking people out mid-transaction. Nine or so of these came in most weeks, background noise. This week, thirty. Last week, seventy six. She wanted Marrowatch rolled back before the end of the day.

The whole incident, start to finish. The middle step, checking before flipping anything, is the one the old runbook skipped straight past.

The on-call playbook had one instruction for a bad release: roll back to the previous version. Someone had already opened the ticket to do exactly that before Zvonko joined the call.

He asked for two hours first. Pull the complaint breakdown by merchant category. It came back fast: almost every one of the new declines sat inside one group, high-frequency micro-ticket merchants, about 640 accounts out of Ironmere's 46,000. The timing lined up exactly with the Monday deploy. Card-testing catch rate everywhere else in the network was still holding at 91 percent, meaning Marrowatch 5's real value was intact, just misfiring on one narrow, unusual-looking slice of traffic.

Then came the part nobody expected. Zvonko asked engineering to confirm Marrowatch 4 could actually be redeployed if it came to that. It couldn't, not cleanly. The feature pipeline it depended on, a device-and-merchant reputation store, had been retired three months earlier, replaced by a new one built for Marrowatch 5's different feature shape. Bringing Marrowatch 4 back meant rebuilding that pipeline from scratch, about thirty hours of work, during which all forty banks would run with Marrowatch 4's six-percent card-testing catch rate, or effectively none at all.

The decision Zvonko would take back sat years earlier, in a much smaller meeting, back when Marrowatch was mostly rule-based code and a version really was just a deploy. Someone wrote the incident runbook then: bad release, roll it back, done. Nobody ever came back to update it once Marrowatch started shipping model versions that carried real learned behavior a redeploy couldn't restore. It was never wrong on purpose. It was just never revisited.

Instead of the rollback, Zvonko's team shipped a threshold override scoped to that one merchant category, adjusting the decline line for micro-ticket accounts without touching Marrowatch 5's scoring anywhere else. It went live in under two hours. By the next morning, the segment's false decline rate was back near 0.5 percent, and card-testing catch rate across all forty banks never moved.

Run the same Tuesday again, with the old one-lever runbook still in place and no ORDER check in front of it. The ticket for a full rollback goes through as filed. Engineering discovers the retired pipeline mid-incident instead of before committing to a fix. Every bank loses card-testing coverage for the better part of two days while it's rebuilt, and Marrowatch 4 comes back with no idea the hole it once had was ever a problem worth reopening.

One version of that Tuesday ends in two hours and a fix nobody had to explain twice. The other ends in thirty hours, forty banks exposed, and a rollback that solves Ironmere's problem by quietly handing it to everyone else.

What Zvonko would tell his past self, the one who wrote "roll back to previous version" as the whole plan: that instruction was honest and complete for the product Marrowatch used to be. It just never got rewritten for the one it became.

ORDER, for a rollback that has to out-think its own last release

Not a way to make "just roll it back" sound more careful. ORDER is what forces you to check whether the old version is even the same kind of thing you think it is, before you hand it the keys back.

The single idea underneath this whole answer. A model version carries more than its own weights. Redeploying the old one doesn't undo any of it.

One door swings back in minutes. The other only looks like a door. Behind it is a thirty-hour rebuild and a fraud pattern walking back in.

The order a gut call skips. Checking whether the old version actually works comes before promising anyone it's the fix.

Three things worth stating directly, since this is where the real judgment sits. The alternative on the table, and the one rejected, was the reflex full rollback Miroslava's team had already filed a ticket for; it lost because it would have taken longer than the actual fix and thrown away real fraud protection to do it. The AI-specific failure mode worth naming by name is silent vulnerability re-exposure: a model rollback can quietly reopen a fraud pattern the newer version had learned to close, with nothing in a normal deploy log flagging it. The guardrail is a known-closed vulnerability regression set, a labeled archive of confirmed fraud patterns each model version is scored against, so any candidate rollback gets checked against known holes before it's allowed to take traffic, not after losses show up. And the trade-off is real and accepted on purpose: the micro-ticket segment's decline rate stayed mildly elevated, around 1 percent instead of an instant 0.5, for the day it took to verify the override was scoped correctly, a small, deliberate quality cost paid to protect card-testing coverage for the other 45,000-plus accounts across forty banks.

And if you want to be sure it really works, try it somewhere else

Same five letters, a city permits office instead of a bank, and this time the fragile thing isn't a card charge. It's a renovation nobody flagged for structural review.

Tollgate, built by Wardrail Civic Systems, screens building-permit applications for the City of Yewgate's Department of Buildings, deciding which ones need a mandatory structural review before work can start. Dobromir Stanish owns its rollback policy. Tollgate 5 was retrained to catch permit splitting, where one large renovation gets filed as several small "repair" permits, each one just under the dollar threshold that triggers review.

Different city, same order. The thing that can't be undone, a renovation nobody flagged, ships first, not the fastest-sounding fix.

Tollgate 4 had caught permit splitting about 9 percent of the time. Tollgate 5 caught it 84 percent of the time. Two weeks after it shipped, false rejections spiked, almost entirely among homeowners in one older-housing zip code doing genuinely staged repairs, a roof this month, wiring next month, that structurally resembled splitting from the outside.

Same rank, different lever: outcome here isn't permit throughput, it's a genuinely unsafe renovation not slipping through unreviewed. Reversibility splits the same way: rolling back would drop Tollgate's splitting-catch rate straight back to 9 percent citywide, reopening a loophole it took months to close. But dependency inverted: unlike Marrowatch 4, Tollgate 4's old pipeline was still cleanly deployable, the real gap was that no segment-level override existed to fix just one zip code. So the team's targeted fix looked different, a manual review carve-out for that zip code's inspectors, built by hand in a day, while confirming permit-splitting catches elsewhere in the city held at 81 percent before anyone touched Tollgate 5's version at all.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: check what the new version learned that the old one doesn't know, confirm the old version can actually be redeployed, fix the smallest thing that solves the real problem, and only escalate to a full swap if that fails.
Cost: no engineering time this sprint to build anything new. Ship the fix using whatever lever already exists, even a manual one, and log the missing lever as a known gap for next sprint, not a surprise for the next incident.
The model got better, for real: say Marrowatch 6 later lands with a 97 percent catch rate and almost no false declines anywhere. The rank doesn't change. Still check what the new version learned or left behind before treating a future rollback as free. A better model isn't the same state as the one it replaced, it's a different one.

Where people run it wrong.
They treat a model rollback like a code rollback, and reach for it first instead of last.
They assume "we still have the old build" means it's actually deployable, without checking the pipeline underneath it.
They fix the loudest complaint network wide instead of the segment actually causing it, and quietly give back protection everywhere else to do it.

How to use it live. Ask, before naming any fix: "what did the new version learn, or build up, that the old one doesn't know about?" That question alone usually tells you whether a rollback is actually free, or whether it's trading one visible problem for a bigger, quieter one.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0