How long should you keep the old model available after migrating?

How to answer this, stage by stage

Seven moves. The trap in this question is answering with a number of weeks, like it is a policy setting, when the real question is what has to be true before the countdown is even allowed to begin.

Let's learn

Verrian is a tool that listens to a call-center phone call and turns it into text within a minute of the call ending, so a QA team never has to sit and re-listen to find out what an agent actually said.

Before Verrian, Larchmont Contact Solutions checked required disclosures the old way: a QA team pulled a random 5 percent of the day's 2,400 calls, about 120 of them, and listened to each one for around 12 minutes, catching roughly 55 percent of the calls where a disclosure had been skipped, in that small sample alone.

The turn. When Thornquist shipped v3, the extra accuracy was never the problem.

Nothing in the countdown is safe to start until the middle box is checked, and it's the one Thornquist skipped.

Here's what that costs at its worst. A missed disclosure that a regulator finds, that your own system swore didn't exist, isn't a bug ticket. It's the exact thing the automated scanner was built to prevent, failing without ever saying so.

What I would leave alone. Most of Thornquist's customers used Verrian for quick internal call summaries, nothing downstream parsing the output. For them, a fast, fixed cutover is fine. Holding v2 open for a customer who never built anything against its exact shape just spends money protecting nobody.

The lesson. A model version isn't only its accuracy. It's an exact output shape, and someone downstream may have quietly wired real code against that shape. That's a silent degradation risk, not a wrong-answer risk: nothing crashes, nothing looks worse, it just quietly stops working, and an accuracy eval would never catch it. The guardrail isn't a better eval set, it's checking who's still depending on the old shape before you ever touch the switch.

Now here is the same thing as a story

The short version is above. Keep reading if you want to feel why eleven days of silence read as clean.

Emlyn Sowerby has run QA for Larchmont's insurance line for five years. She can catch a skipped disclosure in the first ninety seconds of a transcript, before the agent's even reached the pitch.

Verrian's automatic scanner arrived two years ago, and for most of that stretch it was the best part of Emlyn's morning. Every day at seven, a report was already sitting in her inbox: every one of yesterday's 2,400 calls scanned overnight, flagged calls listed by name, nothing to hunt down by hand.

So she stopped spot-listening to a sample to double-check the scanner's work. Then she stopped reading the full report, just the flagged count at the top. Then, most mornings, she stopped opening it at all. Zero at the top meant clean. It was always zero.

Then, on a Wednesday in March, a compliance officer from the state insurance division called to schedule a routine audit: ten calls, picked at random, from the last three weeks.

Which week the notice period lands in, you can fix next release. A customer's script going dark for days, you cannot un-happen.

Emlyn wasn't worried. Verrian's scanner had caught nearly everything for two years straight.

What she didn't know was that v2, the model her scanner's parser was built to read, had gone dark eleven days earlier, right on Thornquist's 60-day sunset date. Her script's calls had quietly started landing on v3 instead, whose diarization output used a cleaner tag format, one her old regex never matched. The scanner didn't crash. It didn't log an error. It just returned zero flagged calls, every single morning, for eleven days running, right up to the audit call.

The audit turned up two calls, out of ten, missing the required disclosure. Both from inside that eleven-day window. Both should have been flagged the morning after the call. Neither was.

Emlyn wasn't careless. She trusted a system that had earned two years of being right. The real cost wasn't two missed calls out of ten sampled. It was every call in that window that nobody had actually checked, silently, while the report at the top of her inbox kept saying zero.

Back when Thornquist scoped the v3 rollout, in the meeting where Corentin's team picked the sunset date, someone asked whether 60 days was long enough. The room agreed it matched what most customers had needed for the last upgrade. Nobody pulled the usage numbers for the customers still using v2 that week. Nobody asked whether any of them had built something that would break silently instead of loudly.

I'd go back to that meeting and add one step before the 60-day clock starts for anyone: pull usage telemetry per customer, and for the ones still near their old level, call them directly before the clock even begins. Same near miss, same audit, but this time Corentin's outreach catches Larchmont three weeks out. Emlyn updates her parser for v3's tag format before v2 ever goes quiet. The audit that March finds ten clean calls, not eight.

One design let a countdown start the same day for every customer, whether or not anyone had checked who needed it. The other made checking that the first step, not an afterthought.

What I'd tell myself, back in that scoping meeting: a sunset date isn't a length of time. It's a bet that nobody quietly built their business on the exact shape you're about to remove.

ORDER, for ranking what has to be true before the old model goes dark

GUARD would fit if this were about who gets harmed and can't push back. This is a straight ranking of conditions by what's hardest to undo if skipped, which is ORDER's job.

Same order, an orchard drone instead of a call center

Kettleborough Orchards runs a model over drone photos that flags diseased fruit rows before a crew walks the block, so nobody sprays or replants the wrong section by hand.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0