What is the product cost of switching models after launch?

Every evening shift, Imisi Adeyemi took about 40 calls into Whistlecreek Veterinary Network's telehealth line, the first ten seconds usually enough for her to know if a caller was describing a true emergency. PawSignal is the tool that reads each call transcript and shows a severity tag next to it: routine, urgent, emergency. Nobody at Whistlecreek noticed that Imisi had been running her own second model in her head, and on a sticky note, for the better part of a year.

How to answer this, stage by stage

This question rewards whoever names the cost the eval suite can't see. Most candidates stop at "re-testing takes time." That's true and it's the boring half of the answer.

Let's learn

PawSignal reads the words a caller says to a vet telehealth line and shows the triage tech a severity tag: routine, urgent, or emergency.

Before PawSignal, Imisi read every call transcript herself and wrote her own severity call from scratch, about ninety seconds of thinking per call, forty calls a shift.

With PawSignal's first model, Imisi read the tag in about ten seconds and moved on, unless one of nine or so tricky calls a shift needed her own judgment on top of it, phrasings the model tended to under-flag.

Here's the turn: the extra mistakes weren't the problem. The old model made about the same number of mistakes as the new one, on paper. What actually happened is that Imisi's private shorthand, built over a year to patch the old model's specific blind spots, stopped matching the new model's different blind spots, and nobody knew the shorthand existed until it failed.

Same call volume, same tech, same number of tricky calls a shift. Only the direction of her private fix changed.

At its worst, a model swap that looks clean on the eval suite quietly moves the danger from "the model gets it wrong" to "the model gets it wrong in a new place nobody built a fix for yet."

What I would leave alone: the swap didn't need to touch how calls get routed to a vet once flagged, or the storage and audit trail underneath PawSignal. Those parts don't care which model produced the tag.

The lesson: an eval suite tells you how a model performs against your test set. It cannot tell you what your own staff quietly built to survive the old model's specific mistakes. Go find that before you swap anything.

Now here is the same thing as a story

The short version above is what you'd say in a design review. Read this one for the shift the wrong-queue call actually happened on.

Imisi Adeyemi could tell a true emergency from a worried-well call before the caller finished their second sentence.

PawSignal's first model, for most of a year, matched her read closely enough that she stopped double-checking the tag on routine calls entirely. She kept one thing back: a sticky note taped to the edge of her monitor, phrases she'd learned the model tended to miss. "Not himself" from an older owner, usually meant collapse. "Just tired," from the same crowd, usually meant something worse than tired. She'd silently bump the severity up whenever she heard one, before PawSignal's tag ever crossed her mind.

A private fix, built one shift at a time, riding entirely on quirks specific to one model version.

Nobody on the product team knew the sticky note existed. There was nowhere in PawSignal to log an override, so Imisi's year of quiet corrections lived only on a two-inch square of paper.

The swap shipped on a Tuesday, after the new model scored four points higher on the eval suite. Nobody asked what Imisi's sticky note was correcting for, because nobody knew to ask.

A model swap isn't a dial you nudge. It's a switch, and everything built around the old setting has to be checked against the new one.

By the second week, an owner called in describing her senior dog as "not himself since this morning," the exact phrase Imisi's old cheat sheet had trained her to bump up automatically. She bumped it, same as always. But the new model had already correctly flagged that phrase as urgent, on its own, something the old model never did. Her bump pushed a genuinely urgent call past the emergency threshold into a queue meant for less severe cases, where it sat for 22 minutes before a vet caught it on a routine sweep.

The swap itself was a single moment. The gap between her judgment and the new model's grew for weeks before anyone noticed.

When the swap was first planned, someone in the migration meeting said, "the new model scores better across the board, this should be a clean win." Nobody was wrong to say it. Nobody had a way to see the shorthand that score didn't cover.

The real question was never whether the new model was better on average. It was whether it made the same mistakes, in the same places, as the shorthand a person had already built her whole shift around.

What I'd tell myself, hearing about that 22-minute wait: the eval suite compared two models. Nobody compared the notebook taped to Imisi's monitor.

FLIPS, worn down to one swapNot a lesson in caution. A specific log you'd actually build before the next migration ships.

Held up as one page: the order these five questions get asked in, every time.

The recap, one line per letter: find the person is Imisi on the evening shift, locate the habit is trusting the tag on routine calls, identify the flip is a private fix meeting a model it wasn't built for, pinpoint the old decision is shipping with no override log, show the replay is the same phrase caught safely in canary instead of costing 22 minutes.

And if you want to be sure it really works, try it somewhere elseSame five letters, an HVAC dispatch tool instead of a vet triage line. A different flip family this time: the crew stops trusting the tool's routing entirely and goes back to radio.

Devraj Iyer dispatches HVAC field crews for a regional service company, using a model that reads incoming service calls and assigns a priority and a crew. His flip family here is abandonment, not workaround: the first model's assignments matched his own instinct closely enough that dispatch requests dropped from a phone call to a single tap of "confirm." When the company swapped to a cheaper, faster model, its assignments started clashing with which crews actually knew which buildings, something the old model had absorbed from a year of corrections nobody had logged. Devraj didn't complain. He just quietly stopped tapping confirm and went back to radioing crews directly, a slower, invisible reversion that never showed up as a support ticket, only as a usage graph that looked like adoption had "plateaued."

The same four parts protect a triage line and a dispatch line equally well, because neither is really about the model.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "log every override before you swap anything, and diff the new model against exactly those cases," and stop.
Cost: the company can't afford a two-week canary before a contract deadline. Say so honestly, and shrink the canary to the highest-override segment only, not skip it.
The model got better, for real: if the new model is a genuine, well-tested upgrade, the override log still earns its cost, because it tells you which staff-built workarounds you can now safely retire, not just which ones broke.

Where people run it wrong.
They treat a rising eval-suite score as proof the swap is safe.
They assume any manual workaround a person built is a training gap to fix, instead of a signal worth logging.
They roll a swap out to everyone at once because the model "clearly tested better."

How to use it live. When an interviewer asks about the cost of switching models, ask yourself: what has a person quietly built, by hand, to survive the old model's specific mistakes? Name that, and the real cost of the swap follows on its own.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0