Explain the migration risk specific to a fine-tuned model.

How to answer this, stage by stage

Eight moves. The real number shows up in stage five, and the whole answer turns on what a fine-tune actually welds itself to.

Let's learn

What happens when the model underneath a feature changes, but the feature was built on top of exactly one version of it?

Corvent Health makes a tool that turns a doctor's dictated visit note into a clean, structured clinical summary. Feed a general model that dictation cold and it gets the hospital's own shorthand wrong a lot: a local abbreviation, a formulary drug name that isn't in general use, a department template that only exists at three hospitals. On Corvent's own eval of hospital-specific terms, that error rate sat at 18 in 100.

So the team fine-tuned it. They collected forty thousand real note pairs from six partner hospitals and trained the model on the hospital's own language. The error rate on those same terms fell to 4 in 100. A real win, and the summarizer became the reason two hospitals signed within the quarter.

Fourteen months later, the vendor ships a new generation of the base model, not a small patch, a real jump. Every other feature at Corvent, all fourteen of them, prompts the model instead of fine-tuning it, and every one moves to the new generation the same way: swap the model's name in a config file, run the standard eval, ship. One day, all fourteen.

The note summarizer can't do that. Its fine-tune is trained into the specific numbers of the old base model. Point its config at the new one and the training doesn't come along, the model's own numbers changed underneath it. There's no swap. There's only a new fine-tuning run: new hospital data, a new eval, a new safety check. About nine weeks.

Here's the turn. Nine extra weeks is not actually the problem. The problem is what the team does with that gap. Faced with a nine-week retrain and thirteen other launches already on the calendar, nobody schedules it right away. The note summarizer just stays on the old model.

The fine-tune's quality kept climbing steadily. Whether a migration needs a person by hand only ever had two settings.

At its worst, this doesn't stay a scheduling problem. The new base model also cuts a real safety number: it fabricates a lab value the note never mentioned on only 0.3 in 100 notes, against 1.2 in 100 for the old one. Every other Corvent feature already picked that up for free. The note summarizer, the one built to handle exactly this kind of clinical detail, is the one still running the worse number, for as long as the retrain sits unscheduled. And eventually the vendor sets a hard cutoff, ninety days until the old model's programming interface closes for good. Now the retrain runs on the vendor's clock, not Corvent's. Compressed to hit that date, it's tempting to skip collecting a full new set of hospital notes and reuse most of the old set instead, which drags some of the old model's own blind spots straight into the new one. Built badly enough, under enough pressure, the note summarizer can ship worse than if Corvent had never fine-tuned it at all.

Whether a migration rides along for free isn't a dial you read off a clean-looking config. It's a switch with two positions, and a fine-tune only ever gets the second one.

Same kind of note, two models. One grid still invents a value here and there. The other mostly doesn't, and only the note summarizer is stuck reading from the first one.

The choice I would take back. Fourteen months earlier, when the fine-tune was proposed, the team weighed it against a lighter approach: feed the model retrieved hospital terms alongside each note instead of training them in. That lighter version scored 9 wrong in 100 against the fine-tune's 4. Fine-tuning won cleanly on quality, and nobody in the room ever priced what winning would cost the next time the base model changed.

What I would leave alone. Corvent also fine-tuned a small model that only pulls billing codes out of a note, a narrow, structured task that never needed the frontier model's general skill to begin with. It doesn't chase every base-model generation, so it doesn't carry this same bill. Leave it exactly as it is.

The lesson. A fine-tune's real price isn't what it costs to build. It's what it costs every single time the ground underneath it moves, and that bill comes due on somebody else's schedule, not yours.

Now here is the same thing as a story

Pull this one out when there's more time, and you want the interviewer to feel the gap, not just note it down.

Every time Corvent Health's vendor shipped a new base-model release, the migration board refreshed on a Monday, and for a year, Ilsa Vukovic barely had to look at it. She is the only ML engineer at Corvent who owns a model end to end, and she can read the difference between two eval runs and name, before lunch, exactly which hospital's phrasing tripped the model up.

The note summarizer's fine-tune was hers too. She built it fourteen months back, forty thousand real note pairs from six partner hospitals, and watched the hospital-term error rate on Corvent's own eval fall from 18 in 100 to 4 in 100. Two hospitals signed within the quarter, both mentioning the summarizer by name in the deal.

The first minor version bump came three months later. Ilsa opened the note summarizer's row on the board by hand, reran its hospital-term eval herself, watched it hold at 4 in 100, and only then marked it done.

The second bump, four months after that, she skimmed. The automated eval had already run and passed, so she checked the number, not the notes underneath it, and moved on to the next thing on her list.

By the third bump, she didn't open the row at all. Fifteen rows, fifteen green checks, the same Monday ritual it had always been. Why would she. It kept being fine.

Then came Vantage 3, a new generation, not a patch. The platform lead posted in the migration channel at ten past nine: fifteen rows, fifteen green, done before lunch. Ilsa was closing the ticket when she noticed the note summarizer's hospital-term score in that morning's log wasn't a number. It said, plainly, not run.

A missing number isn't a red flag by itself. So she opened the row properly for the first time in three cycles, and found that Vantage 3's config had gone in clean, same as the other fourteen, and the summarizer had returned nonsense for every single dictated note in the automated eval. Not slightly worse. Nonsense. The fine-tune had been trained into Vantage 2's own numbers, and Vantage 3's aren't the same numbers wearing a new name. There was nothing to swap. There was only a new fine-tuning run, from the data up: new hospital notes, a new eval, a new safety check. Ilsa's own estimate: nine weeks.

Nine weeks is a number. It was never the real cost. Ilsa never had a migration-cost number for this feature in her head. She had a belief, the same one the whole team shared, that migrations were basically free, because for fourteen other features and two prior version bumps, they always had been. Three clean cycles is what it takes to build a belief like that. It only ever had two settings: something to check by hand, or something that just works. By the third cycle, hers had flipped to just works, and nothing was going to flip it back on its own.

So here is the decision she would take back.

Fourteen months earlier, in the meeting where the fine-tune got approved, someone had actually asked what happens when the base model changes. The answer in the room, Ilsa's included, was reasonable at the time: we'll cross that bridge when it comes, the quality win is worth it now. The lighter option on the table that day, feeding the model retrieved hospital terms alongside each note instead of training them in, scored 9 wrong in 100 against the fine-tune's 4. Fine-tuning won on the numbers in front of them. Nobody in that room priced what winning would cost the next time the ground moved.

I would put a number on that decision the day it gets made, not the day it gets tested. Say Corvent had chosen the lighter, retrieval-based design instead, accepting 9 wrong in 100 rather than 4, or had built a fine-tune small enough to retrain in days, not a full nine-week run. Run Vantage 3 through that version. It migrates the same Monday as the other fourteen. The hospital-term number holds inside Corvent's own cutoff. No missing score in the log, no nine-week project landing on Ilsa's calendar alone, and no ninety-day countdown from the vendor deciding how much time she gets to do it in. And the hospitals using it would have picked up Vantage 3's lower fabrication rate, 0.3 in 100 instead of 1.2, the same Monday as everyone else, instead of carrying the higher number for nine more weeks while Ilsa worked it alone.

If I'm honest, the third clean cycle wasn't the mistake. Nobody keeps opening a row that keeps coming back fine. The mistake was fourteen months earlier, the day we picked the version that wins today's eval without ever asking what it would still cost to keep winning after the ground underneath it moved.

The five letters, walked through Ilsa's migration board

The letters matter less than which one snaps first. Here's the same five steps, mapped onto the Monday the note summarizer stopped being routine.

FLIPS, five rows

"They should have retrained faster" is a diagnosis anyone can offer after the fact. The harder part is naming the exact bridge that only ever got crossed once, in one meeting, and showing there was no cheaper fix once a missing score had already shown up in the log.

And if you want to be sure it really works, try it somewhere else

Falkirk County is nowhere near a hospital. Its Building and Safety Department fine-tuned a model to flag permit applications for violations of the county's own, frequently amended building code, the kind of local rule a general model has never seen. Same question, a different flip. Nobody reclaims the migration by hand the way Ilsa did. The team just quietly stops opening the ticket for it.

F. Emrys Greaves, the one data engineer at Falkirk County who maintains the fine-tuned permit-violation model and the only one who can explain why it flags what it flags.
L. Twice before, when the vendor shipped a new base-model generation, someone filed a ticket to schedule the county's re-fine-tune. Both times nothing about the classifier looked broken while the ticket sat open, so it kept getting pushed behind budget season and other requests, until filing the ticket at all stopped happening.
I. A different flip from Ilsa's. He doesn't reclaim the migration and own it as a project. He stops opening the ticket, full stop. One setting: an upgrade ships, and a re-fine-tune gets filed and worked. The other: an upgrade ships, and nothing happens, because the old classifier still runs, so there's no visible reason to.
P. When the fine-tune first shipped, the team decided not to tie its migration to the vendor's own deprecation calendar, only to whether anything looked broken. Reasonable then, deprecation felt far off, and nobody wanted another standing task on a thin backlog.
S. Tie the re-fine-tune trigger to the vendor's deprecation date itself, not to whether anything looks broken. Run it again: ninety days before the old base model's programming interface closes, the ticket files itself and blocks that quarter's roadmap. The retrain finishes with three weeks to spare, instead of the county finding out the classifier is dead the same day the old interface starts returning errors and every permit review in the building department stops cold.

Flashcards (click a card to flip it)

Eight fixed slots, pulled straight from the answer above.

Check yourself Score: 0 / 0