What is the switching cost of a fine-tuned model versus a prompted one?

Cascade Pay is a small business lending platform. RiskLine is the part of it that scores a loan application for default risk before an underwriter ever opens the file. Halima Bassong is the senior underwriter who trusts what it hands her every morning. Tobias Kreft is the engineer who asked, in a planning meeting, what happens the day the vendor changes the model underneath it.

How to answer this, stage by stage

Nobody is scoring whether you know what fine-tuning means. They're scoring whether you priced the cost of getting back out of it before you priced the cost of getting in.

Let's learn

RiskLine is the part of Cascade Pay that reads a small business loan application and scores how likely it is to default, before an underwriter opens the file.

Before RiskLine, Halima and three other underwriters split about 40 applications a day by hand, roughly 20 minutes of reading each, close to 13 hours across the team just to get through the pile. RiskLine now scores all 40 in seconds and hands Halima the roughly 6 it flags as genuinely borderline, about 20 minutes of her actual attention.

The two sides were never competing on accuracy alone. They were competing on how expensive each one is to undo.

Here's the turn: RiskLine's day-to-day mistakes were never the real problem, it cleared its accuracy bar comfortably. The real problem showed up the day the model underneath it had to change, and Halima found out about it only because the flagged pile got stranger for a week and nobody could say why.

At its worst, the cost isn't a bad score, it's a fine-tuned model nobody can afford to move off of, still running on a base model everyone else has already left behind, because the retraining bill keeps getting pushed to next quarter.

What I would leave alone: a task like standard field extraction from a fixed loan form barely changes and the base-model market under it isn't shifting fast. Fine-tuning something that stable is fine, because you rarely need to switch it in the first place.

The lesson: a fine-tuned model isn't a better answer sitting next to a worse one. It's a real commitment with a real bill due the day you need to move, and that bill is the number that should decide whether you take it on, not how good the demo looked on day one.

Now here is the same thing as a story

The short version above is what you'd say out loud in the room. Read this one for what it actually felt like the week the ground moved and nobody had priced it.

Halima Bassong had underwritten small business loans for six years, long enough to spot a shaky cash-flow statement before she finished reading the first page. RiskLine had been live for eight months, and her mornings had settled into something steady: open the flagged pile, work through six or seven real judgment calls, done well before lunch.

Nobody at Cascade Pay had asked which one they were actually signing up for.

The kickoff meeting for RiskLine's second version had gone the same way most of these meetings go. Someone suggested fine-tuning a model on Cascade Pay's own eight years of historical loan outcomes, and it sounded like the serious, grown-up choice. Tobias Kreft, an engineer two seats down, asked the question that reordered the room: "What happens the day the vendor changes the model we fine-tuned this on?" Nobody had a real answer.

They fine-tuned it anyway, and for a while it looked like the right call, ninety-four percent accuracy on the eval set, three points better than the best prompted version they'd tried. Then, five months in, the model provider announced it was retiring the exact base model RiskLine had been fine-tuned against, with a ninety-day migration window.

A prompted sibling feature on the same team patched itself the same afternoon the notice arrived. RiskLine took three weeks, and for ten of those days nobody could prove whether it was still working.

A prompted feature on the same team, one that summarized applicant references, was patched and reshipped within six hours: new prompt, same eval set, done. RiskLine took three weeks to recurate its training data, retrain against the new base model, and get through revalidation, and because there was no versioned golden set from launch, the first ten days of the retrained model ran with nobody able to say for certain whether its accuracy had held.

Halima noticed something was off before anyone told her: the flagged pile that week held two applications she'd have expected RiskLine to catch cleanly on its own, both quietly borderline in a way that felt new. She flagged it to her manager, more out of habit than alarm. It took another four days before engineering connected her note to the retraining that was already in flight.

RiskLine sat exactly where switching cost bites hardest: a model that rarely gets swapped, doing something too custom to just re-prompt.

Fine-tuning RiskLine had made complete sense the week it shipped. The historical data was real, the accuracy gain was real, and nobody expected the vendor to retire the base model within the year. It stopped making sense the moment that notice landed, because by then the cost of moving had already been signed for, three points of accuracy earlier, without anyone pricing what leaving would cost.

None of this is optional once you've fine-tuned. Each row is a real week, not a formality.

Here's the replay: with a versioned eval set built the second time around, the next base-model retirement notice triggered a retrain that finished in eight days instead of twenty-one, and the dip got caught on day two instead of day ten, because there was finally something to check it against. The prompted sibling feature, unchanged, patched itself again in an afternoon.

What I'd tell myself, sitting in that first kickoff meeting when Tobias asked his question: the accuracy number on day one was never the whole price. The real price shows up the day you have to move, and if you haven't priced that day, you haven't actually priced the decision.

PICK, run again on switching cost for a fine-tuneOne line per letter, in case you want to say it fast under pressure.

The recap, one line per letter: position is prompt-first, impact is measured in hours versus weeks for whoever has to do the swap, cost asymmetry is that the fine-tune's real bill is hidden until the vendor forces your hand, and kill criteria is a proven eval gap plus real data, both required before you sign up for the switching cost.

And if you want to be sure it really works, try it somewhere elseSame four letters, a pharmacy chain's refill-reminder tool instead of a lender's risk score.

Ingrid Solberg is the AI PM at Cordell Linguistics, a company that provides real-time interpretation for hospital intake calls. A stakeholder pushes to fine-tune a model on Cordell's own transcripts to catch medical terminology faster. Mapped onto PICK: position is still prompt-first, because a general model already handles medical vocabulary well with the right instructions in the prompt. Impact lands on interpreters mid-call, who need a correction inside two seconds or they lose the caller's trust. Cost asymmetry: a prompted fix ships in an afternoon and a fine-tune here would need constant retraining as new medical terms and drug names enter use monthly, a switching cost that never actually settles down. Kill criteria: only revisit fine-tuning if a full quarter of real call transcripts shows prompting missing the same class of term repeatedly, and even then, retrieval over an updated terminology list gets tried first, since it solves a moving-target vocabulary problem without fine-tuning's ongoing retraining bill.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "prompt by default, because its switching cost is hours not weeks, and only fine-tune once that's proven insufficient," and stop there.
Cost: no time to build a golden eval set before a decision is due. Say so honestly, and make building that set the very next task, since without it nobody can prove a swap even worked.
The model got better, for real: a new base model clears the accuracy bar on a plain prompt alone. Retire the fine-tune. A better prompted option removes the whole switching-cost problem instead of managing around it.

Where people run it wrong.
They price the cost of training a fine-tune and never price the cost of leaving it.
They fine-tune before a versioned eval set exists, so a future swap has nothing to be checked against.
They treat "it scored higher on day one" as the whole decision, instead of asking what day one hundred and eighty looks like.

How to use it live. The moment someone proposes fine-tuning, ask one question out loud: "What does it cost us the day we need to move off this exact model?" That question alone buys real thinking time, and it's usually the number nobody in the room has actually worked out yet.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0