What is the economic argument for a smaller fine-tuned model over a large prompted one?

Verrant Health priced its coding assistant like a bigger version of one call. The real choice was between paying for accuracy every single time, or paying for it once and being done.

How to answer this, stage by stage

Nobody's testing whether you know what fine-tuning means. They're testing whether you'll price two different cost curves, or just say the small model is cheaper and stop there.

Let's learn

What does it actually cost to remind a model of the rules, every single time, instead of teaching it those rules once?

Codeglyph is the coding assistant Verrant Health's billing team uses. It reads a clinical note and hands back the CPT and ICD-10 codes a biller needs to file the claim.

Before Codeglyph, a certified coder read every note by hand and worked out the codes themselves. A complex outpatient note took about twelve minutes to code correctly. At Verrant's first hospital, coding 20,000 notes a month by hand cost the billing team about 4,000 hours.

Codeglyph launched on a big, general-purpose model. Coders still checked its work, but the model did the first pass: reading the note, matching it against the coding guideline, and proposing codes. At launch that cost about $700 a month in model calls. Small money. Nobody watched the line.

The turn: then Verrant added two more hospitals and a clinic network. Volume climbed to 120,000 notes a month, and the bill climbed with it, to $4,200 a month. The extra spend was never really the problem. The problem was that this line would never stop climbing, because a big prompted model has to be handed the coding guideline and a stack of worked examples fresh, on every single note, forever.

The visible cost shows up on the invoice, on schedule, every month. The hidden one shows up on a biller's desk, weeks after nobody was watching the model.

At its worst, saving money by skipping a scheduled retrain doesn't fail loud. It fails quiet. The model keeps answering with confidence, using a code that's already out of date, and nobody finds out until a wave of claims comes back denied.

The choice I would take back: in the year before the switch, every time the bill climbed, Selah trimmed a few of the worked examples and guideline excerpts out of the big model's prompt to bring the number back down. Codeglyph's prompt shrank three times over five quarters, a little each time, and nobody ever measured what any single trim cost in accuracy, only what it saved in tokens.

What I'd leave alone: Codeglyph's "check my note before I sign it" mode, where a single physician pastes one note and asks which codes look right, stays on the big prompted model. It's occasional, it's low volume, and the model's flexibility to handle whatever a doctor happens to paste matters more there than shaving a fraction of a cent.

The lesson: a monthly bill you can see is not the same thing as the real cost. The real cost is whichever number you're not watching, and a big model's bill is easy to watch precisely because it's the one that's already on the invoice.

Now here is the same thing as a story

Read the long version below when you want to feel why skipping one small retrain went wrong, not just be told that it did.

Selah Tolbert could read a denied-claims report and tell you which clinic caused it before finance finished loading the spreadsheet. She'd spent five years building coding automation before Codeglyph existed, and she picked its model architecture herself the month it launched.

The first year, Codeglyph ran quietly in the background. Coders opened their queue every morning around seven, and by nine the easy notes were already coded and waiting for a quick check. When volume tripled and the big model's bill climbed past what finance was comfortable with, Selah built the fine-tuned replacement herself: four thousand certified-coder-labeled notes, a held-out set of five hundred for testing, and a model that matched the big model's best accuracy for about a hundred and fifteenth of the cost per note. The switch went well. The bill dropped from $4,200 a month to under two thousand. Everyone was pleased.

The plan was to retrain twice a year, once for January's CPT update, once for October's ICD-10-CM update, each one checked against the five hundred-note eval set before it shipped. The first cycle went exactly like that. The second one didn't.

It was the last week of the quarter, and finance had asked every team to find something to trim before the books closed. The retrain was the easiest line on Selah's list to cut: ten thousand dollars, and the January change list looked small, a handful of wording tweaks, a few codes retired that Verrant barely used. She read it herself, decided it wasn't worth the spend this cycle, and told her manager she'd catch the real update in July instead.

It came back on an ordinary Thursday, six weeks later, not as a crash but as a number in a report: denials for a specific family of outpatient visits had jumped. A biller flagged it as strange rather than urgent, since nothing about the notes themselves looked different. Selah pulled the claims and found the pattern fast. A handful of the codes on January's "small" change list weren't wording tweaks after all. Two had been retired outright, replaced by a more specific pair, and Codeglyph, trained on the old rulebook, kept confidently assigning the retired ones because nobody had told it otherwise.

She ran the five hundred-note eval set against the model as it stood, on the exact code family involved. Accuracy on that family had fallen from about 94 percent correct to about 79 percent. It had been sitting at 79 percent for six weeks, invisible, because the overall average across every code family barely moved, and nobody was watching one family on its own.

The decision Selah would take back isn't skipping a retrain under budget pressure. That's a normal, fair trade to consider some quarters. The decision she'd take back is deciding to skip it based on her own quick read of a change list, instead of a rule that ran the eval set first, no matter how small the update looked.

Run that Thursday again with one change: no code-set update, however small it looks, gets skipped without the eval set running first, and a floor has to hold, at least 92 percent on every code family, not just the average. January's update would have failed that floor within a day of the update landing, on that exact family. Either the retrain ships that week, or Codeglyph falls back to the big prompted model for that one family until it does. Six weeks of denied claims never happens.

One design trusted a person's fast read of a change list. The other design trusts a fixed number, checked the same way every single time, whether the update looks big or small.

What I'd tell myself the week I skipped that retrain: a change list can look small and still be exactly wrong for the ten codes that matter most to you. The only way to know the difference is to run the eval, not read the summary.

PICK, four moves for picking fine-tune over prompt

This isn't a training question wearing a cost question's clothes. It's a real pick, and PICK is what keeps "fine-tuning is cheaper" from staying a hunch instead of a number you can defend.

Three things worth stating directly, since this is where the real judgment sits. The alternative Selah's team considered and dropped was building the training set by having the big model auto-label the historical notes itself, fast and cheap, instead of paying certified coders to hand-label four thousand of them. It lost because a sample test showed it matched certified coders only about 80 percent of the time on exactly the tricky, less common visit types where a mistake costs the most, which meant the fine-tuned model would have learned to repeat the big model's own errors as if they were correct. The AI-specific failure worth naming by name is silent staleness: a fine-tuned model keeps answering with old rules in the same confident voice after the real rules change, with no error thrown and no alert raised. The guardrail is the five hundred-note eval set itself, run automatically against every code-set update, however small it looks, with a 92 percent floor per code family, not just on average, that blocks a retrain decision, or a skip decision, that doesn't clear it. That guardrail isn't free, it costs about a day of engineering time twice a year, small next to six weeks of denied claims. And the tradeoff being accepted openly: a fine-tuned model costs $50,000 to build and about $20,000 a year to keep current, in exchange for a per-note cost small enough to nearly disappear and a monthly bill that, past month twenty, is always the cheaper one at Verrant's real volume.

And if you want to be sure it really works, try it somewhere else

Same four letters, a law firm's contract taxonomy instead of a hospital's coding guideline, and this time what goes stale isn't a fixed calendar update at all.

Wrayburn is a clause tagging assistant Palmerston Legal built for its contract review team. A lawyer uploads a contract, and Wrayburn tags each clause against the firm's own taxonomy: indemnification, termination, liability cap, governing law, and more. Cindra Onslow is the finance partner who signed off on what it costs to run.

The build-up: Palmerston processes about 3,000 contracts a month. The big model needed the full clause taxonomy and a dozen worked examples in every prompt to tag accurately, at about $0.40 a contract. Wrayburn's fine-tuned model, trained on Palmerston's own tagged history, does the same job for about $0.02 a contract, once it was built for $35,000.

Same rank as before, different lever: for Codeglyph, the thing that goes stale is a fixed calendar update, CPT in January, ICD-10-CM in October. For Wrayburn, there's no calendar at all. The taxonomy changes whenever a practice group decides it should, sometimes twice in a year, sometimes not for eighteen months. When Wrayburn's retrain budget ran dry, it kept tagging a genuinely new clause type as the nearest thing it already knew, confidently, with no flag that it was guessing.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: teach the model the rules once instead of repeating them every call, price the training cost as fixed against a bill that grows forever, and spend to catch the model going stale, not to shave the visible bill.
Cost: there's no budget this quarter for both a wider training set and a faster retrain cadence. The retrain cadence wins for Codeglyph specifically, because a stale billing code is a compliance and revenue problem, not just a missed clause.
The model got better, for real: say the big model's price drops by half next year. That's not proof fine-tuning stops paying for itself, since the fine-tuned model's own cost per note is already close to zero. A cheaper big model narrows the gap. It doesn't close it, and it does nothing about the staleness risk either side of the pick still has to manage.

Where people run it wrong.
They compare the sticker price of one call to the sticker price of a training run, instead of pricing both as the curve they actually are over real volume and real time.
They budget a fine-tune as a one-time build and forget to fund the retraining that keeps it from going stale.
They skip a scheduled retrain based on how small an update looks, instead of running the eval set first, every time, no exceptions for updates that look minor.

How to use it live. Say the real question out loud before quoting a number: "before I pick one, is this asking what today's call costs, or what it costs to keep the answers right after the rules change under it." That buys a beat to think instead of repeating whichever number got written down at launch.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0