Describe the tradeoff between a cheaper model and more retries.

A cheap model that keeps trying costs almost nothing on paper. But a retry loop with no ceiling does not fail loudly. It fails by quietly accepting its own worst guess, and that guess can sit inside a client's own books for weeks before anyone finds it.

How to answer this, stage by stage

Nobody is grading whether you can say "it depends." They are grading whether you can name a real pick, put a number behind it, and say what would change your mind.

Let's learn

What actually happens the day a cheap model gives up trying and just guesses instead?

Formglass is the tool inside Corravue's back office platform that reads a photo of a receipt or invoice and turns it into rows a bookkeeper's ledger can use straight away.

Before Formglass, a data entry team retyped every scanned receipt by hand, about 40 seconds a document, and Corravue kept 22 people on staff just to keep up with what clients uploaded each day.

Formglass reads most receipts in under a second now. It uses a cheap model first, since it costs about a tenth what the accurate model costs, and only calls the pricier model when the cheap one fails.

The turn: the extra misreads that came with picking a cheap model were never the real problem. A few wrong reads a day is the cost of choosing cheap over accurate, and any team that makes that pick should expect it. The real problem was what the system did the moment the cheap model kept failing on the same document: it just kept trying, on its own, until it ran out of patience and quietly accepted its own worst guess.

Same document, two different endings. One error costs a little and gets caught right away. The other costs almost nothing and hides for weeks.

At its worst, a retry loop that finally gives up and guesses is worse than never building the cheap model path at all, because it looks exactly like a system checking its own work, right up until the week it silently is not.

What I would leave alone: clean, typed PDF invoices. The cheap model reads those right on the first try almost every time, so capping retries there, or routing them for review, would just slow down the easy 90 percent of the work to protect against a problem that never shows up there.

The lesson: a retry that costs a cent looks free right up until the moment it is the tenth retry on a document the model was never going to read, and the system quietly decides that guessing counts as finishing.

Now here is the same thing as a story

Read the version below when you want to feel why an uncapped retry loop went wrong, not just be told that it did.

Tobin Winnick could read a scanned receipt faster than most people could open the email it came in. Before Formglass existed, he was one of the engineers who built Corravue's manual review screens, the ones a data entry clerk used to fix whatever the old system got wrong.

Formglass launched in the spring, and the first few months were genuinely good. A receipt that used to take a clerk 40 seconds to retype now got read in under a second, most of the time. Tobin set the cheap model to retry twice on anything it was not sure about, then hand the rare leftover to the pricier model. By June the retry queue rarely held more than a few hundred documents at any hour of the day.

By July, the queue had grown quietly, mostly documents from one client, Tulley's Diner Group, whose thermal receipts faded fast in a hot kitchen and photographed poorly under fluorescent light. Tobin noticed the queue length once, on a slow Friday, and told himself it was probably just Tulley's uploading in a batch that week. By September, an old fallback nobody quite remembered writing, built years earlier for a different reason, had quietly started kicking in: after ten failed retries, the system stopped asking and just kept its last, lowest confidence guess.

The trigger was nothing dramatic. Corravue ran a routine month end reconciliation for every client, and in October, a bookkeeper working through Tulley's books found the ledger sitting $180 over what the bank statement showed.

It took a senior bookkeeper about six hours to trace it back through three weeks of transactions to one receipt: a coffee and lunch order that Formglass had read as $312.40 on its tenth and final retry. The real total, once someone actually looked at the photo, was $132.40.

It was never really about one wrong total on one receipt. Tobin's team had no number for how often this happened, because nothing was counting retries past the second one. They had a switch nobody had noticed was on: keep trying quietly, or stop and ask.

The fallback that finally auto accepted a guess went back to a meeting two years earlier, before Formglass had a single paying client outside the pilot. Someone had asked what should happen if the cheap model just never succeeded, and the answer, reasonable at the time, was do not let a document sit in the queue forever, take its best guess and move on. Nobody expected "best guess" to ever mean a tenth try nobody had checked.

Run that October morning again with Tobin's new design in place. The same faded Tulley's receipt fails the cheap model twice, in under two seconds combined. On the third failure it escalates automatically to the pricier model, which reads $132.40 correctly in about half a second. If the pricier model had not been confident either, the document would have dropped into a queue a person clears the same day, never touching the ledger unchecked.

One design trusted a two year old fallback to know when "good enough" had quietly become "not checked at all." The other design draws that line itself, in code, before a guess ever reaches a client's books.

What I would tell myself, back in that meeting two years earlier: a rule for what happens after ten failed tries is not a small detail to leave for later. It is the whole answer to the question, and leaving it unwritten just means someone else writes it by accident, the day the retries finally run out.

PICK, the four moves behind the retry cap

This is not a story wearing a framework. It is a tradeoff question, and PICK is what stops "it depends" from standing in for a real decision.

Three things worth stating directly, since this is where the real judgment sits. The alternative Tobin's team considered and dropped was paying for the pricier model on every document, since it reads almost everything correctly the first time. It lost because it costs roughly six and a half times more for a small accuracy gain on the 92 percent of documents the cheap model already reads correctly within two tries. The AI specific failure mode worth naming by name is a model retried against a document it is systematically bad at, a faded thermal receipt, not a random noisy scan. More retries do not fix that, because the failure is not luck, it is a mismatch between what the model was trained to read and what it is being shown. The guardrail is two part: a hard retry cap paired with routing anything still unresolved to a person instead of auto accepting the last guess, and tracking failure rate by document type so a systematic mismatch shows up in a chart, not in a reconciliation audit three weeks later. And the bar Formglass holds itself to was never a promise of zero misreads across 800,000 documents a month, no model that cheap can promise that. It is a checked bar: at least 98 percent of automated reads match what a person would enter, sampled weekly, with every read below the confidence cut off routed to a person rather than logged as done.

And if you want to be sure it really works, try it somewhere else

Same four letters, a fishing dock instead of a back office, and this time the thing that decides whether a retry is worth it is not the document at all. It is what time the sun came up.

Catchline is an AI tool Norsund Fisheries Cooperative built to read paper catch tickets, species, weight, vessel, and port, photographed dockside as boats unload. Iida Koskinen runs dock operations for the co-op.

The build up: Catchline handles about 60,000 tickets a month across the fleet. The cheap model costs about a tenth of a cent a ticket, the pricier one about a cent and a half, roughly 15 times more. Tickets logged around midday, in good light, fail after two tries only about 8 percent of the time. Tickets logged before sunrise, when boats come in with the dawn catch and the paper is often damp, fail after two tries about 40 percent of the time. Volume is not what decides whether a retry is worth it here. The hour on the clock is.

That assumption broke when a compliance spot check found Norsund appearing to be about 400 kilograms over its quota for one species, a number that only made sense once someone traced it back to a run of dawn tickets where a retried, low confidence weight had quietly been accepted. The fix looked the same shape as Tobin's: stop treating every failure as equally random, and route the ones with a known, predictable cause straight past the retry loop.

Same method, different lever: for Formglass, the lever worth stress testing first was which document type kept failing. For Catchline, it is what hour a ticket was logged. If Norsund shifted a quarter of its fleet to dawn landings, the failure rate feeding retries would climb from about 8 percent to well over 20 percent almost overnight, the single biggest swing in that system by far.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: cap retries, escalate on the third failure, and never let the loop auto accept its own guess.
Cost: there is only budget this quarter for one fix, the retry cap or the per type failure tracking, not both. Take the cap first. It stops the worst hidden failures on its own, tracking can follow once that guardrail exists.
The model got better, for real: say the cheap model's overall accuracy climbs after an upgrade. That does not mean the hard tail, faded receipts, dawn catches, got any better. The easy majority can get easier while the hard slice stays exactly as broken, so the cap and the escalation stay in place regardless.

Where people run it wrong.
They cap retries but never decide what happens after the cap, so it defaults to auto accept by accident, the exact fallback that caused the problem in the first place.
They set one retry cap for the whole system instead of per document type, so a class of document that is systematically hard keeps hitting the same wall every time.
They watch pass or fail rate, but never watch retries per document, so a queue quietly filling up with tries does not show up on any dashboard until someone goes looking for it.

How to use it live. Ask the real question before quoting a number: is this failure random bad luck, or proof the model cannot read this kind of document at all? That buys a beat to think, and it is usually the difference between the right pick and a guess dressed as a policy.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0