A PM says the model needs more training on our data. Diagnose what they probably actually need.

Palletcast forecasts how many units a warehouse should reorder, SKU by SKU, so nobody has to guess. Sorley Danby owns the reorder model. This is the nine days that taught him "the model needs more training on our data" was the one sentence in the whole incident review that nobody had actually checked.

How to answer this, stage by stage

Nobody is grading whether you know what a fine-tune is. They are grading whether you will check three cheaper things before you agree to one.

Let's learn

Palletcast is a program a warehouse plugs its sales history and stock levels into, and it hands back a number: reorder this many of this item, by this date. Before it, Kettleborn Supply's planners built that number by hand every Friday, mostly from last year's sales plus whatever the planner remembered about the weather. It took a planner about six hours a week per warehouse, and it was often close, but never exact.

Once Palletcast shipped, that six hours dropped to about twenty minutes of review. Sorley Danby, who ran replenishment at a regional hardware chain for ten years before joining Palletcast, built the reorder model to do in seconds what he used to do with a spreadsheet and a hunch about the forecast. For its first year at Kettleborn, the model beat his own old gut-call accuracy on 46 of 50 weekly cycles. Planners stopped double checking it line by line. Why would they, when it kept being right.

Sorley spent a decade forecasting demand by hand before he built software to do it. For a year, Palletcast earned the same trust his own gut used to.

Then a heat advisory hit central Texas. Here is the turn. The 810 extra units Waco never had on the shelf were never really the problem. The real problem is what "needs more training on our data" did to the room the moment Sorley said it out loud in the incident review: everyone stopped asking questions and started scoping a fine-tune, because it sounded like the serious, technical answer. Nobody asked what the model had actually been shown that week, or what it hadn't.

What it costs at its worst: about 186,000 dollars in lost sales at one warehouse, over nine days. And the slower cost was worse. Once planners at other warehouses heard about Waco, they started adding a manual buffer on top of every summer forecast, just in case, which is exactly the six hours a week Palletcast was supposed to remove. A tool that gets quietly double checked again is a tool that stopped paying for itself.

What I would leave alone: everything that isn't weather-elastic. Gaskets, hand tools, fasteners, the bulk of Kettleborn's catalog, none of it moves with a heat advisory, and calendar-based forecasting already handles it fine. Piping a live weather feed into every single SKU's prompt would add cost and latency for a fix most of the catalog never needed.

The lesson: "needs more training" sounds humble, like admitting the model just needs to see more examples. It's actually the most expensive thing you can propose, and reaching for it first, before checking what the model was actually shown that week, skips every cheaper question that would have found the real answer faster.

Now here is the same thing as a story

Say the short version out loud in an interview. Read this one when you want to feel exactly how one lazy sentence nearly cost Kettleborn a fine-tune it never needed.

Sorley Danby spent ten years running replenishment for a regional hardware chain before Palletcast ever existed, standing in a warehouse office every Friday with a spreadsheet, last year's sales numbers, and whatever the ten-day forecast on his phone said. He got good enough at it that other planners called him before ordering anything unusual. He joined Palletcast two years ago to build, in software, the thing he used to do by hand on a Friday afternoon.

The reorder model he built pulled three things into every recommendation: twelve months of sales history, current on-hand inventory, and any scheduled promotion. For its first year at Kettleborn Supply, across six warehouses in Texas and Oklahoma, it beat Sorley's own old accuracy on 46 of 50 weekly cycles. Planners stopped adding their own buffer on top of it. Some stopped opening the spreadsheet at all.

On July 1st, the National Weather Service issued a heat advisory for central Texas: eight straight days above 105 degrees, starting July 9th. Nobody at Palletcast saw that advisory, because nothing in the reorder prompt was built to look for one. The model's regular Tuesday run on July 2nd looked at Waco's fan and window-AC SKUs, checked last July's numbers, and recommended 340 units. A routine week, by the only measure it had.

Eleven days sit between the advisory and the empty shelf. The forecast run in the middle of that gap never moved.

The heatwave landed exactly on schedule. Sell-through of fans and window units at Waco tripled against a normal July week. By July 12th, three days in, the shelf was empty. Kettleborn expedited a restock that didn't land until July 18th. Six days with nothing on hand, during the exact week the whole region wanted a fan.

The incident review happened the following Monday, twelve people on a call, Sorley included. Someone pulled up the July 2nd forecast next to the sales numbers and asked the obvious question: why didn't it see this coming. Sorley, tired and a little defensive, said the line that started everything: "the model clearly needs more training on our data. Let's scope a fine-tune on Kettleborn's full sales history." Two engineers started taking notes on a training-data pull before anyone had actually checked what the model had been shown that week.

"Needs more training" gets said for all four of these. Only the fourth one is actually a training problem.

Sorley caught himself that night, rereading his own message. He hadn't actually checked anything. He'd reached for the sentence that sounded the most serious, not the one that was cheapest to test. So the next morning, before anyone touched a training pipeline, he ruled out the boring explanations first. The reorder prompt's instructions were fine, clearly worded, nothing garbled or ambiguous about what "recommend a reorder quantity" meant. The model had plenty of examples of correctly formatted output from months of normal weeks. Neither of those was the problem.

A fine-tune changes what the model carries with it everywhere, permanently. A prompt change only changes what it's told on this one call.

That left the real candidates, ranked cheap to expensive: fix vague instructions, add a missing example, feed the model a fact it didn't have, or genuinely retrain it. Sorley ran the test that actually mattered. He pulled the exact prompt Palletcast had sent on July 2nd, added a single line, "heat advisory issued: 8 consecutive days above 105 degrees, starting July 9," and reran it against the frozen, untouched model. No new weights. No training run. Just one fact, added where the model could see it.

Nothing about the model changed between these two runs. Only what it was told changed.

The recommendation jumped from 340 units to 1,150, in line with what Waco had actually needed. Same weights, same model, one added fact. That result ruled out a training problem on the spot: if the model genuinely lacked the knowledge, no amount of context stuffed into the prompt should have fixed it. It fixed it instantly, which meant the knowledge was never missing. The fact was.

Sorley considered one other fix before that, and rejected it. He could have told every warehouse to just carry a bigger safety-stock buffer on weather-sensitive SKUs, all summer, no live data needed. He turned it down, because it ties up cash across five warehouses that never had a heatwave, sized for an average July that a genuinely extreme week will blow through anyway. A live weather signal only shows up when there's actually something to react to.

What Sorley would tell himself, back in that first incident review: the fastest-sounding fix and the fastest actual fix are almost never the same one, and the only way to tell them apart is to test the cheap one before you ever agree to the expensive one out loud.

TRACE, for a sentence that sounds like a diagnosis

Not a way to prove a fine-tune was never needed anywhere. TRACE is what stops a team from agreeing to the slowest fix before checking the three faster ones.

Four boxes, cheapest first. Waco's real cause sat third, not fourth, which is exactly why jumping straight to the fourth wasted a week.

Three things worth saying plainly, since interviewers push on this exact spot. Sorley did consider the blunt fix first, a bigger safety-stock buffer on weather SKUs, all summer, at every warehouse, no live data required. He rejected it because it ties up real cash at five warehouses that may never see a heatwave that year, and because a buffer sized for an average July still gets blown through by a genuinely extreme one, while a live signal only shows up exactly when there's something to react to. The AI-specific failure worth naming by name is a grounding gap: the model reasons fluently and confidently from a pattern that used to be true, with no visible sign that a materially different fact now exists. The guardrail isn't a bigger model, it's a coverage check that flags any weather-elastic SKU whose forecast ran without an active regional alert attached, so the gap gets caught before a customer does. And the trade-off, accepted on purpose: the weather-lookup step adds a small per-call cost and about 200 milliseconds to any forecast run it touches, worth it against a single multi-day stockout that costs six figures.

And if you want to be sure it really works, try it somewhere else

Same five letters, an insurance company instead of a warehouse, and this time the confirmed cause isn't a missing fact. It's a missing example.

Corrywell Mutual uses an AI tool to draft the coverage-denial letters its adjusters send to policyholders. Caradoc Vantoor owns it, and hit a version of Sorley's exact mistake four months after launch, on a batch a state regulator flagged during a routine compliance audit.

The regulator's finding was blunt: 40 of 40 sampled letters cited no specific policy clause and gave no appeal deadline, both required by state law. In the review, a compliance lead said the same line Sorley had said: "the model needs more training on our denial letters." Caradoc, remembering Sorley's incident from a conference talk months earlier, asked one question before anyone touched a training pipeline: had the model ever actually been shown a correctly formatted letter, or just told to write one.

Same four candidates as Waco's. This time the confirmed one sits in a different box: no example, not a missing fact.

It hadn't. The system prompt said "explain the denial clearly," with no example attached and no list of required fields. Caradoc's evidence test: add three real, compliant denial letters as examples directly in the prompt, and rerun the same 40 flagged cases against the same, untouched model. 38 of 40 came back with a cited clause and a stated appeal deadline, no retraining involved. The two that didn't were genuinely novel disputes, the one honest case for a narrow fine-tune later, not the whole batch.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to the reframe: "needs more training" is a symptom, not a diagnosis, and it's almost never the cheapest true one.
Cost: no time to trace a real incident. Ask one question instead: has anyone actually put the missing fact or a real example straight into the prompt and reran it, or did the team go straight to scoping a training pipeline.
The model got better, for real: say the next model version ships noticeably stronger reasoning. That helps candidate one and two, vague instructions and missing examples, almost disappear on their own. It does nothing for candidate three, a missing current fact, because a smarter model still can't know something nobody told it.

Where people run it wrong.
They hear "the model got something wrong on our data" and hear "training problem," when the two aren't the same claim at all.
They test the fix by asking the model to try again, instead of testing whether a fact or an example, placed in the prompt, changes the answer.
They scope the fine-tune before running the fifteen-minute prompt test that would have told them whether they needed it.

How to use it live. When an interviewer hits you with this cold, buy two seconds by asking one thing back: "when you say more training, do you mean it's missing a fact, or missing an example of what good looks like?" That question alone usually is the answer this exact shape of question is listening for.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0