How do you handle an engineer who wants to fine-tune when you believe prompting would do?

Hearthloom mails a small personalized box every month, a candle, a tea, a kitchen tool or two, picked for one customer's actual taste by a model called Loomline. Katriona Petranek owns Loomline end to end. This is the week her engineer, Ravendra Novosad, booked three weeks on the training cluster to fine-tune it, and nobody, including him, had tried writing it a real prompt first.

How to answer this, stage by stage

Nobody is grading whether you know what fine-tuning is. They're grading whether you'll let a three-week decision get made on a feeling.

Let's learn

Hearthloom mails out a small box every month, a candle, a tea, a couple of kitchen tools, matched to one customer's actual taste, not a generic starter set. A model called Loomline reads each customer's profile, a taste quiz, what they kept last time, what they sent back, and picks the six to eight items for their next box.

Before Loomline shipped, four in-house curators hand-picked every box. They were good at it. They also never wrote down why a box worked, and a customer who didn't like theirs just quietly unsubscribed instead of saying what went wrong. Once Loomline took over the picking, it did two things the curators never could: it picked instantly instead of one box at a time, and it wrote one short line explaining each choice.

Two things sat on Ravendra's desk that Thursday. Only one of them had ever actually been tried.

Every month, Loomline's picks get checked against a golden set: a hundred fifty old boxes the original curators graded by hand, yes or no, would this box have delighted this specific customer. For three straight months, Loomline scored sixty one out of a hundred against that bar. Hearthloom's own ship bar, the score a model needs before anyone calls it trusted, is seventy five. Sixty one had not moved in three months.

Here is the turn. The stuck score was never really the problem. The problem was what happened next: Ravendra, the engineer who owns Loomline's model, decided the fix was a bigger model trained on Hearthloom's own history, and booked the training cluster for Monday, three weeks of work, before anyone had tried writing Loomline a genuinely strong prompt.

Three real reasons an engineer reaches for a training job. Only one of them turned out to be true here.

What it costs at its worst: say Katriona waves the training job through, no questions. If a three-day prompt sprint would have cleared the bar, Hearthloom spends three weeks and real training budget solving something a Tuesday afternoon could have fixed, and every future model debate skips straight to "let's train something," because that's what got approved last time. Or say she blocks it outright, guessing it's just enthusiasm. If there really is a corner of the customer base a prompt can't reach, that gap sits open for a full quarter, quietly shipping bad boxes to real subscribers with unusual taste combinations, while nobody's even looking.

What I would leave alone: Hearthloom's photo-quality checker, the model that flags blurry customer photos for the referral rewards program. It's already fine-tuned, works well, and its two labels, blurry or not blurry, haven't changed in a year. Making it clear a prompting-sprint gate first would just be paperwork on a model that isn't moving.

The lesson: a good idea and a proven idea can look exactly the same in a Slack message asking to book a GPU cluster. The only way to tell them apart is to make the cheaper idea prove itself first, on the same bar the expensive idea will eventually be judged against.

Now here is the same thing as a story

Say the short version out loud in an interview. Read this one when you want to feel exactly why a training-job request and a real gap can look identical for the first ten minutes.

Katriona Petranek has run product for Loomline since before it had a name, back when it was four spreadsheet columns and a hunch. She built the golden eval set herself, sitting with Hearthloom's four original box curators for two weekends, writing down exactly what made a box work for each of a hundred fifty real subscribers. She knew that data cold.

Loomline shipped, and for the first stretch its score climbed steadily against that golden set. Past forty. Past fifty five. Then it flattened at sixty one and sat there, for three months running. Katriona checked the dashboard every Monday morning at first. It never moved, so eventually she mostly stopped opening it, trusting whoever was working the personalization backlog to flag anything that changed.

The score went stuck in week two. Nobody asked whether a real prompt had ever been tried until week five.

Ravendra Novosad had worked on Loomline for a year, and he was good, he'd shipped three real improvements to the ranking logic before this. So when he messaged Katriona on a Thursday afternoon, she took it seriously. "I think Loomline just doesn't understand taste the way a prompt can fix," he wrote. "I've reserved Monday on the training cluster. I want to run a fine-tune on our history, forty thousand old customer-to-box pairs."

Katriona didn't say yes and didn't say no. She asked a question that wasn't really about the training job at all. "Have we actually tried a real prompt on this? Examples and all. Or are we going straight from 'it's stuck' to 'let's train something'?"

There was a pause. "We've never really tried, no," Ravendra wrote back. "The prompt's basically what shipped on day one."

What "try a real prompt" actually meant, spelled out, so it stopped being a vague ask and became a three-day job with a clear finish line.

So instead of approving the cluster, she asked for three days. Ten of the curators' best old picks, turned into worked examples. A forced output structure, so Loomline had to name which signal it matched for every item and how sure it was. Ravendra built it Monday and Tuesday. By Wednesday afternoon, the golden set score read seventy nine.

The old decision, told straight: a year earlier, back when the personalization team was Katriona and one engineer sharing a desk, they'd talked about writing an approval checklist for model changes, and decided against it. It felt like overhead for two people who talked all day anyway. Nobody wrote it down. Nobody had to, for a year.

Run the same Thursday again, with the gate Katriona wrote after this. Ravendra doesn't book the cluster. He books three days on the calendar for a prompt sprint instead. By Wednesday afternoon the score reads seventy nine. Thursday morning, the GPU reservation gets cancelled before it was ever needed, three weeks and a training budget saved, in exchange for three days everyone agreed to up front.

What I'd tell myself, in that ten-minute meeting a year ago: skipping the checklist felt like trusting the team. It just meant nobody had to prove anything, including me.

TRACE, so a training job has to earn its three weeks

Not a way to prove fine-tuning was wrong. TRACE is what stops a real ceiling and a strong preference from getting the exact same answer by accident.

The question underneath the question: does what you're training on keep changing under you. Loomline's catalog does. That's most of why prompting was the right first move.

Three things worth saying plainly, since interviewers push here. Katriona considered a second option before this one: deny the training job outright, since prompting is almost always cheaper to try first. She rejected it, because a flat no risked being wrong about the one segment where a real ceiling did exist, and it would have taught the team that "no" is always the founder-mode answer to a fine-tune request, which discourages engineers from raising real ceiling cases at all. The AI-specific trap worth naming by name: a fine-tuned model bakes in a fixed snapshot of the catalog it trained on. Hearthloom swaps about thirty percent of what's in the box every six weeks, new candles, new teas rotating in and out. A fine-tuned Loomline would start quietly recommending discontinued items and missing new ones unless retrained every single rotation. The guardrail is simple: log which catalog version served every recommendation, and alert the moment live picks start drifting from what the model actually trained on. And the trade-off, accepted on purpose: a hosted prompt call costs more per box than a fine-tuned small model would at inference, because it's carrying ten examples and the month's catalog candidates every single time. Hearthloom accepts that higher per-box cost to avoid paying a retrain tax every six weeks.

And if you want to be sure it really works, try it somewhere else

Same five letters, a crop photo instead of a candle box, and this time the segment that doesn't clear the bar is a single uncommon crop, not a rare taste combination.

Cropwarden helps smallholder farmers diagnose crop problems from a phone photo: what's wrong, how sure the model is, and what to try next. Nireva Boskovic runs product there, and hit a version of Katriona's exact moment four months into the job, the week an engineer proposed fine-tuning their vision model on twenty five thousand labeled photos instead of trying a stronger prompt first.

Different product, same shape of pipeline: a photo comes in, a prompt does the reading, a person checks the hard calls before anything ships.

Cropwarden's zero-shot diagnosis prompt scored fifty eight percent against a hundred twenty agronomist-graded photos, well under their seventy percent ship bar, stuck there for two months. Nireva ran a two-day prompting sprint instead of approving the training job: eight expert-labeled example diagnoses as few-shot, plus a forced output naming the symptom location and a confidence number. The score jumped to seventy four, clearing the bar. Recut by crop type, one uncommon regional crop, cassava, stayed at fifty one even after the better prompt, the one place her engineer's instinct was actually right.

Mapped onto TRACE: the timeline shows the score flat for two months before anyone asked the question. The recut splits cleanly, an ordinary crop diagnosis against one crop with genuinely different, less common symptoms. The assumption ruled out first, same as Katriona's, was that this was purely a technical preference; it wasn't, not entirely, the cassava gap was real. The cause candidates were the same three reasons: genuine ceiling, technical pull, or simply untried. The evidence test gave the same shape of answer: the two-day prompt cleared the bar everywhere except one named, countable slice.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: ask for a short, real prompting attempt, scored against the actual bar, before approving any training job.
Cost: no time to trace a real incident. Ask one question instead: has anyone actually tried a strong prompt on this, with real examples and a forced output, or are we assuming it can't work?
The model got better, for real: say the hosted model behind the prompt drops in price by half next quarter. The case for prompting first gets stronger, not weaker, since the ongoing cost of never having to retrain now costs even less to carry.

Where people run it wrong.
They treat "the engineer wants to fine-tune" as a technical argument to win or lose, instead of a claim that just needs a cheap test.
They approve or block the whole idea based on one blended number, instead of recutting by segment to see what the average is hiding.
They fine-tune once and never check whether the thing they trained on, the catalog, the labels, the customer base, is still the thing shipping six weeks later.

How to use it live. When an interviewer asks this cold, buy two seconds by asking one thing back: "has a strong prompting approach actually been tried and measured here, or are we starting from a belief?" That question alone is usually exactly what a question shaped like this one is listening for.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0