What is the practical difference between prompting, RAG and fine-tuning for a product decision?

Quarrendon Home sells furniture, and its app Roomcast turns a photo of a customer's room into a redesign, stocked with real pieces they can buy. Ranvir Struzik runs that feature, and inside it are three separate jobs for the model: writing the style caption, picking real furniture from a catalog that never sits still, and keeping every render looking and sounding like Quarrendon. His team picked the right tool for two of the three jobs, and spent six weeks and real money finding out, the hard way, which one they had gotten backward.

How to answer this, stage by stage

Nobody is grading whether you can define three acronyms. They are grading whether you can hold three tools apart under pressure and say, out loud, which job each one is actually for.

Let's learn

Quarrendon Home sells furniture, and its app called Roomcast takes a photo of your room and shows you what it could look like, filled with real pieces you can actually buy.

Five steps. The one drawn in a darker outline, matching real furniture to the photo, is the step this whole answer turns on.

When Roomcast first launched, all three of those middle jobs ran on plain prompting: written instructions telling the model how to write a caption, how to pick items from the catalog, and how to sound. Keeping the furniture-matching instructions correct took an engineer about a day a week. It worked fine while the catalog was small.

Three different memories for three different jobs. The mistake was not picking a bad tool. It was pointing a good tool at the wrong job.

The catalog grew past twelve thousand four hundred items. Keeping a prompt correct across that many products got messy: new arrivals missing, discontinued pieces still showing up, sizes and prices drifting out of sync. So the team fine-tuned a model on the whole catalog instead, hoping to bake the product knowledge in for good. That took six weeks and thirty eight thousand dollars.

The retrain went fine on paper. What nobody built was a way for the model to hear about a new item, or a discontinued one, without running the whole six week process again.

This is a quiet kind of wrong. The model never says "I'm not sure this one's still in stock." It answers the same confident way it answers everything, whether the chair exists or not. What it costs at its worst: within four weeks, nineteen percent of recommended items were already unavailable. By week eight, thirty one percent. A customer taps "shop this look" and half the room is not real anymore.

What I would leave alone: the style caption generator, still running on plain prompting, never caused a single one of these problems, because a caption does not have to be factually current. It just has to sound fresh, and campaign themes only change every few weeks, cheap enough to rewrite by hand each time.

The lesson: a fine-tune is a promise that something will still be true in six weeks. Quarrendon made that promise about the one part of Roomcast that changes every single week. A promise like that was never Roomcast's to keep, and no amount of retraining was going to fix that.

Now here is the same thing as a story

The short version above is what you actually say out loud. Read this one for the meeting that got Ranvir here, and the one small question that finally cracked it open.

For three years, Ranvir Struzik's job was to make Roomcast's recommendations feel like they came from someone who actually knew Quarrendon's floor. He could look at a customer's uploaded room photo and guess, before the render even finished, which three items the app was about to suggest, because he had written the first version of that matching logic himself, mostly by hand, before there was a model in it at all.

Roomcast launched with all three jobs, the caption, the furniture picks, and the overall voice, running on prompting. For the first year, that was fine. Ranvir logged on around seven most mornings to skim the overnight render queue, catch anything odd, tweak a line of instructions if a new seasonal collection needed it. It usually took an hour. He liked that hour.

Then the catalog kept growing. Past eight thousand items, then ten, then twelve thousand four hundred. The prompt describing categories and price rules and style logic got longer and messier every quarter, and every few weeks something slipped through: a discontinued lamp still getting recommended, a new sofa nobody had taught the model about yet.

In a planning meeting that October, someone put it plainly: "We keep hand-tuning this prompt every time merchandising sneezes. Let's just bake the whole catalog into the model, once and for all, so we stop doing this by hand." It felt like graduating from a workaround to a real system. Nobody in that room was being careless. It was, on its face, the more serious-sounding plan.

Nobody decided, on any single day, to ship a model that would go stale. A planning meeting decision made eighteen months earlier just never asked the question again.

The fine-tune took six weeks and thirty eight thousand dollars. It shipped on a Monday, scored ninety seven percent correct against a fresh test set, and for three weeks nothing looked wrong. Nobody was watching the one number that would have told them otherwise, because nobody had set one to watch.

Then, in week eight, a new design intern redoing her own studio apartment as a demo for the team tapped "shop this look" on a walnut accent chair Roomcast had recommended. Sold out, the page said. Actually not sold out, she found a minute later, discontinued two months earlier. She turned around in the open office and asked Ranvir a question nobody could answer on the spot: "Why did it recommend something we don't even sell anymore?"

Small question. No drama. Ranvir pulled the numbers himself that afternoon. Nineteen percent of recommendations, stale, at week four. Thirty one percent, by week eight, the week the intern asked. Every one of them traced back to the same thing: the catalog had kept moving, and the model, once trained, could not move with it.

The decision Ranvir would take back sits in that October planning meeting. Fine-tuning the catalog felt like the mature choice, the one that stopped a team from re-touching a prompt every quarter. Nobody in that room asked the question that actually mattered: is this something that changes on its own, without anyone touching the model? The catalog was exactly that. A fine-tune was never going to hold it still.

One alternative got seriously considered before the rebuild, and it is worth naming, because it looked like the safe middle ground: keep fine-tuning, just do it every week instead of once. It lost. Even a fast retrain cycle takes over a week end to end, and the catalog moves daily, so a weekly schedule would always be trailing the truth, and it would cost thirty eight thousand dollars, every single week, to stay behind.

Run the same six months again, with the furniture picks grounded in RAG from day one: a live lookup against the real catalog, at the moment of each recommendation, instead of a memorized guess. Staleness holds under two percent, checked weekly. The nightly re-index costs about twenty minutes of compute, not six weeks. And the fine-tune budget that would have gone toward retraining the catalog goes instead to the one job that actually needed it, the render's look and the design voice, which had been drifting the whole time on prompting alone and which nobody had gotten around to fixing.

What Ranvir would tell himself, back in that October meeting: fine-tuning is not a graduation from prompting. It is a promise that a behavior will still be true in six weeks. We made that promise about the one part of the product that was never going to keep it.

PICK, three answers per letter

Not a way to rank which of the three tools is best. PICK only works here if you refuse to pick just one, and instead say, out loud, which of the three jobs actually belongs to which tool.

One mistake sits in plain view, fixed before lunch. The other one hides inside a render that looks, on the surface, like it went fine.

Three branches, one question. The catalog answers needs current outside facts. The design voice answers needs the same behavior every time.

The trade-off worth saying out loud: switching the furniture picks to RAG adds a live catalog lookup to every recommendation, about two hundred and twenty milliseconds of extra time compared to an instant memorized guess. That is a real cost, and it was accepted on purpose, because two hundred and twenty milliseconds is invisible to someone scrolling a room render, while a chair that does not exist is not.

And if you want to be sure it really works, try it somewhere else

Same four letters, a repair van instead of a living room, and this time the thing that goes stale is a part number instead of a sofa.

Kettlebrook Mechanical runs HVAC repair calls out of three depots, and its technicians carry an app that suggests likely causes and the right replacement part for whatever unit they are looking at. Leandra Bhaskara runs product there, and her team split into the same three jobs Ranvir's did: seasonal troubleshooting tips, live parts and compatibility lookups, and standardized safety report wording.

Same three-way split as Roomcast, a different trade entirely. The tool that fits each job does not change with the industry.

Kettlebrook fine-tuned its assistant on the full parts catalog, four thousand eight hundred items, hoping to bake part numbers and compatibility in for good. It took five weeks and twenty four thousand dollars. Six weeks later, during a July heat wave, a technician's app confidently cited a compressor part number that had been superseded two months earlier, with a different mounting bracket. He drove forty minutes to a distributor, then had to drive back for the right part. A repair that should have taken half a day took a full week, during the hottest stretch of the year, and by then twenty four percent of the part numbers the assistant cited were already superseded.

Mapped onto PICK: the position is the same shape, seasonal tips stay on prompting, live parts move to RAG, safety wording gets the fine-tune. The impact splits the same way too: a stale seasonal tip is a minor annoyance, a stale part number costs a week during the busiest stretch of the year. The cost asymmetry lands identically, a bad prompt is an afternoon fix, a bad fine-tune was five weeks and twenty four thousand dollars, obsolete before it finished shipping. And the kill criteria transfers without changing a word: does the job need current outside facts, a fixed behavior, or just better instructions.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: ground anything that changes on its own in RAG, before touching a fine-tune, no matter how tempting the permanent fix sounds.
Cost: no budget for a dedicated RAG pipeline this quarter. Fine, but get a written agreement that nobody fine-tunes information that changes weekly, full stop, until the budget exists.
The model got better, for real: say the fine-tuned model's overall accuracy on a fresh benchmark actually improved. Still does not matter. A better score on day one says nothing about day forty, once three hundred more parts have shipped.

Where people run it wrong.
They treat "we already built a fine-tune" as a reason to keep using it, instead of asking whether it was ever the right tool.
They assume a high accuracy score on launch day means the knowledge will still be accurate a month later.
They reach for fine-tuning first because it feels more finished than "just" a prompt or "just" a lookup, when finished has nothing to do with which one is actually right.

How to use it live. Before answering, ask yourself one thing: is the fact about to change on its own, without anyone touching the model? If yes, that is RAG's job, not a fine-tune's, no matter how permanent the fix sounds.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0