Explain how a RAG pipeline's cost structure differs from a single model call.

Fenbrook Research priced its move to retrieval as a bigger version of the same one line. The real cost split into five lines, and the two nobody separately priced turned out to matter most.

How to answer this, stage by stage

Nobody's testing whether you know the word "retrieval." They're testing whether you'll price it as a bigger single call, or actually count what's different about it.

Let's learn

What actually changes in the bill when a tool stops answering from memory and starts answering from real documents?

Citelume is the research assistant Fenbrook Research built for academic researchers. Type a question, get back an answer written from real papers, with citations a researcher can actually check.

Before retrieval shipped, Citelume just asked the model the question directly, no lookup, nothing pulled from a real document. That single call cost about nine tenths of a cent, one line on the bill, tokens in and tokens out.

Boaz Solari, the engineer who owns Citelume's cost budget, priced the move to real retrieval the way most people price a bigger version of something they already know. He doubled the token line. Call it two cents a query, he told finance, still cheap.

The turn: the extra cents were never the real risk. The real risk was two cost lines Boaz's forecast never separately priced at all, reranking and keeping the index current, and neither one moves with token count the way the model call does.

Query embedding costs almost nothing, about a hundredth of a cent. Retrieval, the actual vector search, adds another six hundredths. Reranking, checking the top candidates properly before they reach the model, costs six tenths of a cent, more than embedding and retrieval put together. The generation call itself grows because it now has to read the retrieved passages, not just the question, so it jumps to about two and a half cents. And keeping the index current, embedding new papers as Fenbrook adds about fifteen thousand chunks a month, adds another six tenths of a cent once you spread that monthly cost across every query.

The cheap error shows up on every invoice. The expensive one shows up as a researcher's wasted week, and it never once appears as a line item.

At its worst, an underpriced RAG budget doesn't just cost more than expected. It quietly pressures someone to cut the wrong line to make the number smaller, and the line that's easiest to cut, reranking's fan-out, is also the one protecting a researcher from a citation that was never really the best match.

What I'd leave alone: Citelume's "summarize this one open paper" mode, where a researcher pastes a single PDF and asks for a plain summary, doesn't need any of this. There's no corpus to search and no index to keep current. It's still just one call, one cost line, and building a five-line budget around it would spend effort where there's no real cost to track.

The lesson: a forecast can be completely honest about what it counts and still be the wrong number to plan by. Two cents really was double the token line. It never claimed to be the cost of a pipeline with five moving parts, two of which don't care how many tokens go through them.

Now here is the same thing as a story

Read the long version below when you want to feel why a modest cost cut went wrong, not just be told that it did.

Boaz Solari could price out a new feature to the month before Fenbrook's finance team finished reading his email. He'd spent three years pricing infrastructure for research tools before Citelume existed, and he built its retrieval budget himself the month it shipped.

The first several months were good. Citelume's corpus held about six hundred thousand paper chunks, and a query pulled the top forty candidates by similarity, reranked them properly, and handed the model the best eight. Researchers trusted it. Boaz watched the monthly bill line up close enough to his forecast to feel proud of the number.

Then Citelume grew fast. New labs signed on, the corpus quadrupled toward two and a half million chunks, and query volume climbed with it. Reranking's cost climbed too, since it scores every one of those forty candidates on every query. Nine months in, with the bill running hotter than planned, Boaz made a change that felt small: narrow the vector search to the top twenty candidates before reranking even starts, instead of forty. Fewer candidates to score, lower reranking cost, about four tenths of a cent saved on every query.

Nobody objected. It looked like the kind of sensible trim any engineer would make under budget pressure, and for the topics most researchers searched, the answers still looked fine.

It came back on an ordinary afternoon, and not as an error. A postdoc researching a specific drug interaction asked Citelume for the current evidence, got back a confident answer citing three papers, and moved on with her grant application. A week before the deadline, a co-author mentioned a paper she hadn't seen, directly on point, published two years earlier by a well-known lab in the field. She checked Citelume again. It still hadn't surfaced that paper. She flagged it to Boaz's team, more curious than upset.

Boaz pulled the query and reran it against the old, wider candidate pool. The missing paper sat at rank twenty four by plain similarity, just outside the narrowed top twenty. With the old top forty window, reranking would have had a real shot at promoting it, since a cross-encoder is good at exactly this: catching a paper that's phrased differently but genuinely on topic. With the narrowed window, it never even reached the reranker to get that chance.

He didn't stop at one query. He pulled a golden set, fifty questions with a known, correct paper attached to each one, and ran it against both windows. With the old top forty, the system found the right paper in its top eight results about ninety one times out of a hundred. With the narrowed top twenty, that had quietly slipped to about seventy six, and nobody had checked it since the day the cut shipped.

The decision that opened the door wasn't the fan-out cut itself. Narrowing a candidate pool to save money is a normal, fair trade to try. The decision Boaz would take back is shipping it without a plan to ever check it again. A default chosen once, under pressure, quietly outlived the corpus it was sized for.

Run that afternoon again with one change: before any change to fan-out ships, it runs against the fifty-question golden set, and a recall floor has to hold, at least eighty eight papers found correctly out of every hundred. The narrowed window from nine months back would have failed that check before it ever reached production. Either it doesn't ship, or it ships smaller and safer, trimmed just enough to protect the number without dropping under the floor.

One design let a cost-saving default sit unwatched for nine months while the corpus underneath it quadrupled. The other design treats every change to retrieval as something that has to earn its place against a real, counted number, the same way a change to the model or the prompt would.

What I'd tell myself, back the week that cut shipped: a default that saves money and looks fine on the topics you happen to check is not the same thing as a default that's actually safe. The only way to know the difference is to measure it, on a schedule, not on a hope that nobody notices.

PICK, the four moves behind pricing a RAG pipeline honestly

This isn't a diagnosis question wearing a tradeoff's clothes. It's a real pick, and PICK is what keeps "RAG costs more" from staying a vague feeling instead of a defendable number.

Three things worth stating directly, since this is where the real judgment sits. The alternative Boaz's team considered and dropped was cutting reranking out entirely, feeding the top fifteen raw search results straight into generation. It would have saved the whole six tenths of a cent reranking line, about twenty four hundred dollars a month at Citelume's volume. It lost because tested against the same fifty question golden set, recall fell to about sixty eight papers out of a hundred, worse than even the narrowed fan-out that caused the near miss. The AI-specific failure worth naming by name is silent retrieval drift: recall quietly falling as a fixed, cost-driven default stays put while the corpus keeps growing and getting more crowded within specific topics, with no error thrown and no alert raised, because the system still answers confidently with whatever it did find. The guardrail is the golden set itself, checked automatically before any change to fan-out, the reranker, or the embedding model ships, with a hard recall floor that blocks a change that doesn't clear it. That guardrail isn't free, running it costs a few engineer hours a quarter, small next to the near miss it exists to catch. And the tradeoff being accepted openly: RAG costs about four times a single call per query and adds roughly seven hundred to nine hundred milliseconds of retrieval and reranking time on top of generation, in exchange for answers grounded in real, checkable papers instead of the model's own memory. The bar Citelume holds itself to was never a hundred percent recall, no retrieval system promises that. It's recall at eighty eight percent or better on the golden set, checked before any pipeline change ships, not a number that gets to slide because the topics people happened to test that week looked fine.

And if you want to be sure it really works, try it somewhere else

Same four letters, a wind farm's maintenance floor instead of a university library, and this time the dominant cost line isn't reranking at all.

Trellis is a field assistant Corravin Energy built for turbine technicians. A technician asks about a fault code or a torque spec, and Trellis searches the manufacturer manuals for that exact turbine model and answers with the actual page cited. Anneka Selassie is the finance partner who signed off on what it costs to run.

The build-up: Trellis handles far fewer queries than Citelume, a few hundred a day across Corravin's technicians, so its per-query lines are small, embedding and retrieval together cost under a tenth of a cent, reranking about half a cent, and the bigger generation call about two cents. The number that dominates here is index upkeep, not reranking. Manufacturers revise turbine manuals several times a year, and every revision means re-embedding the changed sections before a technician can trust the answer.

Same rank as before, different lever: for Citelume, reranking's fan-out was the line quietly cut under pressure. For Trellis, it's how often the real world underneath the product changes, a manufacturer's revision schedule, not the software team's own release calendar. When reindexing ran out of budget, Trellis kept answering from the old manual for three weeks. A torque spec question would have returned the outdated number with the same confident tone as a current one, and nobody would have known without checking the physical manual by hand.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: a single call is one cost line, RAG is five, price them separately, and spend to protect against the hidden error, not the visible one.
Cost: there's no budget this quarter for both a wider reranking fan-out and a faster reindex cadence. The reindex cadence wins for Trellis specifically, because a stale torque spec is a safety issue, not just a missed citation.
The model got better, for real: say Citelume's underlying model gets a bigger context window next year. That's not proof retrieval stops paying for itself, since Fenbrook's corpus is still far larger than any window could hold, and the reranking and upkeep lines exist independently of how big the model's context gets.

Where people run it wrong.
They price a RAG pipeline as the single call's token cost times some rough multiplier, and never separately name reranking or index upkeep at all.
They cut a retrieval default under budget pressure and never write down a plan to recheck it once the corpus changes shape.
They treat "the corpus is big" as reason enough to build a full pipeline, without checking whether a much smaller, static slice of it could have just been stuffed into one call.

How to use it live. Say the real question out loud before quoting a number: "before I give you one figure, is this asking what the extra tokens cost, or what it costs to keep the answers actually grounded." That buys a beat to think instead of repeating the doubled number that got written down on launch day.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0