What is a latency budget and how would you allocate one across a RAG pipeline?

The total added up perfectly on a spreadsheet. Five seconds, eight stages, every millisecond accounted for. Nobody asked what the model should say the moment one of those stages ran out of time.

How to answer this, stage by stage

Nobody's grading whether you can list the stages of a RAG pipeline. They're grading whether you can say which one earns the most protected time, and defend it when someone points out it isn't the slowest one today.

Let's learn

Farelight is the trip planner inside Coastmere Travel's app. A traveler types "plan day three in Lisbon," and Farelight reads its own destination guides and calls live flight, hotel, and tour availability, then hands back a day plan with real bookable prices already in it.

Before Farelight, a traveler pieced an itinerary together by hand: a static guide open in one tab, a hotel site in another, a flight search in a third, cross-checking prices themselves. That took about 35 minutes for a rough three-day plan, and only travelers willing to put in the work bothered, roughly 300 planning sessions a week in Coastmere's early beta.

Farelight built the same three-day plan, prices included, end to end, and aimed to do it in under five seconds. Travelers liked it enough that planning sessions grew from 300 a week in the beta to 40,000 a week within about ten months, once it rolled out across the main app.

The turn: the stale prices were never about the model getting worse. The same version of the language model ran the entire ten months. The turn is that the team set a hard five-second budget, and to make eight stages add up to five seconds on a spreadsheet, they gave the smallest, tightest slice to live availability, the one stage checking whether a room or a seat was still actually there.

Here's the build-up behind the budget the team eventually corrected to. Five seconds, split across eight stages, once availability finally got the slice the decision actually needed.

Query rewrite, retrieval, and reranking barely move. Between them they cost 550 milliseconds and they're the easiest part of the whole pipeline to cache, since a destination guide for Lisbon reads about the same on a Tuesday as it did last Thursday. Generation is the heaviest single computation, but once it streams, a traveler sees the first line of the plan under half a second in, long before the full 1,700 milliseconds finishes.

Availability doesn't work that way. It's a call to a hotel wholesaler and two flight providers outside Coastmere's control, and the number that mattered wasn't the average response time. It was how often that call ran out of whatever slice it had been given.

Seven of the eight slices in this budget can be re-tuned next sprint with nobody the wiser. One can't. That's the whole reason it gets the guarded slice even on the week it isn't the slowest stage.

At its worst, a budget that adds up correctly keeps quietly shipping a wrong fact dressed as a confident one, until a customer calls asking why the suite the app promised isn't the suite that's there.

What I'd leave alone: query rewrite and reranking genuinely didn't need this treatment. Together they were 250 milliseconds that never once produced a wrong fact, and shaving another 50 off them wouldn't have caught a single stale price.

The lesson: a total that's correct on a spreadsheet can still be wrong on the one line with the power to embarrass you. The number worth protecting isn't the sum. It's which slice is allowed to fail quietly.

Now here is the same thing as a story

Read the long version below when you want to feel why a budget that balanced perfectly on paper still let a traveler book a room that wasn't there, not just be told that it did.

Katla Vireborg could look at a rough trip request and tell you inside a minute whether Farelight would have an easy time with it or a hard one. She'd run pricing and packaging at a regional airline for five years before Coastmere hired her to own Farelight end to end.

The early months were good, genuinely good. Farelight launched with no hard budget at all, just a promise to wait as long as it took and show a spinner in the meantime. Travelers didn't mind; planning a trip is a lean-forward task, not a chat reply. Sessions grew from the 300-a-week beta to 8,000 a week by month four, and the plans it produced were accurate, because nothing ever got cut short.

It faded in three beats, and none of them looked like a mistake at the time. Beat one: as sessions kept climbing, some responses started taking nine, ten, eleven seconds during busy hours, and travelers began closing the tab before Farelight finished. Beat two: the team set a hard five-second ceiling to stop the drop-off, and split it across eight stages the way an engineer splits any budget, generation first, since it was the heaviest single computation, availability last, squeezed down to whatever was left over, 1,200 milliseconds. Beat three: nobody wrote down what the itinerary should say if that 1,200-millisecond call ran out. It just quietly kept the last price it had.

It surfaced on an ordinary Thursday, not through a dashboard. A support agent, Sinead Kilbrennan mentioned it to Katla afterward, took a call from a traveler asking whether a lakeside suite in Queenstown the app had shown that morning was really still available at that price. Sinead pulled the booking attempt while the traveler waited on hold, and the live check came back different from what Farelight had shown: the room was gone, sold out forty minutes earlier. The traveler hadn't lost anything, she'd called first. But it was the second time that week support had fielded almost the exact same question.

Katla pulled the numbers that afternoon. The blended latency dashboard looked healthy, p95 sitting right at 4.8 seconds, comfortably under budget. Split by how often the availability call actually hit its 1,200-millisecond ceiling instead, the picture changed: during the two busiest hours of each evening, the hotel wholesaler's own response time regularly ran past 1,200 milliseconds, and every one of those timeouts fell back to a cached price with nothing on the screen to say so. Across the month before the call, that was 190 itineraries a traveler had acted on with a price that was no longer real, and about $6,400 in goodwill credits Coastmere had already paid out quietly to make it right.

The decision that opened the door went back to the week the five-second ceiling was set. Someone asked how to split the time, and the honest answer at the time was to give the most milliseconds to whichever stage did the most work, which was generation, waiting on a full completion before showing a single word. Nobody in that meeting asked which stage was allowed to be wrong instead of just slow. It became the rule anyway.

Run that meeting again with one change: availability gets the guarded slice, 2,200 milliseconds, split across two backup providers running in parallel so one slow wholesaler doesn't sink the whole call, and generation streams instead of waiting for a full completion, so the ceiling holding steady at five seconds barely changes what a traveler notices. Same 40,000 sessions a week. The stale-price rate drops from 2.6 percent in its worst week back to about 0.15 percent, and on the rare timeout that still happens, the itinerary says plainly: price may have changed, tap to refresh, instead of staying silent.

One design assumed the slice that computed the most deserved the most time. The other design asks which slice, if it's wrong, a traveler can act on before anyone gets a chance to fix it.

What I'd tell myself, back in the meeting where the five seconds first got split: ask what the app says the moment a slice runs out, before you ever agree on how many milliseconds it gets.

ORDER, the five letters behind the split

Not a story wearing a framework's clothes. This is a time-budget allocation problem, and ORDER is what stops "whichever stage computes the most" from quietly standing in for "whichever stage can't be wrong."

Three things worth stating directly, since this is where the real judgment sits. The alternative Katla's team considered first, and dropped, was raising the total ceiling from five seconds to eight, so availability could get more room without cutting generation's slice at all. It lost fast in a one-week test: abandonment, travelers closing the tab before the plan finished, climbed from about 4 percent to 11 percent, because a longer total wait cost more trust than a reallocated one ever had. The AI-specific failure worth naming by name is a silent fallback dressed as a confident answer: when the live-availability call times out, the model still writes the price into the itinerary in the same certain voice it uses for everything else, so a stale number reads exactly like a true one. The guardrail is a freshness check inside validation, refusing to state a price whose availability data is older than 90 seconds without a visible caveat, and logging every time the fallback path actually fires so it shows up in a weekly count instead of a support call. That guardrail isn't free. Coastmere accepted a slightly higher live-availability bill, calling two backup providers in parallel instead of one, to cut how often that 90-second window gets missed in the first place, because the alternative, a single slow provider deciding the freshness of every price on the page, made the guardrail fire on almost every peak-hour request.

And if you want to be sure it really works, try it somewhere else

Same five letters, an insurance adjuster's coverage lookup instead of a traveler's itinerary, and this time the lever wasn't a wholesaler's API, it was what a timeout was allowed to assume.

ClaimPath is Thornwick Insurance's AI tool. An adjuster asks it a coverage question mid-call, and ClaimPath retrieves the relevant policy language and checks a live third-party lookup for whether the policy is still active, then answers in under four seconds so the adjuster never has to put a caller on hold. Ozan Katsaros owns cost and quality on it.

The build-up: ClaimPath's four-second budget gave the live policy-status lookup 900 milliseconds, most of it eaten by generation's 2,100. When the lookup timed out, about one call in sixty during peak hours, ClaimPath answered as if the policy were active, because "no news" silently became "good news" inside the fallback logic. Over one quarter, that meant 34 coverage answers given on policies that had actually lapsed, caught only when a claims audit cross-checked payment records afterward.

Same rank, different lever: availability-style data goes first here too, not because the lookup is the slowest stage, it wasn't, but because a wrong "yes, you're covered" is a sentence an adjuster can act on immediately, approving a payment, that's much harder to claw back than a slow generation call ever was. The lever this time wasn't which provider to call in parallel. It was what the timeout was allowed to assume by default: Ozan's team flipped it from "assume covered" to "assume unknown, escalate," so a slice running out of time now costs a short hold instead of a wrong approval.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: give the biggest guarded slice to whichever stage touches a live, changeable fact, not whichever stage is slowest to compute, and make its timeout fail safe, never silently confident.
Cost: there's no budget this quarter for both a bigger buffer and a second data provider. Fund the second provider. A guarded slice that rarely times out beats a bigger buffer sitting behind a single point of failure.
The model got better, for real: say Farelight's language model gets meaningfully faster at generation. That's real, and it should let generation's slice shrink further. It does nothing to how often a hotel wholesaler's own API is slow at 7pm, which is what actually decides how often availability's guardrail has to fire.

Where people run it wrong.
They allocate a latency budget by which stage is heaviest to compute, instead of by which stage's mistake a user can act on before anyone can fix it.
They set a timeout number before deciding what the product says the moment it fires, so the fallback quietly becomes silence instead of an honest label.
They raise the total ceiling to buy a struggling stage more room, instead of asking whether the room already in the budget is just pointed at the wrong stage.

How to use it live. Say the real question out loud before naming a single millisecond: "if this stage runs out of time, does the answer just get slower, or does it quietly become wrong." That buys a beat to actually rank instead of reciting whichever stage's benchmark number you remember.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0