How do you budget latency when a third-party API's response time is outside your control?

Quennoc built Trackspan so customers could text "where's my package" and get a real answer instead of a form email. Trackspan doesn't know where any package is. It asks Ferrold, the carrier that actually has the truck, every time. Ferrold is fast almost always, and then, for one call in twenty, it isn't, and Quennoc can't fix Ferrold's servers, only decide what Trackspan does while it waits.

How to answer this, stage by stage

Nobody is grading whether you know a number for Ferrold. They're grading whether you know that budgeting a dependency you don't control means budgeting its worst honest behavior, not its usual one, and that the fix lives on your side of the wall, not theirs.

Let's learn

Trackspan is the chat assistant Quennoc's customers text when they want to know where a package is, and it answers by asking the carrier's own system directly instead of making a person go look.

Three of these four steps belong to Quennoc. The third one belongs to Ferrold, and Ferrold's clock is the one Trackspan actually has to budget for.

Before Trackspan, a Quennoc support agent needed about four minutes and three separate browser tabs, one per carrier site, to answer a single "where is my package" question, and the answer they typed back was usually already a few hours stale by the time it reached the customer.

Four minutes and three browser tabs, for one question, answered by one person, one at a time.

Trackspan answers most tracking questions in under a second now, because it asks Ferrold the moment the question comes in, instead of routing it to a person with three browser tabs open.

The turn: the extra wait was never really the problem on its own, not most days. Ferrold's answer is almost always right and almost always fast. What Trackspan couldn't control was the one call in twenty where "almost always fast" wasn't true, and on those calls, the honest reply was silence, because nobody had told Trackspan what to say instead.

At its worst, before this design shipped, a slow Ferrold afternoon didn't just make Trackspan feel sluggish. It made a growing slice of customers assume the whole assistant was broken, and roughly 1 in 8 of those slow conversations turned into a support ticket asking "is this working?", about 225 tickets a day, near $900 a day, close to $27,000 a month, entirely caused by an API Quennoc never touched a single line of code inside.

What I'd leave alone: Trackspan's own local processing, parsing the question and formatting the reply, already runs around 170 milliseconds and is fully inside Quennoc's control. Shaving that to 140 wouldn't move the total in any way a customer could feel, next to a carrier tail that swings by over 2 full seconds. Effort spent there is effort not spent on the lever that actually matters.

The lesson: you can't budget a number you don't own by hoping it behaves. You budget it by deciding, in advance, exactly how long you'll wait for it before you stop waiting, and what you'll say the moment you do. The API's tail was never Quennoc's to fix. The silence during that tail always was.

Now here is the same thing as a story

Read the long version below when you want to feel why a pipeline that never once returned a wrong answer still trained customers to stop trusting it.

Idriss Doyle had spent his first year at Quennoc cutting the load time on the internal warehouse dashboard in half, entirely by rewriting Quennoc's own code faster. When Trackspan needed its first version of a carrier lookup, Idriss built it the same way he'd built everything else that had worked before: call Ferrold, wait for the real answer, send it back. Simple, and in every test he ran, fast.

For the first two months, that held up fine. Trackspan launched with three smaller regional carriers, none of them handling more than a few thousand packages a week, and their tracking APIs answered in a few hundred milliseconds almost every single time. Customers texted "where's my order," and a real answer came back before they'd even put their phone down. Idriss watched the average response time sit under half a second, day after day, and moved on to other work.

Then Quennoc signed Ferrold, the carrier that actually moved most of its volume, and Trackspan's traffic jumped along with it. Ferrold's API was, on average, just as fast as the smaller carriers had been. But Ferrold ran a much bigger, busier system, and a much bigger, busier system has bad afternoons the small ones never showed. Once or twice a day, a call to Ferrold would take two seconds instead of two hundred milliseconds. Nobody noticed, because the average barely moved. A slow call here and there, buried inside 60,000 questions a day, doesn't dent an average at all.

No single bad call. Just a slow trickle of tickets, building for a week, before anyone went looking for why.

What changed Idriss's mind wasn't one dramatic call. It was a week where the support queue's "is this broken?" tag ticked up a little each day, Monday to Thursday, still a small number against 60,000 daily questions, easy to write off as noise. On Friday, three tickets in, Idriss finally pulled the raw call logs instead of trusting the dashboard's average.

What he found: on the calls that took over two seconds, roughly 1 in 8 turned into a support ticket. Trackspan itself had done nothing wrong on any of them. It had asked Ferrold a question, and waited, silently, for however long Ferrold felt like taking, because that's what Idriss's original design told it to do. A customer staring at a chat window with nothing happening for six seconds doesn't experience that as "the carrier is a little slow today." They experience it as "this thing is broken," and they were right to, because from where they were sitting, nothing was happening at all.

The old design wasn't wrong when Ferrold was small and fast. It never had a plan for what Ferrold would look like at real scale.

Idriss's first instinct, sitting with that log, was to make Trackspan retry the call if it ran long, on the theory that a second attempt might land faster. A colleague talked him out of it inside a day: retrying a call to an endpoint that's already struggling adds more load to that exact endpoint, and in two of the sampled cases it had actually made a slow call slower, not faster. What Trackspan needed wasn't a second attempt at the same question. It needed a plan for what to say the moment the first attempt ran past a line worth defending.

The decision that had opened the door went back to that very first version, three months before Ferrold was even part of the plan. Waiting for the real answer, no matter how long it took, was the obviously correct choice when every carrier Trackspan called answered in a few hundred milliseconds, nearly always. Nobody in that first sprint decided that Trackspan should still be holding an open, silent connection for up to seven seconds a year later, once its biggest, busiest carrier joined the roster. It just kept working, right up until the one carrier that mattered most was also the one with the widest tail.

Run that same Friday again, with the 2 second cap and the fallback message in place from day one. The support queue's "is this broken?" tag never climbs, because a customer whose call runs long sees "still checking, I'll message you" inside two seconds, then gets the real answer by text a few seconds later. Idriss never has to pull a log to find a problem, because there's no slow trickle of tickets building quietly for a week to go looking for.

What Idriss would tell himself, back in that first sprint: waiting for the real answer wasn't wrong when it shipped. It had simply never been asked to survive a carrier ten times the size of the ones it was built and tested against, and nobody had put a date on when to ask it that question.

BOUND, run against Trackspan's actual budget

Not a story dressed up as a framework. This is an estimation problem with a piece you can't move, and BOUND is what stops "the API is slow" from becoming the whole answer.

Three of these four terms are small, stable, and fully owned by Quennoc. One of them is neither small, stable, nor owned by anyone at Quennoc at all.

Three things worth stating directly, since this is where the real judgment sits. The alternative Idriss's team seriously considered and rejected was automatically retrying a slow Ferrold call, dropped once two of the sampled retries came back slower than the original call, adding load to an endpoint that was already struggling instead of relieving it. A second alternative, switching to a different tracking vendor entirely, was never really live: which carrier answers a given question is set by which carrier the customer's package actually shipped with, not something Quennoc can route around. The AI-specific failure worth naming is hallucinated status: Trackspan uses a model to turn Ferrold's structured payload into a plain sentence, and if a call times out with no data, that model must never invent a plausible-sounding status like "it's out for delivery" from nothing. The guardrail is a hard one: the model is only ever handed the fields Ferrold actually returned, and when the timeout fires before any payload exists, the fallback message ships on its own, without the model ever being asked to describe a status it was never given. And the trade being accepted plainly: a cached tracking status can run a few minutes stale, in exchange for a faster typical reply and far fewer repeat calls hitting an API Quennoc doesn't control or pay to overuse.

And if you want to be sure it really works, try it somewhere else

Same three letters worth repeating, a pharmacy refill line instead of a package, and this time the lever that moves the estimate isn't the timeout or the cache. It's which questions ever needed to touch the slow third party at all.

Dosewell is the refill assistant Marlseed runs across its pharmacy chain. A customer texts about a prescription, and for some questions Dosewell has to call Duncastle, the outside benefits system that actually knows what insurance will pay. Sorenne Adair, Marlseed's pharmacy operations lead, owns Dosewell's latency budget.

Duncastle is slower than Ferrold on a bad day: P50 near 500 milliseconds, P95 near a full 5 seconds. But the bigger lever here wasn't Duncastle's own tail at all. It was that Dosewell was asking Duncastle for answers Marlseed already had. "Is my refill ready" never needs Duncastle, that answer sits in Marlseed's own system, current to the minute. "Will insurance cover the new dose" genuinely does, because that answer can change the same day. Sorting questions by which bucket they actually belong in, not by which words they contain, cut the share of refill conversations that ever touched Duncastle's slow tail from about 70 percent to about 25 percent, and the share of conversations running past 3 seconds fell from roughly 4 in 10 to about 1 in 8, without Duncastle getting one millisecond faster.

The lever wasn't speed. It was routing. Most of these questions never needed to leave Marlseed's own record in the first place.

Same method, different lever: for Trackspan, every question genuinely needs Ferrold, there's no version of "where's my package" that Quennoc's own system already knows on its own. The lever there was capping the wait and catching repeats with a cache. For Dosewell, a large share of questions never needed the slow third party at all. The lever there was classification: correctly routing the question before deciding whether to pay Duncastle's tail for it.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: state which part of the third party's own distribution you're defending against, then say what you do the moment you cross it.
Cost: Duncastle charges Marlseed per live lookup call. Routing the easy 70 percent of questions away from it wasn't just faster, it also cut Marlseed's monthly Duncastle bill by roughly the same share.
The model got better, for real: say Duncastle's own latency improves next quarter. That shrinks the P95 tail, which helps, but it does nothing about the 70 percent of questions that never needed to touch Duncastle in the first place. Routing still wins on its own.

Where people run it wrong.
They budget around the third party's advertised average instead of pulling the real P95 and P99 off their own logs.
They fix the timeout, then let it end in more silence instead of an honest message, which solves nothing the customer can actually feel.
They retry a slow call by default, on the hope that trying again will somehow land faster, when it usually just adds load to the exact endpoint already struggling.

How to use it live. Ask the question that actually decides the number before quoting one: "what does this dependency's own distribution look like, not its average, and which part of it am I choosing to defend against?" That's the question that tells you whether a 2 second cap is generous or already too slow.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0