Describe how parallelizing tool calls affects an agent's latency profile.

Cargrave Systems built Groundwire to work an incident the way a good on-call engineer already does: pull the logs, check the metrics, search the runbook, and see what shipped recently, before anyone has to go looking themselves. The model doing the reasoning was never the slow part. The slow part was Groundwire asking each tool one question, waiting for the answer, then asking the next, the exact habit a careful engineer falls into by hand and a program never has to copy.

How to answer this, stage by stage

Nobody is grading whether you know the word "parallel." They're grading whether you can show the actual arithmetic behind it, out loud, including the one case where firing things together is the wrong move.

Let's learn

Groundwire is an agent that gathers the first useful facts about a live incident, so a person doesn't have to go looking themselves.

Before Groundwire, an on-call engineer at Cargrave Systems opened four different tools by hand for every page: the log search, the metrics dashboard, the runbook wiki, and the deploy history. That took about six minutes on an average incident, most of it spent switching tabs and waiting for each one to load, before anyone had actually started fixing anything.

This is what a page used to cost an on-call engineer before Groundwire existed, four tools, opened and waited on one at a time.

Groundwire's first version called all four tools the same way a careful engineer would, one at a time. It replied in about 2,800 milliseconds, still miles faster than six minutes by hand, so nobody thought twice about it.

The turn: The extra milliseconds weren't really the problem on their own. The problem is what happens during a loud, high pressure incident: the on-call engineer, staring at a spinner during the loudest ten seconds of the outage, opens one of those same four tools themselves anyway, just to feel like they're doing something, duplicating the exact work Groundwire is already halfway through.

The old shape: five boxes in a row, each one waiting its turn behind the one before it.

At its worst, that extra 1.5 seconds isn't just slow. During a real outage, in the ten seconds after the page fires, it's the gap that convinces someone to stop trusting the agent and start doing its job over again by hand, at the exact moment their attention is worth the most.

What I'd leave alone: the one pairing where log search and runbook lookup genuinely depend on each other stays chained, on purpose. Forcing that pair into the same wave doesn't make Groundwire faster, it makes it search the runbook before it knows what to search for, which just means a wrong answer arrives at the same fast speed.

The lesson: an agent doesn't get graded on how many tools it can call. It gets graded on whether the wait between the page and the first line of context feels like the agent is already working, or like nothing happened yet. Chain what truly depends on itself. Fire everything else at once. The difference is the whole 1,500 milliseconds.

Now here is the same thing as a story

Read the long version below when you want to feel why a pipeline that never crashed, never errored, and always finished under three seconds still lost an engineer's trust in the middle of an outage.

Ashwin Everleigh had run on-call rotations for six years before he built Groundwire's first version, and when it came time to wire up its four tool calls, he did the obviously careful thing. He made it check the logs, then the metrics, then the runbook, then the deploy history, one after another, exactly the order he'd worked an incident himself for years.

For the first two months, on the small, quiet services, that order barely showed up on anyone's radar. A page would fire, Groundwire would reply in under three seconds, and the engineer on call would already be reading real context before they'd even opened their laptop fully. It felt, genuinely, faster than anything Cargrave Systems had shipped before.

Then Groundwire got wired into the payments service, the one with a page almost every week, and the pages got louder. Three seconds started to feel different when four other engineers were already typing in the incident channel, asking if anyone had eyes on it yet.

What changed, slowly, was who reached for the tools first. On a quiet Tuesday, the engineer on call would wait the full three seconds without noticing. On a loud payments incident, more and more of them started opening the metrics dashboard themselves the second the page landed, not because Groundwire was wrong, but because three seconds of silence during a loud incident felt like nothing was happening.

It surfaced at a retro, three weeks later, when Yumiko Holdrick, who'd taken four of those payments pages herself that month, said something offhand while the team reviewed the timeline. Ashwin had pulled up the dashboard and pointed at the average reply time, which read a healthy 2.9 seconds. Yumiko asked why it still felt slow if the number said it wasn't. Ashwin didn't have an answer ready.

The trigger was small: one honest question at a retro, not a single failed page.

So he watched a real incident live the next time payments paged. What he saw: while Groundwire quietly worked through logs, then metrics, then the runbook, then deploy history, the on-call engineer had already opened the metrics dashboard themselves by the second step, running the exact same query Groundwire was about to run one call later. Two people, one system, doing the same lookup twice, because the wait felt long enough to fill.

Ashwin's first instinct was the sensible one: make the individual calls faster, maybe cache the metrics query. Yumiko talked him out of it fast. A faster metrics call barely touches the total, because the total was never really about any one call. It was about all four of them queued up behind each other when three of the four didn't need to be.

The decision that opened the door went back to Groundwire's very first sprint, when it had exactly one tool and calling things "in order" wasn't a decision at all, it was the only shape there was. Nobody in that sprint planning meeting decided a one-tool habit should still be running a four-tool agent a year later, on a payments incident, at 2am. It just kept working, the way a choice that never once causes an error stops looking like a choice anyone made.

Run that same retro again, with the fan-out design in place from the start. Yumiko never says anything offhand, because there's nothing to notice: the reply lands in about 1.3 seconds, inside the window where silence still reads as "still working," and nobody on the payments incident opens a second tab to check something Groundwire is already checking.

One design let four questions line up and wait their turn. The other lets the three that don't depend on anything go at once, and only makes the fourth wait when it genuinely has to. Same four tools. The difference is whether a person watching the wait ever reaches for a keyboard.

What Ashwin would tell himself, back in that first sprint: building it one tool at a time wasn't wrong, it was the only shape a one-tool agent could have. It was never given a date to be asked whether it still made sense once a second tool showed up, then a third, then a fourth.

BOUND, run against Groundwire's real incident pipeline

Not a story question wearing a framework's clothes. This is an estimation problem, and BOUND is what keeps "parallel is faster" from being a slogan instead of arithmetic you could defend.

The new shape: one dispatch, four calls going out at once, one join bringing the answers back together.

Even sequential's best case sits above parallel's worst case. The two ranges barely touch.

Three things worth stating directly, since this is where the real judgment sits. The alternative Ashwin's first design used was interleaved, one-at-a-time calling: decide, call a tool, read the result, decide the next call, the same shape a careful person uses by hand. It got dropped for three of the four calls once the retro showed it added the same 1.5 second tax as pure sequential, even though metrics, runbook, and deploy history never needed each other's answers in the first place; the interleaved pattern only stayed for the one real dependency, log search into runbook lookup. The AI-specific failure worth naming by name is silent degradation: when the fan-out crosses the concurrency cap, calls don't error, they just quietly queue, and the parallel time creeps back toward the sequential number with nothing in the logs to say why. The guardrail is watching the p95 join time against the expected ceiling, the slowest call plus overhead, in production, and treating a rising gap between the two as the tell that throttling is happening invisibly. And the trade-off Groundwire accepts to protect its budget: every fanned-out call gets a two second timeout, so a slow, thorough 30-day log scan sometimes gets cut short in exchange for an answer that lands while it still matters, and Groundwire says plainly which source it's missing rather than answering as if it had all four.

And if you want to be sure it really works, try it somewhere else

Same five letters, an insurance claims agent instead of an incident responder, and this time the lever isn't a concurrency cap, it's that one call is so much bigger than the other two that firing them together barely helps.

Claimrunner is a claims-intake agent Corribrook Mutual runs on every new auto claim. The moment a claim comes in, it fires three calls: a policy lookup, a fraud-risk score, and a damage estimate built by running a vision model over the uploaded photos. Aditi Solmundsen owns its latency budget.

Firing all three together instead of chaining them still helps, the total drops from about 2,650 milliseconds to about 1,900. But that's a much smaller win than Groundwire's, because one call, the damage estimate, already eats 1,200 to 2,500 milliseconds on its own, while the policy lookup and fraud score barely register at 200 to 400 and 400 to 700. Parallel can only ever get Claimrunner down to whatever that one big call costs. The real lever here isn't firing things together, it's shrinking the damage estimate itself, a smaller image size before it hits the vision model, or a fast, rougher first pass with a slower, sharper one running behind it.

Two calls sit close together near the fast, steady corner. One sits alone, far off toward slow and unpredictable, and that one call sets the floor no matter how the other two are called.

Same method, different lever: for Groundwire, the lever that moved the estimate was whether the fan-out crossed a concurrency cap. For Claimrunner, there's no cap in play, three calls never gets close to any gateway's limit. The lever here is that the calls aren't close to the same size, so parallel buys back the two smaller ones' time and nothing more.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: say which calls are independent, say the total is bounded by the slowest one, then name what actually sets that ceiling.
Cost: there's no time this sprint to both resize photos before the vision model and rebuild the fraud-score call. Resize the photos first, it's the smaller change and the bigger lever anyway.
The model got better, for real: say the vision model doing damage estimates gets meaningfully faster next quarter. That's the one upgrade that actually moves Claimrunner's total, because it's the one call the whole thing is bounded by.

Where people run it wrong.
They copy the order a person used to work through by hand, policy first, then fraud, then photos, without asking whether any of the three actually needs another one's answer.
They celebrate a parallel win that's really just hiding one dominant call, instead of asking whether shrinking that one call would help more than firing things together ever could.
They add a fifth or sixth tool without checking what the gateway allows, and find out about the concurrency cap during a real incident instead of before one.

How to use it live. Ask the real question before quoting a number: "are these calls actually independent, or does one need another's answer first, and if they are independent, which one alone sets the floor on how fast this can ever be?" That question tells you whether parallel is the fix, or just a smaller win hiding a bigger one.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0