How do reasoning models complicate a latency budget?
A reasoning model does not add a few slow seconds evenly across every call. It adds one private phase whose length depends on how hard the problem looks, and a budget built for steady speed cannot see that coming until it is already late.
The direct answer
Reasoning models add a thinking phase before the first visible word, and that phase can run from a few hundred tokens to tens of thousands depending on how hard the task looks, not on how much text goes in or out. A budget built on steady per token speed cannot predict that, so it breaks. Fix it by tiering the budget to task difficulty, logging thinking tokens as their own number apart from output tokens, and putting a hard ceiling on thinking time with a fallback answer ready the moment that ceiling hits.
Do this, in order
Rebuild the budget around thinking time as its own variable, tiered by task difficulty, not one blended number for the whole job.Why: a single blended budget hides the one hard case that is actually running long, the way an average always can.
Log thinking tokens and output tokens as two separate numbers on every call.Why: without that split, a slow report cannot be traced back to its real cause, only guessed at.
Set a hard ceiling on thinking time per task, with a fallback answer ready the moment it hits.Why: one hard case should never be allowed to freeze the whole job while everything else waits on it.
Rule out a timeout, retry, or queueing bug before blaming reasoning depth for a slow call.Why: a misconfigured timeout can look exactly like a real slowdown on a dashboard.
Show real progress or a partial result instead of promising one fixed finish time.Why: the thinking phase cannot be promised in advance, so the interface should stop pretending it can.
Leave the old fixed budget alone for calls that never reason, like a lookup or a formatting pass.Why: adding tiers and ceilings to a call with nothing to think about is complexity nobody needs.
How to answer this, stage by stage
Nobody is grading whether you know the phrase "reasoning tokens." They are grading whether you can see that the extra minutes are not spread evenly, they are hiding inside the one case that was always going to be hard.
1
Scope it to one real product before answering in the abstract
Say it like this
"Let's ground this in one product. Diorite is the AI agent inside Cindercroft Partners, a private equity firm, that reads a target company's data room and writes a twelve section due diligence report. Tindra Dreyfuss is the PM who owns it."
Why this works
Keeps the answer attached to a real budget and real numbers instead of a general complaint about slowness.
2
Say your structure out loud
Say it like this
"I'll answer this in two parts. First, exactly what a reasoning model does to a latency budget that was never built for it. Then what that actually cost Cindercroft the one time it happened."
Why this works
Signals a plan before diving in, so this doesn't read as a vague complaint about the model being slow.
3
Reframe the question before answering it
Say it like this
"This isn't really about the model getting slower. A reasoning model adds a private thinking phase before it says one word, and how long that phase runs depends on how hard the question looks, not on how much text goes in or out. That's the thing a budget built on steady speed can't see coming."
Why this works
Separates a sharp answer from a list of reasons the model might be "a bit slower now."
4
Give the one decision, plainly
Say it like this
"Stop budgeting one blended number for the whole job. Tier the budget by how hard each part actually is, log thinking tokens as their own number, and put a hard ceiling on thinking time with a fallback ready the second it's hit."
Why this works
This is the actual fix, said as something you would build, not a category of thing you would consider.
5
Prove it with the failure, cut to four sentences
Say it like this
"Here's what happened at Cindercroft. Diorite swapped a reasoning model into four judgment heavy sections, and the report's blended finish time barely moved, still around forty five minutes. But the related party transactions section alone ranged from twenty seconds to twenty two minutes on the exact same size report, because its thinking tokens ranged from about twelve hundred to thirty one thousand depending on how tangled the ownership structure was. On a Tuesday, one report froze on that section, and the deal partner was messaging Tindra at eight fifty two asking if it was done."
Why this works
Shows the exact mechanism, not just the conclusion that the report ran late.
6
Say what you would measure going forward
Say it like this
"I'd watch thinking tokens per section as their own number, not folded into total tokens. And I'd gate any reasoning model swap on the p95 of the hardest section, not the report's blended average, before it reaches every deal team."
Why this works
Shows you're designing the fix to catch the next one, not just explaining the last one.
7
Say what you'd leave alone
Say it like this
"Not every section needs this. Diorite's company overview and cap table sections never call the reasoning model, they just pull and format facts, so their old fixed budget is still exactly right."
Why this works
Shows judgment instead of retrofitting tiers and ceilings onto every call in the system.
8
Close on the decision, not the story
Say it like this
"So: a reasoning model doesn't make a latency budget wrong by a little, it makes the one number you were trusting blind to its own worst case. Tier the budget by difficulty, watch thinking tokens on their own, and always have a fallback ready before the ceiling hits."
Why this works
Ends on the rule, which is what a candidate can actually reuse the next time this comes up.
Let's learn
Here is what happens when a model gets better at thinking and no more predictable at anything else. It stops missing a budget by a little. It starts missing it by whatever the hardest case that week happens to need.
Diorite is the part of Cindercroft Partners, a private equity firm, that reads everything in a target company's data room, filings, call transcripts, and market data, and writes a twelve section due diligence report a deal team can bring straight into an investment committee meeting.
For over a year, Diorite used a standard drafting model with no separate thinking step. Every report, all twelve sections, finished inside the same band: thirty five to forty five minutes, almost every single time. The whole review calendar at Cindercroft got built around that number, because it never moved.
Knowledge spark: what's a thinking token?
Before a reasoning model writes its real answer, it writes out its own private working, step by step, checking itself as it goes. Nobody outside the model ever reads those steps, but each one still costs a token and still costs time. A short question might need a few hundred. A tangled one can need tens of thousands, and there's no way to know which until the model has already started.
Three weeks before the incident, Cindercroft swapped a reasoning model into four sections that call for real judgment: customer concentration, related party transactions, revenue quality, and litigation exposure. The goal was fewer factual slips on the sections partners actually argue about. Reports still finished. The blended p95 across the whole twelve section report barely moved, still forty five minutes, same as before.
The blended number stayed calm for three straight weeks before anyone outside the call logs could see what was actually happening.
Report completion time, blended p95 vs the worst section's p95, four weeks
Blended report p95, whole reportRelated party transactions p95, one section
The blended report time never moved off about forty five minutes. The one section's own p95 climbed from five minutes to twenty two in the same four weeks, invisible to the number everyone was watching.
The extra minutes were never the problem. It was that nobody could tell, on any given morning, whether this report would take five minutes on that section or twenty two.
P95 minutes per section, incident week
No reasoning callReasoning, normal caseReasoning, tangled case
Eight of the twelve sections never call the reasoning model and stayed under a minute, same as always. Of the four that do, three stayed under ten minutes. The fourth, related party transactions, reached twenty two minutes on this report, the one slice a blended average could never show.
The choice that mattered
Diorite's original latency budget was set in a planning meeting two years earlier, as one blended p95 across the whole report, back when every section used the same non-reasoning model at close to the same speed. That shortcut cost nothing while it was true. It stopped being safe the day one part of the report started thinking a different amount depending on the case.
What I'd leave alone: Diorite's company overview and cap table sections just pull facts and format them, no reasoning call involved. Their old fixed budget is still exactly right, and giving them a tiered ceiling would be complexity with nothing behind it.
The lesson: a latency budget built on steady speed will look calm right up until the day one part of the job stops being steady. The extra minutes were never the risk. The risk was a number that could not see them building inside one section, while it kept reporting that everything was fine.
Now here is the same thing as a story
Read the short version above for the two minute answer. Read this for why trusting that one blended number felt completely reasonable for two years running.
Cindercroft's investment committee met every Tuesday at nine, sharp. Deal partners walked in expecting a finished report on the target sitting in front of them, and for two years, one thing made that possible: Tindra Dreyfuss's rule that Diorite's twelve sections always finished together, inside the same forty five minute window, no matter which company was under review.
The good months were good. Report after report landed by eight, sometimes earlier, and the investment committee stopped even asking whether Diorite would be ready. It just was. Tindra had built the budget herself, watched it hold for a year, and moved on to bigger problems, because this one had stopped being a problem at all.
The habit thinned in three small beats. First, as Cindercroft ran more deals through Diorite, Tindra stopped personally checking each section's own finish time and let a single "report ready by eight" alert stand in for the whole picture. Then, when the team swapped a reasoning model into four judgment heavy sections to cut down on factual slips, it went through the same weekly release process as any other change, no separate look at whether the old budget's assumptions still held. Then Tindra herself picked up two more products that quarter, and started skimming the "report ready" alert instead of opening the section by section timing underneath it.
The trigger was small, and it landed on an ordinary Tuesday. At eight fifty two, nine minutes before the investment committee call, a deal partner messaged Tindra directly: "is Diorite done or not." The alert hadn't fired. The report was still sitting on the related party transactions section, the one part of the target company with a real web of connected entities behind it.
The report hadn't gotten slower. One section had simply started thinking longer than anyone had ever asked it to.
Tindra and an engineer spent the next two hours pulling the raw call logs instead of the dashboard. Three weeks earlier, the reasoning model swap had cleared its canary review cleanly, looked fine on every quality check the launch actually ran, and shipped to every deal team. Since then, the related party transactions section's own p95 had climbed from five minutes to twenty two, entirely invisible to a budget that only ever watched the whole report at once.
The decision that opened the door went back to that first planning meeting, two years earlier, when tracking one blended number for all twelve sections felt like the obvious way to keep a new dashboard simple. Nobody wrote a rule saying the budget would need revisiting the day one section started thinking a different amount case by case, because at the time nobody could picture a model that worked that way.
Run the Tuesday again with one change. The related party transactions section now has its own six minute thinking ceiling. On this replay, it hits that ceiling, returns its best partial finding flagged "needs a human pass," and the rest of the report ships on schedule at eight forty. The partner has it in hand twenty minutes before the call, and the associate spends those twenty minutes finishing one flagged section by hand instead of scrambling to summarize the whole target company live in the room.
One design trusted a number that could only describe the whole report. The other watched the one phase that could actually run away, and gave it somewhere to land before it took the meeting down with it.
What I'd tell myself, back in that first planning meeting: the day one blended number feels like the simplest way to track a report, ask what happens the first time one part of it starts taking a genuinely different amount of time depending on the case. Nobody asked. That's on the budget, not on Tindra.
TRACE, the five moves that catch a reasoning model before it blows the budget
Not a checklist for a status meeting. Five moves that build toward the one that actually separates a real cause from a guess: the evidence test.
TTimeline. When did it actually start, and what shipped near that date?
The reasoning model swap shipped to four sections three weeks before the frozen Tuesday, and it cleared the launch review's quality bar cleanly. The related party transactions section's own p95 started climbing the same week, quietly, invisible to anything watching the whole report.
A timeline that starts the morning the report froze starts three weeks too late. The real start is the release that looked like a clean, ordinary ship.
RRecut. Slice it by section, not by report, and see what the average was hiding.
Company wide, the blended report p95 never moved off about forty five minutes. Recut by section, eight of the twelve stayed under a minute the entire time, and three of the four reasoning sections stayed under ten minutes. The fourth, related party transactions, climbed from five minutes to twenty two, entirely alone.
A number that looks perfectly flat across a whole report is often hiding one slice doing all the moving.
AAssume nothing. Rule out a timeout or infrastructure bug before blaming reasoning depth.
Before trusting reasoning depth as the cause, the engineer checked whether a load balancer timeout was adding retries, and whether the twelve sections had quietly stopped running in parallel after a recent queue change. Neither had happened. The raw call timestamps showed the calls themselves genuinely took that long, and the sections were still running side by side.
A timeout or a serialization bug can look exactly like a real slowdown on a dashboard. Ruling both out first protects the investigation's most valuable hours.
All three looked plausible on day one. Only one of them survived the evidence test.
CCause candidates. Name three, not everything possible.
One, the drafting model had simply gotten slower under normal load across the board. Two, the reasoning phase on this one section balloons specifically on companies with tangled, multi-entity ownership structures. Three, a recent orchestration change had quietly serialized the twelve sections instead of running them side by side, so one slow section was now delaying every other section behind it.
Three named suspects, not a shrug. This is the step most rushed latency investigations skip, jumping straight to whichever explanation someone guesses first.
EEvidence test. The one check that tells the suspects apart.
The team pulled thinking token counts and output token counts, logged separately, for every call in the past month, and matched them against each section's completion time. Output tokens held flat, close to nine hundred, no matter the case. Thinking tokens ranged from about twelve hundred to thirty one thousand, and every section that blew past five minutes averaged about twenty seven thousand thinking tokens against about twenty one hundred for sections that finished on time. The trace logs also confirmed all twelve sections still ran in parallel, ruling out serialization.
This single check is the strongest move in the whole method. It turned three guesses into one confirmed cause in a single query.
Three things worth stating directly, since this is where the real judgment sits. The rejected alternative was capping every reasoning call at a fixed token ceiling company wide, the moment the mismatch was found. It lost because it would have silently cut off reasoning on exactly the tangled, highest risk cases, the ones where the extra thinking was catching a related party loop a shallow read would miss, trading a confident but shallow finding for a late but real one on the sections partners most need to trust. The AI specific failure worth naming by name is silent reasoning truncation: cap a reasoning model's thinking mid-thought and it still returns a fluent, confident looking answer, with no visible sign it was cut short, which is worse than an answer that is honestly late. The guardrail is tagging every truncated call and routing it to a human pass rather than shipping it as finished. And the trade being accepted on purpose is real: giving the hardest cases a real thinking ceiling before falling back costs genuine minutes on exactly the sections that matter most, a real cost against a budget that had already proven it could hide a seventeen minute swing on one section for three straight weeks.
The five, in one line each: T: the real start is the release that cleared review, not the morning the report froze. R: a flat blended average can sit right on top of one section quietly running away with the time. A: rule out a timeout or a queueing bug before trusting that reasoning depth actually explains the gap. C: name three real suspects, never jump straight to the first guess. E: one evidence test, thinking tokens matched against completion time, settles it.
Same five moves, a cargo manifest instead of a due diligence report
Not every reasoning latency problem is one call thinking too long. Sometimes it's several short calls that won't stop asking each other one more question.
Vandenberge Maritime runs Prynne, a tool that checks a ship's cargo manifest against customs codes, sanctions lists, and safety certificates before the vessel's harbor slot closes. Eloy Achterhof is the engineer who owns it. For most of a year, Prynne made one classification call per manifest, and the check finished in about ninety seconds, almost every time.
To catch subtler mismatches, the team then let Prynne chain several reasoning calls together: check a line item, reason about the result, decide what to check next. On a simple manifest that meant two or three hops. On a manifest with many flagged line items, the chain sometimes ran to fourteen hops, each one taking twenty to forty seconds to think through. The check time on those manifests stretched from ninety seconds to as long as eight minutes, close enough to missing a harbor slot close that it happened twice in one month.
The evidence test that settled it
Eloy's team matched total check time against two different numbers: tokens per hop, and number of hops in the trace. Tokens per hop barely varied between fast and slow manifests. Check time rose by about thirty five seconds for every additional hop, almost exactly, no matter what the manifest was actually flagging. The chain itself, not any one call's depth, was where the time was going.
Same method, different shape: a due diligence section thinking longer on tangled ownership and a manifest chaining fourteen reasoning hops look nothing alike on the surface, but both are the same TRACE move: don't trust the blended average, recut by segment, then run one test that separates a real cause from a guess.
Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to the one line: a reasoning model doesn't slow everything down evenly, it adds one unpredictable phase whose length depends on the problem, and a budget built for steady speed can't see that coming.
Cost: there's no time this sprint for a full tiered rebuild. Do the two cheapest steps only, log thinking tokens as their own number and put a hard ceiling with a fallback on the single worst section, and save the rest for later.
The model got better, for real: say the next reasoning model gets faster at the same depth of thinking. That's exactly when a blended budget looks most trustworthy, right before the next hard case shows up, since nothing about how the budget is built ever got fixed, only its current baseline.
Where people run it wrong.
They watch one blended completion number and never separately watch thinking tokens or hop count, so the budget cannot show them where the time is actually going.
They cap every reasoning call at one fixed token limit to force predictability, and silently cut off exactly the hardest cases the extra thinking was there to catch.
They blame the model for "getting slower" the moment one job runs long, without first ruling out a timeout or a queueing change that only looks like a real regression.
How to use it live. Say the two things a latency budget has to get right before saying anything else: what phase of the call is actually variable, and what happens the moment that phase runs long. That buys a beat of thinking time, and it tells the interviewer you know most broken latency budgets fail on the thinking phase, not the arithmetic.
Flashcards (tap any card to flip it)
1 · THE FRAMEWORK
What framework is this, and what's its one job?
Tap to flip
ANSWER
TRACE: timeline, recut, assume nothing, cause candidates, evidence test. Built for diagnosis questions, when a number, or a budget, has gone wrong and you have to find the real cause before you can fix it.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Tindra Dreyfuss, the PM who owns Diorite, the AI due diligence agent inside Cindercroft Partners, a private equity firm. She built its original latency budget herself.
3 · THE HABIT
What did the team stop doing as Diorite grew?
Tap to flip
ANSWER
They stopped checking each section's own finish time and let a single "report ready by eight" alert stand in for the whole picture, so the reasoning model swap shipped without anyone revisiting the budget's old assumptions.
4 · THE TWO NUMBERS
What are the two token numbers this answer turns on?
Tap to flip
ANSWER
Output tokens, which held flat near nine hundred no matter the case, versus thinking tokens, which ranged from about twelve hundred to thirty one thousand depending on how tangled the case was.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Setting the original latency budget, two years earlier, as one blended p95 across all twelve sections, with no rule to revisit it once one section started thinking a different amount case by case.
6 · THE NUMBER
Fill in the blank: sections that blew past five minutes averaged about ___ thinking tokens, while the blended report time stayed at forty five minutes the whole time.
Tap to flip
ANSWER
27,000. Sections that finished on time averaged about 2,100 thinking tokens, and the worst single case reached 31,000, invisible to the blended number.
7 · THE REPLAY
Same Tuesday, new design, what changes?
Tap to flip
ANSWER
A six minute thinking ceiling on the related party transactions section trips, returns a flagged partial finding, and the rest of the report ships on schedule at eight forty, twenty minutes before the committee call instead of freezing at eight fifty two.
8 · CROSS PRODUCT TRANSFER
Section 4 answers this same question again for a different product. Which product, and what caused the hidden latency this time?
Tap to flip
ANSWER
Prynne, a cargo manifest checker at Vandenberge Maritime. The cause was a chain of sequential reasoning hops, up to fourteen on a flagged manifest, not the depth of any single call.
Check yourself Score: 0 / 0
Fill in the blank
1. Fill in the blank: on Diorite's related party transactions section, thinking tokens ranged from about twelve hundred up to about ___ tokens on the most tangled ownership cases.
Show hint
The number appears in Section 1's story and again in the hand sketch comparison diagram in Section 3.
Show answer
31,000. That single worst case, against output tokens that held flat near 900, is what made the section's completion time swing from twenty seconds to twenty two minutes on reports that were otherwise the same size.
True or false
2. True or false: because Diorite's blended report time never moved off about forty five minutes, that proves the reasoning model swap didn't really slow anything down.
True
False
Show hint
Look at the line chart in Section 1. Compare what the blended line does against what the related party transactions line does over the same four weeks.
Show answer
False. The blended number stayed flat by construction, since it averages across eight sections that never changed. The related party transactions section's own p95 climbed from five minutes to twenty two in the same window, entirely invisible to the blended number.
Multiple choice
3. In the TRACE method, what is the point of the "assume nothing" step, applied to this latency investigation?
A. To double check the reasoning model's training data one more time.
B. To interview every engineer who touched the report pipeline.
C. To rule out a timeout or a queueing change before concluding reasoning depth actually explains the gap.
D. To cap every reasoning call at a fixed token limit as a precaution while the investigation runs.
Show hint
Think about what a misconfigured timeout or a serialization bug would look like on a dashboard, compared to a genuine increase in thinking time.
Show answer
C. A timeout misconfiguration or a serialization bug can look exactly like a real slowdown on a dashboard. Ruling both out first protects the investigation's most valuable hours from being spent chasing the wrong cause.
Short answer, name the rejected alternative
4. What alternative fix did Tindra's team consider and reject once they found the cause, and why did it lose?
Show hint
Look at the paragraph right after the evidence test step in the TRACE recap section.
Show answer
Model answer: They considered capping every reasoning call at a fixed token ceiling company wide. It lost because it would have silently cut off reasoning on exactly the tangled, highest risk cases, where the extra thinking was catching a related party loop a shallow read would have missed.
Multiple choice
5. At Vandenberge Maritime, check time rose by about thirty five seconds per extra reasoning hop, while tokens per hop barely changed between fast and slow manifests. What does that pattern tell you?
A. The manifest format itself is broken and needs to be redesigned.
B. Prynne's classification model has quietly gotten worse at its job.
C. A chain of sequential reasoning calls can compound delay in a way no single call's token count explains.
D. Hop count should be capped at one everywhere, with no further check.
Show hint
Compare what a slow single call would look like against what several short calls chained together would look like, on the same total time.
Show answer
C. Since time per hop stayed roughly constant but total time scaled with hop count, the delay was coming from how many reasoning steps got chained, not from any one step thinking unusually hard.
Short answer, apply it yourself
6. Pick an AI product you use that pauses or streams before answering. What's one sign it might be doing variable, hidden reasoning before the first word appears, and how would you tell that apart from it just being slow?
Show hint
Think about whether the pause tracks with how hard the question actually is, or with something unrelated like time of day or server load.
Show answer
Model answer: A coding assistant that answers instantly on a simple fix but sits silent for a long stretch on a gnarly bug. You'd tell it apart from general slowness because the pause tracks with how hard the question looks, not with time of day, and the delay sits entirely before anything starts streaming, not partway through.
Before you close the answer
Why this works
Tests whether you assume "the model just got a bit slower" is the whole story, or whether you know a reasoning model adds one unpredictable phase that a blended budget structurally cannot see. Most candidates suggest padding the budget, without seeing that padding hides the same problem instead of fixing it.
Follow-up traps
"Couldn't you just pad the whole budget to fifty minutes and call it done?" Response: padding hides the swing, it doesn't remove it. The related party section still ranges from twenty seconds to twenty two minutes, and now the pad just makes every report look artificially slow on the easy days too.
"Isn't capping thinking tokens the simple fix here?" Response: a fixed cap silently truncates reasoning on exactly the tangled cases it was added to catch, so you'd trade a late, correct finding for a fast, shallow one on the sections partners most need to trust.
If pressed
The evidence test that separated a real reasoning cause from a queueing bug was matching thinking token counts against completion time while confirming, from the raw trace logs, that all twelve sections still ran in parallel. Sections that blew past five minutes averaged about 27,000 thinking tokens against about 2,100 for sections that finished on time, and the serialization theory was ruled out entirely by the trace.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
