What is time-to-first-token and why is it the metric users actually feel?

Pathwright kept finishing its answers just as fast as it always had. The wait before it said anything at all had crept past three seconds, and the one speed number Varrow Systems watched had never been built to catch that.

How to answer this, stage by stage

Nobody is grading whether you can define the term correctly. They are grading whether you know which half of the wait a person actually sits and feels.

Let's learn

What does "it's working" actually mean to someone staring at a blank chat window?

Pathwright is the tool Varrow Systems built so a developer can ask a coding question without leaving the editor: type it into a chat panel next to the code, and Pathwright answers using the files actually open and the repo around them.

Before something like Pathwright, a developer stuck on a bug used to alt-tab out to a browser, search around, maybe paste the error into a separate chat window, then come back. That round trip cost about seven minutes on average, and a busy developer made it something like ten times a day.

With Pathwright, the same question gets asked and answered without leaving the editor. On a small, typical repo, the first word shows up in well under a second, and the whole round trip shrinks to under a minute.

The turn here isn't that Pathwright's answers got worse, or slower to finish. Total completion time, the whole answer start to end, barely moved. What changed was the part nobody watched on its own: how long a developer sat looking at nothing before the first word showed up.

Here's what was actually driving that line. Pathwright had shipped a "deep repo search" step: before the model writes a single word, Pathwright searches the whole codebase for the files that matter, then hands that to the model to work with. Search time scales with how big a repo is and how much history it carries, and it happens before generation starts, not during it. On a small repo, the search finishes in well under a hundred milliseconds. On a large one, it can run for seconds, all before the first word appears.

At its worst, a wait nobody measures on its own is worse than no metric at all, because the completion-time number everyone watches keeps saying the product is fine. Once Grendon's renewal risk score flipped to red in that same quarterly review, Varrow was looking at a $410,000-a-year contract nobody had a warning about, weeks after the number that would have warned them had already crossed its own line.

What I'd leave alone: developers working in small repos, a few hundred files or fewer, barely enter this conversation. Their searches already finish in well under two hundred milliseconds, so p95 wait-before-first-word for that group never gets close to either threshold. Building the fix around them would spend engineering time on a group that was never the problem.

The lesson: people don't feel how long an answer takes to finish. They feel how long they sit waiting for it to start. A calm completion-time average isn't proof a product feels fast. It's proof you're measuring the half of the wait a person never actually sits through.

Now here is the same thing as a story

Read the long version below when you want to feel why a calm completion-time number hid so much, not just be told that it did.

The speed dashboard lived on Torin Osterberg's second monitor, propped at a slight angle so he could glance at it without turning his head. He'd built it himself, back when Pathwright answered from a single small index and every pilot repo fit on one screen.

Pathwright had been live with paying customers for a little over a year. In the early months, Torin did more than glance at the dashboard. Every week or two, he'd sit and watch a handful of real request logs frame by frame, checking exactly where the milliseconds were going before the first word appeared.

For a long stretch, that checking confirmed the same thing: search was fast, generation was fast, and the wait before the first word barely existed as its own number. Somewhere around month six, the frame-by-frame checks thinned to once a month. By month ten, Torin was reading one line off the dashboard, average response time, and moving on if it looked like it always did. It always looked like it always did.

It came back during a user research review, a recording of a Grendon Robotics developer working through a hard bug with Pathwright open. A colleague running the session paused the video and said, "Watch this again. She's retyping the same question. She thinks it froze."

Torin didn't have an answer for why. He'd never actually pulled p95 wait-before-first-word on its own, only the blended average that folded it in with everything else.

He pulled ten weeks of logs. Platform-wide p95 wait-before-first-word had climbed from 780 milliseconds to 3.6 seconds. Median total completion time hadn't moved, still sitting near 5.1 seconds, the whole ten weeks, because the model's own token-by-token speed hadn't changed at all. The wait had all piled up in the part before the model said a word.

Grendon Robotics had the worst of it: a forty-thousand-file monorepo, the biggest Pathwright served, which meant the deep repo search took the longest there of any account. Its own p95 wait-before-first-word had gone from about 900 milliseconds to 4.4 seconds in the same ten weeks. The share of Grendon's 280 developers opening the chat panel three or more times a week had fallen from 68 to 41 percent, and it surfaced not in Torin's numbers but in Varrow's quarterly account review, where Grendon's renewal risk score had flipped to red.

It was never really about whether 5.1 seconds was a healthy completion time. There was no single response-time number that could describe a wait split into two very different halves, one a person watches and one they don't.

The decision that opened the door went back to Pathwright's very first architecture review, more than a year earlier. The team decided search and generation would run as one strict sequence, search first, wait for it, then generate, because at the time every pilot repo was small enough that search finished in under a hundred milliseconds, invisible next to the model's own startup. Nobody chose carelessly. It was the right order for the repos that existed that week.

Run those ten weeks again with one change: p95 wait-before-first-word tracked weekly from day one, next to completion time, not folded into it. By week three, the number crosses 1.2 seconds, and Torin's team goes looking, the way they eventually did, four weeks sooner. The fix ships before Grendon's engagement ever drops: the model starts talking from the current file and the last few turns of chat while the deep repo search keeps running in the background, then weaves in the fuller answer as the search results land. Grendon's panel usage never falls below 60 percent, and the account never turns red.

One design let a calm completion-time number speak for a wait it was never built to describe on its own. The other watches the part a person actually sits through, and it would have rung four weeks sooner, before a single account ever had a reason to stop trusting the panel.

What I'd tell myself, back at that first architecture review: the sequence was never wrong, exactly. It was right for the repos that existed then, and nobody gave the day it stopped being right a name.

LEAD, the four letters behind the 3.6 seconds

This isn't a story wearing a metric's clothes. It's a metric question, and LEAD is what separates the half of the wait a person feels from the half a dashboard was already measuring.

Three things worth stating directly, since this is where the real judgment sits. The alternative Torin's team considered first, and dropped, was a smaller default search radius for every account, searching fewer files before answering, to cut the wait everywhere at once. It lost because it cuts grounding on the hardest, most valuable cross-file questions, right where a developer most needs the answer to be accurate, and it doesn't fix why the wait exists, it just hides less of the repo. The AI-specific failure worth naming by name is silent context truncation: shrinking how much of the codebase gets searched to hit a speed target raises the odds the model guesses at a file or function it never actually saw, instead of citing real code. The guardrail is a fixed codebase question-and-answer eval set that any change to retrieval scope has to clear before it ships, so grounding accuracy has a real number attached to it, not a hunch. That guardrail isn't free: the pipelined fix, letting the model start talking while search keeps running, needs a small model kept warm to start the response early, and that raises average inference cost per question from about $0.006 to about $0.009, a real cost accepted for the latency win, not wished away. And the bar Pathwright holds itself to was never zero wait across six thousand seats on repos of every size; no product serving repos from a hundred files to forty thousand can promise that. It's a threshold-specific bar, p95 wait-before-first-word held under 1.2 seconds for the typical week, checked every week against the real number, not one blended completion-time average standing in for a wait it was never built to describe.

And if you want to be sure it really works, try it somewhere else

Same four letters, an AI phone dispatcher instead of an in-IDE coding assistant, and this time it's a growing parts catalog driving the wait, not a growing codebase.

Ringwell is the AI phone agent Aldercrest Home Services uses to answer inbound HVAC calls, book appointments, and route emergencies to a live dispatcher. Chiara Vosburgh runs product on it.

The build-up: for voice, the wait a caller actually feels is time-to-first-sound, how long after they stop talking before they hear anything back. Average full call length barely moved for months, sitting near 132 seconds. Then Aldercrest rolled out a full-catalog lookup, so Ringwell could check real parts, warranties, and technician schedules before answering, the same kind of search-before-answer step Pathwright had shipped.

p95 time-to-first-sound climbed from 420 milliseconds to 2.9 seconds over eight weeks after the full-catalog lookup shipped, while average call length barely moved, 132 seconds to 138. Hang-up-before-response, callers who ended the call before hearing anything real, climbed from 4 to 19 percent over that same stretch, caught in a monthly call-quality audit, not in the call-length number anyone was already watching.

The dashboard's first-sound number can look perfectly healthy the moment a fast filler line plays. The caller is still sitting through real, useless silence right after it.

Same rank as before: track the wait a person actually sits through, on its own, and act before the number everyone already watches is forced to move. The fix is the same shape too: let Ringwell confirm what it heard, or answer the easy part, while the full catalog lookup keeps running, instead of making every caller wait for the whole lookup before a single word comes back.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: the wait before it starts and the time it takes to finish are two different numbers, track the one a person actually sits through, set a real threshold on it, and fix the block, not the metric.
Cost: there's no budget this quarter for both the pipelining rewrite and a dashboard overhaul. The pipelining rewrite wins, since it fixes the real wait instead of just changing how the team watches it.
The model got better, for real: say Pathwright's underlying model doubles its generation speed. That's not proof the wait before the first word gets better. The wait is dominated by the search step that runs before generation even starts, so a faster model does nothing for the part a developer actually feels, until the blocking step itself gets fixed.

Where people run it wrong.
They watch one blended response-time average because it's the number every dashboard already shows, and never split out the wait before the first word.
They notice the complaints and "fix" it with a filler line or a typing indicator that plays fast, instead of shortening the real wait underneath it.
They wait for a business number, renewal, engagement, to move before acting, when the wait itself would have told the same story weeks earlier.

How to use it live. Say the real distinction out loud before naming a metric: "the wait before it starts and the time it takes to finish are two different numbers, and only one of them is what a person actually sits and feels." That buys a beat to think instead of reciting whatever the dashboard already shows.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0