What is the right latency target for an AI search feature, and how did you derive it?

Quorlyn built Seekline so its 6,200 employees could ask a plain question and get a sourced answer, instead of guessing which of forty wiki tabs holds it. The launch team promised the answer would land in under a second, the same promise the old keyword search bar had always kept. An AI answer has to search, check, and write, not just look something up, and chasing that borrowed number quietly cost Seekline the one thing an internal search tool cannot survive losing: whether the source it points to is actually the right one.

How to answer this, stage by stage

Nobody is grading whether you know a number for Seekline. They're grading whether you know an AI answer has two different clocks hiding inside one word, "latency," and that mixing them up is how a fast tool ends up trusted by nobody.

Let's learn

Seekline is a single search box sitting at the top of Quorlyn's own intranet, in the exact spot the old keyword search bar used to sit. An employee types a real question, not a set of keywords, and Seekline answers in a sentence or two, with a link to the actual policy, runbook, or wiki page it pulled the answer from.

Five steps. The fourth one, the first word landing, is the one the whole target actually gets built around.

Before Seekline, the same question meant opening the wiki's home page, guessing which of forty tabs might hold the answer, and searching a long document by hand. Employees at Quorlyn spent close to five minutes on an average question this way, and a fair share gave up and asked a teammate instead, which cost that teammate the same five minutes all over again.

About five minutes, for one question, and a decent share of the time the teammate got asked instead, paying the same five minutes twice.

Seekline answers most questions in a few seconds now, sourced and linked, because it searches, checks, and writes instead of routing the question to a person with forty tabs open.

The turn: the wrong citations were never really the problem on their own. Seekline was almost always fast, and it looked almost always right. What broke it was chasing a speed number borrowed from a tool that never had to write anything, and quietly paying for that speed with the one thing an internal search tool cannot lose: whether the source it points to is actually the right one.

At its worst, the launch design's citation problem turned into real mistakes, not just annoyance. Twice in one month, someone followed Seekline's answer to a stale policy and acted on it before anyone caught it, once on an expense rule, once on a leave rule. The bigger cost was quieter: the share of sessions where someone gave up on Seekline and searched the old way climbed from about 9 percent to about 33 percent within a couple of months, which meant close to 9,600 extra manual searches a week, at about five minutes each, close to 800 hours a week nobody was supposed to spend searching by hand anymore. Once you counted the time spent distrusting it, Seekline had quietly become slower than never building it at all.

What I'd leave alone: the nightly job that keeps Seekline's index in sync with the wikis. It ran on a fixed schedule before any of this and still does. The problem was never how fresh the index was, so touching that would have meant fixing a knob that was never actually broken.

The lesson: you cannot hit a speed target by copying it from a different kind of tool and hoping the new one bends to fit. You derive it from what the person in front of it actually needs to feel: a first word fast enough to hold their attention, and a source they don't have to double check. The old search bar's number was never Quorlyn's to inherit. It belonged to a tool that never had to write anything.

Now here is the same thing as a story

Read the long version below when you want to feel why a tool that tested perfectly could still lose a company's trust one honest question at a time.

Yannis Kirov had spent his first year at Quorlyn trimming the new-hire onboarding checklist from eleven steps down to five, entirely by cutting the ones nobody actually needed. When Seekline needed a launch number to put in the announcement email, Yannis did what he always did: tested the design against the questions people actually asked, and trusted what he saw.

What he tested against was a demo set of the fifty questions the IT helpdesk fielded most: the wifi password, the expense report deadline, how to book a conference room. Against those fifty, a lean design, retrieval trimmed to the top three matches, no separate rerank pass, a smaller, faster model, answered correctly and landed the whole thing in under a second, every time. Yannis stood in front of leadership and said "under a second, easily." He was right. For that fifty.

The first six weeks after launch were genuinely good. Early adopters clustered around the same kind of question the demo set had covered, the obvious, well-worn ones, so the real traffic looked almost exactly like the traffic Seekline had been tested against. By the third week, the #it-help Slack channel had gone quiet most mornings, the easy questions just weren't showing up there anymore, answered a full minute before a person would have typed a reply. Yannis watched the accuracy dashboard sit near 95 percent and moved on to other work.

No single bad answer. A trickle of one-off complaints, easy to wave off one at a time, until a plain question from someone new made Yannis actually look.

Then word spread past the IT-helpdesk crowd. Finance started asking about specific expense-approval edge cases. HR-adjacent questions got more particular, a leave rule for a specific employment type, a policy version that had quietly changed in March. None of that looked like the fifty-question demo set anymore, and the lean design, built to be fast on three retrieved documents with no second check, increasingly grabbed the wrong one, or a stale one, and wrote a confident, fluent answer over it anyway. A department lead spot-checked ten answers for her team and found two wrong. She flagged it in the tool-feedback channel and got a "we'll look into it," easy to file as one more one-off, because the dashboard's overall accuracy number, still weighted by all the easy questions still flowing through, barely moved. The "is this even right?" pings kept trickling in, one or two a day, each one small enough to dismiss on its own.

What actually changed Yannis's mind wasn't one of those pings. It was a new engineer, two weeks into the team, watching a demo where an answer took about four seconds to fully appear. He asked why the team was still fighting so hard to shave the whole thing under a second when the answer he'd just read had taken him about five seconds to actually read. Nobody in the room had a good answer. That was the whole trigger. Yannis went home that night and, for the first time, pulled the real production logs instead of trusting the dashboard's average.

The old design wasn't careless. It was tested honestly, against a set of questions that simply stopped being what people actually asked.

What he found in the logs: across real traffic, not the demo fifty, citation accuracy sat near 74 percent, not 95. His first instinct was to ask for a bigger inference budget, buy enough GPU capacity to make the small model generate faster and afford a bit more context. A colleague talked him out of it inside a day, doing the same reading-speed math the new hire's question had triggered: the model was already streaming words about seven times faster than anyone could read them, so throwing money at generation speed would fix a number nobody was waiting on. What actually needed the budget was retrieval and reranking, run properly, every time, even if the finished answer took longer to fully appear.

The decision that had opened the door went back to that very first demo, three weeks before launch. Proving the design against the fifty most common questions was the obviously correct way to test it, and it would have stayed correct if usage had stayed that shape. Nobody in that meeting decided Seekline should keep quoting a number tuned for fifty easy questions once the other 6,150 employees started asking the other 1.8 million pages' worth of things. It just kept working, right up until it was the harder questions doing most of the asking.

Run that same week again, with the two-clock target in place from day one: first word under about 1.2 seconds, full retrieval and reranking kept on every query no matter what. The department lead's spot check still finds the occasional miss, nothing built by a model is ever perfect, but the miss rate holds near 6 percent instead of climbing to 26. The new hire never has a reason to ask the question, because nothing on screen sits blank long enough to notice, and there's no growing trickle of "is this even right?" pings for anyone to wave off one at a time.

What Yannis would tell himself, back in that first meeting: testing against the fifty most common questions wasn't wrong. It just needed a second promise sitting next to it, a plan for the day the traffic stopped looking like the fifty, and nobody had written that plan down.

BOUND, and the two clocks it forced Quorlyn to tell apart

Not a story dressed as a framework. This is an estimation problem with a real gap hiding in it, and BOUND is what turns "make it fast" into an actual, defensible number.

Three of these terms are small and stable. One of them, the rest streaming in, is the biggest number on the page and the one that matters least to how fast this feels.

Three things worth stating directly, since this is where the real judgment sits. The AI-specific failure worth naming is distribution drift in the query mix itself: Seekline's accuracy looked fine during validation because early real traffic happened to resemble the fifty-question demo set, and it drifted only once genuinely different, harder questions became most of what people actually asked. The guardrail is a hard floor, not a tuning knob: retrieval, rerank, and the permission check always run in full, and if nothing clears a real similarity bar inside the target window, Seekline says it cannot find a confident source and links the top raw candidates instead of writing a fluent answer over a weak match. And the trade being accepted plainly: Quorlyn is paying for a slower finished answer, about 8.5 seconds at the busy hour instead of 1, and for running rerank and a larger model on every query, in exchange for a first word fast enough to hold attention and a source someone doesn't have to double check.

And if you want to be sure it really works, try it somewhere else

Same five letters worth repeating, a dispatch phone line instead of a search box, and this time the lever that moves the number isn't reading speed at all. It's whether there's a screen to read in the first place.

Bellrose is the live-call translation layer Renlow Freight's dispatch center uses when a dispatcher and a driver or warehouse worker don't share a language, about 450 calls a week. Doron Thurne, Renlow's dispatch operations lead, owns Bellrose's latency target.

A live phone call has almost no patience for that. Natural conversation runs on a gap between one person finishing and the next starting of around 200 milliseconds; past about 800 milliseconds to a second, a call starts to feel dead, not slow. Bellrose's old design took about 2.1 seconds to turn around a full sentence, so drivers and dispatchers kept talking over the translated audio, sure the line had gone quiet, in close to 4 in 10 calls. The fix wasn't a faster model. It was a smaller unit of work: chunk the translation by phrase instead of by sentence, and start playing the translated audio the moment a phrase boundary is heard, instead of waiting for the whole sentence. Typical round trip per phrase dropped to about 750 milliseconds, under the line where a live call starts to feel broken, and the share of calls with an overlap moment fell from about 40 percent to about 8.

Seekline's reader can always out-pace the model. A driver on a phone call has nothing to read ahead of, so the true round trip is the only number that matters.

Same method, different lever: for Seekline, the real bottleneck was never generation speed, a person reading text can always out-pace what's already on screen catching up. For Bellrose, there's no screen to read ahead on, a driver hearing a translated sentence can't skim it faster than it's spoken, so the target has to be the true round trip, not a first word. The lever there was the unit of work, cutting a whole sentence down to a phrase, not shrinking any one step's speed.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: name which clock the person actually feels, first content, full content, or on a call with no screen, the whole round trip, then size the target from that, not from the model's fastest possible run.
Cost: if Quorlyn's inference budget got cut and a smaller model became the only option, the first-word target wouldn't move; the corpus and rerank scope would shrink instead, protecting the two terms that decide trust over the one term reading speed already made irrelevant.
The model got better, for real: say the next model streams at 70 tokens a second instead of 38. That's still only about double human reading speed at best, so it doesn't touch the first-word target at all, and the saved time only matters if it's spent on retrieval and rerank instead.

Where people run it wrong.
They copy a different product's promise, a legacy search bar, a competitor's marketing page, instead of deriving the number from what this pipeline and this reader actually need.
They chase the model's raw generation speed because it's the easiest thing to benchmark, even after the reader can no longer keep up with it.
They shrink retrieval or reranking to hit a number, quietly trading away citation accuracy with nothing on screen to tell the reader the answer just got less trustworthy.

How to use it live. Ask the reframing question before quoting any number: what's the clock this person actually feels, the first real content, or the whole thing finished? On a screen those are two different numbers. On a phone call with no screen, they collapse into one, and that alone changes the whole target.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0