What is the relationship between latency and perceived quality?

Speed and trust move together on an easy question and pull apart on a hard one. The job is knowing which ticket you are looking at before you celebrate a latency win.

How to answer this, stage by stage

Nobody is grading whether you know a fast reply feels nice. They're grading whether you'd check two different numbers before deciding a speed win was also a quality win.

Let's learn

What happens when you make an AI feature four times faster and the trust score does not go up?

Tempo is a small tool inside Greenhale Wireless's live chat. A customer types a question, and before the agent even finishes reading it, Tempo writes a reply underneath. The agent reads it, fixes anything wrong, and hits send. Tempo never sends anything on its own.

For eight months, used as is sat at fifty seven percent, easy questions about a bill and hard ones about a cancellation lumped together in one number. Then engineering shipped a real win. A smaller, faster model cut the time a draft took to appear from six point four seconds down to one point three. Nobody expected trust to fall. If anything, the team expected it to climb, because a faster tool should feel like a better one.

Here is the turn. The overall used as is rate did not climb. It barely moved at all, fifty seven percent down to fifty six.

At its worst, that stillness was hiding two real, opposite stories. Easy tickets, order status, a password reset, a quick plan question, jumped from seventy eight to eighty four percent used as is. Hard tickets, a billing dispute, a cancellation save, a refund escalation, fell from fifty one to thirty nine. Complex tickets run about forty thousand a month at Greenhale. A twelve point drop meant close to forty eight hundred more drafts a month getting thrown out and rewritten from scratch instead of lightly edited, at about ninety extra seconds each. That comes to about a hundred and twenty extra agent hours a month, close to one extra full time agent, spent cleaning up after a change that shipped as a pure win.

What I would leave alone: the one point three second draft on easy tickets. Nothing about that segment needs a visible pause. An order status reply doesn't carry enough weight for anyone to wonder whether the tool thought hard about it, and the numbers back that up, easy tickets only got better.

The lesson: a wait can be doing quiet work nobody ever asked it to do. We built that message to fill dead time. It turned out some of that time was never dead. On the hardest tickets, it was the only visible proof that something had actually looked at the case.

Now here is the same thing as a story

Read this when you want to feel why the split mattered, not just know that it did.

Poldine Fask has run trust and quality for Tempo for three years. Ask her what shipped this week and she can list it before her coffee's done.

For most of that first year, Monday mornings were boring, and boring was the whole point. She'd open the used as is number, open the breakdown by ticket type underneath it, glance at the blind quality score, and move on. Three numbers, five minutes, nothing to report.

Tempo had shipped strong. Easy tickets, order status, a plan question, went from fully manual to a one click send almost overnight, and agents said so out loud in the team channel. Used as is climbed fast that first quarter and then settled, steady, dependable.

Somewhere around month five, Poldine stopped opening the breakdown tab. Not on purpose. The headline number had told the truth for so long that checking underneath it started to feel like checking a lock she'd already checked twice.

Then came the latency release. Engineering swapped Tempo onto a faster model, cut draft time from six point four seconds to one point three, and the whole team celebrated in the same channel that used to complain about the wait. Two weeks later, Poldine glanced at the headline number out of habit more than concern. Fifty seven percent. Then fifty six. One point down. Barely a Monday's worth of noise.

Someone else noticed it too, and the instinct was immediate: the new model must be a little dumber. Two engineers had a rollback to the old model half planned by Wednesday lunch, ready to trade the whole latency win back for a point of trust nobody had actually diagnosed yet.

Poldine asked for a day instead, to reopen the breakdown tab she'd stopped checking.

It was never really about whether the model got worse. The blind quality score, graded by reviewers who never see the trust numbers, held at ninety one to ninety, well inside the range it had sat in for a year. What moved was something that score was never built to catch: how bare an instant reply looks on a ticket where a customer is upset, and how much a visible pause had quietly been standing in for care nobody had priced.

The decision that opened the door went back to a much smaller meeting, when the wait was still six seconds and somebody needed the screen to feel less dead. Checking the account. Reviewing the plan. Drafting a reply. It was written to fill time, not to earn trust. Nobody in that room would have called it load bearing. It was load bearing anyway, on exactly the tickets where a customer needed to believe the reply had actually reckoned with their case.

Run the same Monday again with one change. A short, real staged reply, two to three seconds, tied to what Tempo is actually checking, comes back on the highest stakes tickets only. Easy tickets keep their one point three seconds, untouched. Four weeks after the change, hard ticket used as is climbs back from thirty nine to forty eight percent. Not all the way to the old fifty one, agents had learned something in those weeks too, but close, and still rising.

One design assumed a fast reply always reads as a good one. The other design knows a fast reply reads as good right up until the stakes are high enough that speed alone stops being proof of anything.

What I'd tell myself, back in that first small meeting: dead time and deliberate time look identical on a stopwatch. We only ever asked how long the wait was. We never asked what the wait was quietly doing while it lasted.

TRACE: the five checks behind that flat fifty six percent

Not a diagnosis of a bug. TRACE run on a trust number that should have moved and mostly didn't, using the blind score as the ruler instead of the mood in the room.

The three named causes behind the flat fifty six percent. Two are confirmed. One, the model getting worse, is ruled out.

Three things worth stating plainly, since this is where the real judgment sits. The rejected alternative, a full rollback to the old model and the old six second wait, lost because the blind score proved the model's real behavior hadn't changed, so a rollback would have spent real engineering time undoing a proven win to fix a problem sitting in one segment. The AI specific risk worth naming by name is silent capability drift hiding behind an unrelated interface change: a model swap could quietly get worse on hard cases while a removed loading message takes all the blame, or the reverse, a real regression could hide behind a story about vibes. The guardrail is keeping the blind, rubric graded quality score as its own required check on every release that touches the model or its serving path, checked apart from the trust numbers, never folded into one score. That guardrail isn't free. Reintroducing a real, two to three second staged reply only on high stakes tickets gives back part of the latency win on exactly the segment where a wrong feeling reply is most expensive, while leaving the one point three second draft alone everywhere else. And the bar that decides whether a release stays live isn't a demand for zero movement, a system built on a model can't promise that. It's a quality score that holds within two points of baseline on a rolling weekly sample of at least three hundred and fifty drafts, graded blind against the same rubric, checked apart from how anyone feels about the wait.

And if you want to be sure it really works, try it somewhere else

Same five letters, an insurance claims desk instead of a phone carrier, with no chat window anywhere in sight.

Journeyman Claims runs an AI tool called Steadynote inside its adjuster desktop. It drafts the plain language note explaining why a settlement offer landed where it did. The adjuster reviews it, edits what's off, and sends it to the claimant. Halston Solvang runs trust for that tool.

T, timeline. A monthly file audit compared Steadynote's drafted rationale against what an experienced adjuster would have written, on a held out sample. Agreement sat near eighty nine percent for nine straight months. Nothing about that number moved when Steadynote's own draft time dropped from nine seconds to two.
R, recut. Straightforward claims, one line item, no dispute, accepted without edits jumped from sixty two to seventy five percent. Claims with a disputed liability or more than one injury fell from forty four to thirty three.
A, assume nothing. The audit's grading hadn't changed. The harder check, the blind rationale score on disputed claims specifically, held at eighty nine to eighty eight, inside the normal band.
C, candidates. The same three suspects leaned differently here. The dominant one wasn't a missing pacing message, Steadynote never had one. It was the second cause: a rationale that used to take nine seconds now lands as one dense paragraph an adjuster can't visually track against a multi page medical file, so it reads as boilerplate on the one kind of claim where boilerplate is a real liability risk.
E, evidence test. Same move as Tempo: pull the blind score before and after, not the number of notes an adjuster sent back for a rewrite. It hadn't moved. One real perception gap, not a quiet quality drop.

Two different desks, two different files, the same shape underneath: a reply that got faster stopped showing the work it was still doing.

Swap the trigger and it still runs.
Speed: an interviewer gives you ninety seconds. Skip straight to the direct answer and the one evidence check, don't react to a flat number before you've split it.
Cost: there's no blind grading team funded yet. Hand grade a smaller, stratified sample yourself, weighted toward the risky segment, until one exists.
The model got better, for real: say overall accuracy climbed that quarter. That's not proof the highest stakes segment moved with it. A model that improves on average can still leave one exposed slice sitting exactly where it was.

Where people run it wrong.
They read a flat topline number as proof nothing changed, and never open the segment underneath it.
They see a trust dip after a speed win and reach for a rollback before checking the one number that actually tells them if the model changed.
They fix it by slowing the whole product back down, instead of only the slice that needed the pause.

How to use it live. Say the real question out loud before answering it: did the model get worse, or did the wait stop doing quiet work nobody ever asked it to do. That line buys you a breath instead of a guess.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0