How do you handle latency variance rather than average latency?

Sentrywave scores every card swipe for Cascabel Bank before it clears. Its average latency never once broke budget the day a stolen card cleared fourteen gas pumps in six minutes.

How to answer this, stage by stage

Nobody is grading whether you can say the phrase "tail latency." They're grading whether you'd trust a healthy looking average, or go find the six minutes it was quietly wrong.

Let's learn

Here is what happens when a number stays true for a year, and then, for one specific six minutes, it quietly stops being the number that matters.

Sentrywave is a fraud scoring service from Coriolis Systems. Every time a Cascabel Bank customer swipes a card, Sentrywave has less than half a second to say how risky that swipe looks, before the payment network decides whether to let it through.

For the first year, that half second was plenty. Coriolis published one number on Cascabel Bank's dashboard: average scoring time, a steady two hundred eight milliseconds, comfortably under the four hundred millisecond budget. The dashboard stayed green, month after month.

Here is what the average never showed. If Sentrywave can't return a score inside that four hundred millisecond budget, the payment network doesn't wait. It approves the swipe anyway, with no score behind it at all, so a slow check never holds up somebody's legitimate coffee order. That's a sensible rule for the rest of the traffic. It just means a scoring call that runs late doesn't fail loud. It fails silent, and it looks, on Coriolis's own dashboard, exactly like a swipe that was checked and passed clean.

One stolen card cleared fourteen gas pumps across two states in six minutes, twenty five hundred ninety dollars total. Two early swipes hadn't shown a pattern yet and passed cleanly, correctly, no score needed to catch them. The other twelve ran past budget and were waved through with no check behind them at all, because the same shared record, how many times this card has been used in the last ten minutes, was being read and rewritten by swipe after swipe within seconds of each other.

At its worst, this doesn't cost one bank one ring's worth of fraud. It costs the whole reason a bank bought real time scoring in the first place: a green light nobody can actually trust.

What I would leave alone: the overwhelming majority of swipes never come anywhere near the four hundred millisecond budget. A single card used once, at one store, has nothing to contend with. Watching those any more closely spends effort defending a risk that was never there.

The lesson: an average can be completely honest and still hide the only six minutes that ever mattered. Two hundred eight milliseconds was true, all day, every day. It was never close to the whole story.

Now here is the same thing as a story

Read the short version above if you're using this to answer out loud. Read the story below for how a two year old dashboard decision nearly cost Cascabel Bank the one thing real time scoring was supposed to buy them.

The dashboard sat on the wall of Coriolis's ops room, one line, always green: average scoring time, two hundred eight milliseconds. Rasa Solaas had built almost this same dashboard once before, at a checkout risk startup where every request cost about the same to score, so watching one blended number had never once let her down. She brought that same instinct to Sentrywave without questioning it, because there had never been a reason to.

For the first year, the instinct held. Cascabel Bank signed on with about six hundred thousand cards, mostly routine grocery and gas station swipes, and Sentrywave scored almost all of them in under two hundred fifty milliseconds. The four hundred millisecond budget, and what happened if a call ever missed it, wasn't something anyone on the team had spent real time on. Nothing had ever come close.

Cascabel Bank grew past two million cards in its second year with Sentrywave, and its fraud team leaned harder on it, trusting a green dashboard to mean a green day. Nobody was watching how many scoring calls ran past four hundred milliseconds, because nobody had ever built a chart for it. Nobody was counting how many of those calls got waved through with no score at all, because that number lived nowhere anyone looked.

Then, on a routine monthly call about a batch of fraud losses, Cascabel Bank's fraud lead mentioned one ring almost as an aside: fourteen gas pumps, two states, six minutes, twenty five hundred ninety dollars, one stolen card. "The average looked fine on that one, by the way, in case that's useful." It wasn't dressed up as a problem. It was one line in a longer list.

Rasa almost let it go. Then she pulled the raw trace for those fourteen swipes herself.

Swipe one and two came in clean, genuinely, no pattern yet for the model to catch. Swipe three took six hundred ten milliseconds, over the four hundred millisecond budget, and the payment network approved it without waiting for a score, the same rule that let a coffee order clear on a slow network day. Swipe four took past a second. Swipe fourteen took three point nine seconds. Every one of the twelve late swipes cleared with nothing behind it, and every one of them looked, on Coriolis's own dashboard, exactly like a swipe that had been checked and passed.

The decision that opened the door went back two years, to a meeting nobody thought twice about at the time. Coriolis needed one number for Cascabel Bank's onboarding deck, something simple enough to put on a slide. The team picked average scoring latency, under two hundred fifty milliseconds, and built the alerting entirely around it. Percentile tracking got raised once, and got cut, because nobody could yet point to a reason a client would care.

Rasa's first instinct was the fast fix: add capacity to the feature store, the shared record every scoring call reads and rewrites, so it could handle more at once without falling behind. She modeled it, and it helped, cutting the worst case from three point nine seconds to around one point two. It didn't fix it. A ring that hit sixteen pumps instead of fourteen, a little faster, would still find the edge of whatever headroom got added. The tail wasn't a capacity problem. It was a question of what Sentrywave did the moment it couldn't answer in time.

What she asked engineering for instead: a second, much smaller model, using only the transaction amount, the merchant type, and the velocity count itself, that could always return a real verdict inside eighty milliseconds. The moment the full pipeline ran past budget, this fallback would fire instead of letting the network default to approved. Checked first against a fraud eval set, tuned to flag more often than the full model would, on purpose, because a wrongly frozen card costs someone a phone call, and an unscored fraud swipe costs the money outright. And on the dashboard, a fallback scored swipe would show its own color, not folded into the same green as everything else.

Run the same ring again with the redesign live: swipe one and two clear the same way. By swipe three, the velocity count is climbing fast enough that the full pipeline starts to lag, crosses budget, and the fallback fires in eighty milliseconds instead of nothing at all. It flags elevated risk. The alert reaches the analyst's queue while swipe three is still settling. The card gets frozen before swipe four clears. The ring stops at five hundred fifty five dollars instead of twenty five hundred ninety, and it stops in the time it takes to read one line in a queue, not five weeks later on a call.

What I'd tell myself, back at that first meeting about the onboarding slide: we picked the number that fit cleanly in one line. We never once asked what the system would do on the six minutes it stopped being true.

GUARD, the five checks an average never asked

This isn't a speed question wearing a safety word. It's a risk question, and GUARD is what stops a healthy looking average from standing in for whether anyone actually got checked.

Rasa's team held the thresholds. A cardholder mid ring held nothing, and never even knew a check was supposed to happen.

Three steps that worked, and a fourth that was never built, right where a real check should have sat.

Three things worth stating directly, since this is where the real judgment sits. The alternative Rasa modeled first and set aside was adding capacity to the shared feature store. It helped, cutting the worst case latency by more than half, but it lost because it doesn't bound the tail, a slightly bigger or faster ring can always outrun whatever headroom gets added. The AI specific failure worth naming by name is a silent fail open under a real time scoring budget, a pipeline that, when it can't finish in time, defaults to no check at all instead of a slower but real one, and it happens to correlate with exactly the traffic pattern most worth catching. The guardrail is a bounded, eval gated fallback scorer that always returns a verdict inside budget. And the trade off is real: the fallback model catches less subtle fraud than the full model does, and it flags more often than the full model would on its own. That's the cost being accepted for a guaranteed, fast, real verdict instead of a fast, silent default. The bar was never zero missed fraud. It's a recall and false alarm rate on the fallback, checked against an eval set, set so a genuinely fast decision beats no decision at all.

And if you want to be sure it really works, try it somewhere else

Same five letters, a factory floor instead of a card network, and this time the shared record isn't a stolen card's history, it's one vibration reading several machines are all trying to read at once.

StampWatch is a press line safety monitor from Foundrywell, running at Ironbrook Stampworks. It watches vibration and pressure sensors on a stamping line and warns a floor supervisor before a jam turns into someone's hand catching between dies. Reine Vautrin is the plant engineer who owns it.

The build up: Ironbrook's safety rule said StampWatch had five hundred milliseconds to raise a flag, or the line kept cycling, the same shape of default as Sentrywave's fail open rule. A single press running alone almost never came close. But during a scheduled changeover, when three or four presses on the same line share one vibration baseline reading, that shared value gets read and rewritten by every press at nearly the same instant, and the check can run well past budget, right when a jam across several stations at once is most likely.

G, groups. The floor supervisor holds the estop, and can stop the line the second an alert lands. The operator standing at the press, hands near the die, holds nothing, and the line won't pause itself just because a check is running slow.
U, unequal. The slow checks concentrate on synchronized changeover runs, exactly when several presses share one baseline, and exactly when a jam across stations is most likely to matter.
A, ability to contest. The status board shows the same all clear light whether a cycle was actually checked or just defaulted through, so nobody, not the supervisor, not the operator, can tell the difference in the moment.
R, reduce. A fast fallback check using only that one press's own last reading, skipping the shared baseline, that always returns inside budget, with its own distinct light on the status board when it fires.
D, detect. Track P99.9 alert latency and the count of defaulted through cycles as their own numbers, split by whether a synchronized changeover run was active.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: never trust an average latency number to mean everyone got checked, watch the tail, and never let a slow check default to silence.
Cost: no budget this quarter for the fallback model. Build the fail open counter first, since you can't safely fix a gap you can't yet see broken out on its own.
The model got better, for real: say Sentrywave's core model gets faster across the board. That shrinks the tail some, it doesn't remove it, because fourteen swipes on one card in six minutes still contend for the same shared record, no matter how fast any single check runs.

Where people run it wrong.
They alert on the average and call the system healthy the moment it stops moving.
They let a slow check default to whatever's cheapest to assume, approved, all clear, no alarm, instead of guaranteeing a real answer.
They fix a tail latency problem with more capacity alone, without asking whether the load causing it can simply grow to outrun it.

How to use it live. Ask the real question out loud before answering: "what does this system do the moment it can't answer in time?" That buys a beat to think, and it's almost always where the real gap is hiding.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0