How do you build an early warning system for silent quality degradation?

A silent quality problem never trips a switch by itself. You have to build the switch, and if you set it too close to normal noise, the person watching it turns it off before it ever catches anything real.

How to answer this out loud

Seven moves. This is an estimation question wearing a monitoring costume, so most of the work is showing your arithmetic, not telling a story.

Let's learn

FirstCall is a chat tool inside Ashfield Mutual's claims app. After a crash, a customer opens it and types out what happened, and FirstCall pulls out the pieces a claims file needs: when it happened, where, who was involved, whether anyone was hurt, the policy number.

Before FirstCall, an intake specialist rang the customer and typed the same fields into a form by hand, about 14 minutes a call. FirstCall dropped that to under 3 minutes of specialist time per claim, mostly spent glancing over what the chat already caught.

Every week, Ashfield grades a random sample of 200 real FirstCall conversations against a fixed rubric, did it correctly capture every required field, and reports a pass rate. For most of the last year that number sat around 91 percent, and it moved a little each week the way any sampled number does.

Here is the turn. The score does not crash when the model quietly gets worse. It slides, a point or two a week, and every single week it is still inside the range a person would call normal on its own. Say this plainly: a quality drop that never trips an obvious line is not a smaller problem than one that does. It is a worse one, because nobody looks until a customer or a regulator does it for you.

At its worst, this costs more than a bad week of data. Ashfield's claims examiners rely on FirstCall's captured fields to route a claim correctly, fast-track the ones with an injury flag, and set reserves. If the chat quietly stops catching injury mentions, those claims sit in the slow lane, and the customer who needed a fast response does not get one, for weeks, with nobody's dashboard saying anything is wrong.

The choice I would take back is building the very first version of this alert around a flat pass rate target, "stay above 90 percent," instead of a moving baseline. It felt simpler to explain in a meeting. It also meant the alert stayed silent through the entire 9 day slide from 91 to 84, because 84 still looked like a passing grade to anyone reading it cold.

What I would leave alone: the daily count of conversations FirstCall handles. That number swings by 30 percent depending on the day of the week and the weather, and nobody needs an alert on it, because volume moving around is not a quality signal, it is just Tuesdays being busier than Sundays.

The lesson: a metric that is allowed to drift a little every week needs a watcher that knows how much "a little" normally is. Otherwise you cannot tell a bad week from a bad model, and you end up either blind to the real thing or drowning in false alarms about the ordinary one.

Where a threshold set too tight actually breaks

The short version above gives the rule. This is the part that explains why the rule cannot be tighter than it is.

Two alerts built off the same score. One rings every ordinary week. Only one of them is worth keeping switched on.

Say Ashfield had set the cutoff at 1 point instead of 4. A quiet Thursday where the day's sample happened to draw a few more multi car pileups, which are naturally harder to summarize, would drop the score by a point and a half against the trailing average. Nothing is wrong. Nobody changed the model. The mix of that day's cases was just a little harder than usual.

That alert fires anyway. Iosefa's team checks it, finds nothing, and closes it. It fires again the next week. By the third false alarm, someone mutes the channel "until we can tune it properly." The tuning never happens, because there is always something more urgent, and the alert stays muted through the week it actually needed to ring.

This is the trap worth naming directly, since it is the whole reason the threshold cannot just be "as sensitive as possible." An alert wired to fire on the model's ordinary day to day wobble teaches the exact people who need to trust it that it cries wolf. The guardrail is not a smarter alert, it is a threshold sized against a measured noise band instead of a guess, backed by a human actually reading the flagged transcripts whenever it does fire, so a real hit gets confirmed and a false one gets logged as evidence the threshold needs revisiting, not muted in silence.

4 points is the working default. The real incident that mattered dropped the score by 7, well past the line either way, which is part of what makes the backtest reassuring.

The five letters, mapped onto this alert

This is an estimation and architecture question, so the framework is BOUND, not FLIPS. There is no habit to trace here, just an equation that has to earn its numbers.

B, break it down. The alert fires when today's score falls more than a set gap below its own trailing 4 week average, and that gap has to be bigger than the score's normal week to week wobble.
O, own numbers. Trailing average 91 percent. Normal wobble measured at 2 points. Working cutoff, 4 points below average, or 87 percent.
U, use a range. 3 to 5 points below the trailing average, depending on how expensive a false alarm is versus how expensive a missed week is.
N, nail the sanity check. Backtested against a real 9 day incident, it fires on day 4, five days before a person noticed by hand, and it stays silent through a calm month where the worst real swing was 2.3 points.
D, direction. The noise band itself is the assumption that moves this the most. Underestimate it and the cutoff is too tight, and a too-tight cutoff is what causes alert fatigue in the first place.

And if you want to be sure it really works, try it somewhere else

Same five letters, a completely different industry, and this time the number the whole thing hinges on is a plant disease call, not a claim field.

Loamwell AgTech runs CropScout, a chat tool a farmer texts a photo and a few lines to, and it comes back with a likely diagnosis and a next step. Mirembe Quennel runs product for it.

B, break it down. Fire an alert when the week's diagnosis accuracy against a graded set of known cases falls more than a set gap below its trailing average.
O, own numbers. Trailing 6 week average accuracy, 88 percent, graded against 120 known cases a week. Normal week to week wobble, measured at 3 points, wider than Ashfield's because plant symptoms photograph less consistently than a written claim.
U, use a range. 5 to 7 points below the trailing average, since the wider natural wobble here means a tight 4 point rule borrowed straight from FirstCall would fire almost every other week.
N, nail the sanity check. Backtest it against the week CropScout's photo model got swapped for a faster, smaller version without regrading: accuracy slid from 88 to 79 over 11 days, and a 6 point rule fires on day 5, four days before a county extension agent flagged a run of bad blight calls.
D, direction. Here it is the mix of crops in each week's photo sample. A week heavy on a crop the model handles worse naturally drags the score down with no real degradation behind it, so the sample has to be balanced by crop type before the alert can trust its own number.

Swap the trigger and it still runs.
Speed: an interviewer caps this at ninety seconds. Skip straight to the rule: alert on a drop against the trailing average, sized bigger than the normal wobble, never a fixed floor.
Cost: engineering says a full daily golden set is too expensive to run. Fall back to a smaller weekly sample and widen the noise band and the cutoff to match, do not shrink the sample and keep the old tight cutoff.
The model got better, for real: accuracy climbs from 88 to 93 over two months. That is not a reason to relax the alert, it is a reason to update the trailing average it compares against, or a real improvement starts looking like a permanent alarm.

Where people run it wrong.
They pick a round number for the cutoff because it sounds reasonable, without ever measuring what a calm week actually looks like.
They set the alert and never revisit it, so it stays tuned to the noise band from the week it launched, not the noise band today.
They let the alert fire straight to a dashboard with no person reading the actual flagged cases, so a real hit and a false one look identical from the outside.

How to use it live. Say the reframe before naming a single number: "The hard part of this isn't picking a threshold, it's measuring how much this number naturally moves on a normal week before I pick one." That buys you room to walk the arithmetic instead of guessing a percentage out loud.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0