How do you avoid alert fatigue with probabilistic metrics?

How do you avoid alert fatigue with probabilistic metrics? Commit to a side, then say who pays for each kind of wrong answer.

How to answer this, stage by stage

Nobody is grading whether you know the word "threshold." They are grading whether you can name who eats the cost on each side and commit to one before you're asked to. Six moves get you there.

Let's learn

What happens when the thing meant to warn you gets so loud that everyone stops hearing it?

Fenrose Capital is an investing app that manages your money for you. Its engine, called Ballast, checks every client's portfolio against a target mix, say sixty percent stocks and forty percent bonds, and rebalances it when the real mix drifts too far away. Ballast doesn't just rebalance quietly. It also raises an internal alert any time a portfolio's drift crosses a line, so a human can glance at anything unusual before money actually moves.

The engine used to alert on a simple rule: if any account drifted more than two percentage points from its target mix, page the on-call risk desk. That rule made sense on a quiet day. Markets aren't quiet every day. On a day with a Fed announcement or a sharp rate move, ordinary portfolios drift two or three points just from normal price swings, nothing wrong with any of them.

Here is the turn. Say it plainly: the extra pages were never the real problem. The real problem is what Camren Sundvik, who runs the on-call risk desk, started doing about them. For the first few weeks she opened every alert and checked it. Within two months she was opening maybe one in twenty, and acknowledging the rest in a batch without reading them, because on most days, ninety-seven of every hundred pages were normal noise.

At its worst, this looks like nothing at all, right up until the day it doesn't. Nobody at Fenrose saw a crisis building. They saw a risk desk staying on top of its queue, alerts cleared fast, no backlog. The real cost sat one layer down, in how carefully any single page was actually being read.

What I would leave alone: a portfolio drifting because the client added new money last week, or because a single stock in it had one unusually good day, doesn't need any of this. Those moves are already explained by an obvious event sitting right next to them in the log. The mistake was never having any alert. It was having exactly one alert setting for everything, whether the day was calm or the whole market was moving at once.

The lesson: a probabilistic signal needs a band that respects how much noise is normal that day, not a single line drawn once and left there. Draw the line too close to normal, and the team that reads it will teach itself, honestly and reasonably, to stop reading it.

Now here is the same thing as a story

The short version sits above. Read on for the Thursday a client's real problem sat inside 1,800 pages nobody was actually reading anymore.

Camren had run the risk desk at Fenrose for two years, and she was good at it. Give her an alert and she could tell inside thirty seconds whether it was a client who'd just added money, a stock having a wild day, or something Ballast itself had gotten wrong. She trusted her own read more than she trusted the dashboard, because the dashboard, at the start, was quiet enough to actually look at.

The first few months of Ballast's launch, the pager buzzed maybe forty times a week. Camren opened every one. Most were nothing. A few needed a quick note to a client's advisor. It felt like the system was doing its job, and so was she.

Fenrose grew fast that year. More accounts, more money moving, and markets got louder too, a string of rate decisions that had half of Wall Street's portfolios drifting on the same Tuesday. The fixed two-point rule didn't know any of that. It just kept counting drift the same way it always had, on a quiet March morning or a chaotic September one.

The habit that had kept Camren sharp thinned out in three beats. First, the page count crept from forty a week to four hundred, and she started skimming subject lines instead of opening every one. Second, she noticed that on the loudest market days, nearly everything in her queue was noise, so she started clearing those days in bulk, five or six clicks and the queue was empty. Third, a "mark all reviewed" button that engineering had added as a convenience, meant for the rare day the whole team was out sick, quietly became the thing she reached for most Fridays.

One side of this tradeoff costs a few extra hours. The other side costs the whole habit of actually reading the page.

The trigger wasn't a crash. It was a Thursday like any other. A data feed that fed Ballast a client's bond holdings broke quietly overnight, and for three days the account's real allocation drifted eleven points away from target while the feed kept reporting stale numbers underneath it. Ballast alerted on it, correctly, twice. Both alerts landed inside a batch of 1,800 that week, a week the whole market had been jumpy anyway, and both got cleared with the rest.

A client called on the fourth day, confused about a statement that didn't match what her advisor had told her. That's how it surfaced. Not a dashboard. A phone call.

Before anyone panicked, the team checked the boring explanation first: was the alert itself wrong, some bug in Ballast's drift math? It wasn't. The two pages that Thursday were accurate, clearly labeled, sitting in the log exactly where they should have been. The system had done its job. The queue around it had made doing that job cost too much attention to actually pay.

Fenrose's engineering lead pushed for the obvious fix first: add a second, stricter alert layer on top, so anything Ballast was really sure about got flagged twice. Camren pushed back. A second layer of alerts, on the same noisy signal, just means twice as many pages to learn to ignore. That idea got named and set aside, on purpose, before the real fix.

The real fix widened the band instead. Ballast's threshold stopped being a flat two points and became a number that moved with how much the broad market had swung that day, calculated off a rolling twenty-day volatility window. On calm weeks it tightened back down near two points. On loud weeks, when a two-point drift meant nothing, it opened up to four or five.

The old decision that set the two-point rule in place went back to Ballast's earliest days, when Fenrose had a few hundred accounts and markets moving in unison across the whole book was rare enough to ignore. Nobody wrote that rule down as temporary. Nobody revisited it once the account count, and the market's habit of moving as one, both grew past what a flat number could sensibly judge.

Run the same three days through the new band. On day one, the account's drift crosses the volatility-adjusted line the same as before, real and accurate. But that week the queue holds ninety alerts instead of eighteen hundred, and Camren is back to opening nearly all of them. She catches it same-day instead of on a phone call four days later. The data feed bug gets fixed by that afternoon. Three days of drift become a few hours.

One design asked the desk to read everything and made that impossible. The other asked it to read less, and made that actually work.

What I would tell myself, back when that two-point rule first shipped: a threshold that assumes every day is quiet will, on the first loud day, teach the person reading it that the alert means nothing. By the time you notice, the lesson has already been learned.

PICK, in one screen

This is a tradeoff wearing an operations question's clothes. Both a tighter band and a wider one produce a wrong answer sometimes, so PICK forces a side before it asks why.

Two things worth naming directly, since this is where the real judgment sits. The rejected alternative was stacking a second, stricter alert layer on top of the existing one, so the surest cases got flagged twice. That got set aside on purpose: a second layer on the same noisy signal doubles the pages without fixing the reason Camren stopped reading them, it treats the symptom and leaves the actual cause, a threshold that doesn't know a loud day from a quiet one, untouched. The AI-specific failure worth naming by name is exactly the one this story lived through: a probabilistic signal treated like a deterministic one. Ballast's drift score is a confidence-weighted read on noisy market data, not a fact, and pinning a single hard cut-off to a number that moves for reasons having nothing to do with the client's account is what manufactured the false alarms in the first place. The guardrail is a rolling volatility-adjusted band, checked against a golden set of confirmed past incidents on a schedule, not a threshold set once and trusted forever. There's a real trade-off underneath the fix too, not a free lunch: a wider band means some real, smaller problems sit uncaught for a few extra hours before the next check catches them. Fenrose accepted that cost, on purpose, in exchange for a desk that still actually reads its queue.

And if you want to be sure it really works, try it somewhere else

Same four letters, a wind farm instead of a trading desk, and the same shape shows up with turbines instead of portfolios.

Cairnbrook Wind runs an AI system that watches vibration sensors on two hundred turbine gearboxes and pages a maintenance tech when a bearing's vibration signature drifts from its own healthy baseline. Neve Marlow leads the predictive-maintenance team.

P, position. Widen the vibration-alert band to account for normal wind-gust noise instead of adding a second sensor layer on every gearbox.
I, impact. Too tight, and techs get paged on every gusty afternoon, ordinary wind noise shaking readings past a flat cut-off, and learn to close tickets without a site visit. Too wide, and a bearing that's genuinely starting to fail gets caught a few days later than it could have been, on the next scheduled inspection instead of same-week.
C, cost asymmetry. A few extra days before a scheduled visit catches a wearing bearing is a bounded cost, the part isn't going to fail in a week. A crew that's learned to close every vibration ticket without checking is the same crew that closes the one ticket that was a real bearing about to seize, and that failure takes the turbine offline for a month instead of an afternoon.
K, kill criteria. Tighten the band back down the moment a back-test against confirmed past failures shows it would have missed more than one in twenty-five of them, or immediately, with no debate, for any offshore turbine where a failure means a boat crew and a multi-week repair window instead of a same-day site visit.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to the pick, state the position, name the asymmetry in one line, give the kill criteria, done.
Cost: engineering says a real volatility-adjusted pipeline is eight weeks out. Ship a coarse three-tier band, calm, normal, loud, off the current day's market data by hand, as the interim fix, not the tight fixed rule as a stopgap.
The model got better, for real: say Ballast's drift-scoring model itself gets more accurate this quarter. That's still not the same claim as "the band is calibrated." A better model can still sit behind a band tuned for the old model's noise level, and it needs its own recalibration pass, not a free pass because the underlying score improved.

Where people run it wrong.
They widen the band once and never back-test it against real incidents again, so a genuinely dangerous drift starts sliding through unnoticed.
They treat "fewer alerts" as the goal instead of "the real ones get read," and quietly trade away detection they never meant to trade.
They add a second review step instead of fixing the band, which just moves the fatigue to a different person on a different screen.

How to use it live. Open with the position, not the reasoning: "widen the band, don't add more alerts." That buys you the room to walk through the asymmetry calmly instead of talking yourself into a hedge halfway through.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0