What counts as an incident for an AI feature but not for a normal one?

How to answer this, stage by stage

Nobody is grading whether you can say "monitor the model." They are grading whether you can name the specific, checkable thing that makes an AI mistake different from a normal outage. Six moves get you there.

Let's learn

Say a phone company builds a chat assistant that answers billing questions, roaming questions, and questions about bringing your number in from another carrier. Before it existed, a person on the support line pulled up your account and read the real record before saying anything final, like "yes, your port is done." That took a while: on a bad evening, the wait to reach a person ran past eleven minutes.

Then Pellworth Mobile built Switchboard. It answers in seconds instead. About twenty-two thousand chats a day go through it, checking the same account systems a person would, just faster.

Here's the part that matters. A few extra wrong answers are not the real problem. The real problem is what kind of thing gets it wrong. Getting your data balance off for a minute costs you a refresh. Telling you your number port is finished when it isn't costs you your phone service, because the sensible next move is cancelling the carrier you're leaving.

Two dials on the same product. One of them was moving the entire time. Nobody was watching it.

At its worst, Switchboard tells a run of customers their port is complete while the losing carrier's system still shows it pending. They cancel their old line, the way anyone would. For a few days, until the real port finishes, they have no working phone at all. No calls out, no calls in.

The choice I would take back. Switchboard's transcripts only ever got a human read when a customer filed a complaint about one, roughly three in every thousand chats. That made sense at launch, when Switchboard only handled billing and data balance, and a wrong answer meant a customer tapped refresh. It stopped making sense once porting and SIM swaps became categories where a wrong answer doesn't just annoy someone, it strands them.

What I would leave alone. Not every category needs this kind of watching. If Switchboard misquotes this month's data balance by a gigabyte, the customer taps refresh and it corrects off the same account system in real time. Nobody loses anything a second look doesn't fix in five seconds. That category can stay on the old occasional spot-check; there's no reason to page anyone over it.

The lesson. A number that looks healthy in total can be hiding exactly the mistake that matters, because the categories where a model is quietly wrong are rarely where most of the volume sits. If the harmful mistake is rare on purpose, tucked inside a category most people never touch, watching the average will never find it.

Now here is the same thing as a story

The short version is above. Read on if you want to feel how a number can climb for five weeks while every other light on the board stays green.

Idara Merrow has run the platform team behind Switchboard for two years, the group that owns what happens once the model gets something wrong, not just whether it answers fast enough. She wrote the current complaint-triggered review process herself, back when Switchboard only handled two things: "what's my bill" and "how much data do I have left."

Switchboard grew. Roaming status. Discount eligibility. And, eighteen months in, porting: telling a customer whether their number had finished moving over from another carrier. For most of that year the complaint queue stayed thin, three or four a week, mostly people annoyed about tone, not facts. Idara read it Monday mornings with her coffee and moved on.

When the complaint count first crept past five in a week, Idara's instinct wasn't to build something new. She proposed tightening the existing process instead: drop the count that triggered a deeper look from five down to two. She dropped that idea within a day. A queue that only fills up when someone notices they were lied to doesn't get more honest by watching it more closely, it just means reading the same few complaints about tone more often, while the one category that could actually strand someone stays exactly as invisible as before.

She used to read every complaint line by line. Then she skimmed for anything with the word "wrong" in it. Then, most weeks, she just checked the count stayed under five and closed the tab.

There wasn't a Tuesday where it happened. Nobody can point to the day it started. Sometime over about five weeks, in a stretch nobody was watching closely, Switchboard started telling a small slice of porting customers their number transfer was complete a little early, before the losing carrier's system had actually released it.

Customers did the sensible thing. Told their port was done, they cancelled the carrier they were leaving. Most real ports finished within a day of that message anyway, so most people never noticed a gap. A few didn't. For a handful of people, the real port dragged an extra two, three, four days behind what Switchboard had told them, and during that stretch they had no working line at all.

It was never really about how many chats got a wrong answer. Switchboard answered every one of its twenty-two thousand daily chats exactly the way it was built to: instantly, in full sentences, with no error. The failure wasn't in whether it answered. It was in whether the specific thing it said was true, and nothing in the pipeline was built to ask that question on its own.

The complaint-triggered review process went back to a planning meeting eighteen months earlier, when Switchboard only handled billing and data balance. Someone asked how they'd catch a bad answer. The answer in the room was, "if it's actually a problem, someone will complain, and we'll see it in the queue." Nobody was wrong to say it. At the time, a wrong answer meant a customer refreshed a number. There was nothing porting-shaped in the product yet for anyone to worry about.

Run the same five weeks through the sampled unsupported-claim rate instead. It reads zero point four percent in week one, same as always. By week three it's crossed one point five percent on the porting category, held for two days straight, and the rule fires automatically, before Idara has opened her laptop that morning. Porting answers freeze to a human handoff by lunch. Six customers got an early port confirmation before the fix shipped two days later. Not sixty-three.

What I'd tell myself, back in that first planning meeting: "someone will complain if it's a real problem" is only true for mistakes people can tell happened to them. A wrong port confirmation doesn't look like a bug from the other side of the chat window. It looks like good news, right up until your phone stops working, and by then most people are calling their old carrier, not us.

LEAD: the four things worth tracking instead of uptime

This is a metric question, what actually tells you the AI feature is failing, not a system-health question, so LEAD fits and a normal incident checklist doesn't.

L, link. The real outcome underneath a "port complete" message: whether a customer who trusts it still has a working phone line a week later, not the model's own confidence. Every category Switchboard answers ties back to some outcome like this; porting's is unusually unforgiving because the customer's next move, cancelling the old line, can't be undone.
E, early signal. The unsupported-claim rate: how often Switchboard states a specific, checkable fact, no hedge, that the real account record disagrees with, sampled continuously and checked automatically. This is the number that moved for five straight weeks while uptime sat at 99.98 percent the entire time.
A, abuse. The rate drops if the model just hedges more, "not sure, please check," on questions it used to answer confidently and correctly. That would look like an improvement on the dashboard and be a worse product underneath it.
D, decision. Below 1.5 percent, sustained: nothing changes, that's normal model noise. At 1.5 percent held for two days on a category a customer can't undo: page the on-call model owner, freeze that category to a human handoff, same severity ladder as a system outage. Below that line nobody gets woken up over a stray wrong data-balance guess; above it, someone does, on purpose.

One more thing worth saying: 1.5 percent isn't a number pulled from nowhere. It's roughly four times the rate the same check found on Switchboard's lowest-stakes category, data balance questions, where the model runs the exact same setup and trips over the exact same kind of confusing phrasing. If porting's rate matched that quiet-day baseline, nobody would call it an incident. It's sitting four times above its own noise floor that makes the bar real instead of made up.

A metric can be satisfied on paper while the thing it stands in for is still going wrong. That's why the abuse check is its own line, not an afterthought.

And if you want to be sure it really works, try it somewhere else

Ardmoor Terminal runs Dockline, a chat assistant dispatchers and truckers use to check whether a container has cleared customs and can leave the gate. Roan Selkirk runs dispatch operations there.

L, link. Whether a container that leaves the gate is actually allowed to, not whether Dockline sounds confident about it. Moving an uncleared container off the terminal is a real compliance violation, with fines attached to the trucking company and the terminal both.
E, early signal. The share of "cleared for release" answers Dockline gives that don't match the customs system's own record, sampled continuously, not caught by asking whether the chat widget loaded or answered fast.
A, abuse. During a customs-system integration change, the rate climbed from a baseline of 0.6 percent to 2.8 percent over three weeks. Nobody was gaming it on purpose; the new integration was quietly serving Dockline a cached status a few hours stale, and stale looked exactly like current on the screen.
D, decision. Under the old setup, with no separate signal for this, 11 containers left the gate before their customs release was actually confirmed, each one a real fine. Running the same 1.5-percent, two-day rule Idara wrote for Switchboard against Dockline's numbers, the bar gets crossed in the first week of the integration change; the projected count under that rule is 2 containers, not 11, because clearance answers would have frozen to a live check against the customs system itself instead of trusting a cached number.

Swap the trigger and it still runs.
Speed: an interviewer caps the answer at ninety seconds. Skip straight to the reframe: an AI incident is a sustained rise in confident, wrong, unfixable claims, not whatever uptime happens to say.
Cost: engineering can only afford to check a small slice of daily chats. Don't drop the high-stakes categories out of the sample to save budget; shrink the sample size on the low-stakes categories instead, since a wrong data-balance answer costs a refresh either way.
The model got better: a new Switchboard version turns out to be more accurate on porting than the one it replaced. That doesn't make the signal pointless, it's still the only way anyone would know that for sure instead of assuming it from a quiet complaint queue.

Where people run it wrong.
They watch uptime and error rate and call that "monitoring the AI feature," when those checks only catch the system stopping, not the system confidently lying while it keeps running.
They wait for complaints to define the problem, when most people hurt by a wrong AI answer assume the mistake was theirs, not the bot's, and never file one.
They set one blended number across every category the assistant answers, so a small, dangerous category disappears inside a big, healthy average.

How to use it live. Say the reframe before naming a single number: "the incident isn't whether it's up, it's whether it just stated something false about an action the person can't undo." That buys the room to ask what this specific product's unfixable categories actually are, instead of reciting a generic uptime checklist.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0