What does an AI product's error handling reveal about its maturity?

Duskvane reads a unit's sensor data plus a technician's spoken description and suggests the likely fault and fix. Perpetua Lindqvist dispatches technicians for a regional HVAC service company and reviews every call Duskvane got wrong.

How to answer this, stage by stage

Seven moves for a diagnosis question. The fifth is the strongest move in the whole framework.

Let's learn

Duskvane listens to a technician describe a unit's symptoms, reads its sensor data, and suggests the fault it thinks is most likely, along with a fix.

Before Duskvane, a technician worked through a printed troubleshooting checklist by hand, roughly 45 minutes per call, checking each likely cause in order.

With Duskvane, most calls drop to about 20 minutes, since the checklist starts at whatever fault Duskvane suggests instead of at the top of the list every time.

The turn: the extra wrong diagnoses aren't really the maturity problem. The maturity problem is that Duskvane never once flags a case as one it's unsure about, right or wrong.

At its worst: a technician trusts a confidently-stated wrong diagnosis on a rare compressor fault, replaces the wrong part, and the customer's unit fails again two days later, on a call that should have been escalated to a senior tech from the start.

What I would leave alone: Duskvane's suggestions on dirty filters and thermostat faults, the vast majority of real calls, stay accurate and don't need an uncertainty flag they'll almost never use.

The lesson: a product's error handling is the actual maturity test, not its average accuracy. A mature product tells you when to stop trusting it. An immature one just sounds the same either way.

Now here is the same thing as a story

The short version above is what you'd say under time pressure. Read this one for how Perpetua actually traced it back.

Perpetua has dispatched HVAC technicians for eleven years and can usually tell from a two-sentence phone description whether a call needs a senior tech.

The suggestion step, third in line, is where an uncertainty signal should surface and, since the redesign, doesn't.

For months, Duskvane's suggestions looked reliable across the board. Techs stopped double-checking dirty-filter and thermostat calls entirely, since Duskvane was right on those almost every time.

The redesign, not the launch, is the actual moment the maturity problem started, weeks before anyone noticed a pattern.

The trigger was one rare fault: a compressor short-cycling in a pattern combined with a slow refrigerant leak, a combination Duskvane had seen fewer than five times in training. It suggested "replace the start capacitor" with the exact same confident tone it used on a routine dirty filter.

Duskvane's unfamiliar-case branch runs straight into the "guesses anyway" box, the same one a genuinely immature product runs into.

The technician replaced the capacitor. The unit failed again two days later. Perpetua pulled the call log and found nothing in Duskvane's own output distinguishing that call from a routine, high-confidence one.

The rare short-cycle fault sits in the exact corner where escalation should happen most and happens least.

She ran the evidence test herself: twenty deliberately novel cases, faults Duskvane had seen only once or twice in training. It escalated one of them. Nineteen got a confident, specific-sounding suggestion, several of them wrong.

All four of these existed in Duskvane's first release. Only the logged record survived the redesign.

The old interface asked technicians to read a small badge that rarely appeared and decide for themselves whether to trust it. The new one, once fixed, brings that badge back but only on cases below a set confidence line, so it stays out of the way on the calls where it isn't needed.

The redesign team simplified the screen because the badge cluttered a mostly-empty space and rarely fired. It took a technician replacing the wrong part twice on the same unit to see that "rarely fires" and "safe to remove" were never the same thing.

TRACE, five checks for what error handling actually revealsNot a vibe about polish. TRACE is what finds the exact behavior a maturity claim rests on.

Five letters, and the evidence test is the one that actually proves which cause is real.

The recap, one line per letter: timeline is the redesign months before anyone noticed, recut is escalation collapsing on rare faults specifically, assume nothing is confirming the confidence score still existed internally, cause candidates is three real hypotheses, and evidence test is twenty manual cases proving it's a removed display, not a broken model.

And if you want to be sure it really works, try it somewhere elseSame five letters, a pharmacy instead of an HVAC dispatch office.

Mornbright flags potential drug interactions for pharmacists filling prescriptions. Corvin Adeyanju leads the pharmacy tech team at a regional chain, reviewing flagged interactions before dispensing.

Mapped onto TRACE: timeline traces a drop in "consult pharmacist" flags to a software update that merged two separate alert types into one generic banner. Recut shows the drop concentrated entirely in rare, multi-drug interactions, not common two-drug ones, which stayed accurate. Assume nothing rules out a data feed issue first, confirming the underlying interaction database itself was never touched. Cause candidates: an intentional simplification to reduce alert fatigue, a genuine gap in rare-interaction training data, or a broken confidence threshold. Evidence test: manually checking twenty known rare interactions shows the system still flags them internally, in a log nobody reads, but no longer displays a distinct "high-stakes, consult required" banner, the same removed-signal pattern as Duskvane.

A different product, a different fault type, and the same missing piece: a signal the system still computes but no longer shows.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "it never says it's unsure on rare cases, and the evidence shows the confidence score still exists internally, just not on screen," and stop.
Cost: there's no engineering time this quarter to rebuild the full confidence UI. Say so honestly, and start with a simple text line, "uncommon case, verify manually," triggered whenever the hidden score falls below a threshold.
The model gets better, for real: if Duskvane's overall accuracy genuinely improves next quarter, that's still not proof of maturity, a more accurate model that still never says "unsure" is just a quieter version of the same gap.

Where people run it wrong.
They judge maturity by overall accuracy alone, missing that the real signal lives in the rare, sliced-out segment.
They assume a missing uncertainty signal means the model can't calibrate, without checking whether it's just not being displayed.
They fix the model's accuracy first, when the real gap is a design decision that removed a signal that already existed.

How to use it live. When someone asks what error handling reveals about maturity, ask yourself: does this product ever say "I don't know," and if not, is that because it truly can't, or because someone decided not to show it. Those are two very different problems with two very different fixes.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0