What kinds of errors are unrecoverable, and how does that rule out AI?

Thrale Midstream runs six big reciprocating compressors at Station 14, pushing natural gas down the pipeline day and night. CompressorGuard reads their vibration, heat, and pressure every two seconds, and it can recommend, or in some cases decide, when one needs to slow down or stop. Reliability lead Devraj Sallust had a harder question to answer than whether the model was accurate enough: which of its mistakes were the kind nobody, model or person, ever gets to take back.

How to answer this, stage by stage

Nobody's grading whether a candidate can list scary things that could go wrong. They're grading whether the rule they name would have actually stopped a compressor from acting alone the one night it mattered.

Let's learn

Say we build a system that watches a big gas compressor's shaking, heat, and pressure, and tells someone when to slow it down or shut it off.

Before it existed, someone walked the floor once an hour and read six gauges by eye on every machine. That catches a problem that builds for half an hour. It does not catch one that builds in three seconds. In one year, the station had four shutdowns nobody saw coming, and each one meant close to a week of repairs.

Four readings, checked every two seconds instead of once an hour.

Now the system reads every sensor every two seconds and flags anything strange. Unplanned shutdowns dropped from four a year to one.

Two ways to catch trouble. Only one of them can see something building in seconds.

Here's the turn. The extra flags were never the real problem. The real problem showed up once the team let the system act on its own, not just flag things for a person to see.

At its worst, this costs more than a week of repairs. One night, the system eased a compressor's load down instead of stopping it, because it read a rare kind of trouble as an ordinary one. A technician doing his rounds was standing at that exact machine. He stepped away to check a different gauge a couple of seconds before the cylinder head cracked. Nobody outside the plant ever heard about it. But a couple of seconds either way, and this would be a very different kind of answer.

What I would leave alone: a note that says a bearing is wearing down and should get checked next week. Nobody's hurt if that note is wrong for a day. Run that fully on its own, no person needed.

The lesson: "the model is usually right" and "a wrong call here can be fixed" are two different questions. We kept answering the first one and assuming it answered the second.

Now here is the same thing as a story

The short version is above, and it already answers the question. Read this one for the three seconds that made it true.

Devraj Sallust has run reliability engineering at Station 14 for five years. He can tell a compressor is about to have a bad week from the sound of it, before he's looked at a single gauge.

CompressorGuard went live in his second year. For the first fourteen months, it only recommended. It would flag something strange on a technician's handheld, and a person decided whether to walk over and pull the trip. Real catches were rare, maybe two a month out of thirty flags, but when it caught one, it caught it hours before a person walking the floor once an hour ever would have.

In month three, the technicians opened every flag and read the detail. By month eight, most got a glance and a dismiss. By month thirteen, a flag on Unit 4 got closed within four seconds nine times out of ten, without anyone opening what it actually said. Nobody was being careless. Thirty flags a month, twenty eight of them nothing, wears a habit down whether or not anyone notices it happening.

The meeting where that changed was not dramatic. Ops leadership wanted fewer nuisance flags and fewer of the unplanned stops that were still costing close to two hundred thousand dollars each. Someone pointed out that a ramp-down, easing a compressor's load rather than stopping it outright, is reversible. You can always bring the load back up. So the team gave CompressorGuard the authority to ramp down on its own, above a set confidence, with no person in that loop at all, not even to review it afterward.

It ran clean for eight months. Then, on a Thursday at 9:46:57pm, the vibration on Unit 4 turned strange in a way CompressorGuard had almost never seen in training, a liquid slug starting to form. It had four real examples of that exact shape in forty thousand training runs. Read against everything else it knew, the shape looked closest to ordinary bearing wear. At 9:46:57.9pm, it eased the load down instead of tripping the unit.

Two and a half seconds between the wrong call and the crack. That's the whole window there ever was.

Reyansh Chikwendu was doing his rounds at Unit 4 that hour. At 9:46:58pm, a minute into a routine check, he stepped over to Unit 3 to jot down a reading. At 9:47:00.4pm, the cylinder head on Unit 4 cracked.

Nobody outside the plant ever heard about it. The head, the seals, six days offline: about three hundred forty thousand dollars, and the kind of story that gets told at shift change for a year. If Reyansh had still been standing at the access panel, the story would have been a different one entirely.

Devraj wasn't the one who wrote the confidence threshold that let CompressorGuard act alone that night. He was the one who'd argued, fourteen months earlier, that ramp-down was the safe one to automate first, because it was the one you could always undo. He wasn't wrong that a ramp-down can be undone. He was answering the wrong question. The question was never whether the action could be undone. It was whether the thing the action might fail to stop could be.

Same night, redesigned. Any anomaly shape with fewer than a set minimum of confirmed detections behind it, and any shape whose fastest confirmed case gave less than ten seconds of warning, routes straight to a hard-wired trip, no model authority over it at all. The same vibration signature at 9:46:57pm trips Unit 4 by 9:46:57.3pm, a third of a second, no confidence score involved. The slug still enters the cylinder. It enters a unit that's already stopped. Reyansh never needs the luck.

One design asked whether the action was reversible. The other asks whether the harm is. Those sound like the same question until the night they aren't.

The thing I'd tell myself, standing in that meeting fourteen months earlier: reversible is a word about what you did, not about what happens next. I used it to mean both, and for eight months, nobody found out I was wrong.

ORDER, sized to the one question that actually matters: can this be undone?

This isn't a straight two-way tradeoff between speed and safety. Five checks decide whether CompressorGuard gets to touch a decision alone, has to hand it to a person, or shouldn't be trusted with it at all. That's what ORDER ranks.

Reversibility isn't about the action itself. It's about what the action might fail to stop.

Nobody decides a category can act alone until the warning window is actually measured, not assumed.

The order that decides whether the model ever gets the final call at all.

One alternative is worth naming and rejecting directly: scoring every anomaly purely by the model's own confidence number, and letting anything above ninety percent act on its own, since a high confidence score looks like the safety check. It lost, because confidence measures how sure the model is against what it's already seen, not how reversible the result is if it's wrong. CompressorGuard had seen the liquid-slug vibration shape four times in forty thousand training runs, so its confidence that night meant almost nothing: it was matching a rare case to the nearest common one, not truly recognizing it. That's the specific failure worth naming, a rare-case blind spot, not a general accuracy problem, and the guardrail isn't "train it more," it's routing any anomaly shape with too few confirmed real examples straight to a hard-wired trip regardless of how confident the model claims to be. The trade-off is real and worth saying out loud: a hard-wired trip is less precise than the model. It will trip on some harmless spikes too, which costs more nuisance downtime on exactly the fastest category. That extra cost is accepted on purpose, in exchange for a guarantee that no model judgment, however confident, sits between a rare fast event and the equipment actually stopping.

And if you want to be sure it really works, try it somewhere else

Same five checks, a hospital ICU instead of a gas plant, and the honest ranking doesn't change shape.

Cairnhollow General Hospital runs GlucoPilot, a system that watches a patient's continuous glucose monitor and can suggest, or in one mode actually deliver, insulin through a pump. Clinical informatics lead Zbigniew Milanovic had three candidate actions to sort: a trend alert when glucose is heading somewhere bad, a small automatic tweak to the pump's background insulin rate, and a full bolus dose delivered on the system's own authority when a reading crosses a set line.

The item near the top of the chart is the one that never gets to run on its own.

Same steps, mapped onto Cairnhollow. Outcome: protect against a dose that can't be undone once it's in the bloodstream, not a smoother glucose curve on a chart. Reversibility: a wrong basal-rate tweak gets caught and corrected at the next fifteen-minute reading, nobody harmed. A wrong bolus dose is already absorbed by the time anyone reviews it. Dependency: a bolus delivery would need a real, fast clinician check before it fires, genuinely fast enough given how quickly insulin acts, and if that check can't happen in time, the model doesn't get to deliver it alone either. There's no hard-wired backup for a dose the way there's a mechanical trip for a compressor, which is exactly why this category never gets a fully automatic version, full stop. Evidence: the cheap check is the hospital's own log of bolus suggestions a nurse overrode, and how many minutes of real warning a glucose trend usually gives before a dangerous low. Rank: trend alerts and predictive low warnings run fully automatic, informational only. Basal-rate tweaks get a fast confirm loop, since they're smaller and genuinely correctable within the next reading. Bolus delivery never runs on the model's own authority. A nurse or clinician is always the one who presses deliver.

Swap the trigger and it still runs.
Speed: an interviewer cuts you to sixty seconds. Skip straight to Rank, name the four kinds of harm, and say plainly that none of them get the model's final call alone.
Cost: say the hospital can't staff a clinician to confirm every basal-rate tweak. That's real, but it means basal-rate suggestions stay informational-only until the staffing exists, not that GlucoPilot gets the extra authority instead.
The model got better, for real: say GlucoPilot's dosing model now beats clinicians on accuracy in trials. The rank doesn't move. Being right more often doesn't make a wrong dose more reversible.

Where people run it wrong.
They treat "the model is usually right" and "the error is recoverable" as the same fact. They're different questions entirely.
They assume a person is really in the loop because a screen shows a confirm button, without checking whether anyone can actually read and decide inside the real time window available.
They let a small, reversible-sounding action run fully automatic without asking what it could set off downstream, the same way a ramp-down interacted badly with the one rare event nobody had modeled well.

How to use it live. When you get a version of this question cold, buy yourself two seconds. Say it out loud: "can a person catch this after it happens, or is the harm already done the moment it starts." Then answer that one sentence for the exact feature in front of you. That's the whole method, and it buys you a second to actually think.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0