Critique an incident process that treats all model errors as sev-3.

How to answer this, stage by stage

Nobody is grading whether you can say "we should improve our incident process." They're grading whether you can take a policy apart and show, with real numbers, exactly where it stops making sense. Seven moves get you there.

Let's learn

The ticket queue is one page, sorted by time. Every model related bug lands on it, no matter what it actually is. Say a property tech company builds a tool that reads recent home sales nearby and prints an instant price for any address, then uses that same price to build real cash offers for homeowners who want a fast sale.

How it went before. When the tool only printed a number on a page, wrong meant small. Maybe one price in twenty landed outside a fair range, off by a few thousand dollars, and a homeowner who saw a strange number just shrugged and moved on. A single sev-3 bucket, checked inside two days, covered every ticket there was, because every ticket really was that same size.

How it goes now. The same tool prices real offers on hundreds of homes a month. A wrong price does not sit on a page waiting to be shrugged at anymore. It becomes an offer, sent, sometimes accepted, sometimes closed, before anyone has looked at the ticket that flagged it.

Say plainly, more tickets is not the problem here. The company still gets the same trickle of tickets it always did. The problem is what the team does with each one once it lands. Every ticket, big or small, waits in the exact same line, because the rule never learned to tell them apart. So the team keeps doing what always worked: read the queue in order, forty eight hours, done.

At its worst, that costs real money and real trust in a hurry. A data problem that quietly overprices homes in one part of town can run for most of two days inside the normal queue, feeding real offers the entire time, because nothing about its ticket looks any different from a wrong bedroom count on a page nobody will act on for a week.

What I would leave alone. A wrong bedroom count on a page nobody has acted on yet does not need paging anyone at two in the morning. Most valuation errors really are that small and harmless, and building four speeds of urgency for a typo would just spend the same attention it's trying to protect.

The lesson. A severity label is a promise about how fast someone looks, not a description of what broke. When every promise runs at the same speed, you have built a lot of tickets and no severity at all.

Now here is the same thing as a story

The short version is above. Read on if you want to feel how ordinary the forty one quiet hours actually looked from Esti's desk.

Every Monday morning, Esti Renshaw opens the on call queue before her coffee finishes brewing, the way she has for three years running incident response on Redbrook's InstaValue team.

She built that habit back when the queue was short enough to read in full. If something looked odd to her, wrong direction, wrong shape, she'd walk over and ask about it that same afternoon, whatever the ticket's label said. The label was a formality. Her own read of it did the real sorting.

For a long stretch, that habit and the flat sev-3 rule got along fine, because nothing serious ever hid behind the label long enough to matter. Then Redbrook grew. InstaValue started pricing real cash offers, a few a week, then dozens a day, then hundreds a month. Esti kept walking the floor for a while, out of habit, even as the queue got longer and the tickets got harder to tell apart just by reading them.

The trigger was small. A Monday morning data feed swap, moving Redbrook's comparable sales provider onto a faster pipeline, quietly started pulling comps that were current everywhere except one submarket, the Piedmont Basin metro, where a recent slowdown in sale prices hadn't synced yet. InstaValue kept printing numbers about twelve percent too high there, an average of fifty four thousand dollars a home, and it printed them with the same flat confidence it used everywhere else. An automated monitor caught the odd spread and filed a ticket at nine oh five that Monday morning. Sev-3, same as always.

The rule was written once, for a product with nothing real to lose. Nobody rewrote it when that stopped being true.

Esti didn't see it that morning. She had eleven other sev-3 tickets ahead of it, a rounding bug on a listing photo, a missing school district field, a stale zip code on a demo account, and the queue's own rule said read them in order. By the time a data analyst happened to open the comp regression ticket, the clock read forty one hours. In that window, thirty one homes in the Piedmont Basin metro had already gone out with an inflated cash offer attached. Twelve homeowners had already said yes.

Redbrook could not simply cancel twelve accepted offers. Renegotiating even a few of them meant real conversations with real homeowners about a number the company itself had put in writing. Total exposure by the time anyone looked: a bit over one point six million dollars.

One alternative got seriously considered and dropped: score severity by the technical cause itself, any model regression counts as sev-1, any data pipeline issue stays sev-3. It sounds tidier. It falls apart the same afternoon you check it against Redbrook's own ticket history, because a model regression on a field nobody uses can be worth zero dollars, and a boring pipeline bug feeding the live offer engine can be worth two million. Cause type never lined up with what mattered. Only downstream consequence did.

All three are real. The evidence in the Piedmont Basin ticket confirms the second one: a genuine sev-1 wearing the same clock as a typo.

The replay, run the same regression forward with a rewritten rubric in place. Every valuation ticket now carries an automatic dollar count, tallying money already committed through generated offers tied to that ticket's anomaly. Cross fifty thousand dollars and it pages someone immediately, no matter what part of the pipeline caused it. In the replay, the counter crosses that line after the second inflated offer goes out, about forty five minutes in. Two homes, not thirty one. Neither offer has been accepted yet. Redbrook catches it before anyone even gets the chance to say yes.

The thing Esti would tell her past self, from the week she first started trusting the label over her own instinct: a rule that never learned to tell a typo from a live offer was never really deciding anything. It was just picking a number and hoping the world stayed small enough for that to be enough.

TRACE, aimed at the policy instead of the incident

The question hands you a broken rule, not a mystery drop in a metric, so TRACE runs here as a teardown of the policy itself: rule out what it gets right, then narrow to exactly where it stops working.

And if you want to be sure it really works, try it somewhere else

Portage Freight runs RouteSense, a model that predicts delivery ETAs for the dispatch board and, on the same output layer, decides whether a shipment gets classified and placarded as hazardous material before a truck leaves the yard.

T. The severity rule got written three years before hazmat classification got bolted onto the same model. Back then RouteSense only forecasted ETAs, an internal number nobody outside dispatch ever saw.
R. The flat bucket never learned to split a soft internal estimate from a decision already printed on the side of a truck. An ETA running twenty minutes late and a placard that's wrong file into the exact same queue.
A. It assumes severity should follow the technical layer, a labeling bug or a routing bug, instead of what the output actually regulates, an FYI number or a DOT placard.
C. Alert fatigue. A real hazmat mismatch getting a lazy SLA. Dispatchers learning to ignore sev-3 tickets outright, since almost all of them turn out to be ordinary ETA noise.
E. A quarter of tickets pulled and checked: one truck left the yard placarded as non-hazardous while carrying a flagged corrosive. It sat in the normal queue for over a day, caught by a roadside inspection, not by Portage's own process.

Swap the trigger and it still runs.
Speed: an interviewer caps the answer at ninety seconds. Skip straight to the core move: split severity by what the wrong number already did, not by which layer broke, and attach an automatic exposure count to every ticket.
Cost: the on call team is short staffed and can't handle a stricter rubric's extra pages. Don't fall back to flat sev-3. Calibrate the dollar or count threshold higher until page volume matches real capacity, a stated tradeoff, not a silent return to the old rule.
The model actually got better everywhere: suppose the comp feed migration made InstaValue more accurate on average. That still wouldn't remove the need for the rubric. An improving blended average says nothing about whether one submarket's dispersion quietly got worse underneath it.

Where people run it wrong.
They rewrite severity purely by technical cause, any model regression is sev-1, and end up paging for a harmless regression while a catastrophic pipeline bug still hides behind a boring label.
They shorten the flat SLA to 24 hours for everyone instead of splitting it, which still lets a high exposure incident run unpaged for most of a day while also waking someone for a caption typo at 2am.
They build the exposure counter once and never recalibrate the threshold as the product's stakes grow, so the new rubric quietly goes stale the same way the original flat rule did.

How to use it live. Say the reframe before naming the fix: "the question isn't whether this label is technically accurate, it's whether it's asking the right question at all. A severity system built around what broke instead of what the wrong number already did will always treat a typo and a two million dollar mistake the same." That buys the room patience while you actually rebuild the rubric instead of reciting "we'll improve our incident process."

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0