What feedback should route to a prompt change versus a model change versus a product change?

Brindlecrest Telecom runs a network operations center that watches fiber rings, routers, and cell towers around the clock. RootCause reads the alarm feed and drafts a likely cause and fix for whoever is on shift. Farai Chikonzo has worked the night desk there for five years.

How to answer this, stage by stage

Nobody is grading whether you know all three fix types exist. They're grading whether you can tell, on the spot, which one a real piece of feedback needs.

Let's learn

For five years, Farai Chikonzo could tell a real fiber cut from a noisy, self-clearing blip in under a minute, no tool involved, just the shape of the alarm feed.

Before RootCause, a first diagnosis on a tricky, multi-alarm outage took a sharp analyst about forty minutes of cross-checking logs by hand. RootCause cut that to under five: it reads the same alarms and drafts a likely cause and fix before Farai has finished her coffee.

The turn: the extra mistakes were never the real problem. The real problem was that Brindlecrest had no shared rule for where a wrong RootCause answer should go next, so every analyst made that call alone, in their head, on the spot.

One mistake costs a delayed release. The other one costs months, and it's the one that hides.

At its worst: the same failure pattern gets "fixed" with a slightly different prompt three separate times by three different engineers, none of whom knew the other two had already tried, because nothing was written down anywhere all three could see.

What I would leave alone: a one-off phrasing complaint that nobody else has ever reported doesn't need this whole routing process. Log it, patch it, move on. The routing rule exists for patterns, not for every single ticket.

The lesson: a good routing rule isn't about picking the right bucket once. It's about writing the decision down clearly enough that the next tired analyst on shift doesn't have to make the same judgment call from nothing.

Now here is the same thing as a story

The short version above is what you'd say defending the rule to Brindlecrest's engineering lead. Read this one for how the gap actually got found.

Farai Chikonzo has run the night desk, eleven at night to seven in the morning, for five years. She can read an alarm storm the way a mechanic reads an engine.

For RootCause's first eight months, the good nights were very good. It flagged the obvious stuff fast, and Farai's job shifted from digging through logs to double-checking a draft that was usually right.

Four steps, and the highlighted one is the step that shouldn't have lived only in one notebook.

With no shared place to record a routing call, Farai started keeping her own tally in a spreadsheet on her second monitor: which wrong answers looked like wording problems, which ones looked like the model just didn't know something, and which ones looked like the screen never gave RootCause a fact it needed. Nobody asked her to. It was the only way she had to remember what she'd already flagged.

One night, RootCause confidently labeled a strange multi-site alarm pattern "known flapping link, standard reseat fix," phrasing it had used correctly a dozen times before. Farai almost closed the ticket on that read. Something about the timing didn't sit right, so she dug in herself, by hand, one more time.

She caught it with four minutes to spare before a routine failover would have made the outage far worse. Her spreadsheet already had two earlier tickets that looked like the same shape, both logged under "unsure, maybe wording," because nobody had ever told her that repeating pattern was the actual signal to escalate.

Three branches, and the middle one is the branch Farai's near miss should have taken two tickets earlier.

After that night, Brindlecrest built the shared triage log Farai's spreadsheet had been standing in for: every routing decision, the reasoning behind it, and how many times that pattern had shown up before.

Run that same near miss forward under the new log: the moment the second occurrence of that pattern gets entered, the log itself flags "two prior sightings, both routed as wording, still unresolved: escalate." Farai finds it in under two minutes, without needing to trust her own memory, and without needing to be the one analyst who happened to keep a spreadsheet.

I built RootCause's early feedback habits around whoever happened to notice a pattern, because for the first eight months that was Farai, and she was good enough that it worked. It took one near miss, caught with four minutes to spare, to see that a rule which only works because one specific person remembers things isn't a rule. It's luck with a name attached.

PICK, in one screenNot a filing system. PICK is what tells you which layer actually owns a piece of feedback, and which mistake to prefer when you're not sure.

The ones worth a shared rule aren't the ones seen once. They're the ones sitting high and to the right.

The recap, one line per letter: position is routing by where the fix lives, impact is naming what both kinds of misrouting cost, cost asymmetry is preferring the cheap, visible mistake over the hidden, expensive one, and kill criteria is the two-repeat rule that actually forces the call.

And if you want to be sure it really works, try it somewhere elseSame four letters, a classroom grading tool instead of a telecom control room. Nothing else about the two jobs is alike.

Marrowdell School District uses an AI tool that drafts feedback comments on student essays for teachers to review before sending them home. Torstein Lien has taught eighth-grade English there for eleven years.

Mapped onto PICK: position is the same rule, applied to comment drafts instead of network alarms, route by where the fix lives. Impact names both sides: over-escalating a single awkward phrase to "the model needs retraining" wastes an engineering cycle on one teacher's pet peeve, while under-escalating a real gap, like the tool consistently missing thesis-statement problems in five-paragraph essays, means hundreds of students get the same shallow comment for months. Cost asymmetry favors the same side: the hidden, repeating miss costs far more than one delayed model update. Kill criteria: if two different wording tweaks to the prompt still can't get the tool to notice a weak thesis statement, that's a model gap, not a phrasing one.

Four fields, and the third one is what keeps three different people from trying the same failed fix.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "route by where the fix lives, prefer the mistake that's cheap and visible over the one that hides," and stop.
Cost: the model team says a retrain costs three engineer-weeks. That raises the bar for escalating, but it doesn't change the rule, it just means the kill criteria might move from two repeats to three.
The model gets better, for real: if RootCause's underlying accuracy improves overall, that's still not a reason to skip the routing rule. A better model just means the mistakes that do slip through are rarer and easier to mistake for one-offs, which makes the shared log more necessary, not less.

Four fields around one entry, and none of them are optional if the next person on shift is going to trust it.

Where people run it wrong.
They route by who complained loudest instead of by where the fix actually lives.
They let "we already tried a prompt fix" excuse an unlimited number of further prompt fixes on the same pattern, instead of setting a real limit.
They treat a shared log as paperwork instead of the thing that keeps the whole rule from depending on one person's memory.

How to use it live. When someone hands you a "which layer" question, ask yourself first: which mistake, made in the wrong direction, would I rather be caught making. Say that answer out loud before you say anything else.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0