Describe acceptance criteria that account for the severity of different error types.

Six moves for saying the bar out loud

A severity question is a tradeoff wearing a spec's clothes, so PICK does the work here, not a checklist of rubric fields.

One more thing before the walkthrough moves on: this bar covers one severity tier, not the whole spec. Most candidates hear "the worst case matters more" and answer by tightening everything across the board. Say which tier gets the strict bar and which one stays on the pooled average, and you've shown real judgment instead of reciting a checklist.

Let's learn

Every Red Flag Warning morning, grid engineers at Talus Ridge Power & Light used to spend about three hours deciding which of the utility's 1,800 line segments to shut off, checking wind and dryness reports by hand for the riskiest 380 of them.

Then came Emberline, a tool that reads wind speed, humidity, how dry the brush is, and each segment's equipment history, and scores every one of those 1,800 segments for fire risk every fifteen minutes.

One score, one dispatcher, one call before dawn

With Emberline sorting segments and flagging only the ones worth a closer look, usually around 40 a morning, that three-hour review fell to about 20 minutes.

Here is the turn. Those 20 minutes were never the risk. What was hiding inside the number that let Emberline ship was. The team had set one release bar: overall fire-risk classification accuracy above 92 percent, pooled across every segment-score, calm days and Red Flag Warning mornings, all averaged together. Emberline cleared it at 92 percent. Nobody asked, separately, how good it was on the mornings that actually matter.

At its worst, that looks like this: a segment under real Red Flag Warning conditions, dry brush, wind past 35 miles an hour, scores clear, the line stays energized, and a fault arcs into the brush below it. Or, in the version nobody wants to say out loud, nobody catches it until the smoke is already visible from the highway.

What I would leave alone. The criteria for ordinary, low-wind days. Those are common, cheap to get slightly wrong, and a false flag there costs a dispatcher a few minutes and a customer a few hours without power. Tightening that bar to 99.5 percent too would only trigger more shutoffs nobody needed.

The lesson. We wrote the criteria to answer "is this tool accurate," and it answered that well. We never wrote a separate bar for "is it accurate on the one kind of morning that can start a fire." Those are two different questions, and only the second one has a canyon full of houses riding on it.

The wind report Junko almost waved off

You don't need this to answer the question. Read it if you want to feel why the bar has to sit on the rare tier, not the average.

The control room at Talus Ridge Power & Light goes quiet a little before 4 a.m., which is exactly when the wind starts moving over the Widow's Peak corridor. Junko Hirano has run the overnight shutoff desk for eight years. Ask her which segment will be trouble before dawn and she can usually tell you before her coffee's done, corridor 14, always corridor 14, because the canyon funnels the wind straight down it.

Emberline arrived the spring after the 2019 Hollow Creek Fire, and for most of two seasons it was just good. A segment scored in under a second, flagged or clear, and the flagged ones were almost always right. She'd still call the field crew to confirm conditions on anything scored clear near corridor 14, out of habit, the way she'd done it for years before the tool showed up. It agreed with her. Every time.

So she started calling on half of them. Then only the ones Emberline flagged. By the second season, if the screen said clear, she moved on to the next segment.

Same tool, two very different kinds of wrong

Then, one morning in October, she was about to clear corridor 14. Emberline had it scored clear, nothing on the screen looked wrong. But the field radio crackled with a downed-branch report two corridors over, wind gusts already past what the seven a.m. forecast had called for. She paused, and made the call to hold the segment and send a crew out anyway, the way she used to on every uncertain morning.

The crew found a fault on a splice fitting, arcing, sitting eight feet above brush dry enough to have gone up in seconds if the wind had shifted. Junko hadn't caught it because Emberline told her to look. She caught it because eight years of habit hadn't fully left her, and a downed-branch report happened to still be in her ear that morning.

Halvard Skeie had owned Emberline's release criteria since before it shipped, six years into a career writing acceptance bars for grid safety equipment before AI ever entered the picture. When Junko's near miss reached him, the header number still looked fine. Pooled accuracy, checked that same week, read 92 percent. He almost logged it as one lucky catch.

Then he pulled the corpus the launch decision had been made against: about 600,000 segment-scores across the prior season, only 600 of them under confirmed Red Flag Warning conditions. One in a thousand. He reran the recall number on those 600 alone. Emberline had correctly flagged 408 of them. Sixty-eight percent.

Sixty-eight percent had been sitting inside a 92 percent average the entire time, because 600 extreme-tier scores can barely move a number built from 600,000.

So Halvard split the bar in two. Before any pooled accuracy number counts for anything, Emberline has to clear 99.5 percent recall on a test set built to actually hold enough confirmed Red Flag Warning segment-scores to measure it, stacked well past the natural one-in-a-thousand rate. Calm and moderate days stay on the old pooled bar, since getting those slightly wrong has never cost anyone more than a few hours without power.

The part he'd take back isn't the launch. It's picking one number, simple to explain, simple to gate a release on, and never asking whether that one number could actually see the mistake that mattered. A number that good on 600,000 segment-scores hid exactly the tier it needed to prove itself on.

Junko's flagged-segment review now runs about 25 minutes a shift, five minutes more than the old average-based bar, because the new one flags a few extra borderline calm-day segments on purpose rather than risk waving off a borderline fire-day one. And corridor 14 can't slip through silently the same way again, because nothing scoring below the new bar on a Red Flag Warning morning reaches her marked as clear.

The four letters, and why this isn't a rubric

This is a tradeoff dressed as a spec, so PICK is the tool. A "how would you measure whether this feature is working" question would reach for LEAD instead.

Run the four letters again, on a hospital ward

Lindenfield General Hospital runs a model, VitalArc, that reads a patient's vitals and labs continuously and flags a rising risk of deterioration for the charge nurse. Same shape of question, a different ward, a different kind of harm.

P. Gate any release of VitalArc on its recall for one tier, a patient heading toward septic shock within the next four hours, at 99 percent or higher, on a set stacked with confirmed real cases. General deterioration-score accuracy is reported, but it never gates a release by itself.
I. A false alarm on a stable patient costs a nurse about ten minutes to recheck and clear, folded into a normal round. A missed case of oncoming septic shock isn't caught until the patient's own vitals crash on their own, and by then it's an emergency response, days added to the stay, and in the worst case, a death.
C. Confirmed septic-shock progressions are a tiny slice of every monitored patient-hour next to ordinary stable readings, the same shape of imbalance as Emberline's extreme fire days. A pooled deterioration-accuracy number will always be dominated by how well the tool reads ordinary vitals, never by how well it catches the rare crash.
K. Drop the special bar the day septic-shock-tier recall, tracked quarterly, catches up to the pooled deterioration accuracy. Until then, any patient flagged near the threshold gets a nurse's eyes regardless of score.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0