How do you measure whether the human in the loop is adding value?

Kestrel Regional Stroke Center runs an AI tool over every incoming CT scan, flagging which ones show signs of a large-vessel occlusion, a blockage that needs a specialist within hours. Ingunn Marsalis is the radiographer who does the second look on every flagged scan before it goes to the on-call neurologist.

How to answer this, stage by stage

The interviewer isn't grading whether you can name a metric. They're grading whether the metric you name would catch a reviewer who has stopped reviewing, before a patient pays for it.

Let's learn

Picture a reading room with forty flagged CT scans a day, and one number everyone upstream watches: how fast each scan gets a final read.

Kestrel's stroke tool reads an incoming CT scan and flags it as likely showing a large-vessel occlusion, needing a specialist fast, or clear, safe to route normally. Before the tool, every scan waited in a queue for a radiographer to read it cold, slower, but never once skipped.

The whole point of the second look lives in that third box. Everything upstream of it is just routing.

Now most clear scans move on in minutes, and Ingunn still opens every flagged one, checking the AI's marked region against what she sees herself, free to override whenever the reasoning doesn't hold up.

Here's the turn: the tool wasn't the problem. Nobody was watching whether Ingunn's second look was still a real second look, only whether flagged scans kept moving. A three-second glance and an eight-minute read look identical on a throughput dashboard.

At its worst, an entire scan category gets a rubber-stamped second look for months, and the first anyone learns of it is a peer-review audit finding a delayed diagnosis.

Same underlying problem, two very different clocks. One of them had a five-month head start nobody used.

What I would leave alone: clean scans with no ambiguous shadowing have always had a low override rate, and repeat audits confirm that's correct. A low number there is a fact about the case, not a warning sign about Ingunn.

The lesson: a number that only counts finished reviews can't tell you when the reviewing itself quietly stopped.

Now here is the same thing as a story

The short version above is what you'd say defending this measurement plan to Kestrel's chief of radiology. Read this one for how the number actually crept down.

Ingunn Marsalis has read stroke scans for nine years, and she used to be the radiographer other shifts asked to double-check a borderline case, the one who could see a subtle occlusion nobody else had caught yet.

In her first months with the tool, Ingunn overrode about one in seven flagged scans, usually catching a small occlusion the model's marked region had missed by a few millimeters. Over the following months, as the model's highlighted regions got more consistently close to what she'd have marked herself, she found herself overriding less, then rarely, then almost never.

Three months sat between the number crossing its own warning line and anyone outside the reading room finding out why.

A locum radiographer covering her shift one week remarked, half-admiring, "you clear these fast, does the model just nail it every time now?" Ingunn laughed and said something like that. She hadn't actually timed how long she'd spent on each one lately.

The signals worth building live in the hidden-and-early corner, exactly where today's dashboard never looks.

Six months in, a routine peer-review audit sampled twenty recent cleared scans and found one with a genuine small occlusion that should have triggered an urgent referral, the exact kind of case Ingunn used to catch without a second thought.

With the redesigned dashboard, a per-scan-type override-rate trend sits next to throughput, not instead of it, and any category falling below a quarter of its baseline rate triggers a mandatory sampled re-read that same week. Run the same six months forward: the month-three warning triggers a re-read, catches the drift, and a short refresher on what the model's confident-looking regions can still miss brings the override rate back to a level the audit confirms is genuinely earned.

The old dashboard asked "are scans getting cleared." The new one asks "is the second look still a second look."

I built the throughput dashboard first because it was the easiest thing to show the board was working. It took a peer-review audit to see that "fast" and "still being checked" were never the same claim.

LEAD, in one screenNot a lecture on picking a north star metric. LEAD is what tells you which number rings first.

Four numbers. Only one of them would have rung early on its own, which is why override rate leads the list.

The recap, one line per letter: link is correct, timely referrals rather than raw throughput, early signal is override rate falling below its baseline, abuse is a rubber-stamped rate looking identical to a genuinely earned one without an outside check, and decision is a mandatory sampled re-read the moment the floor is crossed.

None of these four require reading a single scan yourself. They just require watching the checking, not just the throughput.

And if you want to be sure it really works, try it somewhere elseSame four letters, a marketplace trust and safety queue instead of a reading room. A completely different kind of checking, the same quiet drift.

Palisade Marketplace uses an AI tool that flags seller listings likely to be counterfeit goods, so a trust and safety analyst doesn't have to open every new listing cold. Ingrid Castellanos reviews flagged listings most mornings and used to remove or request proof on roughly one in five of the AI's flagged items.

Mapped onto LEAD: link is genuine counterfeit removals that hold up on appeal, not listings reviewed per hour; early signal is Ingrid's override rate, which had quietly fallen from one in five to nearly zero as the AI's flagged reasons started reading as more specific and confident, whether or not the underlying detection model had actually improved that much; abuse is that a falling override rate could mean the model genuinely got sharper, or that Ingrid, under a growing review queue, started clearing flagged listings faster than she was reading the evidence behind them.

Swap "scan" for "listing," and the same missing number shows up in a completely different queue.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "track the reviewer's override rate over time, by category, since a falling one means checking has quietly stopped, weeks before a bad outcome ever surfaces," and stop.
Cost: there's no budget for a full audit team this year. Say so honestly, and start with a small, cheap monthly sample on whichever category has the steepest override-rate drop, since that's where the risk is concentrated.
The model gets better, for real: if the underlying model genuinely improves, override rate should still fall, but slowly and evenly, not in the sudden, sustained way a real behavior change looks like. The shape of the drop, not just its direction, tells you which one happened.

Where people run it wrong.
They build a dashboard that measures throughput and call the human-in-the-loop question answered, since throughput is the easiest number to show a stakeholder.
They treat any low override rate as unambiguous good news, without ever checking whether it reflects trust or fatigue.
They wait for a downstream audit to tell them something's wrong, when the whole point of a leading indicator is to ring before that.

How to use it live. When someone asks how you'd measure the human's value, ask one question first: what number would have moved weeks before the harm became visible to anyone outside the room. Build the measurement around that number, not the one that's easiest to screenshot for a board deck.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0