What failure telemetry does design need in order to improve the experience?

The City of Alder Falls uses an AI tool that pre-screens building permit applications, flagging each one as likely compliant or needing a closer look. Wilhelmina Otieno is the permits clerk who reviews the flagged ones, and who first noticed something felt off almost a year before anyone could prove it.

How to answer this, stage by stage

Nobody is grading whether you can name a metric. They're grading whether the metric you name would have rung before the actual problem did.

Let's learn

Picture a review queue with three hundred permit applications a week, and only one number anybody upstream watches: how many got approved on time.

Alder Falls' permit-screening tool reads a building permit application and flags it likely compliant, ready for a fast approval, or needing a closer human look for something like a zoning setback or a load-bearing change. Before it, every application waited in a queue for a person to check it by hand, a slower process that never once skipped a check.

The whole design depends on that third box actually happening, every single time.

Now most likely-compliant applications move through in under a day. Wilhelmina still reviews every flagged application, and can override the AI's call whenever the reasoning doesn't hold up.

Here's the turn: the tool wasn't the problem. The problem is that nobody was watching whether Wilhelmina's checking was actually still happening, only whether applications kept moving. A review that opens a reasoning panel and clicks approve without reading it looks identical, on a throughput dashboard, to a review that read every word.

At its worst, a whole category of applications gets waved through on autopilot for months, and the first anyone learns of it is a state auditor's letter.

Same underlying problem. One clock had a three month head start nobody used.

What I would leave alone: simple applications, like a fence permit with no zoning question at all, have always had a low override rate and audits confirm that's correct. A low number there isn't a warning sign, it's just an easy category.

The lesson: a number that only counts what happened can't tell you when people stopped actually checking it.

Now here is the same thing as a story

The short version above is what you'd say defending this telemetry plan to Alder Falls' city council. Read this one for how the number actually crept down.

Wilhelmina Otieno has reviewed building permits for the city for eleven years, and she used to be the person who caught the setback variance nobody else noticed, the kind of thing that saves a homeowner from tearing out a wall two years later.

In her first months with the tool, Wilhelmina overrode about one in eight of its likely-compliant recommendations, usually catching a zoning detail the model's summary had glossed over. Over the following months, as the model's write-ups got more confident and specific-sounding, she found herself overriding less, then rarely, then almost never.

The rate looked healthy from a distance. What it was actually measuring had already changed.

A colleague reviewing a batch alongside her one afternoon remarked, half-joking, "you haven't overridden one of these in weeks, has it gotten that good?" Wilhelmina laughed it off. She hadn't timed how long she'd actually spent reading each one.

Three months sat between the number crossing its own warning line and anyone outside the queue finding out why.

Seven months in, a routine state compliance audit sampled twenty recent likely-compliant approvals and found four with a genuine setback issue that should have triggered a closer look, the exact kind of case Wilhelmina used to catch without a second thought.

With the redesigned dashboard, a per-category override-rate trend sits next to throughput, not instead of it, and any category falling below a quarter of its historical rate triggers a mandatory sampled review that same week. Run the same six months forward: the fourth-month warning triggers a review, catches the drift, and a short retraining session on what the model's confident write-ups can still miss brings the override rate back to a level the sampled audit confirms is genuinely earned.

The old dashboard asked "is work getting done." The new one asks "is the checking still real."

I built the throughput dashboard first because it was the easiest thing to prove the tool was working. It took a state auditor's letter to see that "fast" and "still being checked" were never the same claim.

LEAD, in one screenNot a lecture on picking a north star. LEAD is what tells you which number rings first.

Four numbers. Only one of them would have rung early on its own, which is why override rate leads the list.

The recap, one line per letter: link is correct, durable approvals rather than raw throughput, early signal is override rate falling below its historical level, abuse is a rubber-stamped rate looking identical to a genuinely earned one without an outside check, and decision is a mandatory sampled review the moment the floor is crossed.

None of these four require reading a single permit application yourself. They just require watching the checking, not just the output.

And if you want to be sure it really works, try it somewhere elseSame four letters, a public library's book-tagging assistant instead of a permit office. A completely different kind of override, the same quiet drift.

Larkspur Public Library Network uses an AI assistant that suggests subject tags and shelf categories for newly catalogued books, so librarians don't have to tag every arrival from scratch. Beatrix Solheim catalogues new arrivals most mornings and used to correct the AI's suggested tags on roughly one book in six.

Mapped onto LEAD: link is patrons actually finding the right book under the right subject heading, not just how many books get catalogued per day; early signal is Beatrix's tag-override rate, which had quietly fallen from one in six to nearly zero as the AI's suggestions started sounding more specific and confident, whether or not the underlying tagging model had actually improved that much; abuse is that a falling override rate could mean the model genuinely got better, or that Beatrix, under a cataloguing backlog, started accepting suggestions faster than she was reading them.

Swap "permit" for "book," and the same missing number shows up in a completely different building.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "track override rate over time, by category, since a falling one means checking has quietly stopped, weeks before a wrong decision ever surfaces," and stop.
Cost: there's no budget for a full independent audit team this year. Say so honestly, and start with a small, cheap monthly sample on the category with the steepest override-rate drop, since that's where the risk is concentrated.
The model gets better, for real: if the underlying tagging or screening model genuinely improves, override rate should still fall, but slowly and evenly, not in the sudden, sustained way a real behavior change looks like. The shape of the drop, not just its direction, tells you which one happened.

Where people run it wrong.
They build a dashboard that measures throughput and call it done, since throughput is the easiest number to show a stakeholder.
They treat a low override rate as unambiguous good news, without ever checking whether it reflects trust or fatigue.
They wait for a downstream audit to tell them something's wrong, when the whole point of a leading indicator is to ring before that.

How to use it live. When someone asks what failure telemetry design needs, ask one question first: what number would have moved weeks before the problem became visible to anyone outside the room? Build the dashboard around that number, not the one that's easiest to screenshot.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0