What should you do when the canary looks fine but you have a bad feeling?

How to answer this, stage by stage

Eight moves. This question tempts a vague answer about "trusting your gut" or "trusting the data," so most of these stages exist to show exactly what a bad feeling is a stand-in for, and the one check that turns it into proof.

Let's learn

Braithford Bank built Ironclad to watch every card transaction the moment it happens and decide, in under a second, whether to let it through, block it, or hold it for a person to check. Whatever it decides, it also writes one short line explaining why.

Before Ironclad's newest version, the old model, v3, had run on all of Braithford's traffic for a year. When it blocked a genuinely fraudulent transaction, fraud ops could act on the written reason directly in about 92 out of every 100 blocks: a device changed twice in ten minutes, then a $2,400 charge two states away, that kind of thing, specific enough to escalate, specific enough to teach the next case.

Now version four, v4, is the canary. It's been running on 5 percent of Braithford's transactions for three weeks. On the dashboard, it's clean: block rate 1.9 percent, matching the old model's 1.8. False-positive rate 4.0 percent, slightly better than the old model's 4.2. Every week, the numbers say the same thing: ship it.

Here is the turn. Those two flat, clean lines were never the story, and they were never going to be. The dashboard was built to check one thing: did the model land on the right side of block-or-allow. It was never built to check whether the reason the model gave for that decision was the real one.

At its worst, this costs Braithford the thing fraud ops actually depends on the reason field for: the ability to escalate a real case fast, to build the pattern library the next investigation runs on, to tell a customer something specific. And underneath that, a model that lands on the right outcome without the right reason isn't guaranteed to keep landing there. If the cue it's actually using stops correlating with fraud once it sees the full population, not just this canary slice, a rare, high-value pattern gets waved through, and the dashboard would say the block rate is fine for weeks before anyone noticed, because a rare miss barely dents an aggregate number.

The gap between when Ironclad's dashboard first read clean and when anyone actually read what the model was writing

The choice I would take back. When Braithford first wrote the canary graduation checklist, a year earlier at v3's own launch, the only gate was the dashboard's two aggregate numbers. Nobody wrote in a mandatory hand-read of a random case sample, because back then everyone still read the narrative export out of habit, so it felt like the checklist would just be repeating something people already did. I would take that back. I'd put the hand-read into the checklist itself, not leave it to whoever still happens to be in the habit.

What I would leave alone. Ironclad's transaction-speed alerts don't need any of this. They're a pure latency number, under budget or not, with no generated explanation behind them to check against reality. Reading case narratives for a metric that has no narrative would just be motion.

The lesson. A dashboard only checks what someone thought to measure. Block rate checks whether the model decided right. It was never built to check whether the model's own account of that decision was real. A bad feeling, most of the time, is a person noticing that second question is still open.

The week Freya stopped opening the case file

Read the short version above if you're short on time. This is the long version, for the part where you feel exactly how a clean dashboard can talk someone out of their own instincts.

Freya Osagie can read a fraud case in under a minute and tell you, before she finishes the paragraph, whether ops missed something or the model did. Five years running fraud product at Braithford Bank will do that to a person.

Ironclad's v4 canary launched on a Monday, 5 percent of live traffic, the rest still on v3. That first week, Freya did what she always did at the start of a canary: she pulled the full narrative export herself and read fifteen or twenty cases end to end, checking the model's stated reasons against what she already knew about each account. All fifteen checked out. The dashboard agreed. Block rate 1.9, false-positive rate 4.0, both fine.

By the second week, she'd handed the full narrative review to a junior analyst and only skimmed the dashboard summary herself. It kept being fine. She had three other launches that quarter.

By the third week, she didn't even open the export. She just watched for the green tick in the Thursday review meeting, and it was there, same as always.

Then, on a Wednesday, Callixte Mbeki said something in the hallway that wasn't even really addressed to her. He'd been reading canary cases as part of his normal rotation, and he said, half to himself, "Feels like it's blocking the same stuff, just with different words on it." Then he kept walking.

Freya couldn't let it go. Not because Callixte had proof of anything, he didn't, he'd said it like a shrug. But it matched something she hadn't let herself notice: she hadn't actually read a case in two weeks.

So she pulled a real sample, 50 of v4's blocked transactions, and read every reason against the actual account facts. Nineteen named something specific. Thirty-one all read some version of "pattern inconsistent with typical account behavior," no matter what the transaction actually was.

Before she trusted that gap, she handed the same 50 cases, unlabeled, to a second analyst who'd never heard Callixte's remark. He flagged the exact same 31. Then she checked whether any of it actually cost anyone anything, or whether it was just clumsy wording. Customers disputed and won reversal on 42 percent of the generic-reason blocks, against 16 percent of the specific-reason ones. It was real.

So here is the decision I would take back. When Braithford wrote the canary checklist a year earlier, at v3's own launch, the dashboard's two numbers were the only required gate. Nobody wrote in a hand-read, because at the time people did it anyway, out of habit. The habit faded. The checklist never noticed, because it was never built to.

And the part I'd want to tell myself, if I could go back: we built a dashboard that could tell us whether the model decided right. We never built one that could tell us whether it was deciding right for a reason we could trust. I mistook a clean dashboard for a clean model.

What the hand-read actually showed

Before trusting the 31-case gap, Freya's team checked whether v4 was even getting the underlying decisions right. They checked the block-or-allow call on all 50 cases against known fraud outcomes: 47 of 50 were correct calls, in line with what the dashboard's own 1.9 percent block rate suggested. The model wasn't guessing badly. It was explaining badly.

That left the question of what a generic reason actually cost. Freya's team split the same 50 cases by whether the reason was specific or generic, and checked how often each group got disputed and overturned on review.

The generic-reason blocks weren't just harder to act on. They were wrong more often, and nobody could tell without investigating each one from scratch, because the reason field that was supposed to say why gave nothing to check against.

Three reasons a clean canary metric can miss, and the one the test confirmed

Not because anyone was careless. Each of these, on its own, looks like a small, forgivable rough edge in a new model. Together, they're why a canary that never once looked shaky could still be quietly failing the people who depend on its reasons, not just its outcomes.

Three separate, checkable causes, only one of them confirmed by the hand-read

TRACE, when the dashboard has nothing left to tell you

This reads like a question that wants a general answer about trusting instincts. The real job is diagnosis: work out whether a bad feeling is pointing at something real that two clean numbers structurally can't show, and prove exactly what that something is.

The same gap, on a crop photo instead of a card swipe

Silvergrain Cooperative pilots a plant-disease flagging tool for its member farmers: upload a photo of a leaf, get back a likely diagnosis and a one-line reason. Nomvula Khumalo leads the product. A new model version runs as a canary on 8 percent of uploads. Detection accuracy reads 91, 92, and 91 percent across three weeks, matching the old model's 90 percent baseline every time.

T. The canary launched three weeks earlier. Aggregate accuracy read clean every week. In week three, a regional agronomist mentioned to Nomvula that the captions on the newer model's alerts had started sounding "the same no matter what's actually wrong with the leaf."
R. Recut by kind of evidence, not by region. Reading 40 flagged photos by hand against lab-confirmed diagnoses: the old model's captions named the real disease marker in 33 of 40 cases. The new model's canary named it in 14 of 40, defaulting instead to "leaf discoloration consistent with early stress" regardless of the confirmed disease.
A. A second agronomist, unbriefed, read the same 40 photos and flagged the identical 26 generic captions. Farmers who got a generic caption acted on the alert, sprayed or didn't spray, correctly only 38 percent of the time, against 81 percent for a specific caption. The gap was real, not a wording quirk.
C. Three candidates: a new fungal strain this season showed up in only a handful of uploads, too rare to move aggregate accuracy; generic captions erode farmers' willingness to act at all, a trust problem no accuracy number tracks; and the model may be flagging based on a lighting or soil-color cue that correlated with disease in its training photos, not the actual disease marker, a shortcut that won't hold once farmers photograph in different light and soil across regions.
E. Take photos the model captioned generically, and check the visual cue it actually weighted most heavily using a saliency check: in 22 of 26 cases, the heaviest weight sat on background soil tone, not the leaf itself. Reshoot the same leaves against a neutral background: accuracy on those cases jumps to 88 percent, close to the specific-caption rate. The background cue, not a rare disease or a caption-writing problem, was the real driver, a different confirmed cause than Ironclad's, found the same way.

Flashcards (click a card to flip it)

This is a question about a bad feeling surviving a clean canary, worked as a diagnosis, so these eight test the TRACE moves and the real numbers behind them.

Check yourself Score: 0 / 0