What is the leading indicator you would add first to an under-instrumented AI product?

Rank what you would build first inside an AI product that has been running with almost no instrumentation at all, and defend the order out loud.

How to answer this, stage by stage

Seven moves. The trap in this question is answering with a list of dashboards you would like to have, when it is really asking which missing piece of data disappears for good the moment the blind period ends.

Let's learn

Callixta Drummond can tell you, without checking, exactly how many identity documents get bounced to a human each day, and why. She built the review-queue staffing plan herself, back before PassGate handled almost everything on its own.

PassGate is a tool built into Verrow Financial's signup flow. A new customer holds up a driver's license or a passport, takes a selfie, and PassGate decides in about four seconds whether the two match and whether the document looks real.

Before PassGate, every new account got a five-minute check from someone on Verrow's identity team: does the photo match the ID, does the ID look genuine, does the address line up. At the volume Verrow does now, around 42,000 new signups a month, that is roughly 3,500 hours of review work a month. Nobody could staff that.

PassGate cut it to almost nothing. Today it clears 96 percent of new signups on its own, in seconds, no person involved. The other 4 percent, the ones its own confidence score comes in low on, get routed to Odalric and the small review team.

Four of these five steps cannot start until the first one exists. The dashboard on the far right could, in theory, be built any day of the week.

Here is the turn, and it is not that PassGate gets some of the 96 percent wrong. Every model does. The real problem is that nobody at Verrow could say how many, or which ones, because from day one PassGate was only ever wired to report one number: how many signups it cleared that day. Not its own confidence on any of them. Not what happened to the account afterward. Nothing that would let anyone go back and check.

At its worst, this costs Verrow the one thing an identity check exists to protect: knowing, later, whether it worked. A forged passport that PassGate waves through with high confidence looks, on every dashboard Verrow had, identical to a real one that PassGate waves through with high confidence. Both just count as cleared.

What I would leave alone: Odalric's review queue and workflow for the 4 percent PassGate already flags don't need touching. That pool is fine, staffed, and reasonably well understood. The blind spot sits entirely inside the 96 percent nobody ever built a way to check.

Fixing it wasn't free. Keeping images and scores around longer than 30 days costs more storage and more compliance overhead, the exact cost the original 30-day purge was built to avoid. Callixta's team narrowed that cost instead of ignoring it: full images now get kept for 180 days only for a random 2 percent weekly audit slice and for every flagged case, not for every one of the 40,000-plus signups a month. Less coverage than keeping everything, but enough to catch a drifting rate, at a storage bill Verrow could actually defend.

Once the gap finally got closed, here is what the numbers looked like. Out of a 300-case sample checked by two reviewers, PassGate's confident-wrong rate came back at 1.8 percent overall, but that hid a much bigger split underneath it.

The lesson: a number that only tells you how many, never how sure or how it turned out, isn't really instrumentation. It's a receipt.

Now here is the same thing as a story

The short version sits above. Read on for the Tuesday a routine audit found the gap nobody had been tracking as a gap.

For the first year PassGate ran, Callixta's morning routine was one glance: open the internal dashboard, see the "cleared today" counter, close the tab. Onboarding time had dropped from two days, back when a human reviewed every ID by hand, to under a minute. Complaints about slow signups had all but stopped. She had no reason to look closer, and for a long while, nobody did.

The gap opened quietly, in three small decisions that never felt like one big one. At launch, logging PassGate's own confidence score for every decision got cut from the build, to hit the launch date two weeks earlier. "We can always add it later, once we know what we actually need it for," someone said in that meeting, and it sounded sensible. Six months in, someone in leadership asked for a fraud-pattern dashboard, and the answer came back: not enough data yet, next quarter. By month twelve, next quarter had been said three times, and the gap had stopped being a live topic at all. It just wasn't anyone's job anymore.

The trigger wasn't a fraud alert. It was a Tuesday, and an unrelated payments audit. Odalric pulled fifty old approved signups to spot-check formatting, nothing to do with fraud. One of them, a passport-based signup from three weeks earlier, stopped him cold. The kerning on the passport number was off in a way a trained eye catches in about two seconds. PassGate had scored it 91 percent confident and waved it straight through. The account had already moved money out twice.

One of these gaps was always going to be there whenever anyone got around to it. The other one had an expiration date nobody had been watching.

Callixta asked the obvious next question. How many others like this got through in the last three weeks? The honest answer was nobody could know. PassGate had never logged its confidence anywhere durable, and the ID image from three weeks back was already gone under the 30-day purge. The blind period wasn't a mystery waiting to be solved. It was just gone.

Leadership's first instinct, right after the near miss, was to ask for exactly the dashboard they'd been asking for all along: a live fraud-pattern view, built fast. Callixta talked them out of building that first. You cannot chart a number you never captured, and a rushed dashboard would have been back-filled from the only detailed records that existed, the 4 percent Odalric's team had already flagged by hand, which would have produced a confident-looking chart with almost no real signal underneath it.

So instead, the first thing her team shipped was the least exciting one: every PassGate decision, going forward, now carries its confidence score and, once available, a downstream outcome flag, whether the account was later reported for fraud or reversed by a manual review. Two days after that pipeline went live, they ran it against a golden set of 300 real production signups, checked by two senior reviewers instead of PassGate's own opinion of itself.

That golden set is what surfaced the 4.6 percent versus 0.9 percent split, passports failing far more often than licenses. Verrow raised the passport auto-approval bar that same week, instead of waiting for a fourth "next quarter."

Run the same near miss with that pipeline already in place a year earlier, and the difference isn't that the forged passport never gets approved. Models still miss things. The difference is what happens the next morning. Callixta pulls the last ninety days of logged decisions instead of a purged folder, finds the passport gap inside a week instead of inside a scramble, and raises the bar before the next forged document, not after it.

What I'd tell myself, back in the meeting where we cut confidence logging to hit the launch date: the whole point of an automatic decision is that nobody's watching it happen. If you don't write down what it decided and how sure it was, you haven't saved that watching for later. You've deleted it.

ORDER, and why the blind period ranked first

This is a ranking question wearing a metrics question's clothes. Everything competes for one engineer's time, so ORDER decides which candidate leading indicator is hardest to live without, not which one sounds best on a roadmap slide.

Same rank, a telehealth triage line instead of a signup flow

Understory Veterinary Telehealth built FirstLook Triage, a tool that reads the photo and short description an owner submits and sorts each case into a callback within the hour or a routine callback within a day. Solvig Marchetto runs clinical operations there.

Understory had the same blind spot Verrow did. FirstLook auto-sorts about 92 percent of the roughly 900 submissions it sees a day, and for over a year the only number anyone tracked was how many got sorted, not how sure FirstLook was or what actually happened to the pet afterward. Call transcripts got purged after 48 hours.

The near miss looked different, but the shape was identical. A dog's owner described restless pacing, retching without vomiting, refusing to lie down, textbook early signs of bloat in a large-breed dog. FirstLook scored it 88 percent confident, routine, mild digestive upset. The owner called back panicked six hours later. The dog survived emergency surgery, barely. Solvig asked how many other routine-labeled cases that week had quietly turned into the same kind of emergency, and the honest answer, again, was nobody could know. The transcripts were already gone.

O. Every version of FirstLook's build order protects one thing: catching a pet FirstLook wrongly called routine, before it becomes a middle-of-the-night emergency nobody saw coming.
R. A triage decision made without logging its confidence and later outcome, once the transcript purges at 48 hours, is gone for good. A friendlier clinician dashboard can be added any month from data that will already exist by then.
D. Nothing works until every triage decision, its confidence, and an outcome flag, an ER visit within 24 hours, yes or no, gets logged and kept past 48 hours. Everything else assumes that table exists.
E. Cheap and fast: within the first two weeks of logging, pull the routine-labeled cases that turned into ER visits and see how big the gap actually is, before spending a month building a full golden set.
R. Raw capture first. Then the confident-wrong rate, routine calls that became real emergencies. Then a small vet-reviewed golden set built from real transcripts, not scripted test cases. Then the same rate split by symptom category, since GI symptoms in large-breed dogs carry a different bloat risk than GI symptoms generally. The nicer clinician dashboard goes last.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to the rule: raw capture first, because nothing else can be built without it.
Cost: engineering says the pipeline linking a triage call to a real ER outcome is six weeks out. Don't wait on it. Pull a smaller weekly sample by hand from insurance claims in the meantime.
The model got better, for real: FirstLook's underlying model genuinely improves at reading symptom photos. That still isn't the same claim as "the confident-wrong rate is zero." A better model can still be wrong in a new, differently shaped way, and without capture, nobody would ever see that either.

Where people run it wrong.
They build the friendlier dashboard first because leadership asked for it right after a scare, and it looks like progress.
They add confidence logging going forward only, and never notice the same blind period keeps recurring every time nobody looks back.
They test the model once against a golden set and call it settled, instead of re-checking it whenever the model underneath changes.

How to use it live. Open with the outcome before naming a single metric. "Every leading indicator I'd rank here protects one thing, catching a real problem before it compounds silently." Then rank from there. Naming the outcome first turns a list of dashboards into something you can defend line by line.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0