Give three leading indicators of AI feature health and the lagging metric each predicts.

Three numbers move before an agent ever stops opening the panel. Here is what each one is actually warning you about, and where the same alarm gets rung by accident.

How to answer this, stage by stage

The interviewer is not grading whether you can name three metrics. They are grading whether each one earns its place with a real lagging number attached, and whether you know what to do the day it crosses a line. Eight moves get you there.

Let's learn

What tells you a coaching tool is dying before the chart everyone watches ever moves?

Signalwell sits inside Coldharbor Roadside Assistance, an auto club's contact center. It listens to a live call, scores it, and drops a short coaching note into a side panel: what to say next, roughly how long the call should take, a flag for whether the model is sure or just guessing.

Three separate numbers, all answering the same real question: will the agent still be leaning on this thing in month six.

Before Signalwell, an agent handling a tow request worked from memory and a printed script taped to the monitor. The average call ran about six minutes, and a third of tricky ones needed a supervisor pulled in. In its first months, Signalwell cut that down close to four minutes, and pulled-in supervisor calls dropped by half. Agents liked it. Nadiya Boruc, who runs the coaching program, watched one number on her weekly report: the share of calls where the panel was open. It sat at ninety-one percent, and it stayed there.

Here is the turn. The rising override rate was not the real problem. It was the earliest honest signal Coldharbor had. What mattered was what caused it: a new towing procedure for electric vehicles had rolled out two weeks earlier, and Signalwell had barely seen it, but a bug in how the model judged its own certainty still tagged its guesses on those calls as "high confidence." Agents on the overnight shift, who handle most tow requests, got burned by a few wrong green flags and stopped trusting green flags generally, not just on EV calls.

At its worst, this cost more than an awkward number on a report. The gap between what Signalwell's note promised ("wrap this in about ninety seconds") and how long the call actually took widened from six seconds to thirty-five seconds over five weeks, on EV tow calls especially. Supervisors started fielding complaints about "the tool making calls take longer," rising from two a week to thirteen a week. And repeat callbacks on EV flatbed dispatch, where the customer had to call back because the first call went sideways, climbed from six percent to nineteen percent.

The choice I would take back is not the prompt update itself. It is that nobody put a check on it before it shipped. Coldharbor tested the new, friendlier wording on a general sample of calls and it read fine. Nobody built a slice of the test set from the newly onboarded EV procedure, because at the time of testing, almost no EV calls existed yet to sample from.

What I would leave alone: the panel-open auto-expand behavior itself. That default is genuinely fine, it just cannot be the metric anyone trusts, because "present" and "used" are not the same claim.

The lesson: a metric that is watching the panel instead of the person will read healthy for months after the person has already quit trusting what's on it. The gap between those two dates is exactly the gap the three earlier signals are built to close.

Now here is the same thing as a story

The short version sits above. Read on for the six weeks Nadiya spent explaining a number that hadn't moved yet.

Nadiya Boruc has run Coldharbor's coaching program for five years. She built the training deck every new agent gets on day one, and she can tell from the shape of a call transcript, before she checks a single score, whether an agent is having a good week or a bad one.

When Signalwell launched, the good months looked steady. Morning shift-change, she'd pull the weekly report, see panel-use sitting at ninety, ninety-one percent, and move on to the next thing on her list. The number never gave her a reason to look closer.

Underneath that flat number, something was thinning out. In week one after the EV procedure rolled out, an overnight agent got a green-flag note that told her the customer's flatbed request needed no special handling. It did. The customer's EV had a battery isolation switch the note never mentioned, and the call ran nine extra minutes while the agent figured it out live. She mentioned it to a coworker at the next break. By week three, three more agents had their own version of that story.

The number on the report and what the agent actually did with the tool can drift apart for weeks without anyone noticing which one is lying.

None of them filed a ticket. They just started reading the whole note more slowly, then skimming it, then, on calls that felt routine, not opening it at all. Nobody decided this out loud. It built the way distrust always builds: in private, one call at a time.

The moment anyone outside the overnight shift noticed was not dramatic. A regional manager forwarded Nadiya a spreadsheet: repeat callbacks on flatbed dispatch, up from six percent to nineteen percent over six weeks. Nadiya's first instinct was to check panel-use. Ninety-one percent. Healthy. She almost closed the spreadsheet.

It was only when she pulled call recordings, not dashboard numbers, that she heard it: agent after agent glancing at an open panel and doing the call from memory anyway. The panel was there. Nobody was reading it.

A year earlier, when Coldharbor built the coaching dashboard, watching panel-use made sense. Back then the panel didn't auto-open, so a low number meant exactly what it looked like: agents choosing not to use the tool. Nobody in that early design meeting pictured a future where the panel opened itself by default and the real signal moved somewhere the dashboard couldn't see.

Run the same six weeks through the fixed design. Override rate on green flags crosses twelve percent by week 3. The threshold fires a hand review that week, not week nine. A QA analyst pulls twenty overridden EV calls, finds the bug in an afternoon, and Signalwell's next prompt version ships with a golden set built specifically from the new procedure. Callback rate on flatbed dispatch never gets past eight percent. Nadiya spends her Tuesday on something else.

What I would tell myself, before any of this: the day a number stays flat by design, because of how the interface defaults, is the day to stop trusting that it means what it used to mean.

LEAD, the four moves behind this answer

This is a metric question, so LEAD runs it, not FLIPS. Each letter gets its own answer here, then the same four letters run again in Section 4 on a completely different product.

Two things worth naming directly, since this is where the real judgment lives. The AI-specific failure mode here is the model's own "sure" tag going quietly wrong on a kind of call it hasn't seen much of: a new call type the model has barely seen gets tagged high-confidence anyway, because its sense of "sure" was learned on the old mix of calls. The guardrail is a golden set that always carries fresh examples from any newly onboarded call type, and a rule that a new prompt or model version only ships once it agrees with hand-graded specialists on that set most of the time, checked before rollout, not after an agent gets burned. The trade-off worth stating plainly: putting the caveat sentence back and having the note briefly explain its own confidence adds real generation time mid-call, on the order of a second and a half, plus a bit more cost per call. That is a cost worth accepting, because the alternative is agents deciding on their own, silently, that green flags cannot be trusted, which costs far more than a second and a half ever will.

And if you want to be sure it really works, try it somewhere else

Same four letters, a grocery chain's self-checkout loss-prevention tool instead of a call center, and this time the leading signal is a dismiss rate instead of an override rate.

Corrales Grocery runs Loomstack at its self-checkout lanes. It watches the scanner feed and flags carts that likely have an unscanned item, sending an alert to a loss-prevention associate's radio. Zahra Pellerin runs loss prevention across the chain's twelve stores.

The break happens at step three, long before anyone can see it in the store's shrink number at the far end.

L, link. Not flag volume. Actual shrink dollars recovered per store per month, the number that decides whether Loomstack keeps its budget.
E, early signal. The share of high-confidence flags an associate dismisses without walking over to open the cart. That predicts a rise in shrink dollars at that store, roughly three to four weeks later, once word gets around that flags don't need checking.
A, abuse. A store manager can force the dismiss rate down by making associates walk over to every flag on principle, even ones they already know are false, which looks like trust and changes nothing about real theft. Misread the other way: dismiss rate can rise because a new store-brand barcode rollout spiked false positives, which is model drift, not associate distrust, so you check the false-positive rate before blaming the person.
D, decision. Under 8 percent dismiss rate, healthy. 8 to 20 percent, spot-check 15 dismissed flags by hand. Above 20 percent sustained, pull that store's threshold back to human-reviewed samples only until the model is retaught what the new barcodes look like.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to the fix: whatever adoption number you're watching, pair it with a share that shows whether the tool is actually being trusted, not just present.
Cost: engineering says a real live-reference metric can't ship for two months. Do not fall back on raw panel-open rate as a stopgap in the meantime, and hold off on any new alert built from that number until the real one exists.
The model got better, for real: say Signalwell's suggestions genuinely improve next quarter. That alone does not prove override rate falling means renewed trust, agents who already stopped checking may never notice the improvement. Only a rising live-reference rate proves people are actually engaging with the better output.

Where people run it wrong.
They treat "the panel is open" as "the note is being used," when it just means the interface auto-expanded.
They fix the number by loosening what counts as high confidence, instead of asking why agents stopped trusting the real ones.
They wait for the client's scorecard or a regional audit to notice, instead of checking these three signals on their own schedule.

How to use it live. Say the reframe before naming a single number: "A tool being open on someone's screen isn't the same as someone trusting what's on it, especially when it opens itself by default." That buys you room to give the real answer instead of reaching for a generic engagement metric.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0