How do you measure whether your data flywheel is actually turning?

Here is what happens when a loop that is supposed to feed itself quietly stops feeding itself, and every dashboard keeps saying it's fine. Cascade Mobile is a regional wireless carrier. ReplyLine is its tool that drafts replies to customer support chats, and is supposed to get better each release as support agents' edits feed back into training. Selin Marek is the senior AI PM who has to prove, every quarter, that this loop is actually turning and not just spinning.

How to answer this, stage by stage

Nobody is scoring whether you can name a metric. They're scoring whether you can catch the one that's lying to the room.

Let's learn

Here is what happens when the metric everyone watches can go up for a reason that has nothing to do with the thing it's supposed to measure.

Before ReplyLine, a Cascade Mobile support agent read each incoming chat and typed a reply from scratch, about ninety seconds a ticket, several hundred tickets a shift split across a team. With ReplyLine, a draft reply appears the moment a ticket opens, and the agent edits it or sends it as is. The whole pitch of the tool was that every edit an agent makes gets logged and folded into the next training run, so the drafts should need less and less rewriting over time. That loop, working, is the flywheel.

The third step is where Cascade Mobile's loop actually broke, quietly, for three releases running.

Here's the turn: the metric Cascade Mobile actually watched, the share of drafts agents sent without opening the editor at all, kept climbing. Release over release it looked like exactly the story everyone wanted: the model learning, agents trusting it more. Nobody was watching the number underneath it.

At its worst, a flywheel that has quietly stopped turning still ships releases, still shows a rising accept rate, and still gets treated as a success story right up until a lagging number finally moves and forces the question nobody asked earlier.

What I would leave alone: I wouldn't throw out accept rate everywhere. On a genuinely healthy loop it moves together with edit-depth, and watching it costs nothing extra once edit-depth is already being tracked properly.

The lesson: a flywheel metric has to measure the wheel, not the person standing next to it. The moment your number can rise because someone got tired instead of because something got better, you're not measuring the loop anymore.

Now here is the same thing as a story

The short version above is what you'd say defending a metric choice in a planning review. Read this one for how a whole floor's trust quietly outran the thing it was trusting.

Every shift on Cascade Mobile's support floor starts the same way: headsets on, queue open, the first ticket usually a billing question that answers itself in the first line.

Bryndis Halvorsen has worked that floor for three years. When ReplyLine launched, it was, for months, the best thing that had happened to her shift. A draft appeared, she'd read it, fix a word or two, hit send. Ninety-second tickets became twenty-second tickets. She still read every draft closely those first weeks, the way you'd watch a new hire before trusting them with the register.

Then she started reading a little less closely. Then barely at all. By the fourth release, she was hitting send on most drafts the instant they appeared, the way you stop double-checking a coworker who's never once let you down.

Nobody decided, on any one day, to stop checking whether the number underneath was actually moving.

Nobody at Cascade Mobile noticed a single Tuesday where anything changed. There wasn't one. The floor's trust in ReplyLine had been quietly climbing for a year, and the loop it was supposedly built on top of had been quietly flat for almost as long.

What finally surfaced it wasn't a complaint. It was Marcus Teal, a data analyst three weeks into the job, asking a question in a release review that should have been trivial: "has edit-depth actually gone down since launch?" Nobody in the room had the number ready. When they pulled it, it hadn't moved.

Accept rate and edit-depth both move early. Only one of them is telling you the truth.

The audit that followed found the real break: a retraining pipeline bug had been silently deduplicating agent edits by an old ticket ID scheme, discarding roughly nine out of every ten logged edits from the last three releases before they ever reached the training set. The model had barely trained on real corrections in nine months. Agents had simply gotten used to a tool that had quietly stopped improving.

All three of these are true at once, and only the middle one is the actual root cause.

Two years earlier, when the pipeline was first built, someone had said, "let's keep the ingestion logic simple, we can always add monitoring once we see how it behaves in practice," and it sounded reasonable, since the whole system was new and nobody had edits to lose yet.

One clock had already rung. Nobody was listening to it, because the other one hadn't rung yet.

Rerun the same nine months with an ingestion audit in place from day one: the dropped-edit bug gets caught at release two, when edit-depth first fails to fall as expected. The retrain pipeline gets fixed within a sprint. Edit-depth resumes its slide, cost per contact never spikes, and Bryndis keeps a tool that's actually still earning the trust she's giving it.

What I'd tell myself, watching a new hire ask the one question the whole room should have been asking for a year: the rule was never wrong to trust agents' own behavior as a signal. It was wrong to trust it alone, with nothing underneath checking whether the thing they were trusting was still true.

LEAD, the signal that rang firstNot a dashboard of everything you could track. LEAD is what tells you which single number to actually watch.

The recap, one line per letter: link is cost per contact staying low as the model genuinely improves, early signal is edit-depth ratio catching a stall three releases before the cost number did, abuse is accept rate quietly rising while nothing underneath actually got better, and decision is a two-release flat reading triggering a mandatory pipeline audit.

And if you want to be sure it really works, try it somewhere elseSame four letters, a fishing fleet instead of a support floor. Different flip family entirely, the same lying accept rate.

Saltmark Fisheries runs an onboard catch-classification tool that sorts a haul by species and estimated weight from a conveyor-belt camera, so deckhands can log the catch faster. Mapped onto LEAD: link is accurate catch logs without slowing the sort. Early signal is the correction rate, how often a deckhand overrides the model's species call, tracked per trip, not the raw percent of hauls confirmed without any override. Abuse is that confirmed-without-override climbed for months as hauls got bigger, not because the model got better at identifying species, but because deckhands started trusting a busy conveyor belt they no longer had time to watch closely. Decision is auditing the correction-logging step itself the moment confirmed-without-override rises faster than haul volume does. The flip underneath it is a scope one, not an over-trust one: as catch volume grew past what one deckhand could review haul by haul, Callum Bardsey stopped checking each fish and started spot-checking one crate in five, and once he did, the model's actual error rate on rare species had no way to reach him at all.

The same five-step loop, wearing a conveyor belt instead of a headset. This time the break is at step two, not step three.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "don't trust the number that measures behavior, trust the one that measures the thing behavior is supposedly responding to," and stop.
Cost: no budget for a full ingestion-audit pipeline this quarter. Say so honestly, and at minimum log a monthly manual sample of ten edits checked against the live training set.
The model actually got better, for real: if edit-depth genuinely falls alongside rising accept rate, that's the flywheel working as designed, and the honest move is to say so plainly rather than assume every good number is hiding a bug.

Where people run it wrong.
They watch acceptance or usage as if it were quality, instead of watching what the model would need to be doing to earn that acceptance.
They treat a flat lagging metric as proof nothing is wrong, instead of checking whether the early signal underneath it has already broken.
They build the logging step and assume it's also the training step, without ever auditing that edits actually cross that gap.

How to use it live. The moment someone asks how you'd know a flywheel is turning, ask yourself: what number would rise even if the model never actually improved, and what number could only rise if it genuinely did? Name the second one, out loud, and the rest of the answer follows.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0