Explain why thumbs-down rate is a lagging indicator disguised as a leading one.

A number that reacts the second someone taps it feels like the fastest thing on the dashboard. It can still be the slowest thing to tell you the truth.

How to answer this, stage by stage

Nobody is grading whether you can spot that a metric is misleading. They're grading whether you can show your work getting there, one check at a time, instead of just announcing the twist. Seven moves.

Let's learn

What happens when the number a team built to catch bad answers fast turns out to be the slowest number on the whole dashboard?

ScrapChef is an app inside Coalridge. Point your phone at what's left in the fridge, and it suggests a recipe, with a swap for anything you're missing, out of buttermilk, use milk and a spoon of lemon juice.

Before the swap, checking a substitution used to mean stopping mid-recipe to look it up online, a couple of extra minutes hunting on a phone with sticky hands. ScrapChef cut that to zero. The suggestion just appeared, and thumbs-down rate sat around two percent, an honest number, because the model was good enough that almost nobody needed to push back.

Ten weeks ago, Coalridge swapped in a cheaper model for generating substitutions, about a third the cost per call, a real saving at their volume. Thumbs-down rate today reads two point six percent. Basically the same as before.

Here is the turn. The flat line is not proof nothing changed. It is proof the number was never built to catch what happens when a suggestion goes wrong mid-recipe. Most cooks don't click anything. They wing it, or they quietly close the app and order takeout, and neither one shows up as a thumbs-down.

At its worst, this costs the exact users a subscription app can't afford to lose. Seven-day return rate for cooks who hit a bad suggestion in the swapped-model group fell from sixty four percent to forty one, and not one of them left a review or tapped a thing. The dashboard stayed green the whole time.

The choice I would take back is that gate. It made sense while everyone believed thumbs-down was a real-time gauge, a click, fired instantly, right there under the recipe. Nobody had worked out that a real-time click and a reliable one are not the same thing.

What I would leave alone: for a low-stakes swap, like table salt for kosher salt, thumbs-down really is a fine, honest signal. Nobody gets that wrong even distracted at the stove, so a flat near-zero rate there genuinely does mean fine. Not every metric here needs a behavior proxy standing next to it.

The lesson: a number that's easy to click isn't the same as a number that's easy to catch. The three steps between a bad answer and a thumbs-down are exactly the three steps most people skip when they're standing at a stove with wet hands and a timer running.

Now here is the same thing as a story

The short version is above. Read on for the Tuesday sync where a one-line remark turned into a two-week audit.

Cosmin Fassbinder has run recipe quality at Coalridge for three years. He built the golden set the old substitution model was checked against, four hundred pantry combinations, hand-scored by a food scientist named Reya who used to test recipes for a cookbook publisher. He trusts numbers. He also trusts Reya more than any dashboard.

When the cheaper substitution model shipped ten weeks ago, Cosmin watched thumbs-down rate the way you'd watch a smoke alarm, not because he expected it to go off, but because if it stayed quiet, that meant the room was fine. Two point three percent the first week. Two point four the second. He stopped checking it daily around week four. It kept being fine, so he had other things to look at.

Three separate ways a genuinely bad suggestion can happen and still never reach the dashboard as a click.

The remark came on a Tuesday, in a weekly sync, from Zinaida Trevino, who runs support. She wasn't presenting anything. She just said it, almost as an aside, on her way to the next agenda item: "App Store reviews mentioning 'wasted ingredients' or 'recipe didn't work' have nearly tripled this quarter. Nobody's filing a support ticket, they're just leaving a star rating and moving on." Then she moved on too, to a question about return labels.

Cosmin almost let it go. The dashboard was green. Thumbs-down rate had not moved. But he pulled a sample that afternoon anyway, four hundred sessions from the swapped-model group, and had Reya score them against the same golden set she'd used for the old model.

Two hundred and thirty two of the four hundred suggestions were genuinely bad, ratios that didn't work, pairings that ruined a dish. That's fifty eight percent, roughly a quarter of all sessions overall. Cosmin pulled the session recordings for those two hundred and thirty two. Most cooks didn't stop. They adjusted on the fly, tasted, added more of something, or just plated whatever came out and moved on with their evening. A handful closed the app outright, mid-recipe, and never came back that night. Eighteen tapped the thumbs-down button. Eighteen, out of two hundred and thirty two confirmed-bad suggestions.

He ran the return-rate check the same afternoon, comparing that group against a small control group Coalridge had quietly kept on the old model for exactly this kind of comparison. Sixty four percent of the control group opened the app again within seven days. Forty one percent of the swapped-model group did. That gap had been sitting there the entire ten weeks, visible the moment anyone thought to look at it next to thumbs-down instead of looking at thumbs-down alone.

It was never that the model got a little worse and nobody minded. It was that almost nobody who minded ever told the dashboard.

A year before any of this, when the thumbs-down button was first designed, the team debated where to put it, prominent under every step, or tucked under the finished card so it didn't clutter a recipe someone was actively cooking from. They chose tucked away, a fair call at the time, because the model was good enough that almost nobody needed it. That call was never revisited once the model changed underneath it.

Run the same ten weeks through the fixed design. The model swap still ships, same savings. But it ships behind Reya's golden set first, which would have caught the ratio problem in a two-day check instead of a ten-week silence, and return rate rides alongside thumbs-down on the same dashboard from day one, not discovered after a hallway remark. The gap between sixty four and forty one gets caught in week one, not week ten, and Coalridge fixes the substitution logic before a single subscriber quietly stops coming back.

What I would tell myself, before any of this: a metric that reacts instantly to a click is not the same as a metric that reacts quickly to a problem. Those are two different kinds of fast, and only one of them is the one you actually need.

TRACE, and the letter that actually cracked it

This is a diagnosis question dressed as a critique. TRACE is built for exactly this: rule out the boring answer, then narrow to the real one, one letter at a time.

Two things worth naming directly, since this is where the real judgment sits. First, the rejected fix: make the thumbs-down button bigger and easier to find. Turned down on purpose, that only widens the "find the button" filter, leaves "notice" and "care" untouched, and would inflate the click count for reasons unrelated to quality, muddying the exact signal you're trying to clean up. Second, the AI-specific failure worth naming by name: a cheaper model swap can quietly change behavior on the long tail of pantry combinations it saw less of in training, a form of distribution shift the golden set exists to catch before launch, not after. The guardrail is that same golden set, a batch of hand-checked substitutions a food scientist scores, required to clear a near-total pass rate before any model swap ships broadly, with a behavior-proxy dashboard running in parallel for the first few weeks regardless of what thumbs-down says. That guardrail has a real cost: the cheaper model still saves roughly two-thirds of the cost per suggestion, but the eval-and-monitor step adds about two weeks before the savings reach every user. That delay is the price of not finding out the hard way, from a support lead's aside in a Tuesday meeting, that a quiet number was never quiet because things were fine.

And if you want to be sure it really works, try it somewhere else

Same five letters, a completely different job, a technician's "not helpful" tap instead of a cook's thumbs-down, and this time the pressure comes from speed, not cost.

Larchdon runs FixFlow, a tool that points an HVAC technician's phone at a unit and an error code and suggests a diagnostic step. Idony Northrup runs product for it.

T, timeline. Two months ago, Larchdon shortened FixFlow's prompt template to cut response time from six seconds to two, so a technician isn't standing there waiting with a wrench in hand. "Not helpful" rate has stayed under one percent the whole time.
R, recut. A technician has to notice the fix was wrong, sometimes not until a callback the next week. Then care enough to log it instead of just moving to the next job, techs are paid per job, not per feedback. Then find the icon, buried behind a settings menu on a rugged tablet, gloves on.
A, assume nothing. Job volume and tablet sessions stayed flat over the same two months, ruling out "fewer sessions" as the reason the rate looks steady.
C, cause candidates. Silent workaround, techs diagnose it themselves and never reopen the tablet for that job. Trust erosion, senior techs skip the suggestion entirely and call a colleague instead. A UI regression from the last app update that moved the icon behind a menu, suppressing the click independent of the fix's quality.
E, evidence test. Compare "not helpful" rate against fourteen-day callback rate, same technicians, shortened-prompt group against a control still getting the longer prompt.

Same three filters as ScrapChef's cooks, a different tool, a different job, gloves instead of flour.

Swap the trigger and it still runs.
Speed: an interviewer caps the answer at ninety seconds. Skip straight to the recut: whatever "negative feedback" number you're watching, ask what three things a person has to do before that click fires, then check whether most of your users clear all three.
Cost: the behavior-proxy check can't ship for a month. Don't read the raw negative-feedback rate alone as a stopgap in the meantime, hold off on trusting any "still healthy" read from it until the real check exists.
The model got better, for real: say FixFlow's suggestions genuinely improve next quarter. That still isn't proof the "not helpful" rate would have caught it if they hadn't. A better model just makes the next quiet regression harder to spot, because the same three filters are still standing there.

Where people run it wrong.
They treat a flat, low rate as proof of health, on reflex, without asking what a person had to do to produce it.
They respond to a hunch that something's off by watching the same metric harder, instead of building a second one that doesn't depend on a click at all.
They wait for a support lead's aside in a meeting to go looking, instead of running the behavior-proxy comparison on a fixed schedule whether anyone remembers to ask or not.

How to use it live. Say the reframe before naming a single fix: "A click-based number can only be as fast as the person clicking it, and most people never clear all three things it takes to click it at all." That buys you room to give the real answer instead of reaching for "it's a lagging indicator" as a label with nothing under it.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0