Tell me the three numbers you would check every morning as an AI PM.

Not a wish list. A method for landing on exactly three morning numbers, and a way to prove they're the right three.

How to answer this, stage by stage

Nobody's grading whether you can name three metrics that sound sensible. They're grading whether you can say, out loud, why it's three and not two, and why it's three and not ten. Eight moves get you there.

Let's learn

What happens when the one number on your morning dashboard stays perfectly calm while something real is already going wrong underneath it?

Provendr is a grocery delivery app. When something you ordered runs out, its AI model quietly picks a replacement and ships your order, instead of texting you to ask and making you wait.

Before that swap model existed, an out-of-stock item meant one of two things: the picker left it out of your bag, or someone texted you and you sat there waiting. That wait added close to nine minutes to the average order, and it was the single biggest reason people gave up on the app partway through checkout. The swap model fixed that inside its first month.

Adeline Corradini runs growth analytics at Provendr. She built the morning dashboard back when the company ran one warehouse and did a few hundred orders a day. The number she checks first each morning is substitution reversal rate, the share of AI-picked swaps a customer sends back or asks refunded within a day. For a year it sat near 3 percent. Steady. Boring. Exactly what a healthy number is supposed to look like.

At its worst, a number like this can sit still for weeks while something real breaks underneath it, because a blended average has no way to point at the one piece of itself that's actually cracking. The whole company looks fine. One slice of it isn't.

What I would leave alone: the blended number itself isn't wrong to keep. It's still the right single line for a monthly update, "reversal rate held under 4 percent, company-wide." Leave it there. The mistake was making it the only number anyone reacted to at 8am.

The lesson: an average can only tell you how the whole company is doing. It can't tell you which one piece of the company is about to cause a problem. Those are two different questions, and one blended number can only ever answer the first.

Now here is the same thing as a story

The short version sits above. Read on for the Tuesday a single one-star review made Adeline pull nine weeks of logs by hand.

Adeline has run Provendr's growth numbers since the company had one warehouse and forty employees. She built the substitution reversal metric herself, back when it was genuinely the only number that mattered, and for most of three years it did exactly its job. It sat near 3 percent, and on the two occasions something really broke, it moved.

Provendr opened its fourth warehouse that spring, in a city two states over, to cut delivery times for a whole new stretch of customers. The launch went well. Orders from the new warehouse grew from a trickle to a real slice of the business inside two months, and nobody on the growth team gave it a second thought, because the one number that mattered kept doing what it always did. It sat still.

The habit that had quietly held the dashboard together thinned in three beats nobody noticed at the time. First, Adeline had once spot-checked twenty or so substitution logs by hand every week, back when Provendr was small enough that felt necessary. She'd stopped, because the automated dashboard felt reliable enough once volume grew. Second, a plan to add a warehouse-level cut to the dashboard got pushed twice in planning, since the top-line number never gave anyone urgency to build it. Third, the new warehouse's product catalog had been loaded fast to hit its launch date, and the substitution model's candidate list for that warehouse's shelf was thinner than the older ones. Nothing on the dashboard measured that directly, so nobody flagged it.

The trigger wasn't a crisis. It was one line in a weekly digest of app reviews: a parent said their kid's lunch order arrived with a peanut granola bar swapped in for an oat one, and their family avoids peanuts. One star. Sitting between a complaint about a late delivery and someone unhappy about a missing eggplant.

Adeline didn't have a warehouse cut to check, so she couldn't just look. She spent most of a day pulling nine weeks of raw substitution logs and cutting them herself, warehouse by warehouse, order by order, something a real dashboard should have shown her in one glance.

What she found, once she cut the numbers by warehouse instead of blending them: the new warehouse's own reversal rate had climbed from about 5 percent in its first week to 22 percent by week eight, the week the review landed. The company-wide blended number, over that same stretch, moved from 3.0 percent to 3.4 percent. A number small enough that nobody watching it would ever have called it a warning.

Before trusting any of it, Adeline checked whether the "swap reversed" event had changed its definition or started firing twice around the launch date. It hadn't. The drop in quality was real, and it belonged to one warehouse.

The old decision that set this up went back to a planning meeting in Provendr's first year, when there was exactly one warehouse and cutting the reversal number by warehouse would have meant cutting it into one piece. Someone said, reasonably, that the blended number was fine as is. Nobody put a date on revisiting that call. Nobody did, through three more warehouses.

What made it worse, and what a segment cut alone wouldn't have explained: the new warehouse's orders were also getting more expensive to serve. Its thin candidate list meant the substitution model couldn't find a confident match on the first pass as often, so it fell back to a slower, pricier re-ranking call that checked more candidates before picking one. Cost per order at that warehouse crept from about half a cent to almost a cent and a half. Small money. But it moved for the exact same reason the quality number moved, and it moved earlier.

Run the same eight weeks again with the three-number dashboard I'd build instead. The new-warehouse segment cut crosses double its normal reversal range by week three. The fallback cost for that warehouse ticks up the same week. The warehouse's share of volume crosses 5 percent around the same point. Three numbers, agreeing in the same week, which is enough to open a look before a single customer has to complain. The catalog gap gets patched by week four. Five weeks of exposure become one.

One dashboard could only tell her the company was fine. The other could have told her which single week to worry about.

What I would tell myself, back in that first-year meeting: the day you build a metric that only exists as one blended number, write down who comes back to cut it once there's something worth cutting it by. Someone has to remember, on purpose, or nobody does.

The method behind picking exactly three

This isn't a diagnosis of a single incident. It's TRACE run on the question of which numbers deserve a permanent spot on the dashboard, using one real incident as the ruler.

Plotted against the same two questions this whole method asks: does it warn you early, and can you actually read it in one glance. The three chosen numbers cluster together. The rejected ones don't.

Three things worth naming plainly, since this is where the real judgment sits. The rejected alternative that mattered most was model eval accuracy, the substitution model's own offline test score. It's a real, trustworthy-sounding number, and it stayed flat through the entire episode, because it's measured in a lab against a fixed test set, not against a live warehouse's actual shelf. A number that would have stayed calm through the real failure is worse than no number at all, it's false reassurance with a decimal point on it. The AI-specific failure worth naming by name is a form of distribution shift: the substitution model's candidate pool for the new warehouse didn't match the pool it was trained and tuned against, a kind of cold start that a company-wide accuracy score can't see because it never isolates one warehouse's traffic. The guardrail is concrete: route a newly onboarded warehouse's low-confidence swaps to a stricter automatic threshold, holding some of them for a person to confirm, until that warehouse has enough live outcomes for its own segment-cut reversal rate to be trusted on its own. There's a real trade-off behind that guardrail, not a free lunch: holding new-warehouse swaps for confirmation is slower to launch and costs a few cents more per order in the first weeks than letting the model decide alone every time. Provendr took that trade on purpose, a slower and slightly pricier first month for a new warehouse, against another allergy reaching someone's kitchen. And the bar for treating a segment number as a real signal, not noise, isn't a single bad morning. It's a segment's reversal rate sitting more than eight points above its own four-week trailing average, for two straight days, before anyone gets paged.

And if you want to be sure it really works, try it somewhere else

Same five letters, a long-haul trucking company instead of a grocery app, and the same shape shows up with no swap model anywhere in sight.

Truvane Freight runs an AI system that scores every truck each morning for how likely it is to break down, before dispatch sends it out. Wendeline Yancey runs fleet operations there.

T, timeline. Truvane acquired a smaller regional carrier's fleet in month one, older trucks running different sensor firmware, folded into a new depot the team called Depot 9. The one dashboard number, post-clearance breakdown rate, the share of AI-cleared trucks that break down within fourteen days anyway, held near 1.1 to 1.3 percent company-wide the whole time. Depot 9's own trucks were breaking down far more, invisibly, inside that blend.
R, recut. Breakdown rate, cut by depot instead of fleet-wide. Depot 9's segment climbed from 3 to 17 percent over six weeks, while the blended number moved from 1.1 to 1.3.
A, assume nothing. Wendeline's team checked first whether the "breakdown" event had changed its definition after the acquisition. It hadn't. Then, the harder check: the one number they'd always watched, blended breakdown rate, genuinely wasn't enough on its own, because it carried no signal about Depot 9's different sensor calibration.
C, cause candidates. Chosen: Depot 9 segment-cut breakdown rate (quality), fallback telemetry reconciliation cost per truck scored (cost), Depot 9's share of the total fleet (mix). Rejected: total trucks scored, which only grows. Driver satisfaction survey, which lags by a full quarter.
E, evidence test. Back-tested against the actual highway breakdown that stranded a shipment: Depot 9's segment breakdown rate would have crossed its own trailing baseline by week two. Depot 9's fleet share crossed 8 percent by week three. Fallback reconciliation cost per truck at Depot 9 nearly tripled by week two. All three moved in the same short window, weeks before the truck that actually failed on the highway.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to the three, and say the count matters as much as the names: one segment-cut quality number, one per-unit cost number, one mix number, no more.
Cost: engineering says a real per-depot cost pipeline is eight weeks out. Don't wait on it as an excuse to fall back to one blended number, pull the fallback rate from raw logs by hand once a week until the pipeline ships.
The model got better, for real: say the fleet-wide clearance model's accuracy genuinely improved that quarter. That's still not the same claim as "every depot is fine." A model that gets more accurate on average can still be badly wrong for one segment it wasn't trained on, and the blended number will hide that the whole time either way.

Where people run it wrong.
They add the segment cut, then still only glance at the blended tile out of habit, because that's the one they've checked for years.
They see the blended number holding steady and call that proof the acquisition went fine, instead of asking whether it's holding steady only because the new depot is still small.
They respond to a bad segment number by adding a fourth or fifth metric instead of asking whether the real fix is a guardrail on the new segment, not a new number to watch.

How to use it live. Open with the count, not the names: "I'd watch exactly three numbers, one for quality, one for cost, one for mix, and here's why three and not two." That buys you the room to defend the count on its own terms before the interviewer can push you toward a longer list.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0