What is the difference between rolling out a new feature and rolling out a quality improvement?

How to answer this, stage by stage

Seven moves. The number that matters shows up in stage five, and the whole answer turns on whether a dashboard was ever built to answer this launch's question.

Let's learn

What happens when a launch has nothing on screen for anyone to react to? Mossgate Market is an online marketplace, and its search bar runs on an AI model that sorts millions of listings down to the ten or so a shopper actually sees.

For over a year, every launch that went through the search team's dashboard had a button behind it. A camera icon that opened visual search. A bell for saved searches. Stage it to 5 percent, then 25, then everyone, and watch two numbers: how many shoppers used the new thing, and whether they stuck around longer because of it. The last one, a "Find Similar" button, got tapped by 18 percent of shoppers who saw it, and their sessions ran 42 seconds longer. Clean win. Shipped in five days.

Then the team fixed something nobody could see. About one search in nine on Mossgate turns up nothing good, usually a misspelled brand, an odd size, or a search typed the way a person actually talks and not the way a catalog is labeled. The ranking team shipped a fix for exactly that. Same staged rollout. Same two-week dashboard read. And at the two-week mark, the numbers hadn't moved. Click-through sat at 34.1 percent before, 34.3 percent after, small enough to be noise.

So somebody finally ran one. They pulled the 500 real searches the ranking team had flagged as their hardest, the ones that used to come back with nothing good, and ran every one through the old model and the new model.

Here's the turn. A flat dashboard is not the problem. The problem is what the team almost did about it: read the flat line the same way they'd read a failed button, and scheduled a Thursday meeting to roll the fix back.

The model's own quality kept climbing quietly. Whether anyone checked it directly only ever had two settings.

At its worst, this costs more than one bad meeting. Roll the fix back, and the marketplace goes right back to showing shoppers nothing good for the exact searches that were already driving the most frustration. And the team learns the wrong lesson from it: that fixing the model quietly doesn't pay off, so stop trying.

Whether a launch's proof method fits isn't a dial you read off a clean-looking number. It's a switch with two positions, and only one of them was ever built for this launch.

Same 500 hard searches, two models. One grid is mostly a miss. The other mostly isn't.

The choice I would take back. Eighteen months earlier, when the launch-readout dashboard was first built, someone actually asked whether launches with nothing to click would need a different kind of proof. The answer in the room: every launch we've ever shipped has had a button in front of it, build one dashboard, keep it simple. Reasonable, because at the time it was true.

What I would leave alone. For the "Find Similar" button, and anything else with a badge or a new screen, the adoption dashboard is exactly the right tool. A shopper tapping something new is precisely the reaction it was built to catch. Nothing about a visible feature needed to change.

The lesson. We treated "rollout" as one word for one process. It isn't. A launch someone can see and a launch where only the model's output changed need two different questions, asked of two different kinds of data.

Now here is the same thing as a story

Pull this one out when there's more time, and you want the interviewer to feel the gap, not just note it down.

Every Monday, Dalia Vega opens the same three tabs before her coffee's done: the launch dashboard, the query logs, and a spreadsheet she's kept since her first month running search launches at Mossgate Market. Three years in, she can look at a launch's day-three curve and tell you by lunch whether it's healthy or quietly dying.

The "Find Similar" button arrived on a Tuesday in April. Dalia staged it the way she staged everything, 5 percent, then 25, then everyone, over five days, and watched two numbers climb: how many shoppers tapped it, and how much longer they stuck around after. Eighteen percent tapped it. Forty-two seconds longer. She pulled ten real sessions that first week anyway, just to see the tapping with her own eyes. They matched the dashboard exactly.

The next launch matched too. So did the one after that. By the sixth clean launch in a row, she'd stopped pulling sessions before she even opened the dashboard. Why would she. It kept being right.

Then came the ranking fix.

Nothing about the launch looked different going in. Same staging, five days to everyone. Same two-week readout on her calendar, the way it always was. On day fourteen, she opened the dashboard the way she'd opened forty of them before: click-through, 34.1 percent before, 34.3 after. Flat. Sessions, flat. Nothing to see.

She'd seen this exact shape before, and it always meant one thing. A launch that didn't move the numbers was a launch that didn't work. She put a rollback meeting on the calendar for Thursday.

It was the newest person on her team, three weeks into the job, who asked the question at Tuesday standup. "Wait, if the search results just quietly got better, would anybody actually click differently because of it? What would that even look like on the dashboard?" Dalia didn't have an answer. Standing there, she realized she hadn't had one in a while.

That afternoon she pulled the 500 real searches the ranking team had flagged as their hardest, and ran every one through the old model and the new model, side by side. The old model found a real match for 220 of them. The new one found 355.

I want to say the dashboard was wrong. It wasn't. Click-through really did sit at 34.3 percent, and eighteen months earlier, when Dalia's predecessor built that dashboard, every launch really did have a button behind it. Dalia never had a number telling her the ranking fix needed a different kind of proof. She had a habit, and by launch six the habit only had one setting left: trust the dashboard, whatever it says. Six clean launches in a row had switched it off. Nothing was going to switch it back on by itself.

So here is the decision I would take back.

Eighteen months earlier, in the meeting where that dashboard first got built, someone actually asked whether launches with nothing to click would need a second kind of proof. The room's answer was reasonable at the time: every launch we've shipped has had a button in front of it, keep one dashboard, keep it simple. Nobody voted to make that permanent. It just never came up again, because for eighteen months, it never needed to.

I would put a second gate on that same checklist: before any launch with nothing new on screen clears its first stage, run the old model and the new model on the same real inputs and grade the difference directly. Run the identical ranking fix again with that gate in place. On day one, at 5 percent, the paired comparison already shows 220 becoming 355. Dalia doesn't wait for a two-week dashboard that was never going to move. The fix ramps to everyone by day five. No Thursday meeting ever gets scheduled, and the newest person on the team never has to be the one who asks the question that should already have been on the checklist.

If I'm honest, trusting the dashboard on launch six wasn't the mistake. Six clean launches in a row will talk anyone out of pulling ten sessions by hand. The mistake was eighteen months earlier, the day nobody wrote down what that dashboard could actually prove, so that on the one launch where it mattered most, nobody in the room could point to the sentence that said: this only tells you whether someone reacted, not whether the model got better.

The five letters, run against Dalia's almost-rollback

The letters matter less than which one breaks first. Here's the same five steps, mapped onto the week Dalia nearly killed a real win.

FLIPS, five rows

"They should have watched the numbers more carefully" is a diagnosis anyone can offer after the fact. The harder part is naming the exact question that only ever got asked once, and showing there was no cheaper fix once a flat readout had already scared the room.

And if you want to be sure it really works, try it somewhere else

Redthorn Insurance is nowhere near a marketplace search bar. It processes auto and home claims, and its AI reads photos of damage and flags claims worth a second look for possible fraud. Same question, a different flip this time. Nobody stops sanity-checking a dashboard. A senior fully hands a judgment call down to junior staff, and stops checking their work against what actually happened.

F. Esteban Rivas, team lead over the flagged-claims queue at Redthorn Insurance, three years reading a week of junior decisions and telling which one needs a second look.
L. He used to personally re-check a sample of every junior's flagged-claim calls against the claim's real, confirmed outcome, every week. Two launches in a row, the team's "resolved without escalation" rate held above 90 percent and matched what his own checks found, so he let the weekly check lapse.
I. A different flip from Dalia's. He doesn't stop trusting a dashboard, he stops personally auditing a judgment he'd already handed all the way down. Authority for flagged-claim quality moves from a senior checks it against real outcomes to a junior handles it, unaudited, because the resolved-without-escalation number says the queue is fine.
P. When that number held steady through two straight launches, the team decided it alone was proof of a healthy queue, and never built a second check for the one launch where nothing on that dashboard would move: an invisible fix to the fraud model's own precision.
S. Require the audit-against-outcomes to run on a fixed sample before any invisible model change clears its first stage. Run the identical precision fix again with that gate in place. The false-positive rate on legitimate claims shows its real drop, 6.2 percent to 3.1 percent, in the first days of the audit sample instead of a compliance review finding it five months later. Esteban's Monday spot-check habit, the one that had lapsed, comes back by design instead of by accident.

Flashcards (click a card to flip it)

Eight fixed slots, pulled straight from the answer above.

Check yourself Score: 0 / 0