What metrics gate each stage of a phased rollout?

How to answer this, stage by stage

Seven moves. Say what a gate actually protects before the story, or the answer sounds like a checklist instead of a decision.

Let's learn

What number should let a rollout move to its next stage, and what number should stop it?

Say a ride-share company builds a tool that looks at live supply, weather, and local demand, and suggests the fare a rider sees, instead of one fixed surge formula used everywhere.

Before Tarifly, Loxmere's old formula produced a cancel-after-quote rate of 4.2 percent, citywide, and had for years. Teodoro's team set a gate: no stage of the new tool could advance past a citywide cancel-after-quote rate of 5.2 percent, one point above baseline, checked over a full two-week window.

Every stage passed that gate. Five percent of rides, then twenty-five percent, then all of Loxmere. The citywide number sat between 4.3 and 4.6 percent the whole time. By any measure the team was watching, Tarifly worked.

Here's the turn. The citywide number being healthy was never really the open question. The open question was whether every meaningful slice of riders was doing fine, and nobody's gate ever asked that, because the gate only ever looked at the whole city blended into one line.

At its worst, that gap doesn't stay small. Airport pickups are about nine percent of Loxmere's roughly 40,000 weekly rides, and their cancel rate had climbed from 4.0 percent to 21 percent by the time the city hit full rollout, about six hundred extra riders backing out every single week. Multiply that same blind spot across forty more cities, and a problem nobody had named yet was about to ship everywhere at once.

The cost that line was quietly building only became visible once someone finally sliced the complaint numbers the same way.

The gate was telling the truth. So was the rider walking away from the car. Both were true numbers from the same week.

What I'd leave alone. Pre-booked, scheduled rides in Loxmere are a small slice, and every one already gets a human-reviewed price before it's confirmed. A separate gate there would be checking a number that can't move the real outcome even at its worst. The extra gate only earns its place where a slice is big enough, or costly enough, to actually hide inside the average.

The lesson. A rollout doesn't fail because nobody was watching a number. It fails because everyone was watching the one number built to look calm no matter what was happening underneath it.

Now here is the same thing as a story

Use this version when you've got a few minutes. The short version is above. This is for when the decision needs to survive more than one meeting.

Teodoro Vasco had run four rollouts at Farelane before this one, and he was good at the part everyone worries about: writing the gate, picking the cut-off, getting sign-off from finance before a single ride went live. Tarifly was supposed to be the clean one. One clear metric. One clear number. Nothing exotic.

At kickoff, in Farelane's Loxmere office, Teodoro wrote the gate on the whiteboard himself: cancel-after-quote rate, citywide, can't cross 5.2 percent, checked over two full weeks, before any stage moves forward. Everyone nodded. It was the kind of gate you could explain to a VP in one sentence.

The first stage was genuinely good. Five percent of Loxmere's rides went live on Tarifly, and the dashboard Teodoro checked every morning held steady, 4.3, then 4.4 percent. Two weeks later, twenty-five percent. Steady again. Two weeks after that, all of Loxmere. Still steady. Teodoro stopped opening the full dashboard every morning and started glancing at just the top card, the one number, the one that mattered for the gate.

Denby, the second city, cleared the same gate the same clean way. By week fourteen, Teodoro was drafting the brief to take Tarifly to the other forty cities in Farelane's network, all at once. The whole rollout, on paper, had never once come close to tripping its own cut-off.

Then, on a Tuesday two weeks before that brief was due, Nasrin Okada, a data scientist two desks over, messaged Teodoro about something else entirely. She was trying to work out why airport driver supply seemed to shrink every Friday evening, and while pulling ride data by pickup zone, she noticed something odd sitting right next to it: the cancel rate for airport pickups alone was sitting at 34 percent in Loxmere and 29 percent in Denby. Nowhere near the citywide gate. Nowhere near anything anyone had written a cut-off for.

Teodoro pulled the full history that afternoon. Airport pickups, about nine percent of Loxmere's rides, had gone from a 4 percent cancel rate to 21 percent over the eight weeks of that city's rollout, and kept climbing after. Roughly six hundred riders a week, quietly backing out at the airport curb, the entire time the citywide gate kept reporting green.

Two months earlier, at that same kickoff whiteboard, nobody had asked the question that would have caught this. "Should we check any slice of the city separately?" Teodoro doesn't remember anyone raising it, including himself. One number, citywide, was simple, and simple felt like the responsible choice.

I would take that back. I'd have said, in that same kickoff meeting: "before we write one cut-off, let's name the two or three slices of this city that are big enough, or expensive enough, to hide a real problem inside the average. Airport pickups are one. Late-night rides might be another. Each of those gets its own number, checked at every stage, not just the citywide one."

Here's the replay. Same rollout, same 5.2 percent citywide cut-off, but this time airport pickups also carry their own gate, an 8 percent cancel-rate cut-off, checked at every stage. Week four, five percent live: airport sits at 7 percent, under the line, stage clears. Week six, twenty-five percent live: airport hits 15 percent. The segment gate trips. Loxmere's rollout pauses there, two stages before it ever reaches full city, while the team finds out why: Tarifly had been borrowing its demand pattern for thin-volume zones like the airport from downtown's much busier one, and downtown's pattern pushed airport fares too high, too often. Two weeks to retrain that piece, and the rollout resumes with the fix already in place, in Loxmere, in Denby, and in every city after.

One design let a healthy average stand in for the whole rollout. The other made every slice earn its own answer before the next stage got to start.

And the thing I'd tell myself, back at that whiteboard: "one clear number" isn't simplicity. It's agreeing in advance not to look at anything smaller than the whole city.

LEAD, so a green gate can't hide a red slice underneath it

This sounds like a question about picking the right metric. Underneath, it's still asking which number you're willing to be fooled by. That's LEAD, run on a rollout's own gate instead of a dashboard.

And if you want to be sure it really works, try it somewhere else

Halveston Pharmacy Group runs about a hundred and twenty branches. Amadika Ferrows leads pharmacy operations there, rolling out a tool that flags risky drug combinations before a pharmacist fills a prescription, branch cluster by branch cluster.

Once Ferrows split the override rate by drug class and branch type, the anticoagulant-heavy branches stood out immediately. The network-wide number, the only one anyone had a cap written for, had never once suggested there was a problem.

The segment number was ready to ring in week five. The network average was still sitting quiet in week fifteen, telling the truth about everything except the part that mattered.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0