Design the rollout plan for an AI feature going to two million users.

How to answer this, stage by stage

Seven moves. This is a case question at a size most interview stories never reach, so the hard part is saying what changes with size, not what stays the same. Each stage has the words you'd actually say.

Let's learn

Here is what happens when a system that always worked meets exactly the group it was never tested on.

Say an airline builds a helper that lives in its app. When your flight gets cancelled or badly delayed, it looks at what's left and proposes a new flight for you, right there on your phone.

The number that barely moved

Before the assistant, a cancelled flight meant calling the airline or finding a gate agent. Selin, who flies for work about twice a month, once waited forty minutes on hold for a rebooking, twice in the same year. With the assistant working the way it was built to, that same cancelled flight turns into a new boarding pass in about ninety seconds. Tap the card, done.

The airline ran this with a pilot group first: forty thousand of its most frequent flyers, people who opted in, for seven months. Then it went to general availability, all two million app users, in three widening phases.

Here's what the pilot group never showed anyone. Frequent flyers who fly for work mostly fly alone. The forty thousand people in the pilot were the airline's most reliable, most solo travelers. General availability is a different crowd entirely: families, couples, people flying with their kids or their parents.

At its worst, that's worse than never building the tool. A tired gate agent, calling out two names at the counter, pauses and asks, "are you two traveling together?" The assistant never asks. It answers in ninety seconds, and the answer can be wrong in a way a person would have caught on instinct, without even trying.

What I would leave alone. Solo travelers with a normal connection are exactly the case the pilot tested, ninety one times out of a hundred. The one-tap Accept stays for them. Slowing that down to protect against a case that almost never touches them would cost millions of people time for nothing.

The lesson. We tested the assistant on the group most likely to forgive it, and least likely to ever need it to think about anyone but themselves. That group proved the model works. It never had the chance to show us who the model was missing.

Now here is the same thing as a story

Use this version when you have room to let it land, not just list it.

The rebooking card lives in an app Selin already had open, one tap behind the boarding pass she pulls up forty times a year. She's a marketing director for a regional distributor, and she flies for client visits about twice a month, the kind of traveler an airline builds a whole loyalty tier around.

She joined the pilot in March. The first few times a flight got cancelled, she read the whole card: her old flight on the left, the proposed new one on the right, gate and time and seat all lined up so she could check it before tapping "Use this." It was always right. By June she was skimming the card, glancing at the new departure time and tapping through. By August she wasn't opening it at all. The notification would land, and her thumb would tap Accept before she'd even read the airline's name on it.

Competent alone, the good months, the habit thinning, the trigger

It had never once been wrong. Not in five months, not in nine cancelled flights. So why would this one be different.

In October, general availability finished rolling out, and Selin's mother flew with her for the first time, for her niece's wedding in Portland. Chicago to Denver to Portland, connecting through a hub that gets thunderstorms most October afternoons.

Thursday, 4:50pm. A ground stop at Denver. Their connecting flight, and about six thousand other people's, cancelled at once.

The notification landed the way it always did. Selin's thumb was already moving.

The assistant had found the fastest way to get one traveler named Selin Aydin to Portland that evening. It was a good answer, for one traveler. It put her on a 6:40 connection through Salt Lake City. It put her mother, booked separately in the same reservation but not read as "this person is with her," on the next open seat anywhere: a 9:15 through Phoenix, landing three hours later, alone, at a gate her mother had never stood at before.

People are switches, not dials

I want to say the problem was one wrong booking. It wasn't, not really. Selin never had a percentage in her head. She had a feeling, and the feeling only had two settings: read the card, or trust the button. There was no setting in between, no "skim it a little more carefully this time." Once the tool had been right for seven months straight, the only lever left was whether she opened the card at all.

So here's the decision I'd take back. In July, once pilot satisfaction passed ninety eight percent with almost no complaints, someone in a planning meeting pointed out that the comparison card felt like a leftover from when the model needed watching. Nobody had asked for a mandatory extra tap on something that was reliably right. It looked like friction with no upside. So the default changed: one big Accept button, the comparison tucked under a small "view details" link almost nobody tapped.

That was a sensible call in July. A comparison card built for a model that still needed checking is, in fact, clutter once the model stops needing checking. It stopped being sensible the day two million people, most of them not traveling alone, started tapping that same button.

I'd put the comparison back, not for everyone, just for the itineraries the pilot group never had: more than one traveler on the reservation, or a connection under the minimum time. If Selin's booking gets flagged the moment it includes her mother, she sees the two options side by side, taps the one that keeps them on the same connection, and they land in Portland together by 6:40, not three hours apart in two different airports.

That's the whole difference. One design hands two million people a single button and asks them to trust it the way forty thousand hand-picked people learned to. The other hands the ninety one percent who fly alone the same fast button, and gives the other nine percent one honest look before it moves them.

And the part I'd tell myself, if I could go back: we measured whether the flight the assistant picked was a good flight. We never asked whether it was picking for one traveler when two were standing at the same gate.

The five steps, sized for two million people

This is a Perturbation question wearing a design question's clothes: something changed size, not shape, and FLIPS runs straight down the line, F to S.

FLIPS, in five rows

A small move in the model. A big snap in what she does with her thumb.

And if you want to be sure it really works, try it somewhere else

A hospital's chest X-ray tool flags scans by severity for a radiologist's second read. Same question shape, a different product, a different flip entirely.

Only one letter changes

F. Kenji Sato, staff radiologist at a mid-size hospital, eleven years reading chest films, about ninety a shift.
L. He stopped writing down his own first impression before checking the tool's severity score. The two agreed often enough that his own note started to feel like paperwork.
I. A different flip. He doesn't check less or delegate the read. He starts keeping a private spreadsheet of patients he's personally worried about, cross-checking each new flagged film against his own notes by hand, off the record. No middle setting: either the hospital's system tracks a patient over time, or Kenji does it himself on paper.
P. We built the tool to score each film on its own, with no link back to the same patient's earlier scans, because at launch it only had to handle a single read with no history to draw on.
S. Give the tool a patient's last three scans, and let it flag a small nodule that's individually unremarkable each time but trending up across visits. Kenji doesn't need his own spreadsheet. A hospital-wide chart audit finds the same slow-growing cases even for doctors who never built a private system at all.

Flashcards (click a card to flip it)

Eight fixed slots, pulled straight from the answer above.

Check yourself Score: 0 / 0