ConceptIntermediateShipping & Model Lifecycle / Rollout strategy and phased launches / #3
Explain the difference between a feature flag rollout and a model rollout.
The direct answer
A feature flag turns a piece of code on for some people while the code underneath stays exactly the same. A model rollout swaps which model is actually doing the writing, so the output itself changes for whoever gets it. Treat any change to the model behind an AI rewrite tool as a model rollout, never as a feature flag, even when it ships behind the same on and off switch: compare it against the current model on real input first, and build the rollback around what it already wrote, not just around the switch position.
How to tell the two apart, in order
Classify the change by what actually flips, not by which switch ships it.Why: a feature flag turns code on for some people; a model rollout changes what the model writes for them. The same switch can hide either one.
Shadow test any model swap against the current model on real input before it writes to anything real.Why: this is the only way to know how big the change is before your users find out for you.
Build the rollback plan around what the model already produced, not just the switch position.Why: turning a flag off stops the next mistake. It does not undo the ones already saved or sent.
Keep the fast, light rollout for changes that never touch model output.Why: a new button or preset is genuinely safe, and slowing it down protects nobody.
Set the kill line before the rollout starts, as a real number.Why: without one, "it's basically the same as last time" quietly becomes the whole justification.
Watch for the kind of damage a crash dashboard cannot see.Why: a changed voice does not throw an error, so the usual graphs stay green while the real thing goes wrong.
How to answer this, stage by stage
This is a yes or no about which kind of change one release actually is, not a rule for every AI feature, so PICK carries the answer.
1
Scope it to one concrete decision
Say it like this
"Let's make this real. Say Penbrook is a note-taking app, and inside it there's a Tone Rewriter: pick a tone, Crisp, Warm, or Formal, and it rewrites your note in that voice. The team's about to ship two changes in the same release, and I need to work out which kind of rollout each one actually is."
Why this works
Stops the answer floating at "it depends on the change" and gives the interviewer one real switch to push on.
2
Say your structure out loud
Say it like this
"I'll pick a position first, then say who gets hurt on each side if I call it wrong, then name which kind of miss actually costs more, then say what evidence would flip my answer. That's PICK, and I'll go in that order."
Why this works
Shows a method instead of a ramble, and tells the interviewer what's coming before you start.
3
Reframe what the question is really testing
Say it like this
"This isn't really about how big the change looks on the roadmap. It's one question: does this switch change what the model decides to write, or does it just decide who sees a button. That's the whole split."
Why this works
Shows the interviewer you see past the surface question to the judgment being tested.
4
State the position, with the risk line in it
Say it like this
"My pick: the new 'Make it playful' preset is a feature flag rollout, plain and simple, same model underneath, just a new option in a menu. But swapping the model that writes every existing tone from version 2 to version 3, that's a model rollout. Even shipped behind the exact same on and off switch, I'd run it differently: shadow test it against the old model on real notes before it ever touches a saved one."
Why this works
PICK rewards a real line drawn between two specific changes, not a blanket rule about AI features.
5
Name who feels each kind of miss
Say it like this
"Here's the split. Run the model swap like a flag, ramp it, watch the crash dashboard, flip it back if something looks wrong, and the dashboard stays clean, because a changed voice isn't a crash. A user's note gets rewritten warmer than they meant, they don't notice, they paste it into an email to their manager, and now the switch is off but that email's already sent. Flip it the other way, and some PM is stuck running a three-week shadow test for a menu item that was never going to change a single word anyone already wrote."
Why this works
Turns "the stakes are different" from a claim into something the interviewer can picture happening to a real user.
6
Name the cost asymmetry, plainly
Say it like this
"The playful-button mistake is cheap and it's loud: you ship a little late, someone grumbles, you loosen the process next time. The model-swap mistake is quiet and it's expensive: nothing broke, nothing errored, and by the time anyone noticed the voice had changed, ninety thousand notes had already gone out written by the new model instead of the old one. I'm optimizing against the one nobody would catch on a dashboard."
Why this works
Names which miss is which instead of leaving "asymmetry" as an unexplained word.
7
Name the kill criteria and close on one line
Say it like this
"I'd flip the model swap over to a lighter process once a shadow test on real notes shows well under ten percent of outputs come out meaningfully different from version two. Right now that number's thirty eight percent, so it stays a full model rollout: compare it side by side first, ramp by serving percentage, and assume the rollback only stops the next mistake, not the one already sent."
Why this works
Ends on the line the interviewer remembers, and shows the pick can move if the evidence does.
A last note before the walkthrough ends: this pick is about matching the process to what actually flips, not about treating every AI feature as dangerous. Most candidates hear "AI feature" and reach for extra caution across the board. Name the one piece that changes what the model writes, and you've shown judgment instead of reciting a rule of thumb.
Let's learn
Penbrook is a note-taking app. Inside it sits a button called Tone Rewriter: pick a tone, Crisp, Warm, or Formal, and it rewrites your note in that voice.
Every change at Penbrook ships the same way. A flag goes out to 5 percent of users, then 25, then everyone, over about three weeks. If a graph moves the wrong way, someone flips the switch and it's gone within the hour. That plan has shipped over forty changes in two years: new buttons, a redesigned settings page, a different shade of blue. Every single one of them behaved exactly the same after the switch went back off.
This time, two changes went out together in the same release: a new "Make it playful" preset, and a swap of the model that writes every rewrite, old version out, new version in.
At the 25 percent stage, about 10,000 notes a day were running through the new model. Nine days in, nobody had looked closely enough to notice anything wrong. By then, about 90,000 notes had already been rewritten by it.
Here is the turn. The crash dashboard never moved. The new model didn't error more, or run slower, or return anything that looked broken. That's exactly why nobody caught it for nine days: nothing on the usual graphs was built to notice a voice changing.
We did not roll back a mistake. We rolled back a switch, nine days after the notes it changed were already gone.
At its worst, a Penbrook note doesn't stay inside Penbrook. People copy a rewritten note into an email, paste it into a shared doc, forward it to someone else. Once that happens, flipping the switch back doesn't reach it. The switch only decides what happens next.
The choice I would take back
We built one rollout plan and used it for two different kinds of change. A new preset and a new model went out under the exact same plan, because both of them shipped behind a switch. I'd split it: any change that can change what the model writes gets shadow tested against the current model on real notes, before it ever goes out behind any switch at all.
What I would leave alone. The playful preset genuinely deserves the fast plan. It's a new option sitting next to the old ones, not a new voice for them. Turning it off leaves nothing behind that needs fixing.
The lesson. The size of the change was never the real question. The real question was whether the thing writing the note had changed, because that's the part a switch cannot put back the way it found it.
Same switch, two different machines. One decides who sees it. The other decides what gets written
The Tuesday the dashboard stayed green
You don't need this to answer the question. Read it if you want to feel why the split has to happen before the ramp starts, not after someone finally reads the notes.
Zoya Linhart can read a rollout dashboard the way some people read weather. Four years running launches at Penbrook, and she has never once let a bad change reach more than a quarter of the app's users before catching it.
Every launch at Penbrook goes through the same three gates: 5 percent, 25 percent, everyone. For two years, Zoya opened the dashboard the morning after each push and checked three things: crash rate, load time, support tickets. If all three sat still, she waved the next stage through. It worked every time, because almost everything Penbrook ever shipped was a button, a colour, a layout.
For the first year, she also did one more thing. During every 5 percent stage, she opened ten real rewritten notes from the test group and read them herself, start to finish, just to see if the tone actually landed. Slow, but it caught things the dashboard never would.
By year two, she had stopped. The dashboard had never once lied to her. Reading ten notes by hand started to feel like double checking something that had already checked itself.
Then came the release with two changes bundled into it: the playful preset, and the model swap. Petros Adamou, the engineer running it, scoped both under the same plan. Five percent. Twenty five. Everyone. Same three weeks. Same dashboard.
The dashboard never turned red. The only thing that changed was what forty thousand notes sounded like, every single day.
At the 25 percent stage, nine days in, Petros pinged Zoya about something unrelated, then added one line at the end: "Also, ran my own notes through Warm just now. Feels different. Not wrong exactly. Just not us anymore."
Zoya almost didn't open the sample. The dashboard was clean. She opened it anyway, because it was the one habit she'd let go of, and something about Petros's message made her want it back.
Ten notes in, she stopped counting how many read differently. Most of them did.
She pulled the pause that same afternoon. By then, the rewrite had been running at 25 percent for nine days. Roughly 90,000 notes had gone out written by the new model. A quick audit found that only about 9 percent of them, the ones nobody had opened, copied, or shared since, could still be regenerated in the old voice. The rest were already somewhere else: an email, a shared doc, a message to a manager, written in a voice their owner never picked.
Same tool, same on and off switch, two very different sizes of wrong
I want to say the mistake was bundling two changes into one release. It wasn't, not exactly. The mistake was scoping both of them by the same question: is this off by default, and can we switch it back. That question is right for a button. It has never once been the right question for a model.
So here is the choice I would take back.
Back when Petros scoped the release, in a fifteen-minute planning call, both changes went on the same line of the same ticket: "ship behind flag, ramp 5/25/100, roll back if anything looks wrong." Nobody in that call asked what "looks wrong" would actually look like for a model that hadn't errored once. I would have split that ticket right there, and put one question in front of the model swap that the preset never needed: run it against the old model on real notes first, and tell me the difference before either of us ever sees a dashboard.
And the replay. Same two changes, new design: the shadow test runs before stage one even starts. On the morning the release is scheduled to go out, the report is already sitting in Zoya's inbox: 38 percent of the new model's rewrites came out meaningfully different from the old one, on the same 500 real notes. The model swap never reaches 5 percent that day. It goes back to engineering with a number attached to it instead of a feeling.
One design tells you on day nine. The other tells you before day one.
And the thing I'd tell myself, back on that fifteen-minute call: the switch was never the risky part. We just never asked what was standing behind it.
PICK, once you can see both machines behind the switch
This is a yes or no about which kind of change one release actually is, not a rule for every AI feature Penbrook ever ships, so PICK carries the weight here.
P, position. Fast, light rollout for the "Make it playful" preset, ship it under a normal flag, three weeks, done. But swapping the model that writes every tone gets treated as a full model rollout: shadow tested against the current model on real notes before it ever writes to a saved one.
I, impact. A wrong preset button is felt by nobody until someone chooses it, and it's just an odd tone they can undo by picking a different one. A model swap is felt by everyone already using Crisp, Warm, or Formal, and it lands in notes they never opened again, in tone they never asked for.
C, cost asymmetry. The over-cautious mistake is cheap and it's loud, a slow ship and an annoyed team, fixed by loosening the process next time. The under-cautious mistake is hidden and expensive, it survives quietly inside a clean crash dashboard, because employees have no way to flag a voice that just feels slightly off, they just live with whatever the model wrote.
K, kill criteria. Flip the model swap to the fast process once a shadow test on real notes shows well under 10 percent of outputs come out meaningfully different from the current model. Below that, the risk is close enough to a button. Above it, and it stays a full model rollout, because right now the only reason this one was caught was a Slack message, not the pilot's own numbers.
Knowledge spark: what is a shadow test?
Running the new model on real input without letting it write anything real yet. You put the old model's answer and the new model's answer side by side, on the same notes, and see how different they actually are before anyone's saved note changes.
Knowledge spark: why not just treat every AI change as a model rollout, to be safe?
Because most changes to an AI product never touch the model at all. A new preset, a redesigned menu, a colour change, all of it can ship fast and reversible. Treating a harmless button like a model swap just means a good idea waits three extra weeks for a process it never needed.
Share of the mistake that's stuck, once you catch it
Holding the preset back too long costs nothing once you loosen up, it's a 0 percent stuck rate. Running the model swap too loosely leaves 91 percent of the affected notes permanently in the new voice, already read, copied, or shared before anyone paused it. The other 9 percent, still untouched, were the only ones the team could actually regenerate in the old voice.
Where the 10% kill line sits, against real shadow tests
Below the line, treat it like a feature flag
Above the line, treat it like a model rollout
Kill line: 10% divergence
Every past model-level change at Penbrook shadow tested under 10 percent divergence, which is why the light process always felt safe. The version 2 to version 3 swap is the first one to clear it by a wide margin, 38 percent, which is exactly the number that should have kept it off the fast plan from the start.
Try the same four letters at a vet clinic
Costin Barlas runs product for Clawmark, a note tool three hundred veterinary clinics use. A vet talks through an exam out loud, and Clawmark turns it into a written visit record, medication doses included.
Two changes are queued for the same release. A new template built for large animals, horses and cattle, that formats the record differently. And a swap of the model that turns speech into that written record, old version out, new version in.
Costin split the pilot the same way Zoya eventually did.
P. Fast, light rollout for the large-animal template: a new layout, same model underneath, ship it in two weeks. Full model-rollout treatment for the transcription swap: shadow test it against real recorded exams before it writes to a single visit record. I. A wrong large-animal template is felt by one vet who reformats a page by hand, mildly annoying, fixed in a minute. A transcription model that phrases a dosage line differently is felt by whoever reads that record next, a technician, another vet, an owner, and it's never checked twice unless something already looks wrong. C. The template miss is cheap and it's loud, someone notices the layout is off the same day. The transcription miss is quiet and it's expensive, it sits in a permanent medical record, phrased with slightly different confidence than the vet actually had, and nobody rereads an old visit note unless something has already gone wrong. K. Flip the transcription model to the lighter process once a shadow test on real recorded exams shows well under 10 percent of its phrasing comes out meaningfully different from the current model, dosage lines included. Above that, it stays a full model rollout, compared side by side before it ever writes into a real record.
What I would leave alone, at Clawmark
The large-animal template doesn't need any of this. Getting a layout wrong costs a vet one minute with a pen. That's not a risk level, it's a formatting choice.
Swap the trigger and it still runs
Speed: if Clawmark needed the transcription swap live in a week instead of a full quarter, the pick doesn't move, the shadow test still runs first, it just runs faster, on a smaller sample, not skipped.
Cost: if the shadow test turned out to cost more engineering time than expected, the pick still doesn't move, that cost was never the question, whether a mistyped dosage could reach a real record was.
The model got better: if the new transcription model already matched the old one on 500 real exams, under 2 percent different, that's exactly the evidence that flips it to the fast, light process.
Where people run it wrong
Treating "it ships behind a flag" as proof a change is reversible, when the flag only controls who sees it, not what already happened.
Blending a new preset and a model swap into one release plan, so the safe half makes the risky half look safer than it is.
Running the full model-rollout process on every AI-adjacent change anyway, so harmless presets wait behind a shadow test they never needed.
Buy yourself two seconds, out loud
Say the reframe before you answer with a rule of thumb. "Give me a second, I want to check whether this switch changes what the model writes, or just who sees a button." That's true, it's already stage three of the walkthrough, and it buys you the time to find the real split instead of reciting "it's all AI, better be careful."
Flashcards (click a card to flip it)
1 · THE FRAMEWORK
Which framework fits this question, and what's the hardest step to nail?
Tap to flip
ANSWER
PICK, for a tradeoff. The hardest step is C, the cost asymmetry: naming why a quiet model rollout costs more than a loud, cheap over-caution mistake.
2 · THE PERSON
Who is this answer about, and what does she already do well?
Tap to flip
ANSWER
Zoya Linhart, a product manager who has run launches at Penbrook for four years and has never let a bad change reach more than a quarter of users before catching it.
3 · THE HABIT
What did Zoya stop doing once the dashboard never let her down?
Tap to flip
ANSWER
Reading ten real rewritten notes by hand during every early rollout stage. She let it go in year two and started trusting the crash dashboard alone.
4 · THE ASYMMETRY
What are the two ways to get this backwards, and who gets hurt by each?
Tap to flip
ANSWER
Running a model swap like a flag hurts the user whose notes quietly change voice. Running a harmless feature like a model rollout hurts the team, who waste weeks shadow testing a button that changes nothing.
5 · THE POSITION
State the pick in one sentence, the way you'd say it out loud.
Tap to flip
ANSWER
The new preset is a feature flag rollout, ship it fast. The model swap is a model rollout, shadow test it against the current model before it ever writes to a real note.
6 · THE NUMBER
By the time anyone paused the ramp, about ______ notes had already been rewritten by the new model.
Tap to flip
ANSWER
90,000. Ten thousand notes a day, at the 25 percent stage, for nine days, before anyone read the sample closely enough to notice.
7 · THE KILL CRITERIA
What evidence would flip the model swap down to the lighter process?
Tap to flip
ANSWER
A shadow test on real notes showing well under 10 percent of outputs differ meaningfully from the current model. The actual test came back at 38 percent, so it stays a full model rollout.
8 · THE TRANSFER
Section 4 runs PICK again on a different product. Which one, and where does the position land there?
Tap to flip
ANSWER
Clawmark, a note tool for vet clinics. The large-animal template stays on the fast plan. The transcription model swap gets shadow tested first, because a wrong dosage line can sit in a permanent record unnoticed.
Check yourself Score: 0 / 0
True or false
1. True or false, with why: since the new preset and the model swap both ship behind the same on and off switch, they should roll out on the same plan.
True
False
Show hint
Think about what the switch actually controls in each case.
Show answer
False. A switch only decides who sees a change. It doesn't decide whether the change itself is a new button or a new voice for the model. Those need different plans regardless of which switch ships them.
Fill in the blank
2. Fill in the blank: the shadow test comparing the old and new model on 500 real notes found that ______ percent of the rewrites came out meaningfully different.
Show hint
It's the number from stage seven of the walkthrough, the one that decided the model swap stayed a full rollout.
Show answer
38. Well above the 10 percent kill line, which is why the swap kept the slower, shadow-tested process instead of moving to the fast plan.
Multiple choice
3. Which of these is the actual mechanism behind this answer's pick?
A. Ship every AI-related change the fast way, since it's all behind the same switch anyway.
B. Run the full shadow-test process on the entire release, since some of it touches an AI feature.
C. Split the release: fast and light for the preset, a shadow test and staged serving swap for the model change.
D. Cancel the model swap entirely until the crash dashboard flags something.
Show hint
Three of these either treat the whole release as one risk level or wait for a signal that won't come.
Show answer
C. A ships a real behavior change with no comparison first. B slows down a button that was never risky. D waits on a dashboard that a changed voice never touches. Only C matches the process to what each change actually does.
Short answer
4. If Petros had never sent that message about his own notes feeling off, would the model swap eventually have been caught anyway? Walk through it.
Show hint
Think about what the crash dashboard was actually built to notice, and what it wasn't.
Show answer
Maybe, but much later and by someone outside the team. The crash dashboard was never built to notice a changed voice, only errors and speed. Without Petros's remark, the swap likely runs to 100 percent on schedule, and the first real signal comes from a user complaint or a support ticket, weeks after far more than 90,000 notes had already gone out in the new voice.
Multiple choice
5. Why did the crash dashboard stay clean the entire time the model swap was running, even though something real had changed?
A. Because the new model was actually working correctly the whole time, so there was nothing to catch.
B. Because the dashboard measures errors and speed, and a changed tone doesn't throw an error or slow anything down.
C. Because Zoya had turned off alerts for that release by mistake.
D. Because the new model was rolled out too slowly for the dashboard to register the change.
Show hint
Think about what kind of thing a crash dashboard is actually built to notice.
Show answer
B. The dashboard tracks crashes, load time, and support tickets, none of which a changed voice touches on its own. The model was doing exactly what it was built to do, just differently, and "different" isn't the same as "broken."
Short answer, apply it yourself
6. Pick a product you use yourself. Name one change to it that would be a feature flag, and one that would be a model rollout.
Show hint
Look for the part that just decides who sees something, versus the part that decides what the thing behind it actually produces.
Show answer
Model answer: "A photo app I use has a filter picker and an auto-enhance button. Adding a new filter to the picker is a feature flag, same processing underneath, just a new option in a menu. Swapping the model that decides what auto-enhance actually does to my photo is a model rollout, because the same tap on the same button would start producing different photos for me overnight." Any answer works if it names the part that changes what gets produced, not just who sees a button.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
