ConceptAdvancedShipping & Model Lifecycle / Rollout strategy and phased launches / #15
Explain the role of a holdback group after full rollout.
The direct answer
Keep a small group of users on the old system running well past full rollout, because a model doesn't hold still the way ordinary code does: it gets retrained, and the spammers it grades keep adapting to beat whatever filter is live. Check the same outcome number on both groups on a schedule, and treat the gap between them, not the change in the rolled-out group alone, as the model's real effect. Don't shut the holdback group down the day the launch numbers look good, because a static filter is the one thing in the picture that isn't also moving.
Do this, in order
Keep a holdback group alive well past full rollout, not just during it.Why: the model keeps changing after launch, retrained on new data and up against senders who adapt to beat it, so a one-time launch check goes stale fast.
Check both groups against the same outcome number, on a fixed schedule.Why: one before-and-after check misses anything else moving in that same window.
Size the model's real effect as the gap between the two groups, not the change in the full-rollout group alone.Why: the full-rollout group's own change can come from anything happening at the same time.
Write down what else could move the number, before launch, not after.Why: you can't ask a holdback group to rule out a cause nobody named.
Shrink the holdback group over time instead of deleting it all at once.Why: a smaller live comparison still catches real drift, and it costs less than running it at full size forever.
Skip the holdback group where being wrong about the cause costs nothing measurable.Why: comparing forever against a low-stakes feature is ceremony, not protection.
How to answer this, stage by stage
Eight moves. Say what a holdback group actually protects before the story, or the answer sounds like a definition instead of a decision.
1
Scope it to one product and one open question
Say it like this
"Let's make this concrete. Nuru Vantongeren runs product for Postglade, an email company. Her team built a model that tries to stop real emails from landing in the spam folder by mistake. It's fully live now, except for a small slice of mailboxes kept back on purpose. The question in front of Nuru is what that leftover slice is actually for."
Why this works
A question about a holdback group stays abstract until it's tied to one real launch with one real number that group is protecting.
2
Say your structure out loud
Say it like this
"I'll walk this through LEAD. L is the real outcome a holdback group protects. E is the early signal it gives you, the counterfactual, measured the whole time, not assumed once. A is how a launch's impact story gets gamed once there's no holdback left to check against. D is what even a real holdback group still can't tell you."
Why this works
Naming the four letters up front tells the interviewer you're about to size a real effect, not describe a testing ritual.
3
Reframe what the question is actually checking
Say it like this
"This sounds like a question about a leftover test group. It's really asking who gets to take credit for a number that moved. Almost any metric can look better right after a launch. A holdback group is the only way to tell whether the launch is why."
Why this works
This moves the answer from "what's a holdback group" to naming exactly what it protects against: a coincidence wearing the model's name.
4
Give the decision straight
Say it like this
"Here's the answer. Keep a small holdback group running well past full rollout, and check it against the same number on a set schedule. The gap between the two groups, not the change in the rolled-out group by itself, is the model's real effect."
Why this works
This names exactly what the holdback measures and for how long, instead of a vague "keep testing it."
5
Prove it with the number that shows the gap
Say it like this
"Here's why that matters. Postglade's false-positive rate had sat at 1.8 percent for over a year. Six weeks after full rollout, the 98 percent of mailboxes on the new model were down to 0.9 percent, and the team said so at the next all-hands. But the 2 percent still on the old filter, no model at all, had dropped too, from 1.8 to 1.1 percent, over that same six weeks."
Why this works
One real number that moved on its own, with no model touching it, does more work than a paragraph about the importance of controls.
6
Name the gaming path
Say it like this
"Nobody faked this. Postglade's infra team had migrated to stricter sender checks that same quarter, for reasons that had nothing to do with spam. That change cut false positives for every mailbox, model or no model. Full rollout with no holdback group left running would have handed the model credit for all 0.9 points, when it earned about 0.2."
Why this works
Naming the exact mechanism, a real number moving for a real reason that isn't the model, is what separates this from a generic "measure more things" answer.
7
Say what you'd leave alone
Say it like this
"Not every feature needs this. Postglade also shipped a small wording change to its reply suggestions that same month. Getting that wrong costs nobody anything they'd notice. I wouldn't hold back a comparison group for that. Save the holdback for a number where being wrong about the cause actually costs something."
Why this works
Shows judgment instead of blanket caution. A holdback group is expensive to keep running and shouldn't be reflexive.
8
Close on the hard limit
Say it like this
"And here's the limit, worth saying out loud. Even a clean holdback group only tells Nuru that the model was worth about 0.2 points, and that the rest came from somewhere else. It doesn't tell her why the model earned those 0.2 points, which senders, which emails, or whether that number holds once the model retrains again and the mail it's grading keeps shifting under it. So if I could change one thing about how her team planned this: don't set a kill date for the holdback group before you launch. Set a re-check date instead, and shrink the group rather than deleting it, once the number's been steady for a full quarter."
Why this works
Naming the boundary and closing on a concrete replacement policy gives the interviewer both the honest limit and something they could actually go build.
Let's learn
Here's what happens when a number gets better the same week something else changes too, and almost nobody thinks to check which one did it.
Say an email company builds a model that decides whether a message is spam. Not the whole spam filter. Just the part of it that tries hard not to send a real email to the junk folder by mistake.
Knowledge spark: what's a false positive here?
A false positive is a real email wrongly marked as spam. A bill. A job offer. A message from a kid's school. It never gets seen, and the sender never finds out it failed to arrive.
For over a year, Postglade's false-positive rate had sat still at 1.8 percent of real mail. Out of every thousand real emails sent to a Postglade inbox, about eighteen never reached anyone.
A model like this doesn't hold still once it ships. Postglade retrains it every week on freshly reported spam, and the senders trying to slip past it keep changing their tricks to beat whatever version is live. A button or a screen stays put once you ship it. This does not. That's part of why checking the number once, right after launch, was never going to be enough.
The team rolled the new model out to 98 percent of mailboxes. The other 2 percent, chosen at random, kept the old filter on purpose, as a holdback group to check against.
Six weeks after full rollout, the 98 percent's false-positive rate had fallen to 0.9 percent. Almost cut in half. The team put it on a slide and said so at the next all-hands.
Here's the turn. The 2 percent who never got the new model dropped too. From 1.8 percent to 1.1 percent, over that same six weeks, running the exact same old filter they had always run.
The model didn't cut the rate in half. Something else did most of that work, and the model got all the credit.
False-positive rate, full-rollout mailboxes vs. the holdback group, same six weeks
Full rollout, had the modelHoldback, no model, same mail change
Both lines fall together right from the week the routing migration ships. The gap between them at week 8, not the drop itself, is the part the model actually owns.
The gap between those two lines is the model's real size. Not 0.9 points. About 0.2.
The number the launch claimed vs. the number the holdback group leaves for the model
0.9 pts
Claimed: the full-rollout group's own drop
0.2 pts
Real: the model's share, holdback-adjusted
Four-fifths of the win the team announced belonged to the mail-routing migration, not the model. Only the holdback group could have shown that split.
The gauge was telling the truth. So was the quiet infrastructure change standing right next to it. Both were true numbers from the same week.
The choice I'd take back
When the launch was planned, months earlier, Nuru's team wrote the holdback group's kill date into the launch doc: two weeks after full rollout, once the numbers looked clean, fold those 800,000 mailboxes back onto the new model so nobody stayed on the worse filter longer than they had to. I'd take that back and set a re-check date instead of a kill date: keep a small holdback group running for a full quarter, long enough to catch a slower-moving cause, like a routing migration, whose effect hadn't finished landing yet.
What I'd leave alone. Postglade also shipped a small wording change to its reply suggestions that same month. Getting that wrong costs nobody anything they would ever notice. A holdback group there would be checking a number that can't hide a real cost even at its worst. Save the comparison for numbers where the cause actually matters.
The lesson. A number moving right after a launch isn't proof of anything. It's a claim. The only way to test a claim like that is to keep a few people the change never reached, and keep watching them for as long as the number matters, not just until the launch party's over.
Now here is the same thing as a story
Use this version when you've got a few minutes. The short version is above. This is for when the decision needs to survive more than one meeting.
Nuru Vantongeren had shipped four launches at Postglade before this one, and she had a habit the rest of her team teased her about: she wrote the holdback group's kill date into the launch doc before she'd even picked the metric. Two weeks past full rollout, once the numbers hold, fold it back in. Simple, fair to the mailboxes still stuck on the old filter, and it had worked every time before.
This launch was supposed to be the easy one. One clear number: the false-positive rate, the share of real mail that lands in spam by mistake. Cut it, and Postglade's support queue gets shorter, and its biggest customers stop asking why their invoices vanish.
Full rollout landed on a Thursday. Nuru checked the dashboard every morning that first week, the way she always did. The following Wednesday: 1.6 percent. The Wednesday after that: 1.2 percent. By the start of week six it was 0.9 percent, and it had held there for three days straight. She wrote the update herself: false positives down almost 50 percent since full rollout. The company-wide channel filled with the kind of replies that make a launch feel finished.
The 2 percent holdback group was due to fold back in the following Monday, right on schedule, exactly as the doc said.
On the Friday before that, Farshad Kian, an infra engineer two floors down, messaged Nuru about something else entirely, a routing question for a different team. Near the bottom of the message, almost an afterthought, he mentioned that his team had finished rolling out stricter sender-alignment checks across all of Postglade's inbound mail that same quarter, model or no model, holdback group included.
Nuru pulled the holdback group's own numbers that afternoon, mostly to rule it out. She didn't rule it out. The 2 percent still on the old filter had gone from 1.8 percent to 1.1 percent over the same six weeks, with nothing about their filter touched at all.
Nobody had lied about the number. The number was true. It just wasn't the model's number, not most of it.
Two months earlier, at the kickoff, nobody had asked whether anything else in the pipeline might move that same rate during the rollout window. Nuru doesn't remember raising it either. The kill date felt like the responsible choice: don't make 800,000 mailboxes wait on a worse filter longer than they had to.
I would take that back. I'd have said, at that same kickoff: "before we set a date to fold this group back in, let's name what else could move this number in the next two months, and set a re-check date instead of a kill date. If nothing else moved it by then, we fold the group back in on schedule. If something did, we need the comparison a lot longer than two weeks."
Here's the replay. Same rollout, same 0.9 percent for the full-rollout group at week six, but this time the holdback group is scheduled to shrink, not disappear, dropping from 2 percent to 0.2 percent of mailboxes at the two-week mark instead of folding in completely. That smaller group, about eighty thousand mailboxes, keeps running through week ten. It shows the same climb toward 1.1 percent that the full group would have shown alone, and this time Nuru's quarterly report credits the model with 0.2 points instead of 0.9, and flags the mail-routing change as the real source of the rest, in writing, before anyone builds next quarter's roadmap on the wrong number.
One design let a good-looking number stand in for the whole launch. The other made the launch earn its own number, the whole time, not just on launch day.
And the thing I'd tell myself, back at that kickoff: a kill date isn't fairness. It's agreeing in advance to stop checking right when checking might start to matter.
LEAD, so the model doesn't get credit for someone else's fix
This sounds like a question about a leftover test group. Underneath, it's still asking who gets to take credit for a number that moved. That's LEAD, run on a holdback group instead of a dashboard.
L, link. The real outcome a holdback group actually protects. Not a slide that says the launch worked. A true, causal read on what the model itself did to the number, kept honest even as the model keeps retraining and the senders it grades keep adapting to it, not a coincidence that happened to land in the same window and got credited to the launch. → Here, that's whether Postglade's false-positive drop was really the model's doing, not the mail-routing migration's, and whether that's still true after the next retrain.
E, early signal. The counterfactual: what would have happened without the model, measured the whole time, not assumed from a single before-and-after snapshot. A static, un-retrained filter is the one part of the picture that isn't also moving. → The 2 percent of mailboxes kept on the old filter, checked against the same false-positive rate, every week, for as long as the model's real effect still needs proving.
A, abuse. How a full-rollout launch with no holdback group left running gets its impact story gamed. Any movement in the number after launch gets credited to the model, even movement that would have happened anyway, and even movement from the model's own drift or a retrain gets folded into "the launch" instead of examined on its own. → The team announced a 0.9-point win. Four-fifths of it belonged to a routing migration that had nothing to do with the model.
D, decision. What a holdback group's read genuinely cannot tell you. Not whether the effect is real, and not how big it is, the group answers both of those. What it can't tell you is why, and it can't promise the number holds once the model retrains again. → Nuru now knows the model is worth about 0.2 points, this month. She still doesn't know which senders, which kinds of mail, or whether that 0.2 survives the next retrain.
The check that proves a holdback group is real
Ask, before a launch: "if this number gets better right after rollout, what else, besides the model, could have moved it in that same window? Is our holdback group still running when that other thing would show up?" If the honest answer is "it's already folded back in," the holdback group protected the launch party, not the number.
And if you want to be sure it really works, try it somewhere else
Ferncrest Unified School District runs an early-warning tool that flags ninth graders at risk of falling off track to graduate, based on attendance and grades. Endrit Sallaku, who leads student support there, rolled it out to every school except a holdback group of six campuses, kept on the old biweekly counselor check-ins on purpose.
L. Whether the tool is the real reason more freshmen stayed on track to graduate, not just a number that rose the same semester it launched.
E. A holdback group of six campuses with no tool at all, checked against the same on-track rate, all semester.
A. Full-rollout campuses' freshman on-track rate rose from 76 to 84 percent, eight points, and the district was ready to expand the tool statewide on that number alone. The same semester, a new state law started auto-texting parents after three unexcused absences, no tool required. The six holdback campuses, with no early-warning tool at all, rose from 75 to 82 percent, seven points.
D. Even knowing the tool's real share, about one point, doesn't tell Endrit which specific students it caught that a blanket text to every parent wouldn't have caught anyway.
Knowledge spark: why would a rollout-only number hide this?
A state law that reaches every school at once moves the number the same way a genuinely great feature would: it goes up everywhere, right after the change. Only a group that got the state law but not the tool can tell the two apart.
Once Ferncrest checked the six holdback campuses against the rollout campuses, the truancy law's fingerprints were obvious. The district's own rollout number, the only one anyone had been watching, had never once suggested there was anything left to check.
Almost the same needle, on a campus that never got the tool at all. The law moved both gauges. Only one of them had a model to blame it on.
Swap the trigger and it still runs
The model gets cheaper to run. Doesn't help. A cheap model can still be getting credit for somebody else's fix.
The rollout takes twice as long, in smaller stages. Doesn't help either. More stages just means the same untested credit gets assumed more times, not fewer.
The model genuinely gets more accurate the longer it runs. Still needs the holdback group. A model that's actually improving is exactly when a team is least likely to go looking for a second explanation.
Where people run it wrong
Folding the holdback group back in the moment the launch numbers look good.
Comparing the full-rollout group only to its own past, never to a live group that never got the change.
Assuming a holdback group proves why the model works, when it only proves how much.
How to use it live
Say the split out loud, early: "Before I say what the holdback group is for, I want to separate two things: whether the number moved, and whether the model moved it. A holdback group is the only way to tell those apart, and it only works for as long as it keeps running." That's not stalling. It names the exact gap a launch-day number always has, and it buys you a beat to build the rest of the answer around a real comparison instead of a victory lap.
Flashcards (click a card to flip it)
1 · THE FRAMEWORK
Which framework fits this question, and what does each letter stand for here?
Tap to flip
ANSWER
LEAD. L is the real outcome, a true causal read on the model's actual impact. E is the early signal, the counterfactual, measured continuously through the holdback group. A is how it gets gamed, any movement after launch getting credited to the model. D is what it can't settle: why the effect happens, only that it does and how large.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Nuru Vantongeren, who runs product for Postglade, an email company, and led the full rollout of its false-positive-reducing spam model.
3 · THE HABIT
What let the mail-routing migration's effect almost get credited entirely to the model?
Tap to flip
ANSWER
The launch plan set a kill date for the holdback group, two weeks after full rollout, before the routing migration's full effect had finished showing up in the numbers.
4 · THE EARLY SIGNAL
What's the E step here, in one line?
Tap to flip
ANSWER
The holdback group's own reading, the counterfactual, checked against the same false-positive rate every week, for as long as the model's real effect still needs proving.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Writing a fixed kill date for the holdback group into the launch doc, two weeks after full rollout, once the numbers looked clean. Replace it with a re-check date and a shrinking, not vanishing, holdback group.
6 · THE NUMBER
The full-rollout group's false-positive rate fell from 1.8 to ______ percent. The holdback group, no model at all, fell to ______ percent over the same six weeks.
Tap to flip
ANSWER
0.9 percent, full rollout. 1.1 percent, holdback. The model's real share of the drop is the 0.2-point gap between them, not the full 0.9-point change the launch announced.
7 · THE REPLAY
Same rollout, a shrinking holdback group instead of a two-week kill date. What changes?
Tap to flip
ANSWER
The smaller group, about 80,000 mailboxes, keeps running through week 10, still shows the climb to 1.1 percent, and the quarterly report credits the model with 0.2 points instead of 0.9, in writing, before the next roadmap gets built on the wrong number.
8 · THE TRANSFER
Section four runs this same question again for a different product. Which one, and what does its holdback group reveal?
Tap to flip
ANSWER
Ferncrest Unified School District's dropout early-warning tool. Its holdback group of six campuses, with no tool at all, rose almost as much as the rollout campuses, revealing a new truancy law as the real driver.
Check yourself Score: 0 / 0
True or false
1. True or false: because the full-rollout group's false-positive rate fell from 1.8 percent to 0.9 percent, we know the model caused that entire drop.
True
False
Show hint
Check what the holdback group, which never got the model, did over that same window.
Show answer
False. The holdback group, still on the old filter, dropped almost as much, from 1.8 to 1.1 percent, because of a mail-routing migration that had nothing to do with the model. The model's real share was about 0.2 points, not 0.9.
Multiple choice
2. Which of these is the decision Nuru says she'd take back?
A. Hiring another engineer to watch the false-positive dashboard.
B. Tightening the false-positive target from 1.8 percent to 1.5 percent.
C. Setting a fixed two-week kill date for the holdback group instead of a re-check date and a shrinking group.
D. Announcing the launch results one week later than planned.
Show hint
Look for a decision about how long the comparison kept running, not a dial turned tighter on the same metric.
Show answer
C. The other three are dials: more watching, a stricter number, a different date on the same announcement. None of them would have caught a slow-moving cause the holdback group was folded away before it could reveal.
Fill in the blank
3. The launch was announced as a ______-point win. Once the holdback group's own drop was subtracted out, the model's real share of that was only about ______ points.
Show hint
One of these numbers is the full change in the rolled-out group. The other is the gap between that group and the holdback group.
Show answer
0.9-point win claimed. About 0.2 points real. The gap between those two numbers is the whole argument: a launch with no holdback group left running has no way to see the difference.
Multiple choice
4. Which of these Postglade changes genuinely doesn't need an ongoing holdback group?
A. The false-positive-reducing spam model, which shipped the same quarter as a mail-routing migration.
B. A small wording change to reply suggestions, where getting it wrong costs nobody anything they would notice.
C. A new fraud-detection model shipping alongside a separate change to how flagged accounts get reviewed.
D. A pricing model change rolling out during a quarter when a competitor also changed its prices.
Show hint
Look for the one change where being wrong about the cause has no real cost.
Show answer
B. Nobody is hurt if the wrong thing gets credit for slightly better reply wording. The other three all have a real number, and a real confound running alongside them, worth telling apart.
Short answer, apply it yourself
5. Think of a product change you've seen credited with an improvement right after launch. What else was happening in that same window that could explain some of the gain, and what would a small holdback group have told you?
Show hint
Look for something that shipped, changed, or happened company-wide around the same time as the launch you're thinking of.
Show answer
Model answer: "A retailer launched an AI product-recommendation widget and conversion rate rose 6 percent. The same month, the site also fixed a slow checkout bug. A holdback group with the bug fix but no widget would have shown how much of that 6 percent was the checkout fix, not the recommendations."
Fill in the blank
6. At Ferncrest, the freshman on-track rate rose by ______ points at full-rollout campuses and by ______ points at holdback campuses with no tool at all, because of a new ______ law.
Show hint
This is Section 4's number, not Postglade's.
Show answer
8 points, full rollout. 7 points, holdback. A new truancy law that auto-texted parents after three unexcused absences. The rollout number alone never suggested a problem. Only the six holdback campuses made the law's effect visible.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
