Artifact critiqueAdvancedShipping & Model Lifecycle / Model migration and version changes for users / #16
Critique a migration plan that relies only on benchmark comparisons.
The direct answer
A published benchmark score is a reason to test the new model against your own product, never a reason to skip that test. Before any migration plan gets approved, build a golden set from the real, common case your product actually has that the benchmark never covered, and require the new model to clear a threshold on that set, not just on someone else's leaderboard.
Do this, in order
Build and test against the product's own golden set before trusting a benchmark chart to justify a migration.Why: this is the one check the whole plan is missing, and everything else follows from it.
Pull that golden set from the real, common case the benchmark never tested, not another random sample of everyday traffic.Why: a random sample just re-measures what the benchmark already measured.
Set a threshold on that slice, like catching a fixed share of what the old model caught, instead of a promise to catch everything.Why: model output is probabilistic, and a deterministic bar isn't a bar anyone can honestly pass.
Keep both models running on any account that hasn't cleared the bar yet, even though it costs more to run two at once.Why: the moment nobody can afford to miss is exactly the moment the plan was about to skip.
Migrate the traffic that already looks like the benchmark's own test data on the original timeline, no extra gate.Why: the new model is a real improvement there, and holding it back protects against a risk it doesn't have.
Watch the golden set's own catch rate every week, not the platform wide accuracy number.Why: the blended number can stay healthy for months while the one slice that matters quietly breaks.
How to answer this, stage by stage
Seven moves. Naming what the benchmark never tested, before naming the framework, is where the real answer starts.
1
Ground it in one real artifact before naming a framework
Say it like this
"Say Driftcast is a live streaming app. Gamers chat under a stream while it's happening, and a classifier scores every message for harassment before it shows up on screen. The team just wrote a migration plan to swap in a new model. It's four slides, and the only one anyone remembers is a bar chart showing the new model beating the old one by nine points on a published benchmark."
Why this works
Grounds a broad "critique this plan" question in one specific artifact, a four slide deck, before naming any method.
2
State your structure in one line
Say it like this
"I'd use GUARD here, because critiquing a migration plan is a risk question wearing a project review costume. Who the benchmark's test set actually represents versus who our real users are, where the plan's blind spot costs the most, who can't tell this decision was made on borrowed evidence, the actual fix, and how I'd catch it if I'm still wrong."
Why this works
Two seconds that name a plan before diving into the deck, and it signals the benchmark chart isn't being taken at face value.
3
Name what the benchmark tested, and who it left out
Say it like this
"The public benchmark's test set is built mostly from generic web comments. Plain insults, slurs, threats, the kind of thing you'd find anywhere. Driftcast's real chat has that too, but it also has raid chat: coordinated pile ons using gaming slang and emoji strings built to dodge exactly this kind of filter. None of that is in the benchmark's test set."
Why this works
This is G. It's not "our users," it's the gap between what the benchmark's test data looks like and what the product actually has to score.
4
Show where the gap costs the most, with numbers
Say it like this
"On everyday chat, the new model is a real win, it catches 87 messages out of 100 instead of 78. On raid chat specifically, it drops from 74 out of 100 to 42. And raids aren't rare. They hit about one in five small and mid size streamers every month, and raid reports make up close to a quarter of the weekly abuse queue."
Why this works
This is U. It names the exact slice the plan's evidence never touched, and shows it's common enough to be a real cost, not an edge case.
5
Name who can't tell the evidence was borrowed
Say it like this
"Baltazar Kask signed the rollout off in a Tuesday steering review. He saw a nine point jump on a real published benchmark and a slide that said cheaper and faster. Nobody in that room could tell the nine points came from a test that never once included a raid message, because a benchmark score doesn't come with a label saying what it left out."
Why this works
This is A, GUARD's hardest step here. Baltazar didn't skip diligence. He had no way to see the gap in evidence that looked complete.
6
Give the actual fix, not a general "test more"
Say it like this
"Here's the rule I'd write in. Before any model becomes a streamer's default, it runs against Driftcast's own golden set: real raid messages, hand labeled by moderators. It has to catch at least 90 percent of what the old model caught on that set. Anyone who doesn't clear that bar keeps both models running during a raid, and the stricter verdict wins, until the new one earns the switch on its own."
Why this works
This is R. A benchmark score becomes a reason to test the product's own hardest case, not a reason to skip testing it.
7
Say how you'd catch it anyway, then close
Say it like this
"I'd watch the raid specific catch rate every week against the golden set, not the blended accuracy number, because the blended number stayed at 91 percent the entire time this was broken. And I'd watch how often a moderator manually pulls a message the model marked clean during a live raid, since that number climbing is the earliest sign the fix isn't holding. So: build the golden set, gate the switch on it, and watch the number the average was always going to hide."
Why this works
Closes on the direct answer in one breath, and shows the fix gets checked against real raids, not just written into a deck and shipped.
Let's learn
Driftcast is a live streaming app. Gamers chat under a stream while it's happening, and a classifier reads every message and scores it for harassment before anyone else sees it.
Before the migration, on Driftcast's own everyday chat, the old model caught 78 harmful messages out of every 100. On the vendor's published benchmark, that same old model scored 81 out of 100.
Knowledge spark: what's a benchmark, really?
A public test with its own set of example messages, built by someone outside your product. A model's score on it says how well it does on those examples. It says nothing about a kind of message that was never in the set.
The vendor's new model scored 90 out of 100 on that same public benchmark, a nine point win, and it runs about 35 percent cheaper. On Driftcast's own everyday chat it really is better too, up to 87 out of 100. That's most of the deck. That's most of the plan.
Catch rate, everyday chat vs. raid chat, old model vs. new model
Same swap, same week. One kind of message got better. One kind fell off a ledge.
Everyday chat, old model
78%
Everyday chat, new model
87%
Raid chat, old model
74%
Raid chat, new model
42%
The published benchmark and the everyday chat number both said the same thing: a clean win. Neither one had a single raid message in it.
Here's the part the deck never showed. On raid chat, coded harassment built to slip past a filter, the new model doesn't improve. It falls from 74 out of 100 down to 42.
The extra misses aren't really the problem. A missed message just sits in the chat log. The real problem is what a raid does in real time. Three hundred people can post in six minutes, and once the model is mostly waving them through, there's nobody left checking who's supposed to catch it.
We didn't test whether the new model was better. We tested whether it was better at the test.
At its worst, this reached Radoslav Hren, a streamer running a sponsored segment when his channel got raided. In six minutes, 420 messages hit his chat, most of them coded harassment. The new model flagged about 55 of them. Backtested against the same messages, the old model would have caught roughly 310. Screenshots of the raid reached the sponsor's own social team before Driftcast's moderators had even finished clearing the thread.
The choice I would take back
We only ever tested a new model against the public benchmark and a general sample of Driftcast's own chat. We never built our own set of real raid messages to check a candidate model against, because the benchmark already showed a clean win, and a hand labeled raid set felt like extra work for a question we thought was already answered.
What I would leave alone. Everyday chat moderation doesn't need any of this. No golden set, no dual run, no gate, just migrate it on the plan's original date, because the new model is already a real improvement there, and holding it back protects against a risk it doesn't have.
The lesson. A benchmark score tells you the model got better at the benchmark's version of the job. It never promised anything about the version of the job your product actually has.
Now here is the same thing as a story
Read the short version above if you're pressed for time. Read this one when you want to feel why a real published number and a real product gap can both be true at once.
The migration plan is four slides. Slide two is a bar chart, the new model beating the old one by nine points on a public benchmark, and it's the only slide anyone in the room that Tuesday actually remembers.
Ffion Marsh has run trust and safety product at Driftcast for three years. She built the harassment classifier's current review process herself, and she can read a week's abuse queue and tell you which spike is a real raid and which is just a slow Tuesday, usually before she's finished her coffee.
When the vendor pitched their new model, Ffion ran the standard checks. She pulled the benchmark numbers, then ran the model against a general sample of Driftcast's own chat, and both came back clean. Nine points better on the public benchmark. Nine points better on Driftcast's own everyday traffic too. She built the four slide plan, and every number on it was true.
For the six weeks between the plan and the rollout, that felt like enough. The general sample kept coming back clean every time she reran it. She stopped pulling a second sample from raid heavy channels specifically, the way she used to early in her testing process, because the general number never once moved.
Radoslav Hren had been streaming on Driftcast for two years, a small channel with steady growth, exactly the kind of account a raid likes to find.
One of them has a hand on the switch. The other only has a chat window filling up fast.
Then Radoslav's channel got raided.
He had a hardware brand running a sponsored segment on his stream that night, his first sponsorship of that size. Six minutes in, a coordinated pile on hit his chat. Four hundred and twenty messages, most of them coded harassment built out of gaming slang and emoji strings the classifier had never been taught to read as anything but noise. The new model flagged about 55 of them. Ffion's team backtested the same messages against the old model afterward: it would have caught roughly 310.
Nobody on Ffion's team saw it happen live. The sponsor's own social team did. Screenshots of the raid reached them before Driftcast's moderators had cleared the thread, and for about a day, Ffion didn't know whether they'd still have a hardware sponsor by Friday.
We didn't lose 58 messages. We lost the six minutes where somebody needed the number to already be true.
I want to say the problem is that the new model got worse. It didn't, not everywhere. A model that's wrong 58 out of 100 times on one rare message type is still, on paper, mostly right. The real problem was that nothing in six weeks of clean testing had ever put a raid in front of the new model, so the plan's confidence and the model's actual coverage had nothing to do with each other.
The team had considered a lighter fix earlier, back when the plan was still just a bar chart: leave the old model running behind an opt in toggle, for any streamer who noticed something off and asked for it back. It's a day of work instead of three weeks of labeling raid transcripts. They turned it down for the reason it was tempting: it only helps a streamer who already knows a model swap happened and already knows there's a toggle to reach for. Radoslav didn't know either of those things six minutes into a raid. A toggle protects the person checking their settings. It does nothing for the one buried in chat trying to keep a sponsor from walking.
Months earlier, in the meeting where the plan got approved, Baltazar Kask asked one question about the benchmark chart: is this number real. It was. Nobody in that room asked the second question: real for what. The nine points were real for the version of the job the benchmark tested. Nobody had checked whether that was the same job Driftcast actually had.
I would put that second question back in the room. Before any streamer's account switches over, the new model runs against a golden set built from real raid messages, hand labeled by moderators who've worked an actual raid shift. Anything that doesn't catch at least 90 percent of what the old model caught on that set keeps both models running during a raid, side by side, and the stricter call wins.
Same Tuesday, same slide deck. This time the golden set check runs before the meeting, not after the raid. It comes back at 51 percent, nowhere near the 90 percent bar. Radoslav's account, and every account like his, stays on dual run. Six weeks later, when his channel gets raided again, both models score the chat in real time, the old model's stricter read wins on 90 of the disputed calls, and by the time the sponsor's team is refreshing the stream, the worst of it is already gone.
What I'd tell the Ffion who built that first four slide deck: "the number is real" and "the number is about our product" sound like the same sentence in a Tuesday review. They aren't. One was true for every test we ran. The other was true for everything except the six minutes that actually mattered, and we never built a way to tell which was which until Radoslav's chat did it for us.
GUARD, when the evidence is somebody else's
This is a risk question, so the framework is GUARD. Critiquing a migration plan sounds like a documentation exercise, which is exactly why it's easy to answer with "add a review step" instead of naming the actual gap: a benchmark score is evidence about the benchmark's version of the job, and the plan treated it as evidence about Driftcast's.
G, groups. The public benchmark's test set, built mostly from generic web comments, insults, slurs, threats anyone would recognize. And Driftcast's actual chat, which includes that, plus raid chat, coded harassment built out of gaming slang and emoji strings tuned specifically to slip past a filter that's never seen it before.
U, unequal. The new model is a real win on everyday chat, 87 out of 100 instead of 78. The cost lands on raid chat specifically, dropping from 74 out of 100 to 42, and raids aren't rare: they hit about one in five small and mid size streamers a month and make up close to a quarter of the weekly abuse queue.
The step that should sit here, and doesn't.
A, ability to contest. Baltazar Kask signed the plan off on a nine point published win and a cheaper, faster model. Nothing on that slide told him the nine points came from a test that never included a single raid message. A benchmark score doesn't come labeled with what it left out, and there was no product specific number in the room to check it against.
R, reduce. Before any streamer's account defaults to the new model, run it against Driftcast's own golden set, real raid messages, hand labeled by moderators, and require it to catch at least 90 percent of what the old model caught. This isn't free: the new model runs about 35 percent cheaper and roughly half the latency of the old one, which is exactly why the migration is worth doing at all, and keeping both models live during a raid on any account that hasn't cleared the bar spends roughly double the inference cost during those windows, on purpose, to protect the moment that actually matters instead of pretending speed, price, and coverage all come free together.
D, detect. Track the raid specific catch rate against the golden set every week, not the blended accuracy number, since the blended number sat at a healthy 91 percent the entire time this was broken. And watch how often a moderator manually pulls a message the model marked clean during a live raid, because that number climbing is the earliest sign the fix isn't holding.
Knowledge spark: why 90 percent, not 100?
Model output is probabilistic. Two models can read the same coded phrase differently and both be defensible. A bar of "catch every message, always" is a promise nobody can keep. A threshold on a real, product specific set is a bar you can actually clear, and actually miss.
Where this answer would fail
If the fix is "add a review step to the launch checklist" or "have someone double check the benchmark," none of it counts. This is a case of a model looking better everywhere a test happened to check, and quietly worse somewhere it never did, a kind of drift no dashboard catches because the dashboard was built from the same distribution as the benchmark. The guardrail has to be a product specific eval, run before cutover, on the exact case the benchmark skipped, not a stricter read of someone else's numbers.
And if you want to be sure it really works, try it somewhere else
Bellcairn's permit office runs an AI tool that reads new permit applications and flags likely fraud for a human reviewer, nothing to do with chat or streaming.
G, groups. Most of Bellcairn's applications come in typed, through the online portal, close to what the published fraud detection benchmark's test set looks like. And the downtown intake office's walk in applications, about 15 percent of its volume, filled out by hand, often by older residents and non native English speakers who never touch the portal. U, unequal. Typed applications get genuinely better fraud detection under the new model. The downtown office's handwritten applications start getting flagged as likely fraud far more often, not because they're more suspicious, but because the benchmark's fraud examples were built entirely from typed digital forms. A, ability to contest. Lazaro Delgado, the downtown office's fraud review lead, has no way to see it coming. The dashboard reports one office wide flag rate, and typed applications outnumber handwritten ones six to one, so a spike in the handwritten slice barely moves it. R, reduce. Check the new model against a golden set of real handwritten applications before it becomes the office's default, and keep manual second review on handwritten submissions until it clears a threshold on that set, instead of switching every intake channel on the same rollout date. D, detect. Track the false fraud flag rate on handwritten applications specifically, every week, against its own baseline, not the office wide number blended across six times as many typed forms.
Handwritten applications' false fraud flag rate, five weeks after the swap
The office wide dashboard barely moved. The handwritten slice, blended into a number six times its size, climbed the whole time.
Office wide stayed at a healthy 8 to 9 percent the whole time. The handwritten slice, about one application in seven, climbed from 6 percent to 24 percent, invisible in the blended number because six times as many typed applications never moved at all.
Swap the trigger and it still runs
Speed: if Driftcast rolled the swap out to every streamer at once instead of in batches, the raid gap would surface in the first week instead of six, which only shrinks the window a golden set check would need to run in. It doesn't remove the need for one.
Cost: a cheaper model makes it more tempting to skip the golden set and lean on "the benchmark already proved it," which means more raid heavy accounts switched with nothing checked, not fewer.
The model gets better: a new model that's right 99 percent of the time instead of 90 makes skipping the check look even safer, and that's backwards. The rarer the miss, the longer a real gap runs before the pattern in it is visible.
Where people run it wrong
Treating one published benchmark number as proof every part of the real traffic is equally safe to migrate.
Building a golden set, but only after the first bad raid, instead of before any account switches.
Gating every streamer the same way, including accounts that never get raided, which just delays a real win for people who never needed protecting from it.
How to use it live
Ask one question before answering: "what's the one common case in our real traffic that the published benchmark never included, and did the plan test against that specifically, or just against the benchmark again?" That's usually the exact gap the interviewer wants named, and it buys a few seconds to think.
Flashcards (click a card to flip it)
1 · THE FRAMEWORK
Which framework fits critiquing a migration plan that only cites benchmark comparisons, and why?
Tap to flip
ANSWER
GUARD, for risk. The real question isn't whether the new model beat a published benchmark, it's who that benchmark actually represents, where the gap costs the most, who can't tell the decision rested on borrowed evidence, the actual fix, and how to catch it if it's still wrong.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Ffion Marsh, the trust and safety PM who built Driftcast's migration plan. She's run the team's classifier work for three years.
3 · THE HABIT
What did Ffion's team stop doing once the general chat sample kept coming back clean?
Tap to flip
ANSWER
Pulling a second sample from raid heavy channels specifically to retest. Early in her testing process she did this by habit. Once the general number never moved, she stopped.
4 · THE SWITCH
What's the two setting switch this answer turns on?
Tap to flip
ANSWER
Either a candidate model gets checked against a golden set built from the platform's own hardest real case, or it only ever gets checked against someone else's benchmark and a general sample. There's no middle setting where a high blended number quietly covers for the one slice nobody tested.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Driftcast never built its own golden set of real raid messages. Every candidate model was tested only against the public benchmark and a general chat sample, because the benchmark already showed a clean win and a hand labeled raid set felt like extra work for a question the team thought was already answered.
6 · THE NUMBER
Fill in the blank: raid chat catch rate dropped from ______ to ______, while everyday chat catch rate rose from ______ to ______.
Tap to flip
ANSWER
74 to 42 for raid chat. 78 to 87 for everyday chat. Same swap, opposite direction, because nobody tested the slice that actually mattered before switching everyone over.
7 · THE REPLAY
Same rollout, the golden set check gates it this time. What changes for Radoslav?
Tap to flip
ANSWER
His account fails the check at 51 percent and stays on dual run. Six weeks later, when he's raided again, the stricter of the two models' calls wins on 90 of the disputed messages, and the worst of it is gone before the sponsor's team is even watching.
8 · TRANSFER
Section four runs GUARD again on a different product. Which one, and what does the reduce step become?
Tap to flip
ANSWER
Bellcairn's permit office fraud classifier. Reduce: check the new model against a golden set of real handwritten applications before it becomes default, and keep manual second review on handwritten submissions until it clears that bar, instead of switching every intake channel on one rollout date.
Check yourself Score: 0 / 0
Multiple choice
1. Which two groups does the G step name, and what separates them?
A. The generic web comments the public benchmark tests, and Driftcast's raid chat, coded harassment tuned to slip past a filter that's never seen it.
B. Ffion's team and Baltazar's leadership, who disagree about the rollout date.
C. Radoslav and his sponsor, who disagree about how serious the raid was.
D. Driftcast and the benchmark's publisher, who disagree over licensing terms.
Show hint
Look for what kind of message each source of evidence was actually built from.
Show answer
A. B, C, and D are real relationships in the story, but not the split G names: what the benchmark's test data looks like, and what the product's real hardest case looks like.
Fill in the blank
2. Radoslav's raid hit ______ messages in ______ minutes, and the new model flagged about ______ of them.
Show hint
Both numbers sit in the same paragraph describing the sponsored stream.
Show answer
420 messages, 6 minutes, about 55 flagged. Backtested against the old model, roughly 310 of those same messages would have been caught.
True or false
3. True or false: the new model performed worse than the old one across all of Driftcast's chat.
True
False
Show hint
Check the everyday chat number, not just the raid chat number.
Show answer
False. It was a real improvement on everyday chat, 78 to 87. The drop was specific to raid chat, 74 to 42, which is exactly why one blended number hid it.
Short answer, apply it yourself
4. Pick an AI product you use that was recently upgraded to a new model or version. What's one real, common way you use it that a general product review probably never tested, and how would you want the company to check it before rolling the change out to you?
Show hint
Think about a use case that's common for you specifically, not rare, just uncommon in a general test sample.
Show answer
Model answer: "I use a grammar checker that most people use for plain emails. I write technical documentation in it, full of code snippets and jargon it probably wasn't tested on. If they swapped models, I'd want them to run it against a sample of real technical writing first, not just their general customer sample, and tell me plainly if the catch rate on that specific kind of writing actually held up."
Multiple choice
5. Why doesn't "Baltazar should have asked more questions before approving the plan" fix the actual gap?
A. The benchmark chart looked complete and gave him no way to know what kind of message it left out, no matter how many questions he asked about the number itself.
B. Baltazar doesn't have the technical background to read a benchmark chart.
C. The plan was approved without any benchmark data at all.
D. Asking more questions would have delayed the migration past the deadline.
Show hint
Ask what information was actually missing from the room, not how careful the questions were.
Show answer
A. B, C, and D miss it. The fix has to put a product specific number in front of the approver, not ask them to interrogate a number harder that never contained the missing case in the first place.
Short answer
6. If raid chat's catch rate had dropped from 74 to 65 instead of 74 to 42, would the same fix, a golden set plus a 90 percent threshold, still be the right call? Why or why not?
Show hint
Check where 65 out of 74 sits against the 90 percent bar named in the GUARD recap.
Show answer
Model answer: "The golden set itself is still worth building, but 65 out of 74 is about 88 percent of what the old model caught, close to the 90 percent bar. It might fail narrowly or pass, depending on the exact set, and that's the point: the threshold decides the call instead of a gut sense that the drop 'isn't as bad this time.' A small miss and a big one get judged by the same bar, not by how alarming the number happens to feel."
If they push back
Why this works
Tests whether you'll treat a published benchmark win as the whole answer, or ask what kind of message it never had a chance to test. Most candidates stop at the bar chart.
Follow-up traps
"Only about a fifth of streamers get raided in a given month, is a whole golden set really worth building for that slice?" Response: raid reports are close to a quarter of the weekly abuse queue on their own, and the accounts that get raided are disproportionately the growing ones, exactly who Driftcast can least afford to lose to a bad night.
"Isn't keeping both models running during a raid just wasted spend once the new model eventually gets fixed?" Response: it's not waste, it's sequencing. The dual run exists to protect the six minutes that actually matter while the golden set number is still below the bar, not to avoid ever finishing the migration.
If pressed
The golden set can't just be old raid transcripts pulled once and reused forever. Raid language shifts every few months as one filter starts working, so the set gets a fresh batch of moderator labeled messages from the last quarter's raids each time a new model gets tested against it, or the threshold ends up measuring a raid style that's already out of date.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
