CaseAdvancedShipping & Model Lifecycle / Incident management for AI products / #15
What should the product do automatically when a safety filter fires at ten times the normal rate?
The direct answer
The product should automatically throttle the exact pathway generating the spike and pull a stratified sample of the newly flagged content into human review within minutes, not flip one hard switch for the whole app. Ten times the normal rate has to mean ten times the percent of messages flagged, held for a short confirmation window, because a raw headcount alarm cannot tell a real attack apart from an ordinary busy night, and the human sample is what decides whether the next automatic step is disabling one pathway or rolling the model back.
Do this, in order
Automatically throttle the one pathway producing the spike and pull a review sample, not the whole app.Why: a single platform wide switch punishes every real user for a problem that usually lives in one corner of the product.
Define the normal rate as the percent of messages flagged on an hour of week baseline, never a flat daily average.Why: raw counts rise every evening on their own, so a flat count threshold fires falsely on ordinary busy nights.
Require the elevated rate to hold for a set window, about ten minutes, before the throttle fires.Why: a short burst is often a network hiccup or a small batch of new signups, not an attack, and persistence is the cheapest filter for that.
Route the pulled sample to a human reviewer automatically, before anyone gets paged.Why: whether the spike is a real attack or a broken model decides which automatic action fires next, so the sample has to already be moving by the time the threshold confirms.
Hard disable that one pathway only once the sample confirms real, repeated violations and the rate has not fallen.Why: hard disable costs every legitimate sender on that pathway, so save it for confirmed harm, not a spike arithmetic alone cannot explain.
If the sample instead shows ordinary, varied content wrongly flagged, roll the filter model back automatically instead of throttling harder.Why: throttling users cannot fix a broken model, and the fix belongs on the model side the moment the sample says so.
How to answer this, stage by stage
Nobody is grading whether you can say "we would alert and investigate." They are grading whether you can turn "ten times the normal rate" into real arithmetic, hold it against a range of other thresholds, and land on one automatic action with a reason. Seven moves get you there.
1
Scope it to one real product with real numbers
Say it like this
"Let's ground this. Say we're Duskline, a dating app. Every message and photo goes through a safety filter before the other person sees it. On a normal day it moves about two point four million messages, which spreads out to around a hundred thousand an hour most of the day, closer to three hundred thousand an hour in the evening rush."
Why this works
Grounds the whole answer in real numbers before any policy talk starts, which is what an estimation question is actually testing.
2
Say your structure out loud
Say it like this
"Here's how I'll answer it. First I'll pin down what ten times the normal rate actually means in real numbers, because that phrase hides a trap. Then I'll walk through a few thresholds and what each one costs before I pick one. Then I'll say exactly what the product does on its own at that point."
Why this works
Two seconds that tell the interviewer you have a plan, so the rest of the answer reads as a method instead of a ramble.
3
Reframe the question and name the trap
Say it like this
"Here's the trap. Ten times the rate has to mean ten times the percent of messages getting flagged, not ten times the raw count of flags. If you use raw count, a busy Friday night trips the same alarm as a real attack, because more messages sent means more flags even when nothing about the filter changed."
Why this works
This is the line an interviewer is actually listening for. It shows you didn't take the question's wording at face value.
4
Break down the equation and own the numbers
Say it like this
"So: flags an hour equals messages an hour times the percent flagged. Duskline's normal flag rate is about zero point five percent. At the hundred thousand messages an hour we see most of the day, that's five hundred flags an hour. Ten times the rate is five percent, which is five thousand flags an hour at that same volume."
Why this works
Shows the arithmetic instead of asserting a number. An estimation answer with no numbers on the table is a guess with a confident tone.
5
Show the range you actually considered
Say it like this
"I looked at three times the rate, five times, ten times, and twenty times before I picked one. Three times fires almost weekly from ordinary noise, too twitchy to act on. Five times is rare enough to page a sample, but two of the last three times Duskline hit five times it turned out to be a new ad campaign, not a safety problem, so I wouldn't throttle real senders on five times alone. Ten times is the level where, looking back over eighteen months, every single time it held for ten minutes straight, it was a real incident."
Why this works
Shows you tested the question's own number against real cost instead of just accepting it, and it happened to hold up.
6
Give the one decision, staged
Say it like this
"So here's what fires on its own. At ten times the rate held for ten minutes, the product throttles the exact pathway making the noise, say new opening messages, and pulls a stratified sample straight to a reviewer, no page required yet. If the sample says real attack, the product hard disables that one pathway. If the sample says the flagged messages are actually a wide mix of ordinary stuff, meaning the filter itself broke, the product rolls back to the last model version that worked, automatically."
Why this works
This is the direct answer, said out loud in under thirty seconds, with the branch that makes it AI specific kept intact.
7
Sanity check it, and say what you'd leave alone
Say it like this
"Quick gut check. Five thousand flags an hour is about ten times what our whole review team could physically read by hand in an hour anyway, so there was never a version of this where a person reads all of them. That's exactly why the product has to act first and a person confirms second, on a sample, not the whole pile. And I'd leave three in the morning alone entirely. Normal there is already so small that even a real attack won't look dramatic on a raw count, but the rate still catches it, so the design doesn't need a special case for the quiet hours."
Why this works
Proves you understand both why automation is required here, and where the same design would not need to be touched.
What happens when a safety alarm can go off just because Friday got busy
What happens when a safety alarm can go off for the exact same reason a really good night can happen? Say we build a safety filter for a dating app. Every message and photo gets checked before the other person sees it, for things like harassment, scam bait, and messages trying to move someone off the app fast, before anything worse can happen there.
Before the team had a real threshold, the on call playbook used the simplest number anyone could think of: the flat, trailing daily average count of flags, sitting around five hundred an hour. The rule was, alarm at ten times that count, five thousand flags an hour, and the product locks new message sends across the whole app until someone clears it by hand.
Flags an hour: baseline building up to the ten times trigger, day average volume
Zero point five percent of a hundred thousand messages an hour is five hundred flags. Ten times the rate, five percent, adds forty five hundred more, for a total of five thousand. Nowhere in this build up is a raw headcount alone, only volume times rate.
Then came Valentine's Day. Message volume on Duskline genuinely surges that night, real people, real matches, nothing wrong. Volume climbed to about one point two million messages an hour at its peak. The filter's actual rate never moved off its normal zero point five percent. But zero point five percent of one point two million is six thousand flags an hour, past the flat five thousand trigger.
A raw count going up ten times is not the same fact as the filter getting ten times worse. One of those is a Friday night. The other is an attack.
The product locked new messages across the entire app on the single biggest revenue night of the year, for about forty minutes, before anyone worked out why the whole thing had tripped. Nothing was wrong with a single message. The team's own estimate afterward put the lost upgrade revenue from that one lockout around a hundred and ninety thousand dollars.
Knowledge spark: what is a stratified sample?
A stratified sample pulls messages evenly across different slices, different account ages, different regions, instead of grabbing whichever ones happen to load first. A plain random pull can miss a problem that is only showing up in one slice. A stratified one cannot miss a whole slice by accident.
The choice worth taking back. Someone in an early planning meeting said, let's alarm at ten times the normal count, simple, everyone understands a count. The room agreed, because working out a percentage felt like extra work nobody had a sprint for. That was a fair call with the information they had. It also meant the alarm could not tell a busy night from a broken filter, because it was never built to look at the difference.
The decision that mattered
Defining normal as a flat daily average raw count, then multiplying that count by ten. Once volume was allowed to move the alarm on its own, any night with real, legitimate traffic growth could trip the same switch as a genuine attack, and the product could not tell them apart without a person untangling it by hand.
What I would leave alone. The quiet hours, around three in the morning, do not need a special rule. Normal there is a trickle, so even a real attack produces a small raw count that would never look dramatic next to a five thousand number. But because the trigger is defined as a rate, not a count, it still catches a real problem at three in the morning exactly the same way it catches one at nine at night.
The lesson. A number is not usable on its own until you have answered two questions: how big, and compared to what. Ten times only means something once compared to what has actually been fixed. Get that part wrong and the number can be perfectly correct and still tell you nothing true.
Now here is the same thing as a story
The short version is above. Read on if you want to feel how ordinary the six minutes before the near miss actually looked from Ilana's side of the phone.
Ilana Fennwick spent two years reading flagged messages by hand on Duskline's trust and safety queue before she moved onto the automation team. She can tell a scripted scam opener from a clumsy real one in about two seconds, before she has even finished reading the line.
When the flat count alarm first shipped, it did its job quietly. It barely ever fired, and on the rare night it did, something really was wrong. Ilana trusted it completely. Every page, she read the full sample of flagged messages before clearing it, start to finish, usually about twenty minutes of careful reading.
Then came the ad campaign spike, a wave of new signups from a single promotion pushing message volume up for a week. False alarm. Then came Valentine's Day, and the forty minute platform wide lockout. Also a false alarm. Two real pages in one quarter, both nothing.
So her habit thinned, in three beats. First she stopped reading every message in the sample and skimmed a handful instead. Then she stopped reading the messages at all and just glanced at the shape of the graph. Then, by the third page after the two false ones, she started clearing an alert within ninety seconds of it landing, on sight, without opening a single message, because it was probably just a busy night again.
A Tuesday, 9:47 at night. Her phone buzzed on the nightstand, same shape of alert as the last two. Her thumb was already moving toward clear.
We didn't almost lose six minutes. We almost lost the one page that mattered, buried under two pages that didn't.
She paused, only because the message previews were still scrolling past on the screen while her thumb hovered. They looked wrong in a way volume alone never looks wrong. Near identical. She read three of them. All three were the same romance scam script, word for word, sent to different people.
By the time she caught it, six minutes had passed since the page landed. At the rate the attack was running, about two hundred and fifty flagged messages a minute, that was roughly fifteen hundred scripted openers already delivered to real people before she stopped scrolling and started reading.
The old decision, told as a memory of a meeting. Months earlier, when the alarm first got built, the room chose a flat count because a percentage felt like more work than the sprint had room for. Nobody in that room was wrong about what the alarm would catch. They just never priced in what two false alarms in a row would do to the person reading the third one.
The replay, run the same night forward with the redesigned system already in place: a rate based trigger, hour of week baseline, and a staged response. The moment the rate crosses ten times and holds, the product throttles the opener pathway and pulls the review sample on its own, no page required for that first step. A human only gets paged once the sample is ready to read, about two minutes in instead of the six it took Ilana to catch it half asleep. Two minutes at two hundred and fifty a minute is roughly five hundred messages exposed, not fifteen hundred.
One design put the decision on a phone buzzing on someone's nightstand at midnight. The other design put it in ten minutes of arithmetic that already ran before anyone's phone needed to buzz at all.
What I would tell myself, six months earlier: the ten in ten times the rate was never the hard part. The normal was.
BOUND, laid out on one screen
This is an estimation and architecture question. There is no single person whose habit flips, so FLIPS does not fit here, forcing it on would turn a real sizing problem into a story with a framework taped to its side. BOUND fits: show the arithmetic, own the assumptions, land on a threshold you can defend.
B
Break it down. State the equation before touching a single number.
Flags an hour equals messages an hour times the percent flagged.
Ten times the rate means the percent term times ten, not the whole equation multiplied by ten from volume alone.
O
Own the numbers. Say where each assumption came from.
Duskline's real, measured baseline: about a hundred thousand messages an hour by day, three hundred thousand in the evening, and a normal flag rate of zero point five percent, pulled from ninety days of traffic.
Not invented. Pulled from the same logs the on call team already reads every week.
U
Use a range. Never one lucky number.
Three times, five times, ten times, twenty times, each with a real cost attached.
Three times is noise. Five times pages a human but should not touch real senders, two of the last three false. Ten times, held for ten minutes, has been real every single time in eighteen months. Twenty times is already too late to be the first tier.
N
Nail the sanity check. Compare it to something known.
Does five thousand flags an hour survive a smell test against what a human team can actually do?
A twelve person review team reading forty flagged messages an hour each covers about four hundred and eighty an hour. Five thousand is roughly ten times that. A person was never going to read the whole pile, which is exactly why the product has to act first.
D
Direction. Which single assumption swings the answer most.
Not the flag rate assumption, and not the volume assumption. It's whether normal is measured flat or by hour of week.
Ignoring the evening peak turns a real evening baseline of fifteen hundred into a falsely low target of five hundred, the single biggest lever in the whole estimate.
What moves the ten times target most, if the assumption behind it is wrong
Flat baseline instead of hour of week, evening hours10,000 / hr swing
Flag rate assumption off by a realistic margin2,000 / hr swing
Hourly volume assumption off by a realistic margin1,500 / hr swing
Getting the flag rate or the volume slightly wrong moves the target by a couple thousand flags an hour at most. Using a flat baseline instead of an hour of week one moves it by ten thousand, because it silently treats every evening peak as if it were an emergency.
The automatic action is not one switch. It is a small branching tree, and the branch that makes it an AI product decision, not a generic ops decision, is the one that rolls the model back instead of blocking a person.
And if you want to be sure it really works, try it somewhere else
Palletwise runs a marketplace connecting small shippers with independent truck drivers for same day freight. Before a damage claim gets paid out, a fraud filter flags claims worth a closer look.
B. Flagged claims an hour equals claims filed an hour times the percent flagged as suspicious. O. About two hundred and forty claims filed an hour during business hours, a normal flag rate of two percent, both pulled from claims history, not guessed. U. Same range, three times through twenty times, but claims move slower than chat messages, so the confirmation window lands at thirty minutes instead of ten, calibrated to how fast this product actually moves, not copied from Duskline's clock. N. Ten times the rate is about fifty flagged claims an hour. Palletwise's claims review team can read roughly thirty an hour by hand. Close to team capacity, not ten times over it, so a person stays in the loop a little longer here before automation has to take the wheel alone. D. The swinging assumption here is not hourly, it's weekly. Claims filed after weekend damage spike every Monday, so a flat weekday baseline reads every ordinary Monday as an emergency, the exact same mistake as Duskline's evening peak, just on a different clock.
Same idea, different clock
Duskline's mistake was measuring normal against a flat day when the real cycle was hourly. Palletwise's version of the same mistake is measuring normal against a flat week when the real cycle is Monday shaped. The fix carries over exactly. Only the unit of time changes.
Swap the trigger and it still runs.
Speed: an interviewer caps the answer at ninety seconds. Skip straight to acting on rate, staged, not one switch, and say why in one line.
Cost: the fraud review team gets cut in half this quarter. Don't loosen the threshold to cut their workload. Shrink the sample size pulled for each review instead, a stated tradeoff, slower confirmation, not a lower bar for what counts as suspicious.
The model got better, for real: if Palletwise's fraud model genuinely improved and false flags fell, the old normal baseline goes stale, and a rate based alarm built on last year's normal starts calling an ordinary week an emergency. The baseline needs its own refresh schedule, not just the model.
Where people run it wrong.
They alarm on raw counts and get paged every single Monday for nothing, until they start ignoring Mondays entirely, including the one that's real.
They treat the first automatic action as the only one, no later steps waiting behind it, so a real attack either gets ignored because the bar was set too high, or every user gets punished because the bar was set too low.
They skip the human confirmation sample and let the system hard disable on rate alone, so a broken model and a real attack get treated identically, when they need opposite fixes.
How to use it live. Say the reframe before any story: "ten times the rate only means something once you've said ten times what, so the first thing I'd nail down out loud is whether we're talking about the percent flagged or the raw count, because those two disagree on every single busy night." That buys you the room to show real arithmetic instead of reciting "we'd alert and investigate."
Flashcards (click a card to flip it)
1 · THE FRAMEWORK
What framework fits this question, and why doesn't FLIPS work here?
Tap to flip
ANSWER
BOUND. There is no single person whose habit flips in this question, it's an estimation and architecture problem, so the right method is showing arithmetic and owning assumptions, not chasing a flip.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Ilana Fennwick, on Duskline's trust and safety automation team, two years reading flagged messages by hand before that, able to spot a scripted scam opener in about two seconds.
3 · THE HABIT
What did she stop doing because it worked?
Tap to flip
ANSWER
Reading the full flagged sample before clearing a page. After two false alarms in a row, she thinned that habit down to skimming, then to clearing pages within ninety seconds without reading a single message.
4 · THE TRAP IN THE QUESTION
What does "ten times the normal rate" have to mean, and what does it get confused with?
Tap to flip
ANSWER
Ten times the percent of messages flagged. It gets confused with ten times the raw flag count, which rises on its own during any busy, ordinary night, with nothing wrong at all.
5 · THE OLD DECISION
What decision would you take back, and why did it make sense when it was made?
Tap to flip
ANSWER
Defining normal as a flat daily average raw count, then alarming at ten times that count. It made sense because a percentage felt like more work than the sprint had room for, and nobody had priced in what a volume surge alone would do to that alarm.
6 · THE NUMBER
Fill in the blank: at day average volume, ten times the rate works out to ___ flags an hour, which is about ___ times what the whole review team could read by hand.
Tap to flip
ANSWER
Five thousand flags an hour, about ten times the team's real reading capacity of roughly four hundred and eighty an hour, twelve reviewers reading about forty each.
7 · THE REPLAY
Same near miss, redesigned system, what changes?
Tap to flip
ANSWER
The throttle and the review sample fire automatically at the ten minute mark, with a human paged only once the sample is ready, about two minutes in instead of six. Roughly five hundred messages exposed instead of fifteen hundred.
8 · CROSS PRODUCT TRANSFER
Section 4 answers this same question again for a different product. Which product, and what stayed the same versus what changed?
Tap to flip
ANSWER
Palletwise, a same day freight marketplace's claim fraud filter. Same framework, BOUND, and the same underlying mistake, a flat baseline hiding a real cycle, but the cycle is weekly, Monday shaped, not hourly, so the confirmation window and team capacity numbers both change.
Check yourself Score: 0 / 0
Fill in the blank
1. At evening peak volume of three hundred thousand messages an hour, Duskline's normal baseline is ___ flags an hour, and ten times the rate is ___ flags an hour.
Show hint
Zero point five percent versus five percent, applied to the evening volume, not the day average.
Show answer
1,500; 15,000. A flat day average threshold of five thousand would already be well under this real evening number, which is exactly why the baseline has to track time of day, not just time of week.
Multiple choice
2. Why does this answer measure the percent of messages flagged instead of the raw count of flags?
A. Percentages are easier to put on a dashboard than whole numbers.
B. Regulators require rates instead of raw counts for safety reporting.
C. Raw counts rise on their own whenever message volume rises, so a count based alarm cannot tell a busy night from a real attack, while a rate holds steady unless something actually changed.
D. Rates are cheaper to compute than counts at Duskline's scale.
Show hint
Think about what happened on Valentine's Day, when nothing was wrong with the filter at all.
Show answer
C. Volume and rate move for different reasons. Only a rate isolates the filter's own behavior from ordinary traffic swings.
True or false
3. True or false: once the flag rate crosses ten times normal, the product should hard disable all new messages across the whole app immediately.
True
False
Show hint
Ask what a hard disable costs if the sample turns out to be a false alarm.
Show answer
False. The product should throttle only the specific pathway generating the spike and pull a review sample first. Hard disable is reserved for when the sample confirms real, repeated violations and the rate has not fallen, not for the first sign of a crossed threshold.
Short answer
4. What old decision does this answer take back, and why did it make sense when it was first made?
Show hint
Look for the meeting memory, not a setting someone could just turn up.
Show answer
Model answer: Defining normal as a flat daily average raw count and alarming at ten times that count. It made sense because working out a real percentage felt like extra work nobody had a sprint for, and the team had no reason yet to think volume alone could trip the alarm.
Short answer, apply it yourself
5. Think of an alert or alarm at your own job that fires off a raw count. What would change about it if it fired off a rate instead?
Show hint
Look for a count that naturally rises whenever something ordinary and good happens, like more customers or more traffic.
Show answer
Model answer: A helpdesk's "tickets opened today" alarm fires whenever the company has a big marketing push, since more visitors means more tickets, even with nothing broken. Switching it to "percent of visitors who open a ticket" would separate a real product problem from an ordinary traffic spike, the same fix as Duskline's evening peak.
Short answer, the number question
6. If Duskline's real normal flag rate turned out to be zero point three percent instead of the assumed zero point five percent, would ten times the rate at day average volume be three thousand flags an hour or five thousand? Show the arithmetic.
Show hint
Ten times zero point three percent is three percent, applied to a hundred thousand messages an hour.
Show answer
Three thousand. Ten times zero point three percent is three percent, and three percent of a hundred thousand messages an hour is three thousand flags an hour, two thousand lower than the estimate built on the zero point five percent assumption, which is exactly the swing the direction step flagged as the second biggest lever.
Before you close the answer
Why this works
Tests whether you'll question the question's own framing, raw count against rate, instead of accepting ten times at face value, and whether the automatic action you land on can tell a broken model apart from a real attack, not just react to a number crossing a line.
Follow-up traps
"Why not just hard disable the whole message pipeline the second it crosses ten times, and sort it out after?" Response: because two of the last three times Duskline crossed a high multiple it was ordinary volume, not an attack, and the review sample only takes a couple of minutes, which is cheap next to blocking every real user on the app.
"Ten minutes sustained sounds arbitrary. Why not five, why not thirty?" Response: at fifteen thousand flags an hour that's about two hundred fifty a minute, so five minutes is barely enough to pull and read a fair sample, and thirty minutes at that rate is seventy five hundred messages already sent. Ten minutes is roughly where the sample can be read and confirmed before real harm outruns the check.
If pressed
The stratified sample pulls evenly across account age and region, not just the newest accounts. Duskline's one real broken model incident, a retrain that started flagging normal compliments as harassment, hit established accounts hardest, not new ones. A sample that only checked new accounts would have missed it completely.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
