Artifact critiqueAdvancedEval-Driven Specification / Writing a PRD for an AI feature / #19

Write the rollout and kill-switch section of an AI PRD.

The direct answer
Write two things into the section, not one line pointing at a settings page. An automatic trigger with a real number: flagged or reported output crossing a set rate inside a rolling window. And a manual override that gets checked the moment before each action fires, not back when that action was first scheduled. Skip building a full automated rollback-and-replay system on day one. A kill switch's job is to stop new harm fast, not undo what's already out the door.
Do this, in order
  1. Write both halves of the anchor: an automatic trigger with a real number, and a manual override checked at the moment of action, not at scheduling.Why: this is the whole design decision, and checking the wrong step is exactly how a kill switch quietly fails.
  2. Make the override re-check state on every single send, not just on new work entering the queue.Why: a switch that only blocks new work still lets everything already queued go out anyway.
  3. Give the automatic trigger an actual number, not "the team will decide."Why: "use your judgment" is not a spec line, it's a hope with no clock on it.
  4. Name what the section will not cover on day one: no full automated rollback and replay across every channel.Why: stopping new harm fast matters more than undoing harm already done, and quietly rewriting something someone already saw can look worse than owning it out loud.
  5. Leave slow, reversible rollouts, like a color test on 5 percent of accounts, on the normal feature-flag process.Why: not everything needs a switch that stops in seconds, only the parts that act on their own before a person checks.
  6. Hold the section to one number: minutes from hitting pause to zero new actions.Why: a kill switch nobody can verify against a real clock is a promise dressed up as a requirement.

How to answer this, stage by stage

Eight moves. Ground it in one feature that already acts without asking first, before touching the spec.

1
Scope it to one feature that already publishes without asking first
Say it like this
"Say I'm the PM on Cadence, a scheduling tool that writes captions and auto-publishes posts for small businesses at set times. Nobody has to press a button. I'll write the rollout and kill-switch section against that one feature, not against AI rollouts in general."
Why this works
One real feature that acts on its own makes the danger, that you can't call it back once it's live, concrete from the first line.
2
Say your structure out loud
Say it like this
"Here's how I'll walk through it: what turning this off used to mean here, why that's too slow for something that acts on its own, the trigger and switch I'd actually spec, the number I'd hold it to, and what I'm leaving out on purpose."
Why this works
Two seconds of structure tells the interviewer you have a plan before any detail lands, so they're not guessing where you're headed.
3
Reframe what a kill-switch section actually has to promise
Say it like this
"Most rollout sections describe how a feature turns on. The gap is what happens the first time someone needs it off, in a hurry, while it's already mid-publish. 'File a ticket' is not a plan for that minute."
Why this works
This is the actual insight the question is testing. Skip it and you've written a launch checklist, not a kill switch.
4
Give the anchor, both halves named
Say it like this
"So the section names two things. An automatic trigger: flagged or reported posts crossing 2 percent of that window's sends. And a manual override, one button, checked right before each post publishes, not back when it was first scheduled."
Why this works
Naming both halves with a real number and a real check point is something an interviewer can picture as an actual spec line, not a promise to "add safety controls."
5
Prove it with a near miss, in four sentences
Say it like this
"Here's what happens without that second detail. A trending hashtag gets paired with an unrelated sale caption, 312 posts are queued to ride it, someone flags it 12 minutes out, and the PM hits pause. Because the old switch only stopped new captions, not posts already queued, 84 of them go out anyway before it's fully off."
Why this works
A specific near miss with a real count does more work than "this could go wrong" ever will.
6
Say what you'd measure
Say it like this
"The number I'd hold this section to is minutes from hitting pause to zero new posts going out. Not 'the feature can be disabled.' An actual clock. If nobody can tell me that number for the current design, the section isn't done."
Why this works
Shows you're thinking past launch day, and gives the interviewer something they can quiz you on with a follow-up.
7
Say what you would not build on day one
Say it like this
"I wouldn't spec a full system that automatically finds every bad post already published and deletes or replaces it across every platform. Not every platform even lets you delete a post through its API, and quietly editing something someone already saw looks worse than leaving it up and calling them yourself."
Why this works
Shows judgment instead of a wish list, and answers the follow-up almost every interviewer asks next: "what about the ones that already went out?"
8
Close on the one line
Say it like this
"So: a trigger with a real number, an override checked at the moment of publish, no silent auto-rollback on day one, and a clock on how long it actually takes to reach zero new posts. If the section can't say how fast it gets to zero, it isn't a kill switch. It's a setting."
Why this works
Interviewers remember the last line most, and this one hands them a test they can run on any rollout section afterward.
If you remember one thing Stages 4 and 7 are the answer. The two named halves of the anchor, and the explicit refusal to spec a full auto-rollback system before the simple version even exists. Everything else here is proof it works.

Let's learn

The kill switch for Cadence's smart-send feature was one line in a settings file, and in four months of the feature running, nobody had ever needed to touch it in a hurry.

Knowledge spark: what a kill switch actually is A fast, single way to stop a feature the moment it's doing something wrong. Not a settings page with ten options to click through. One switch, one job: stop it now.

Cadence writes captions for small businesses and puts them on a schedule, then publishes them itself at the time it picked, across whichever platforms the business connected. No one has to press publish.

A flow sketch of the old kill switch: PM spots a bad post, messages on-call, on-call opens the config file, edits it and redeploys, about 40 minutes later it is off
What turning it off used to mean, before this section existed

Before this near miss, turning smart-send off meant messaging whoever was on call, who opened the settings file, flipped the value, and pushed a small deploy. About 40 minutes, start to finish, the one other time anyone had needed it, for an unrelated bug.

Then, on a Wednesday afternoon, the caption engine picked a hashtag that was trending for the wrong reason and paired it with a cheerful sale caption. Three hundred and twelve small businesses had that exact pairing queued to fire in the next batch.

Here's the part that's easy to miss. The extra minutes it took to notice weren't the real problem. The real problem was what the kill switch did once someone finally reached for it.

A support rep flagged it twelve minutes before the batch window opened. The on-call PM messaged whoever was on call, who was in a meeting. It took twenty-six minutes to get the flag confirmed off. And because that switch only stopped new captions from being written, not posts already queued to publish, eighty-four of the three hundred twelve went out anyway, all riding a hashtag that had nothing to do with a sale.

Posts that still published after the kill switch was hit
84
Old switch (checked at schedule)
2
New switch (checked at publish)
Same batch, same twelve-minute warning. The old switch only stopped new captions from being written, so everything already queued kept going. The new switch checks the flag right before each post fires, so only the two already mid-send finish.
We didn't build a kill switch. We built a message that took twenty-six minutes to reach a queue that was never listening.

At its worst, this costs a business its own feed for the day: a sale caption riding a hashtag people were using to grieve, sitting there until someone manually deletes it and writes an apology.

The decision that mattered Write two things into the spec: an automatic trigger with a real number, flagged or reported posts crossing 2 percent of a rolling window. And a manual override, checked right before each post publishes, not back when it was scheduled.

The choice I would take back. Months earlier, scoping smart-send's rollout, the PRD's kill-switch line read: "the feature can be turned off via config if needed." One sentence. Nothing had gone wrong yet, and writing two paragraphs about an emergency that hadn't happened felt like effort nobody had asked for.

What I would leave alone. A slow rollout, like testing a new caption tone on 5 percent of accounts, doesn't need any of this. If that goes badly, nobody's published anything yet, so there's nothing already out the door to catch up with. Only the parts of the product that act on their own, with nobody checking first, need a switch that stops in seconds.

The lesson. A rollout section that only describes how a feature turns on has never actually been tested. It's the day someone needs it off, fast, that tells you whether you wrote a real switch or just a sentence that sounded like one.

Now here is the same thing as a story

Read this one when you've got three minutes, since a Wednesday afternoon says more than any spec section ever could.

For four months, Camden Wu's biggest worry about Cadence's smart-send feature was whether the captions were funny enough.

Camden had joined Cadence to build growth features, and smart-send was the one that actually earned its name. Write the caption, pick the platforms, pick the time, and Cadence takes it from there. No app to open at 9 a.m., no button to remember. For a florist running her own feed before breakfast, that mattered.

It shipped in March. Small businesses signed up faster than the team could onboard them, and every week Camden pulled a sample of live captions and read them the way a customer would, checking the jokes landed and nothing read strange.

He kept that up for six weeks. Then he started only reading the ones the model itself flagged as unusual. Then, most weeks, he skimmed those. By August he mostly trusted the weekly error count, which had sat under half a percent since May, and moved on to the next launch.

Close sketch of the two halves of the anchor: an automatic trigger showing a flag rate gauge, and a manual override shown as a button checked at publish time
The anchor: two named halves, not one settings page

On a Wednesday afternoon in September, a support rep named Denise messaged Camden: "is smart-send using #ForMillhaven right now, because that's not what I think it is."

It was. The trending-topic picker had grabbed #ForMillhaven, trending that morning after a wildfire tore through the town, and paired it with the seasonal sale template: "Everything's on fire this week, up to 40% off." Three hundred twelve small businesses had that exact pairing queued for the four o'clock batch, twelve minutes out.

Camden went for the switch. It wasn't a switch. It was a message to whoever was on call, who was in a design review and didn't see it for six minutes, then opened the settings file, changed the value, and pushed a deploy. Twenty-six minutes, start to finish, the first time anyone had needed it under real pressure.

Two panels comparing the old and new kill switch: the old one checked at schedule time lets 84 already queued posts publish anyway, the new one checked at publish time lets only 2 mid-send posts finish
The day it was wrong, with and without the anchor in place

The deploy stopped new captions from being written. It didn't touch the eighty-four posts already sitting in the queue, generated and ready, which fired on schedule anyway, because nothing in that pipeline ever checked the settings file again once a post was already queued. Eighty-four small businesses posted "everything's on fire this week" over a hashtag for a town burying people that same day.

We didn't build a switch. We built a message that took twenty-six minutes to reach a queue that was never listening.

Camden pulled up the original PRD that night. The kill-switch line, written back in February, read: "the feature can be turned off via config if needed." One sentence. Smart-send hadn't shipped yet when that line went in, nothing had gone wrong, and two paragraphs about an emergency that hadn't happened yet felt like effort nobody was asking for. Nobody argued with it. Why would they.

Run the same Wednesday forward with the switch Camden wrote after. Denise flags it at the same minute. Camden hits pause. This time the publish step itself checks the flag, right before each post goes out, not back when it was scheduled. New sends stop inside the next cycle, twenty seconds. Two posts that were already mid-call to the platform finish, because they were physically already sent. Eighty-four becomes two. Twenty-six minutes becomes one.

One design hands you a message and hopes it arrives before the queue does. The other checks itself on the way out the door, every single time.

And the thing I'd tell myself, back in February: we wrote a whole page about how smart-send picks a time to publish. We never wrote a line about how fast it could stop, and that's exactly the day it existed for.

SPARK, built for the minute you actually need to stop it

This question asks for a spec section, not a screen, but it's still a design decision about what happens the exact moment someone needs the product to stop, so SPARK still fits. A question asking how to measure smart-send's quality would reach for LEAD instead.

SPARK laid out as five rows: situation, payoff, anchor, risk, keep out, each with a short phrase about the rollout and kill-switch section
SPARK, aimed at a kill-switch section instead of a screen
S, situation. Any AI feature that acts on its own, before this section existed: kill it by messaging whoever's on call, who edits a config value and redeploys. Nobody had timed it under real pressure before September.
P, payoff. Not "fewer bad posts." The habit worth building: whoever's on call can stop new posts themselves, in under a minute, without opening a ticket and waiting for someone else to act.
A, anchor. Two named halves in the spec. An automatic trigger: flagged or reported posts crossing 2 percent of a rolling window's sends. A manual override: one button, checked right before each post publishes, not back when it was scheduled.
R, risk. The first time it's needed for real, the switch only touches new work and never checks again. Eighty-four posts already queued go out anyway, twenty-six minutes after someone hit the button.
K, keep out. No full system that automatically finds and deletes or rewrites posts already published across every platform. Not every platform's API even allows deleting a post, and quietly editing something someone can already see looks worse than leaving it and calling them directly.
Why the anchor has to survive the risk Check it against that Wednesday: does the override still work the exact moment eighty-four posts are already queued? Only if it's checked at publish, not at schedule. An anchor that only stops new work hasn't touched the actual risk, it's solved the easy case and left the dangerous one exactly where it was.
A decision tree sketch showing the kill-switch spec branching into what is in scope, stopping new posts fast, and what is not day one, auto deleting and replacing posts already live across every platform
What day one covers, and what it leaves out on purpose

And if you want to be sure it really works, try it somewhere else

A dispatch system for HVAC technicians is a different business entirely, and the same gap between "stop new work" and "stop everything already moving" shows up there too, just wearing a different truck.

A flow sketch: job created, auto text queued, GPS check runs, pause checked here, text sends to customer, with the pause step marked in blue
Same framework, a different department, a different queue

S. Delia Fontaine runs field ops for Thistlewood HVAC. Today, without a real kill switch, wrong auto-texts get stopped by emailing the vendor's support line and waiting for a ticket, which once took most of a business day for an unrelated issue.
P. The habit worth building: Delia can stop the wrong-arrival texts herself, the minute she sees GPS and the message disagree, instead of watching bad texts go out for a day while a ticket sits in a queue.
A. Same two halves, different content. Automatic trigger: the rate of texts saying "arriving now" while GPS shows the technician is still 20 or more minutes out, crossing 5 percent of that hour's sends. Manual override: one button, checked right before each text sends, not when the appointment was booked.
R. The first time she needs it, the button only blocks new appointments from entering the text queue. Thirty texts already sitting in that hour's queue still go out, telling customers a technician is outside when he's nowhere near the street.
K. No full system that automatically re-routes technicians and recalculates every promised window live. Not day one. That needs GPS, traffic, and technician availability solved together, and a silent reschedule nobody told the customer about looks worse than a technician running late.

It took one technician joking in the group chat, "tell them I'm arriving now from two towns over," not an actual complaint, to get someone to check whether the button did what everyone assumed it did. It didn't.

Swap the trigger and it still runs

  • Speed: even if the batch fired in one second flat, checking the wrong step, schedule time instead of publish time, is still checking the wrong step. Speed doesn't fix a switch aimed at the wrong moment.
  • Cost: if running the caption model got free tomorrow, that still wouldn't stop a queue that never rechecks itself before it fires. Cheap generation isn't the same as a working switch.
  • The model gets better: if the flagged-post rate drops to one in a thousand, that one still needs a switch that actually reaches it. A rarer failure isn't a smaller one for the business it lands on.

Where people run it wrong

  • Writing a kill-switch line that only covers the model misbehaving, not the pipeline that keeps running after you've told it to stop.
  • Treating "we can turn it off" as true because a setting exists, without ever timing how long that setting actually takes to take effect.
  • Reaching straight for a full auto-rollback system, and never shipping any kill switch at all because the big version felt like the "real" one.

How to use it live

If you're asked this cold, ask what stopping the feature actually checks, and when. Then ask what happens to the work that's already moving the second you flip it. That second question is where the real requirement lives.

Flashcards (click a card to flip it)

1 · THE SITUATION
Before this spec existed, what did "turn it off" actually mean at Cadence?
Tap to flip
ANSWER
Message whoever's on call, who opens a settings file, edits a config value, and pushes a redeploy. About 40 minutes end to end, never timed under real pressure before September.
2 · THE PAYOFF
What's the actual habit a real kill-switch section is trying to build?
Tap to flip
ANSWER
Whoever's on call can stop new posts from going out themselves, in under a minute, without asking anyone or opening a ticket and waiting.
3 · THE ANCHOR
Name both halves of the anchor: the automatic trigger and the manual override.
Tap to flip
ANSWER
Automatic: flagged or reported posts crossing 2 percent of a rolling window's sends. Manual: one override button, checked right before each post publishes, not back when it was scheduled.
4 · THE RISK
What almost went wrong the day Camden actually needed the switch?
Tap to flip
ANSWER
The switch only stopped new captions from being generated, not posts already queued to publish. Eighty-four posts fired anyway before the flag was fully off.
5 · THE PROOF
What actually happened the Wednesday the trending hashtag got paired with a sale caption?
Tap to flip
ANSWER
A support rep flagged it 12 minutes before the batch. It took 26 minutes to get the old switch confirmed off. 84 of 312 queued posts published anyway, riding a hashtag tied to a real wildfire.
6 · THE NUMBER
___ posts still went out after Camden hit pause, because the switch was checked at ___, not at ___.
Tap to flip
ANSWER
84; caption-generation time; publish time. Once the anchor checks at publish instead, the same near miss drops to 2 posts, and 26 minutes drops to about a minute.
7 · THE REPLAY
Same Wednesday, new kill-switch design. What changes?
Tap to flip
ANSWER
The publish step checks the flag right before each post fires. New sends stop within the next cycle, about 20 seconds. Only 2 posts already mid-send finish. Eighty-four becomes two.
8 · CROSS-PRODUCT
Section 4 runs SPARK again on a different product. Which one, and what does its anchor add?
Tap to flip
ANSWER
Thistlewood HVAC's auto-text dispatch system. Its anchor keeps the same shape: a GPS-mismatch trigger, and an override checked right before each text sends instead of when the job was booked.

Check yourself Score: 0 / 0

Fill in the blank
1. In the September near miss, ___ of the 312 queued posts still published after Camden hit pause, because the old switch was checked at ___ time instead of ___ time.
Show hint
Look at what the old kill switch actually stopped, versus what was already sitting in the queue.
Show answer
84, and generation, instead of publish. The old switch blocked new captions from being written but never rechecked posts already queued, which is exactly the gap the new anchor closes.
True or false
2. True or false: the safest way to spec a kill switch is to have it automatically find and delete or rewrite every bad post already published, across every platform, on day one.
  • True
  • False
Show hint
Think about whether every platform's API even lets you delete a post, and what silently editing one looks like to the person who posted it.
Show answer
False. Not every platform supports deleting a post through its API, and quietly rewriting something someone already saw can look worse than leaving it up and telling them directly. That's the keep-out for day one, not a day-one requirement.
Multiple choice
3. Which pair of design decisions matches the anchor this answer argues for?
  • A. A settings page with ten toggles, and a monthly review of flagged posts.
  • B. An automatic trigger with a real threshold, and a manual override checked right before each action fires.
  • C. A manual override only, checked once when the feature was first scheduled to run.
  • D. An automatic trigger only, with no way for a person to intervene by hand.
Show hint
The anchor needs both an automatic half and a manual half, and the manual half has to be checked at the right moment.
Show answer
B. A is a settings page, not a switch. C is checked at the wrong moment, which is the exact mistake that let 84 posts through. D removes the human override entirely, which this answer never argues for.
Short answer
4. What old decision does this answer take back, and why did it make sense when it was first made?
Show hint
Think about what made pointing at "standard config" feel like the obvious choice while the feature was still unbuilt.
Show answer
Model answer: The PRD's kill-switch line read "the feature can be turned off via config if needed," one sentence, written before smart-send had even shipped. That made sense at the time because nothing had gone wrong yet, and writing two paragraphs about an emergency that hadn't happened felt like effort nobody was asking for. It stopped making sense the day a real near miss showed the config change didn't touch posts already queued.
Short answer, apply it yourself
5. Pick an automated feature you've used or built that acts on its own, no person approving each action. If it needed to be stopped fast, would the switch actually reach work already in flight, or only new work? What would you change?
Show hint
Look for the exact moment in the pipeline where the switch gets checked, not just whether a switch exists.
Show answer
Model answer: "A team I worked with had an auto-send email drip. Pausing a campaign stopped new sends from being scheduled, but emails already queued for that hour still went out, because the send worker only checked campaign status when it first picked up a batch, not right before each email fired. Moving that check to right before send would have stopped it in seconds instead of up to an hour."
Multiple choice
6. According to this answer, where would this exact kill-switch design NOT be needed?
  • A. A slow rollout, like testing a new caption tone on 5 percent of accounts, where nothing publishes until a person checks it.
  • B. Smart-send's auto-publish step, which acts on a schedule with nobody approving each post.
  • C. Thistlewood HVAC's auto-text system, which sends arrival texts without a person reviewing each one.
  • D. Any feature that touches a customer directly, regardless of whether it acts on its own.
Show hint
Think about which option has a person checking before anything goes out, versus the feature acting first and a person finding out after.
Show answer
A. Nothing publishes on its own during that kind of test, so there's nothing already out the door to catch up with. B and C are exactly the features this answer argues need the fast switch. D overcorrects into needing this everywhere, which isn't the argument either.
From U2xAI Academy

From answering questions to owning outcomes.

A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.

  • A live AI agent you actually shipped
  • A launch decision you can defend under pressure
  • An interview-ready portfolio, not more flashcards
Know more