The direct answer
Rank the follow-up work by what's hardest to undo if it gets skipped, not by what feels most urgent to write first. The item at the top is always the one that stops the exact same failure from shipping quietly again: turn the failure case into a locked, gated regression test in the eval set. A postmortem document, on its own, explains what happened. It does nothing to stop it happening again, so it never outranks the fix that actually closes the door.
Do this, in order
Turn the exact failure case into a locked test in the release-gating eval set, so the next model update can't ship if it reintroduces the same cut.Why: this is the only item on the list that stops a repeat. Skip it, and the bug can come back the next time anyone touches the model, and nobody will know until a customer flags it again.
Audit everything the bad model touched, not just the clips a customer happened to flag.Why: a customer flag only catches the failure someone noticed. The rest stays wrong, published, and silently uncorrected, and nobody can walk that back once it's been public for weeks.
Add a lightweight production check that watches for this exact failure shape in live traffic between releases, not only at release time.Why: a gated eval set only runs when someone ships a new model. A prompt tweak or a config change between releases can reopen the same hole with no release to catch it.
Write the blameless postmortem, naming the decision that let it ship and the alternative that got ruled out.Why: this teaches the next team what happened and why, but on its own it changes nothing in the pipeline, which is exactly why it sits below the three items that do.
Reach out to every creator whose published clip may have flipped in meaning, and offer to reprocess it.Why: real trust cost, but it's the most reversible item here. You can apologize and fix one creator's episode any day after the fact, unlike a failure mode nobody logged.
Close the loop with a status update to the team and any affected customers.Why: keeps everyone informed, but sending it a day late costs nothing next to a bug that ships again unnoticed.
How to answer this, stage by stage
Nobody is grading whether you can name "postmortem" and "communication plan." They're grading whether you know which piece of follow-up work actually prevents a repeat, and can rank the rest under it without hand-waving. Eight moves get you there.
1
Scope it to one real tool, one real incident
Say it like this
"Let's ground this. Firstcut processes about 85,000 podcast and video uploads a week, cutting silence, pulling filler words, and building highlight clips automatically. Cassidy Vantour runs the edit-integrity desk there, and this is the incident she'd be closing out."
Why this works
Grounds the ranking in a real volume and a real owner before any follow-up item gets named.
2
Say the ranking rule out loud, before naming a single follow-up item
Say it like this
"Here's how I'd frame it. The question isn't what should be on the follow-up list, everyone can name a postmortem and a Slack update. The question is what order they go in, and I rank by what's hardest to walk back if it gets skipped. An unlogged failure that can quietly happen again always outranks a missing announcement."
Why this works
States the ranking rule before any item, so the room hears a method instead of a checklist recited from memory.
3
Name the outcome every item on the list is competing to protect
Say it like this
"Every follow-up item is competing for the same thing: does it stop the same cut from happening again, on its own, with no person watching for it. Items that do that outrank items that only explain or announce what already happened."
Why this works
This is the O step. Without a stated outcome, "rank the follow-up work" is just opinion dressed as a list.
4
Walk the top of the rank with the real incident behind it
Say it like this
"A model update at Firstcut started quietly cutting the word 'not' out of quiet, breathy sentences after a pause, for nineteen days, before anyone caught it. The postmortem for that is worth writing. It is not worth writing first. First, that exact cut, the word, the pause length, the mic profile, becomes a locked case in the eval set that gates the next release. Skip that, and the same cut can ship again the next time anyone touches the model."
Why this works
This is the R step, reversibility, made concrete with the real failure instead of a generic "add a test" line.
5
Name the dependency the rank can't skip
Say it like this
"None of this works out of order. You can't write the eval-set case, audit the batch, or brief the team until you've actually named the exact acoustic pattern that broke, the pause length, the softness, the word. Find that first. Everything else on the list is downstream of it."
Why this works
This is the D step, the forced order underneath the rank, not just a preference for what matters more.
6
Own the real evidence behind the rank
Say it like this
"Firstcut has had nine incidents like this in two years. Five got a postmortem and nothing else, and four of those five had the same failure shape come back within about seven weeks on average. Four got the case locked into the gated eval set on top of the postmortem, and none of those four have recurred in up to fourteen months of releases since."
Why this works
This is the E step. It's a real, checkable number, not a claim that the eval set "should" help.
7
Name what you considered and ruled out
Say it like this
"We looked at putting every clip from that acoustic profile, breathy solo narration, through full human review going forward instead of building a targeted check. That's about six percent of weekly uploads, roughly five thousand shows. Reviewing all of them by hand would need something like three hundred and forty hours of QA time a week, and it would blow Firstcut's same-day turnaround promise. We didn't take that trade. We built a narrow, cheap production check for this exact failure shape instead."
Why this works
A rejected alternative, stated with its own real cost, is what proves this was a judgment call and not the only idea anyone had.
8
Close on the rank, defended in one line
Say it like this
"So the order is: lock the case in the eval set, audit the whole batch, add the live production check, then write the postmortem, then notify affected creators, then close the loop internally. The eval-set case sits on top because it's the only item that changes what the model is allowed to ship next, not just what the team knows or says."
Why this works
Closes on the rule itself, something a reader can apply to an incident they've never seen, not just this one story.
If you remember one thing
A postmortem tells you what happened. An eval-set case stops it from happening again. They are not the same kind of follow-up work, and only one of them belongs at the top of the list.
Let's learn
What happens when a fix works, the ticket closes, and nobody checks whether the exact same break can happen again on the very next release?
Say we build a tool that takes a raw podcast or video recording and hands back a clean cut: silences trimmed, filler words gone, a few highlight clips pulled out automatically. Before a tool like this existed, an editor sat with the raw file for two or three hours per episode, listening for every dead pause and every "um" by hand. Firstcut cut that down to about twelve minutes of review per episode, across roughly 85,000 uploads a week.
A model update shipped to get better at spotting "crutch phrases," things like "you know" and "kind of," on top of the usual filler words. It cleared the release gate: 400 known clips, catching at least 98 percent of real filler tokens, no more than a small stated share of real words wrongly cut. It shipped clean.
Knowledge spark: what's a golden set?
A fixed batch of clips with the right answer already marked by a person, used to check a model before it ships. A model that clears the golden set can still make a mistake in production. Ninety-eight percent catching real filler, with real words safe, is a bar by design, not a promise of zero.
Now say plainly what actually broke. The golden set had almost no clips of quiet, breathy solo narration, the close-mic style a lot of independent podcasters use, where a speaker pauses, then finishes a sentence softly. On that specific slice of uploads, about six percent of the week's total, the new model started treating a quiet "not," "no," or "don't" after a pause as filler-adjacent hesitation, and cut it clean out.
"This is not going to work" became "This is going to work," and nothing about the audio waveform looked broken. It looked like a perfectly clean cut.
The turn here is not that the model got worse. In every measurable way, it got better. The turn is that better-on-average and safe-in-this-one-slice are different claims, and nobody had checked the second one before shipping.
The decision that mattered
Golden sets got built from whatever clips the team already had on hand, not from a stratified sample across microphone setups and speaking styles. That was fine right up until the one acoustic profile it was thin on happened to be the one where a quiet negation word sounds exactly like a filler pause.
At its worst, this ran for nineteen days before anyone caught it. About 640 episodes in that acoustic slice got processed with the bad model. A later full audit found 71 of them had a meaning-reversing cut, not just a dropped "um," a flipped sentence. Only four of those 71 had been caught by a customer complaint. The other 67 were still sitting there, published, quietly saying the opposite of what the speaker actually said, until someone went looking.
The choice I would take back. When incidents like this happened before, at Firstcut and everywhere, the instinct was to write it up, thoroughly, and move on. A good, honest postmortem, filed and closed. What that instinct missed is that a document doesn't stop a model. Only a test the next release has to pass does that.
What I would leave alone. Not every model change needs this level of follow-up. A filler-word model that starts catching two percent more "ums" than before, with no shift in which words get cut, doesn't need an audit of published clips or a new gated test case. Save the heavy follow-up for the failures that changed meaning, not the ones that just changed how much got trimmed.
The lesson. Closing an incident and preventing the next one are two different jobs. A team that only does the first one will keep having the same incident, dressed in a new postmortem each time.
Nothing on the follow-up list means anything until the exact broken pattern is named. Everything else, the eval-set case, the audit, the postmortem, is downstream of that one find.
Now here is the same thing as a story
The short version is above. Read on if you want to feel how ordinary the nineteen quiet days looked from Cassidy's side of the incident.
Every Monday morning, Cassidy Vantour pulls the weekend incident log before she does anything else, and reads every line twice. Five years running the edit-integrity desk at Firstcut taught her that the incident that reads calm on a Monday is usually the one still running.
The crutch-phrase update had shipped on a Wednesday. It cleared the golden set with room to spare, 98.6 percent of known filler caught, a false-cut rate on real words well under the bar. Nobody threw a party for it. It was a solid, boring release, exactly what a good one is supposed to look like.
For most of the following two and a half weeks, it was exactly that. Firstcut's automated QA sampled clips across the usual mix, studio interviews, two-person banter shows, scripted video essays, and everything looked clean. The sample simply didn't have much of the one thing that had gone wrong: a single speaker, close mic, talking softly, pausing, then finishing a sentence under their breath.
The failure wasn't rare because it was small. It was rare in the sample because nobody had stratified the sample by how a person actually sounds when they're recording alone in a quiet room.
Four creators wrote in over those nineteen days, each with the same complaint in different words: a clip pulled from their episode for a highlight reel said something they never said. One of them, a small true-crime narration show, had a line, "we do not know if she survived," turned into "we do know if she survived," in a clip that got reposted and mocked in a fan forum before the creator even noticed it had happened.
That fourth complaint landed on Cassidy's desk on a Thursday. She pulled the raw audio next to the cut clip, heard the missing "not" land exactly where a breath would have been, and knew inside four minutes what had happened, because she'd seen a smaller version of this shape once before, eleven months earlier, on a different model version.
A late Slack update is a bad Monday. An unlogged failure mode is a bad Monday that comes back, on a random Thursday, months from now, with a different customer's name on it.
That eleven-months-earlier incident had gotten a full write-up. Root cause, timeline, a paragraph on lessons learned, filed and closed within a week. Nobody had turned the specific failing case, a soft negation word after a pause, into a locked test the next model version had to pass. The team that shipped the crutch-phrase update had never seen that old postmortem. Why would they. It lived in a folder, not in the thing that actually decides what can ship.
Cassidy rolled the acoustic guardrail back within the day, a check that a word candidate for cutting needs real voiced structure matching a filler or crutch pattern, not just a quiet pause around it. Then she pulled every one of the 640 episodes processed in that acoustic slice during the nineteen days and ran them back through a corrected model. Seventy-one had a meaning-reversing cut. Sixty-seven of those had never been flagged by anyone.
The old decision, told as a memory of a release review. When the golden set for filler removal first got built, the team pulled 400 clips from whatever shows were already in the test library, mostly studio interviews and two-host banter, because that's what the team happened to have transcripts for already. Nobody decided to leave out quiet solo narration. It just wasn't in the pile anyone grabbed from.
What Cassidy actually wrote into this incident's follow-up plan: the exact case, the audio pattern, the pause length, the acoustic profile, went in as a locked regression test that the release gate checks on every future model version, before anything else on the list got started. The batch audit ran next, because you can't audit what you can't yet describe precisely. The postmortem got written after that, not instead of it.
Same failure shape, recurring within 90 days, by follow-up type
Postmortem only (5 past incidents)4 recurred
Postmortem + locked eval-set case (4 past incidents)0 recurred
Nine incidents at Firstcut over two years. A postmortem alone didn't stop the same failure shape from coming back four times out of five. Locking the case into the gated eval set stopped it zero times out of four, tracked over as long as fourteen months.
The thing Cassidy would tell her past self, back at that first release review eleven months earlier: a postmortem that isn't wired into what the model has to pass next time isn't a fix. It's a very thorough description of a fix that never got built.
The five things ORDER checks before the punch list is real
This is a ranking question, what belongs first on a follow-up list, not a story about a habit switching between two settings, so ORDER fits and a step-by-step recovery framework doesn't.
O
Outcome. What every item on the list is competing to protect.
Not "closing the incident." Whether the exact same failure can reach a customer again with nobody watching for it.
R
Reversibility. Rank by how hard the damage is to undo if the item gets skipped.
An unlogged failure that can silently ship again outranks a late status update every time, even though the update feels more urgent to send.
D
Dependency. What has to happen before anything else on the list means anything.
You can't write the eval-set case, audit the batch, or brief the postmortem until you've named the exact pattern: the word, the pause length, the acoustic profile.
E
Evidence. What you could check before trusting the rank.
Nine incidents, two years: four of five postmortem-only cases recurred, zero of four eval-gated cases recurred.
R
Rank. State the order, defend the top pick in one line.
Eval-set case, batch audit, live production check, postmortem, creator notice, status update. The top item is the only one that changes what the model is allowed to ship next.
The 71 flipped-meaning clips, how each one actually got found
Caught by a customer complaint
Only found once the full batch got audited
Waiting on customers to notice would have left 67 of 71 flipped clips uncorrected. This is the real argument for auditing the whole batch, not just the flagged episodes, second on the rank.
And if you want to be sure it really works, try it somewhere else
Cindermoor Grain Co-op runs an AI tool that scores incoming grain loads for moisture and quality from a handheld scanner, so elevator staff don't have to run every load through a slow lab test.
O, outcome. What every follow-up item competes to protect: whether the same misread load type can get accepted or rejected wrongly again, before a person double-checks it.
R, reversibility. A misread that already triggered payment on a bad load is far harder to undo than a misread caught by the on-site scale operator before the truck leaves.
D, dependency. Priyesh Konar, who owns quality operations there, first had to isolate exactly which grain color and moisture range the scanner was misreading, a specific corn variety scanned under high humidity, before anything else on the list could start.
E, evidence. Past incidents at Cindermoor followed the same split: write-up only, and the same misread type came back within a season; write-up plus a locked test case in the scanner's calibration check, and it didn't recur across three subsequent harvests.
R, rank. Lock the misread case into the scanner's calibration gate first. Re-check loads scored during the bad window second. Add a live spot-check for that grain type third. Write the incident report fourth. Notify affected farmers fifth.
Same shape, different stakes
At Firstcut the unlogged risk was a flipped sentence in a highlight clip. At Cindermoor it's a wrongly priced truckload of grain. The rank doesn't change: whatever can silently recur without a person catching it goes first, no matter how calm the incident felt to close.
Swap the trigger and it still runs.
Speed: an interviewer caps the answer at ninety seconds. Skip straight to the rule: rank by what's hardest to undo if skipped, and name the eval-set case as the top item, not the postmortem.
Cost: the team only has a few hours this sprint for follow-up work, not a full week. Don't drop the eval-set case to save time, that's the one item nothing else can substitute for. Shrink the batch audit's sample size instead, and say so explicitly as the accepted trade.
The model got better, and the incident never recurs on its own: even then, the eval-set case still earns its place. It's cheap insurance against the next update touching the same acoustic path, not a bet that this exact regression will return.
Where people run it wrong.
They write a thorough postmortem, call the incident closed, and never turn the specific failure into anything the next release actually has to pass.
They audit only the clips a customer complained about, and quietly leave the rest of the bad batch live and wrong.
They treat every incident's follow-up list as the same six items in the same order, instead of re-ranking by what this specific failure can silently repeat.
How to use it live. Say the rule before naming a single item: "the top of any incident follow-up list is whatever stops the same failure from shipping again on its own, and a document doesn't do that, a gated test does." That earns you the room to name the real top item for this specific incident, instead of reciting "postmortem, comms plan, retro" on autopilot.
Flashcards (click a card to flip it)
1 · THE FRAMEWORK
What framework fits "what follow-up work should always come out of an AI incident," and why?
Tap to flip
ANSWER
ORDER. It's a ranking question, what goes first on a follow-up list, not a story about a person's habit switching between two settings.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Cassidy Vantour, who has run the edit-integrity desk at Firstcut for five years and closes out incidents on its podcast and video auto-editing pipeline.
3 · WHAT WENT WRONG
What did the model start doing, and why didn't the golden set catch it?
Tap to flip
ANSWER
It started cutting quiet negation words, "not," "no," "don't," after a pause in breathy solo narration, treating them as filler. The golden set had almost no clips of that speaking style, so the release gate never tested for it.
4 · TOP OF THE RANK
What follow-up item goes first, and why does it outrank the postmortem?
Tap to flip
ANSWER
Locking the exact failure case into the gated eval set. It outranks the postmortem because it's the only item that changes what the model is allowed to ship next; the postmortem only explains what already happened.
5 · THE OLD DECISION
What decision would you take back, and why did it make sense at the time?
Tap to flip
ANSWER
Building the golden set from whatever clips the team already had transcripts for, mostly studio interviews and two-host shows. It made sense because it was the fastest way to get 400 clips together, and nobody had a reason yet to think one acoustic style mattered more than another.
6 · THE NUMBER
Fill in the blank: the incident ran ___ days, touched ___ episodes, and a full audit found ___ with a meaning-reversing cut, only ___ of which a customer had flagged.
Tap to flip
ANSWER
19 days. 640 episodes. 71 with a meaning-reversing cut. Only 4 flagged by a customer.
7 · THE EVIDENCE
What does Firstcut's own incident history show about postmortem-only follow-up versus eval-gated follow-up?
Tap to flip
ANSWER
Four of five postmortem-only incidents had the same failure shape come back within about seven weeks on average. Zero of four incidents with the case locked into the gated eval set recurred, tracked up to fourteen months later.
8 · CROSS-PRODUCT TRANSFER
Section 4 answers this same ranking question for a different product. Which product, and what's its version of the top-ranked item?
Tap to flip
ANSWER
Cindermoor Grain Co-op's grain-scoring scanner, run by Priyesh Konar. Its top item is locking the specific misread grain-type and humidity case into the scanner's own calibration gate before anything else.
Check yourself Score: 0 / 0
True or false
1. True or false: writing a thorough, honest postmortem should count as the top-ranked follow-up item after an AI incident, since it's usually the first thing a team does.
Show hint
Ask what a postmortem changes about what the model is allowed to ship next.
Show answer
False. A postmortem explains what happened, but it doesn't stop the same failure from shipping again. The eval-set case that gates the next release does that, so it ranks above the postmortem even though teams usually write the postmortem first.
Multiple choice
2. Why did four of the five postmortem-only incidents at Firstcut have the same failure shape come back within about seven weeks?
- A. Because the postmortems were poorly written and left out the technical cause.
- B. Because the team ignored the postmortems on purpose to save time.
- C. Because the specific failure case never became a test the release gate had to check, so a future model update could reintroduce the exact same cut with nothing stopping it.
- D. Because the customers who flagged the first incident never flagged the second one.
Show hint
Think about what a postmortem lives in versus what an eval set actually gates.
Show answer
C. A postmortem lives in a document. It doesn't automatically become part of what a future release has to pass, so the same failure shape stays possible until someone deliberately closes that gap.
Fill in the blank
3. Of the 71 episodes with a meaning-reversing cut, only ___ were caught by a customer complaint, and the other ___ were only found once the whole batch got audited.
Show hint
Check the stacked chart under "The 71 flipped-meaning clips."
Show answer
4; 67. Relying on customer reports alone would have left the large majority of flipped clips live and uncorrected.
Short answer
4. What alternative did the team consider and reject instead of building a targeted production check, and why did they rule it out?
Show hint
Look at stage 7 of the walkthrough, the rejected alternative.
Show answer
Model answer: Full human review of every upload in the affected acoustic slice, going forward. It got ruled out because that slice is about six percent of weekly uploads, roughly five thousand shows, and reviewing all of them by hand would take about 340 hours of QA time a week, which would break Firstcut's same-day turnaround promise.
Short answer, apply it yourself
5. Think of an AI tool at your own job, or one you use, that had a mistake get fixed at some point. Was the fix ever turned into a test that a future update has to pass, or did it just get fixed once and documented?
Show hint
Look for whether the fix is checked automatically now, or just remembered by whoever fixed it.
Show answer
Model answer: A spreadsheet automation tool once mislabeled a batch of vendor invoices after a template update. It got fixed and written up in a shared doc, but nobody added that specific vendor template to whatever regression check runs before future updates, so the same mislabeling could plausibly happen again the next time the template logic changes.
Short answer, the number question
6. If Firstcut had only audited the four episodes customers flagged, instead of the full batch of 640, would the incident have been considered fully closed? What would that have missed?
Show hint
Compare 4 flagged against 71 actually affected.
Show answer
No, not really. Stopping at the four flagged clips would have missed 67 more episodes with a meaning-reversing cut still live and published. The incident would have looked closed on paper while the majority of the actual damage sat there uncorrected, which is exactly why the batch audit ranks above the postmortem, not below it.
Before you close the answer
Why this works
Tests whether you treat incident follow-up as a fixed checklist or as a real ranking exercise. Most candidates list the same five or six items in whatever order they think of them. The strong answer separates the items that actually prevent a repeat from the items that only document or announce one, and puts the first kind on top every time.
Follow-up traps
"Isn't the batch audit more urgent than the eval-set case, since it fixes the actual live damage first?" Response: the audit fixes what already happened. The eval-set case stops the next occurrence. Both matter, but if you can only do one before the next release ships, the eval-set case is the one that keeps a tenth incident from happening at all.
"Why not just add a human review step for every clip from now on, instead of a narrow eval-set case?" Response: full review across that acoustic slice alone would cost about 340 hours of QA time a week and break the same-day turnaround the product is built around. A locked, narrow regression test catches the same failure at a fraction of the cost, which is the trade the team actually took.
If pressed
The production check added after this incident isn't a full transcript re-read. It's a lightweight acoustic flag: any cut candidate shorter than a stated length, sitting right after a pause longer than 0.4 seconds, with no voiced formant structure matching a known filler or crutch pattern, gets held back from auto-publish and routed to a ten-second human glance instead of going straight out.
