ConceptAdvancedQuality, Cost & Token Economics / Eval design for product teams / #24
Describe the relationship between evals and your incident response process.
Evals are the standing bar checked before every release. Incident response is what runs when something clears that bar anyway. The two only work as one system if every confirmed incident becomes a new, permanent eval case.
The direct answer
Evals and incident response are two ends of one loop, not two separate jobs. Evals are the checklist that runs before a model ships. Incident response is what runs when a real failure gets through anyway, and its job is to find the true cause, using timelines and segment cuts, not guesswork. The loop only closes if every confirmed cause becomes a new, permanent eval case, rechecked on every future release, instead of a ticket closed after the one bad transaction gets refunded. Skip that last step and the same failure comes back the next time the model changes, quietly, because nothing was ever taught to look for it again.
Do this, in order
Treat every confirmed incident cause as a new, permanent eval case, not a closed ticket.Why: an incident fixed in the moment and never saved as a test comes back the next time the model changes, since nothing will ever check for it again.
Run a real diagnosis, timeline first, before touching the model.Why: the release that caused the drift usually shipped weeks before the loss numbers moved, so a timeline that starts the day the alert fired starts too late.
Recut every metric by segment before trusting the company wide number.Why: a blended loss rate can sit flat while one merchant category or one channel is quietly bleeding money underneath it.
Rule out the pipeline before blaming the model.Why: a tagging bug or a stale score cache looks exactly like a real model miss on a dashboard, and chasing the wrong one wastes the incident's first hours.
Rerun the full eval suite, including every saved incident case, on every model swap.Why: a new model version can quietly relearn an old blind spot if the case that caught it once never gets checked again.
Leave ops only mistakes out of the eval suite.Why: a one time human error in a review queue is not a model failure, and folding it into the eval suite buries the real model cases under noise that never repeats on its own.
How to answer this, stage by stage
Nobody is grading whether you know the word "eval." They are grading whether you know that a passed eval and a live incident are the same information, just from two different moments in time.
1
Scope it to one real product before answering in the abstract
Say it like this
"Let's ground this in one product. GateKeep is the fraud model inside Corravane, a payments company. It scores every transaction zero to a hundred for fraud risk in about three hundred milliseconds, before the money moves. Serwaa Marsanne runs risk on it."
Why this works
A question this abstract turns into a definitions exercise fast. One product turns it into a real decision with real numbers.
2
Say your structure out loud before diving in
Say it like this
"I'll answer this in two parts. First, how I'd actually diagnose an incident once it's live, since that's a real method, not a vibe. Then, the one thing that has to happen after, or the eval suite never learns from it."
Why this works
Tells the interviewer you have a plan and a payoff, not just a definition to recite.
3
Reframe the question before answering it
Say it like this
"This isn't really two separate topics. Evals are what you check before you ship. Incident response is what happens when something gets past that check anyway. The real question is whether those two things ever talk to each other, or whether every incident just dies as a one off fix."
Why this works
Stops you from listing what an eval is and what incident response is, two glossary answers stapled together, instead of naming the actual relationship.
4
Give the one decision, plainly
Say it like this
"When an incident hits, I run a timeline first, what shipped and when, then I recut the numbers by segment, then I rule out the pipeline before I blame the model, then I name my top suspects and run the one check that tells them apart. Once I know the real cause, it becomes a permanent eval case, checked on every future model version. Not a ticket. A test that never goes away."
Why this works
This is the actual answer, in one breath: the method for diagnosing it, and the rule for what happens to it after.
5
Prove it with the failure, cut to four sentences
Say it like this
"Here's what happens without that last step. GateKeep's company wide fraud loss rate barely moved, point three five to point three eight percent, after a model update three weeks earlier. But inside gift card transfers alone, the loss rate climbed from about one percent to over nine percent in eleven days, because a fraud ring had found the one pattern the new model scored low. It took a partner merchant emailing about chargebacks to catch it, since the blended number never tripped a single alert."
Why this works
Shows the real cost of a metric that looks fine on average while one slice is on fire.
6
Say what you would measure going forward
Say it like this
"I'd watch loss rate by merchant category and by channel, never just the company wide number, since that's exactly where this hid. And I'd rerun the entire eval suite, including every case built from a past incident, on every model swap, not only when someone remembers to."
Why this works
Shows you're thinking past the postmortem meeting, and names the one signal that would have caught this early.
7
Say what you'd leave alone
Say it like this
"Not every incident needs a permanent eval case. If a reviewer manually overrode a correct model score by mistake, that's a training gap on the ops side, not a model failure, and it doesn't belong in the eval suite."
Why this works
Shows judgment instead of turning every bad day into a permanent addition to the model's test suite.
8
Close on the decision, not the story
Say it like this
"So: evals are the memory of every incident response has ever solved. If the diagnosis doesn't end in a new eval case, the incident wasn't fixed, it was postponed."
Why this works
Ending on the rule, not the anecdote, is what makes this sound like a method you'd actually reuse under pressure.
Let's learn
A payment gets typed in, a card, an amount, a merchant, and in under a second something has to say yes or no. That something is GateKeep, the fraud model inside Corravane's payments platform.
Before GateKeep, Corravane ran a flat rule: block anything over four hundred dollars, and send it to a human. A team of risk analysts reviewed about eight hundred flagged transactions a day, roughly two minutes each, and caught most of the obvious fraud. Anything under four hundred dollars just went through.
Knowledge spark: what does a fraud score actually score?
A number from 0 to 100 that says how likely GateKeep thinks a transaction is stolen money, based on the card, the device, the amount, the merchant, and the time of day. High auto declines. Low auto approves. The middle band still goes to a person.
Now GateKeep scores every transaction, not just the big ones, and the company wide fraud loss rate fell from about 0.9 percent of transaction volume to 0.35 percent in the first two quarters. Everyone was happy. The dashboards stayed green for months.
The company wide number staying flat did not mean nothing was wrong. It meant the damage was small enough, next to everything else GateKeep scored correctly, to hide inside the average.
At its worst, the extra mistakes GateKeep makes are not the real problem. A slightly higher false decline rate on good customers is annoying, but it is visible, and it gets fixed fast because customers complain. The real problem is a pattern the model consistently scores low, on purpose found by someone who wants it to, running quietly inside one slice of the business while the overall number looks fine. That costs real money for days or weeks before anyone notices, and if the root cause never becomes a standing test, it can come back the next time the model changes.
The pattern started right after the release that looked like a win. Nobody knew for eleven days.
Fraud loss rate: company wide vs gift card transfers, the eleven day window
Company wide loss rateGift card transfer loss rate
Company wide, the loss rate barely moved, 0.35 to 0.38 percent the whole window. Inside gift card transfers alone, it climbed from about 1.1 percent to over 9 percent in eleven days, a slice nobody was watching on its own.
The choice that mattered
When GateKeep first launched, an incident postmortem ended with a note on the ticket: "add these transactions to the next training refresh," then the ticket got closed. Turning every finding into a permanent, standing eval case felt like extra process for a problem that already had a fix. That made sense with one small model and a team that remembered its own history. It stopped making sense the day nobody on the team had been there for the original incident anymore.
What I'd leave alone: a one time reviewer mistake, someone approving a transaction they should have declined by hand, does not need to become a permanent eval case. That's a training gap for a person, not a blind spot in the model.
The lesson: a model that passes its eval suite today has only been checked against what someone remembered to write down. Every real incident is a fact the eval suite did not know yet. The only way it learns that fact permanently is if the diagnosis ends in a saved test, not a closed ticket.
Now here is the same thing as a story
Read the short version above for the two minute answer. Read this when you want to feel why closing that ticket felt like the reasonable thing to do at the time.
Before GateKeep, Serwaa Marsanne built and ran the old rules engine by hand for three years. She knew the four hundred dollar threshold, she knew which merchant categories got abused around the holidays, and she could spot a stolen card pattern in a stack of flagged transactions faster than anyone on her team.
GateKeep launched, and the good months were good. Loss rates fell. Analysts spent their time on the transactions that genuinely needed a human, not on a flat dollar cutoff. Every incident that came up in those first months got a fast, sharp postmortem, and Serwaa personally remembered every pattern they had ever caught.
The habit thinned in three small beats. First, as the team grew from three analysts to eleven, postmortems stopped being something Serwaa wrote herself and became a template anyone could fill in, with a free text "root cause" field instead of a required link to a standing eval case. Then, when a quarterly model refresh went smoothly twice running, nobody double checked whether the training data actually included every pattern from the last two incidents, since the notes said it did. Then, as Serwaa moved from analyst into risk lead, she stopped reading every postmortem line by line and started skimming the "resolved" ones.
The trigger was small. A new analyst, three weeks into the job, asked in a team channel: "Is this weird gift card pattern from this week's alerts the same one in that old incident doc from nine months ago?" Serwaa read the doc. She genuinely could not tell.
The postmortem had said "fixed in the next training refresh." It never said how anyone would check, a year later, that the fix had actually made it in.
Three weeks after GateKeep's latest model version shipped, a fraud ring found a pattern it liked: two or three small warm up transfers under fifteen dollars on a stolen card, followed by a fast, larger gift card transfer, timed for the middle of the night. GateKeep scored these consistently low, straight into the auto approve band. Company wide, the loss rate barely moved. Inside gift card transfers alone, it climbed from about 1.1 percent to over 9 percent in eleven days. Nobody caught it internally. A partner merchant emailed asking why their chargebacks had tripled that month.
The decision that opened the door went back to that first year, when a finding closed as "next refresh" felt like enough. Nobody wrote a rule saying a finding also becomes a saved, versioned eval case that reruns automatically, because at the time the team was small enough that everyone just remembered.
Run the eleven days again with one change. The saved case library, built the week after the new analyst's question, now includes 340 confirmed incident patterns, checked automatically before every model swap. On the next swap, it flags the near identical gift card pattern in nineteen minutes, before a single transaction using it clears.
One design trusted a memory that lived in people's heads and a free text field. The other trusted a test that runs whether anyone remembers the story behind it or not.
What I'd tell myself, back at that first postmortem: the day "next refresh" feels like a fix, ask who checks that it landed, and how, a year from now, when the person writing the note has moved to a different team. Nobody asked. That's on the process, not on Serwaa.
TRACE, the five moves that turn an incident into a real cause
Not a checklist for a status meeting. Five moves that build toward the one that actually separates a real fix from a guess: the evidence test.
TTimeline. When did it actually start, and what shipped near that date?
GateKeep's model v14 shipped three weeks before the incident opened, and it looked like a clean win, precision on the standard eval set went up. The pattern started the same week v14 went live, eleven days before anyone caught it.
A timeline that starts the day the alert fired starts too late. The real start is usually the release that looked like good news.
RRecut. Slice it by segment, and see what the average was hiding.
Company wide loss rate, 0.35 to 0.38 percent, basically flat. Gift card transfers alone, 1.1 to over 9 percent in the same eleven days. The average hid a slice that was on fire.
A thirty percent average drop, or in this case a flat average, is often one segment moving a lot and everything else absorbing it.
AAssume nothing. Rule out the pipeline before blaming the model's judgment.
Before trusting the numbers, Serwaa's team checked whether the loss reporting pipeline itself was double counting chargebacks, or whether the merchant category tag had quietly changed meaning in the data warehouse. Neither had.
A tagging bug or a reporting glitch looks exactly like a real behavior change on a dashboard. Ruling it out first saves the incident's first, most valuable hours.
All three looked plausible on day one. Only one of them survived the evidence test.
CCause candidates. Name three, not everything possible.
One, the fraud ring found a genuine blind spot in GateKeep's scoring for this exact pattern. Two, the gift card merchant category had been mis-tagged, so the wrong rules were applying to it. Three, the score cache was serving stale, pre-update scores for a slice of traffic.
Three named suspects, not a shrug. This is the step most rushed incidents skip, jumping straight to a fix for whichever one someone guesses first.
EEvidence test. The one check that tells the suspects apart.
Serwaa's team pulled the actual GateKeep score assigned to each confirmed fraud transaction, not just the outcome. If the scores were genuinely low, it was a real model miss. If the scores were high but the decline never fired, it was a plumbing bug. Every confirmed transaction had scored low, honestly, straight into the auto approve band.
This single check is the strongest move in the whole method. It turns three guesses into one confirmed cause in a single query.
Three things worth stating directly, since this is where the real judgment sits. The rejected alternative was tightening the auto approve threshold across the board the moment the incident opened, so far fewer transactions clear automatically. It lost because it would have pushed thousands of clean, ordinary transactions into manual review overnight, a real cost in review time and customer friction, to guard against a pattern that only lived inside one merchant category. The AI specific failure worth naming by name is adversarial drift: a bad actor doesn't wait for the model to get worse on its own, it actively searches for the exact pattern the current model scores low, and that search gets easier, not harder, as the model gets more consistent. The guardrail is the saved, versioned eval case plus a by-segment loss rate check in production, since together they catch it whether the pattern is found before release or after. And the trade being accepted on purpose is real: rerunning the full incident case library on every model swap adds close to a day to the release cycle, and a few of GateKeep's saved cases now block model versions that are measurably better on the standard eval set but regress on one narrow, high stakes pattern. That's a real cost, traded against a fraud ring that has no reason to ever stop searching.
The five, in one line each: T: the real start is the release that looked like a win, not the day the alert fired. R: a flat average can sit on top of one segment quietly cratering. A: rule out the pipeline before trusting that the model actually failed. C: name three real suspects, never jump straight to the first guess. E: one evidence test, pulling the real score behind the real outcome, settles it.
Same five moves, a permit desk instead of a payments ledger
Not every incident is about money moving. Sometimes the thing that got through is a decision nobody double checks until a building is already half built.
Windemere County runs PermitTrail, a tool that reads building permit applications and decides which ones can auto approve, like a simple fence or a deck, and which ones need a human inspector to look before construction starts. Rooke Adeyera runs permitting operations on it.
PermitTrail's overall auto approve accuracy held steady, around 97 percent, for months. Then a local contractor's accessory dwelling unit applications started failing inspection after the foundation was already poured, four times in six weeks, all from the same contractor, all auto approved. Rooke's team recut the numbers by application type instead of trusting the county wide accuracy figure. Accessory dwelling unit applications from that one contractor had an auto approve rate of 91 percent, while the county wide average for that permit type from every other contractor sat at 34 percent.
The evidence test that settled it
Rooke's team checked whether the county's footing depth code had changed six weeks earlier, and whether PermitTrail's scoring had been updated to match it. The code had changed. PermitTrail had not. The contractor hadn't done anything wrong on purpose, their applications just happened to use the exact phrasing that PermitTrail's now-outdated rules still auto approved.
Same method, different shape: a fraud ring searching for a blind spot and a code update nobody synced to the scoring rules look nothing alike on the surface, but both are the same TRACE move: don't trust the average, recut by segment, then run one test that separates a real cause from a guess.
Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to the one line: evals are the standing bar, incident response is what happens when something clears it anyway, and the loop only closes if the cause becomes a permanent test.
Cost: there's no time this sprint for a full TRACE workup. Do the recut and the evidence test only, the two cheapest, highest value steps, and save the full timeline reconstruction for the postmortem once the bleeding has stopped.
The model got better, for real: say the new fraud model is measurably more accurate on average. That's exactly when a fraud ring gets more dangerous, not less, because a sharper model has fewer easy blind spots left, so the ones that remain are more deliberately hunted for.
Where people run it wrong.
They trust the company wide metric and never recut by segment, so a small slice can bleed for weeks under a flat looking average.
They close an incident with a fix to the current model version and never turn it into a versioned eval case, so the same blind spot can quietly return on the next model swap.
They skip ruling out the pipeline first, and spend the incident's first hours chasing a "model failure" that was actually a stale cache or a mis-tagged field.
How to use it live. Say the two halves out loud before answering: "how would I find the real cause, and what has to happen to it after I find it." That buys a beat of thinking time, and it turns the rest of the answer into naming the one thing most candidates skip, the part after the fix.
Flashcards (tap any card to flip it)
1 · THE FRAMEWORK
What framework is this, and what's its one job?
Tap to flip
ANSWER
TRACE: timeline, recut, assume nothing, cause candidates, evidence test. Built for diagnosis questions, when something broke and you have to find the real cause before you can fix it.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Serwaa Marsanne, risk lead on GateKeep, the fraud scoring model at Corravane. Built and ran the old rules engine by hand for three years before GateKeep launched.
3 · THE HABIT
What did the team stop doing as it grew?
Tap to flip
ANSWER
They stopped requiring every postmortem to link to a standing, versioned eval case, and let a free text "root cause" note and a promise about the "next refresh" stand in for an actual saved test.
4 · THE TWO SETTINGS
What's the two setting switch this answer turns on?
Tap to flip
ANSWER
An incident closed as a one off fix noted on a ticket, versus the same finding saved as a permanent, versioned eval case, rerun automatically on every future model swap.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Letting a postmortem end with "fixed in the next refresh" and no rule requiring anyone to check, later, that the fix had actually landed as a saved, checked eval case.
6 · THE NUMBER
Fill in the blank: gift card transfer loss rate climbed from about 1.1 percent to over ___ percent in ___ days, while the company wide rate barely moved.
Tap to flip
ANSWER
Over 9 percent, in 11 days. Company wide loss rate held at 0.35 to 0.38 percent the whole time, the average that hid the real damage.
7 · THE REPLAY
Same eleven days, new design, what changes?
Tap to flip
ANSWER
The saved case library, now 340 confirmed incident patterns, reruns automatically before every model swap. It flags the near identical gift card pattern in 19 minutes, before a single transaction using it clears.
8 · CROSS PRODUCT TRANSFER
Section 4 answers this same question again for a different product. Which product, and what caused the incident this time?
Tap to flip
ANSWER
PermitTrail, a building permit triage tool at Windemere County. The cause was a footing depth code update that shipped six weeks earlier without PermitTrail's scoring rules being updated to match it.
Check yourself Score: 0 / 0
True or false
1. True or false: because GateKeep's company wide fraud loss rate barely moved after the model update, that means the update caused no real harm.
True
False
Show hint
Look at the line chart in Section 1. Compare the gift card transfer line to the company wide line on its own.
Show answer
False. The company wide rate held near 0.35 to 0.38 percent the whole window, but the gift card transfer loss rate alone climbed from about 1.1 percent to over 9 percent in eleven days. A flat average can sit right on top of a badly broken slice.
Multiple choice
2. In the TRACE method, what is the point of the "assume nothing" step?
A. To double check the model's training data one more time.
B. To interview every analyst who touched the flagged transactions.
C. To rule out a reporting or pipeline bug before concluding the model's judgment actually failed.
D. To lower the auto approve threshold immediately as a precaution.
Show hint
Think about what a tagging bug or a stale cache would look like on a dashboard, compared to a real model miss.
Show answer
C. A pipeline bug can look exactly like a genuine model failure on a dashboard. Ruling that out first saves the incident's first, most valuable hours from being spent chasing the wrong cause.
Fill in the blank
3. Fill in the blank: the saved case library, rebuilt after the gift card incident, caught the next similar pattern in ___ minutes, before any transaction using it cleared.
Show hint
The number is in Section 2's story and repeated on flashcard 7.
Show answer
19 minutes. That's the payoff of turning a finding into a standing eval case instead of a closed ticket: the next time a near identical pattern shows up, a test that already exists catches it fast, instead of waiting for another partner to notice chargebacks piling up.
Short answer, name the rejected alternative
4. What did Corravane's team decide to do with incident findings when GateKeep first launched, and why did that decision make the gift card incident possible months later?
Show hint
Look at the block-key box titled "The choice that mattered" in Section 1.
Show answer
Model answer: They closed each finding with a note that it would be fixed in the next training refresh, instead of a rule requiring it to become a permanent, versioned eval case. So when the team grew and the people who remembered the original incident moved on, nothing automatically retested for the same pattern, and it shipped again undetected.
Multiple choice
5. PermitTrail's auto approve rate for one contractor's accessory dwelling unit permits sat at 91 percent, while the county wide rate for that permit type was 34 percent. What does that combination tell you?
A. That contractor is simply better at filling out paperwork than everyone else.
B. A segment specific gap this large is a signal to recut and investigate, not a coincidence to wave off.
C. The county wide accuracy figure proves the tool is working correctly everywhere.
D. PermitTrail should stop auto approving that permit type for every contractor immediately, with no further check.
Show hint
Look at what the recut by contractor and permit type showed that the county wide figure hid completely.
Show answer
B. A 57 point gap between one contractor's rate and the county wide rate for the same permit type is exactly the kind of segment split TRACE's recut step exists to find, and it turned out to trace back to an unsynced code update.
Short answer, apply it yourself
6. Pick an AI product you use yourself. If it started quietly failing in one narrow way, what's the one number you'd recut by segment first, before trusting the overall stats page?
Show hint
Think about a product with an overall score or rating that could hide a much worse number inside one category, language, or region.
Show answer
Model answer: A language learning app's overall "lesson accuracy" score. I'd recut it by language pair before trusting the app wide number, since a model can be excellent on the two most common language pairs and quietly wrong on a less common one, with the overall average never showing it.
Before you close the answer
Why this works
Tests whether you think evals and incident response are two separate jobs on two separate teams, or one loop where each incident is supposed to make the eval suite smarter. Most candidates describe each process on its own and never say how they connect.
Follow-up traps
"Doesn't rerunning the whole case library on every release slow you down too much?" Response: yes, close to a day per release, and that cost is accepted on purpose, since a fraud ring has no reason to stop searching for the next blind spot and a day is cheap next to eleven days of quiet losses.
"Isn't recutting by segment after the fact just p-hacking, slicing until you find a story?" Response: no, because the slice, merchant category and channel, was chosen for reasons that predate the incident, it's how the business is already organized, not a cut searched for after seeing the data.
If pressed
The evidence test that settled GateKeep's incident pulled the model's actual score against each confirmed fraud transaction, not just the final decision. A score that was genuinely low proves a real model miss. A score that was high but the decline never fired proves a plumbing bug instead, and the two get fixed by completely different teams.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
