CaseIntermediateEval-Driven Specification / Golden datasets and test set ownership / #8
Describe a process for adding newly discovered failure cases to the golden set.
The direct answer
Give every newly found failure one required stop: a shared intake queue where anyone can add the bad transcript and the right answer, and it becomes a real regression check within a day, before anyone's private fix even ships. A quiet, one-off patch is not the same thing as a permanent test. Only the second one stops the mistake from coming back.
Do this, in order
Give every discovered failure one required stop: a shared intake queue, not a private fix.Why: this is the check the whole flip turns on. Skip it and a real mistake can vanish into one person's notes.
Make the intake fast enough that filing it beats fixing it quietly and moving on.Why: a slow, heavy form is exactly why people build their own private tracker instead.
Confirm every submission back to the person who filed it, added, tested, or turned down.Why: a submission that vanishes with no reply is the reason people quietly stop filing anything at all.
Turn every added case into a real regression check that runs before every release.Why: a case sitting in a spreadsheet doesn't stop the same mistake from shipping again.
Track how many discovered failures never make it past someone's private notes.Why: that number tells you the queue is too slow before a customer tells you first.
Leave the full engineering review in place for a case an engineer already found through their own model testing.Why: judgment, not blanket speed. That review is still doing real work there.
How to answer this, stage by stage
Seven moves. Most of the weight sits in stage five: this question is really asking what happens to a real mistake the day nobody makes it easy for the person who found it to report it. Every stage has the actual words to say.
1
Ground it in one product, one person
Say it like this
"Let me make this concrete. Say a telecom runs an AI chatbot on its outage line. A customer texts it to ask if their service is down and when it'll be fixed. A support quality analyst reads a sample of the bot's transcripts every morning, and he's the one who actually finds most of the new ways it gets things wrong, not the golden set itself."
Why this works
Nobody can judge how you'd add a failure case to a golden set without a real person who finds one first.
2
Say your structure in one breath
Say it like this
"Five things, fast. Who actually finds the failure. What he did with it before there was a real process. The switch that has no middle setting. The call I'd take back. And the same storm, replayed with a real intake in place."
Why this works
A named route up front tells the interviewer you have a plan, not a story you're inventing live.
3
Reframe what the question is actually testing
Say it like this
"This isn't really asking for a form to fill out. It's asking what happens to a failure case the day nobody makes it easy to report, because the person who finds it is almost never the person who owns the golden set."
Why this works
That's the difference between a real process and "just add it to the set," which is the assumption the question is already making.
4
Give the one decision
Say it like this
"Concretely: build one shared queue where anyone who spots a new failure case can add it with the bad transcript and the right answer, in under a minute, and it becomes a real regression check within a day. Full engineering review can happen after that, never before it."
Why this works
There's a mechanism and a real time limit in that sentence, not just "improve reporting."
5
Prove it with the compressed failure
Say it like this
"Say the analyst finds 47 wrong zone matches over fourteen months and logs every one in his own sheet, because the official form needs an engineering ticket and a review meeting. Only 6 ever make it into the shared set. He takes three weeks off. The exact zone boundary he flagged five months earlier hits during a real storm, and by the time a colleague traces it herself, 210 customers already got the wrong return time."
Why this works
Four sentences, and it still lands on the exact week the private list and the real golden set split apart.
6
Say what you'd measure, and what you'd leave alone
Say it like this
"I'd track how many discovered failures never make it past someone's own private notes. That number tells you the queue's too slow before a customer ever does. I wouldn't put a fast lane on a case an engineer already found through their own model testing, the full review is still doing real work there."
Why this works
Shows this isn't blanket speed for its own sake. Some checks still earn their slowness.
7
Close on the one line
Say it like this
"So here's the whole thing in one breath. Never let 'I already fixed it myself' be a reason a failure case skips the shared set. One queue, one day to become a real check, and the same mistake stops getting to happen twice."
Why this works
Ends on the exact sentence an interviewer can repeat back to their own team.
If you remember one thing
A discovered failure doesn't need a careless person to get lost. It just needs a slow, heavy form and a faster private way to fix it, and most people will quietly take the faster way every time.
Let's learn
Picture an AI chatbot on a telecom's outage line. A customer texts it to ask if their service is down, and it checks their address against every outage the network is tracking right now and answers in under a second.
Knowledge spark: what's a golden set?
A stack of past conversations where someone already confirmed the right answer: the right outage, the right return time. New versions of the bot get tested against it before anyone trusts it with a real storm.
For the first year, the bot's mistakes were rare and small. It matched customers to the right outage zone almost every time, and the golden set kept every new version honest before it shipped.
Then a new kind of mistake showed up. Some addresses sit right on the line between two outage zones. The bot would pick the wrong one and tell a customer their service was already back, when it wasn't, or that they'd wait until tomorrow, when a truck was already two streets over.
A support analyst who reads the bot's transcripts every morning could catch a wrong zone match within two lines. When one turned up, the analyst wrote it down on a personal tracking sheet and pinged the on call engineer directly. The official form to add it to the shared golden set almost never got filled in. It opened an engineering ticket and needed a review meeting nobody had time to sit in that day.
Where a discovered zone mismatch actually ended up, over fourteen months
Forty seven zone mismatch cases were real, and found, and written down. Only six ever became a check the golden set would run again.
The mistakes weren't lost. They were just kept somewhere only one person could read them.
Three weeks later, while that analyst was out, the exact zone boundary flagged five months earlier came up again, in the middle of a real storm. Nobody else knew it had already happened once. By the time a colleague traced the same root cause alone, 210 customers had already been told the wrong return time that week, some of them twice.
Customers told the wrong return time, weekly, during a three week outage window
210 customers got the wrong return time across the three weeks. The exact case had already been found and written down five months before the storm ever hit.
At its worst, this costs the exact thing the bot was built to prevent. Thirty of those 210 customers called back a second time to ask why the bot had lied, which is more calls than if the chatbot had never answered at all that week.
The decision that mattered
Give every discovered failure one required stop: a shared intake queue that turns into a real regression check within a day. Not a bigger review team. Not a stricter approval meeting.
What I would leave alone. A case an engineer already found through their own model testing can keep going through the full ticket and review, that's still the right amount of care when the person who found it already understands the model.
The lesson. A golden set that's slow to add to doesn't stay small and careful. It just stops being where new mistakes live. If people can't reach it in under a minute, they'll build somewhere else that they can.
Now here is the same thing as a story
Pull this one out when there's room to sit with it, not just tick it off a list.
Sigurd Braathen has read outage line transcripts for four years, the last two of them as the lead quality analyst at Talbrook Wireless, a regional carrier with about half a million lines. Hand him a flagged conversation and he can tell within two lines whether the bot matched the right outage zone, just from how the customer describes their cross streets.
Talbrook's outage bot ships a new version every few weeks, chasing whatever new address pattern is tripping it up that month. Before Sigurd even knew what a golden set was, three reliability engineers built it themselves, one case at a time, and kept it tight with a simple rule: any new addition needed a ticket and a short review meeting, so nothing sloppy got in.
That rule made sense in month one. The three engineers who wrote it were also the ones filing almost every case.
Then Sigurd's morning review started catching more real failures than the engineering team ever did, because he was the one actually reading what customers typed. The first few times, he filed the proper ticket, sat through the fifteen minute review, and watched his case land in the shared set. It felt slow, but it worked.
By the fourth or fifth case, filing the ticket started competing with the rest of his morning. So he built himself a simple spreadsheet: the bad transcript, the right answer, the date. Ten seconds, and he could move to the next one. He still pinged the on call engineer with the fix, so the bot got corrected either way. The ticket just stopped happening.
People are switches, not dials
Fourteen months in, his own sheet held 47 zone mismatch cases. Only 6 of them had ever gone through the ticket and made it into the shared golden set. Sigurd wasn't hiding any of it. If anyone had asked to see his sheet, he'd have shown it in a second. Nobody asked.
He took three weeks off in October, the first real break he'd had in two years.
During his second week off, a real storm rolled through the network's south corridor, right along a zone boundary Sigurd had flagged five months earlier and never got around to filing. A colleague on the review queue, Dagny Aas, started fielding angry callbacks about wrong return times. She had no idea this exact boundary had already broken once. She spent almost an hour tracing the root cause from scratch, the same hour Sigurd had already spent back in May.
We didn't lose Sigurd's list. We lost the five months it sat only where he could see it.
By the time Dagny had it pinned down and pushed a manual override, 210 customers had already been told the wrong return time that week. Thirty of them called back a second time, certain the bot was lying to them.
I want to say Sigurd should have filed the ticket. He knew he should have. But that's not really the story either. Sigurd's only real defense against a repeat mistake was a habit: finishing the form so the case became something the whole team could see. That habit only had two settings. He either finished it, or he fixed it his own way and kept the record to himself. Once the form started losing to his own sheet on speed, there was no smaller version of finishing it left to fall back on.
Here's the call I'd take back. Back when the golden set was young, the three engineers who built it decided every addition needed a full ticket and a review meeting, so the set stayed clean while they were the ones filling it. Nobody revisited that rule once a support analyst started finding more real failures than they did. The same gate that kept the set clean at the start is what kept Sigurd's best evidence out of it for a year.
I'd put a fast lane in. Not instead of review, alongside it. Any case with a real transcript and a confirmed right answer goes straight into a candidate queue and becomes a live regression check the same day. The engineering review still happens, it just happens after the check is already running, not before.
Run the same October again, fast lane in place. Sigurd's boundary case has been a live check since May. When the same storm rolls through in October, the bot's own regression suite already flags the pattern before it ever reaches a customer, because it's been testing for it for five months. Dagny doesn't spend an hour tracing anything. She sees the flag, confirms it by 9am, and about 14 customers get a wrong answer that morning before the override goes out, not 210.
If I'm honest, the mistake wasn't that Sigurd built a spreadsheet. Anyone would have. The mistake was writing a rule that made the fast, private way the only way that actually kept up with his mornings, and never asking what that rule would cost once he got good at his job.
Where the fix actually had to live
The letters matter less than which one breaks first. Here's the same five steps, mapped onto Sigurd's queue.
FLIPS, five rows
FFind the person
Who actually finds the failure?
Not the golden set team in the abstract. Whoever reads the real transcripts and catches a mistake before anyone else does.
In this answer: Sigurd Braathen, lead quality analyst at Talbrook Wireless, who reads outage line transcripts every morning and catches most new failures before the engineers ever do.
LLocate the habit
What did he do with a new failure before the ticket started losing?
Look for the check that quietly went from routine to skipped, not his overall care. A slow form is what buys a private habit its opening.
In this answer: He filed the proper ticket for his first few cases, then stopped, once the ticket and review meeting started costing more than his own tracking sheet did.
IIdentify the flip
What two setting switch snaps, with no middle?
"He got less thorough about reporting" describes the outcome, not the action. Name the exact two states with nothing between them.
In this answer: Adds a new failure straight into the shared golden set with the transcript and the right answer, or fixes it his own way and keeps the only record on his own sheet. Nothing in between once the ticket stopped being the fast path.
PPinpoint the old decision
Which call only made sense before a support analyst outpaced the engineers?
Look for a narrow, defensible call from the golden set's early days. "Add a fast lane" after the fact doesn't count, that's a new dial.
In this answer: Talbrook's three reliability engineers set every addition behind a full ticket and review meeting, back when they were the ones filing almost every case themselves.
SShow the replay
Same storm, fast lane restored. Better ending?
Run the identical trigger through the fixed design and see where it stops. A count or a clock, not an adjective.
In this answer: Sigurd's boundary case has been a live regression check since May. The October storm gets flagged before it spreads, and about 14 customers get a wrong answer instead of 210.
A small move in the count. A hard snap in what he did with it.
"He got less careful about reporting" is a diagnosis anyone can offer after the fact. The harder part is naming the exact habit that had to break first, finishing the ticket, and showing there was no smaller version of it left once it did.
And if you want to be sure it really works, try it somewhere else
Copperbend Sanitation Services runs an AI phone line that tells residents when their bin gets picked up, and keeps a golden set of past calls to certify any new version before it answers real residents. Same question, a waste pickup line instead of an outage bot, and a flip that isn't a private workaround this time.
F. Mirela Costache, route exceptions coordinator at Copperbend Sanitation Services, who fields every call about a missed or wrong day pickup and is usually the first to spot a new bot mistake after a route change. L. For her first few months, she filed a new failure into the golden set intake form the same day she found it, every time, no exceptions. I. A different flip. She doesn't hide the mistake or build a workaround, she stops filing anything at all. Files a new failure the day she finds it, or quietly tells the resident by hand and moves on, with nothing in between once the habit broke. P. The intake form never confirmed anything back. Nobody told Mirela whether a case she filed got added, tested, or thrown out, so after her fourth submission vanished with no reply, filing one more felt like talking to a wall. S. Add one line of confirmation, added, tested, or rejected, sent back within a day. The same missed pickup pattern comes up again after the next route change. Mirela files it, gets confirmation by the next morning that it's now a real check, and keeps filing the next one instead of quietly working around it by hand.
Weeks a repeat missed pickup pattern went unfixed on the same route, Copperbend
No confirmation, submissions felt like they vanished
Old design
11 weeks
Confirmation within a day, filing kept happening
New design
4 days
Old design: the pattern went unreported for eleven weeks, because Mirela had already stopped trusting the form to do anything with what she sent it. New design: she files the same day, gets confirmation the next morning, and the route error is a live check before the following week's pickups.
A second decision worth taking back
Building an intake form and never closing the loop on it is itself a decision, not a fact about how busy people are. A rule that said every submission gets one line back, added, tested, or rejected, would have kept Mirela filing the whole time.
Swap the trigger and it still runs
Speed: if Talbrook only shipped a new bot version twice a year instead of every few weeks, a boundary case like Sigurd's would sit unflagged even longer, because there'd be fewer chances for a regression suite to matter at all.
Cost: if filing a failure case needed real engineering time to write the test, both Sigurd and Mirela would ration what they reported, saving the form for whatever felt worst instead of whatever was actually common.
The bot got better: this is close to what actually happened at Talbrook too. The chatbot wasn't getting worse, it just kept meeting new address patterns nobody had written down yet, and "nothing looks broken yet" is exactly what makes reporting feel optional.
Where people run it wrong
Blaming the analyst for not filing enough, when the form itself was slower than just fixing it by hand.
Building a bigger, stricter review board for new cases, when the real gap was a missing fast lane and a missing confirmation, not too little scrutiny.
Waiting for a customer complaint to reveal the gap, instead of checking how many discovered failures are sitting in someone's private notes right now.
How to use it live
Buy yourself a few seconds by naming the reframe before the fix: "The question isn't whether we have a golden set, we do. It's whether the person who finds a new mistake has anywhere fast enough to put it." Say that, and the rest of the answer is just the mechanism.
Flashcards (click a card to flip it)
Eight fixed slots, pulled straight from the answer above.
1 · THE FLIP FAMILY
What flip family is this?
Tap to flip
ANSWER
Workaround flip. Sigurd doesn't hide the mistake or stop looking for it, he builds a private tracker instead of the slow, official path, because the golden set gave him no fast way to record what he found.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Sigurd Braathen, lead quality analyst at Talbrook Wireless, four years reading outage line transcripts. He can spot a wrong zone match within two lines of a transcript.
3 · THE HABIT
What did he stop doing because it kept working?
Tap to flip
ANSWER
He stopped finishing the official ticket to add a new failure to the shared golden set. He logged it in his own tracking sheet and pinged the on call engineer directly instead.
4 · THE FLIP, HERE
What's the two setting switch in this story?
Tap to flip
ANSWER
Adds a new failure straight to the shared golden set with the transcript and the right answer, or fixes it privately and keeps the only record on his own sheet. Nothing in between once the ticket stopped being the fast path.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Talbrook's three reliability engineers set every golden set addition behind a full ticket and review meeting, back when they were the ones filing almost every case themselves.
6 · THE NUMBER
Sigurd's private sheet held ___ cases. The shared golden set held only ___.
Tap to flip
ANSWER
47 cases on his own sheet, over fourteen months. Only 6 of them ever made it into the shared golden set.
7 · THE REPLAY
Same storm, fast lane restored, what changes?
Tap to flip
ANSWER
Sigurd's boundary case has been a live check since May. The October storm gets flagged before it spreads, and about 14 customers get a wrong answer that morning, not 210.
8 · CROSS-PRODUCT
Section 4 answers this same question again for a different product, with a different flip family. Which product, and which family?
Tap to flip
ANSWER
Copperbend Sanitation Services, a waste pickup phone line, using the abandonment flip: a route coordinator stops filing new failure cases at all once her submissions go unconfirmed.
Check yourself Score: 0 / 0
Multiple choice
1. What was the flip in Sigurd's story, and what were its two settings?
A. He becomes less confident in the outage bot after the storm.
B. He adds a new failure straight into the shared golden set, or fixes it privately and keeps the only record himself, with nothing in between.
C. The chatbot got worse at matching outage zones over time.
D. He asks the on call engineer to review every transcript with him each morning.
Show hint
Look for something Sigurd does with his own hands, not something that happened to the bot.
Show answer
B. C describes the boundary problem itself, not Sigurd's response to it, and the bot never actually got worse. A describes a feeling, and the story never shows him losing confidence, only skipping a specific step. D describes a fix worth making, but it isn't what actually happened in the story.
True or false
2. True or false: the fast lane Sigurd needed should also replace the full engineering review for changes to the chatbot's core zone matching logic.
True
False
Show hint
Ask what the fast lane is actually for: adding one test case, or changing how the model itself decides a zone.
Show answer
False. A case an engineer already found through their own model testing still needs the full ticket and review, that care is still doing real work there. The fast lane exists for a discovered case with nowhere fast to go, not for changing the model's own logic.
Fill in the blank
3. The decision this answer takes back is that Talbrook's three ______ set every golden set addition behind a full ______ and a ______, back when they filed almost every case themselves.
Show hint
It's an old rule that made sense when only a small team used it, and stopped scaling once someone else started finding more.
Show answer
Reliability engineers, ticket, review meeting. Nobody removed the golden set's own quality bar, they just never built a faster path once a support analyst started outpacing the engineers who wrote the original rule.
Multiple choice
4. Why wasn't "ping the on call engineer a bit more reliably" a real fix once Sigurd's habit had set in?
A. Because the golden set only accepts cases an engineer finds themselves.
B. Because there were only two real states once the ticket felt slow: file it properly so the whole team can see it, or fix it his own way and let only he remember it. Nothing forced a quick nudge that would have caught the same case five months later.
C. Because Sigurd was not allowed to talk to the on call engineer about a fix.
D. Because pinging the engineer would not have caught the zone boundary problem at all.
Show hint
This is the flip versus dial mistake. A flip has exactly two settings, not a sliding scale of how reliably someone nudges a colleague.
Show answer
B. D is the trap answer. The engineer's quick patch did fix the bot in the moment, that's exactly why nobody noticed the case never became a permanent check the golden set would run again.
Short answer, apply it yourself
5. Think of a tool at your own work with a way to report a problem. What's one reason a real problem might never get reported through it, and what would you check to catch that?
Show hint
Think about what happens after someone hits submit. Does anything ever tell them what happened next?
Show answer
Model answer: "Our team has a bug tracker, but filing one takes ten minutes of reproduction steps, and nobody ever hears back once it's filed. So people just work around small bugs instead of reporting them, and the tracker only fills up with the biggest ones. Checking how many workarounds exist that never became a ticket would show the real gap, not just counting how many tickets get filed." Any real example counts, as long as it names a plausible reason reporting breaks down and a concrete way to check it.
Fill in the blank, do the math
6. Sigurd's private sheet held 47 cases; the shared golden set held 6. If the fast lane had caught half of those 47 cases instead, about how many would the shared set have held in total, instead of 6?
Show hint
Half of 47 is about 23. Add that to the 6 that already made it through the ticket.
Show answer
About 29 cases, instead of 6. Almost five times as many real failures would have become permanent checks, and the boundary case Dagny had to trace from scratch is exactly the kind of case that math would have caught.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
