InterviewAdvancedQuality, Cost & Token Economics / Eval design for product teams / #25
Design an eval program from scratch for a team that has none.
One hundred and fifty real tickets, agreed on by two people, is what turns "that looks about right" into something you can actually check.
The direct answer
Build one small, real, hand-labeled set of past tickets, and score every change against it before it ships, starting on day one. Pick around a hundred and fifty resolved tickets, weighted toward the queues that hurt most when they're wrong, and have two senior agents agree on the right queue and the right priority for each. Any prompt or model change has to match or beat that same set before it ships, and any real miss found in production gets added to the set within a week, so the bar keeps rising instead of resetting after every launch.
Do this, in order
Build a small, hand-labeled golden set and score every change against it before shipping.Why: this is the one thing that turns "it looked fine when I skimmed it" into something a stranger could check for themselves.
Weight the set toward the categories that hurt most when they're wrong, not evenly across every queue.Why: a wrong call in a security queue costs more than a wrong call in a general question, so the eval's attention should too.
Get two people to agree on every label, with a third to break ties.Why: a golden set that's one person's opinion just moves the guessing one layer deeper, it doesn't remove it.
Add every real production miss to the set within a week.Why: a set built once goes stale the day a ticket type it never saw shows up, so it has to keep growing where it's actually weak.
Keep the rubric to two questions on day one: right queue, right priority.Why: grading tone or writing quality is a real project, but it competes for the same two weeks as the thing that actually put a ticket in the wrong place.
Leave the low-stakes queues with a lighter check.Why: spend the scrutiny where a miss actually costs something, not evenly across every kind of ticket.
How to answer this, stage by stage
Nobody is grading whether you know the word "golden set." They're grading whether you can go from nothing to a trustworthy floor, fast, without waiting six months to build the perfect version first.
1
Scope it to one real team before answering in the abstract
Say it like this
"Let's ground this in one team. Corriehill is a mid-size project management SaaS company, about forty people on support. Deskcompass is the tool that reads every inbound ticket and decides two things: which queue it goes to, and how urgent it is. Iosif Dulac runs support ops there, and until a few months ago, nobody had ever built an eval for it."
Why this works
An abstract "how do you build evals" answer turns into a checklist fast. One real team makes it a real decision instead.
2
Say your structure out loud
Say it like this
"I'll go in order: what happens today with no eval program at all, the one thing I'd actually build first, what breaks the first time that thing is wrong, and what I'd deliberately leave out on day one."
Why this works
Two seconds of structure tells the interviewer you have a plan, so they follow you instead of guessing where you're headed.
3
Reframe the question before answering it
Say it like this
"This isn't really asking me to list eval best practices. It's asking how you go from zero evidence to a trustworthy floor, fast, without spending six months building something perfect while the team keeps shipping changes on a feeling."
Why this works
Stops you from reciting a glossary of eval terms instead of naming the actual judgment call: what to build first, with nothing built yet.
4
Give the anchor, plainly
Say it like this
"So here's the one decision. Pull a hundred and fifty real, already-resolved tickets, weighted toward Security and Billing, since those are the two queues where a wrong call actually costs something. Two senior agents label each one with the right queue and the right priority, on their own, and a third breaks any tie. Every future change to Deskcompass gets scored against that same set before it ships. Not after. Before."
Why this works
This is the actual answer to the question, in one breath, concrete enough to argue with.
5
Prove it with the near miss, cut to four sentences
Say it like this
"Here's what happens without it. A ticket describing repeated failed logins from an unfamiliar country landed in the Account queue at normal priority instead of Security at the top tier. It sat for fourteen hours until a senior engineer happened to notice it while browsing the queue for something else. Eighteen changes had shipped to Deskcompass by then, and not one of them had been checked against anything more solid than a person's morning read."
Why this works
A specific near miss, with a real number attached, does more work than "mistakes might slip through" ever will.
6
Say what you'd keep measuring and growing
Say it like this
"I'd track two numbers, right queue and right priority, against that same set on every change. And I'd add every real miss found in production to the set within a week, so a ticket type the set has never seen gets folded in fast, instead of catching us the same way twice."
Why this works
Shows you're thinking past week one, and names the exact mechanism that stops the set from going stale.
7
Say what you'd deliberately leave out
Say it like this
"I would not try to grade the tone or writing quality of agent replies on day one. That's a real project, but it competes for the same two weeks, and it's not what put a security ticket in the wrong queue for fourteen hours. Right queue, right priority, nothing fuzzier than that, at the start."
Why this works
Shows judgment instead of an instinct to build everything at once, which is how eval programs stall before they ship anything.
8
Close on the one line
Say it like this
"So: one small, agreed-on set, scored before every change ships, growing every time reality finds a hole in it. That's the whole program, at the start."
Why this works
Ending on the design rule instead of the anecdote is what makes it sound like a method you'd actually reuse, not a story about one team.
Let's learn
Before Corriehill built any AI into it, someone on the support desk read every incoming ticket by hand and decided where it went. About twenty two hundred tickets a week, four minutes each, close to a hundred and forty six hours a week just deciding where a ticket should go, before anyone even answered it.
Now a tool called Deskcompass reads a ticket and decides both its queue and its urgency in under a second. That hundred and forty six hours a week mostly disappeared.
The whole check, today. No fixed list, no repeat run, just a feeling.
Knowledge spark: what's a golden set?
A fixed list of real examples with an agreed right answer, used to check every future change against the same yardstick. Not a demo picked to look good. A repeatable test somebody else could run and get the same result from.
The extra mistakes were never really the problem. Eighteen small changes had shipped to Deskcompass since launch, each one checked the same way: someone skimmed a batch of recent tickets, decided it looked fine, and moved on. Not knowing which of those eighteen changes actually made things better, and which one quietly made things worse, was the real problem.
It wasn't that Deskcompass got worse. It's that nobody could say, with anything besides a feeling, whether it had.
Tickets read by hand before each Deskcompass change ships, changes 1 to 18
Tickets read by hand, per change
At launch, the team read fifty tickets by hand before every change. By change ten it had settled at twenty, and stayed there. Change eighteen, checked the same way as every one before it, is the one that let the security ticket through.
At its worst, this doesn't cost a slow reply. It costs a real security ticket sitting in the general queue for half a day, found by luck instead of by design. And if that happens once for real, in front of a customer instead of an engineer scanning the queue for something else, trust breaks hard enough that agents start re-checking every ticket by hand again, on top of the tool. That ends up slower than never building Deskcompass at all.
The choice that mattered
When Deskcompass first shipped, the team decided that a person skimming twenty recent tickets after each change was proof enough that it still worked. That made sense back then: two queues, a few hundred tickets a week, nothing that felt like it could hurt anyone. It stopped making sense once the tool handled four queues, twenty two hundred tickets a week, and one of those queues was security.
What I'd leave alone. The low-stakes queue, general questions like "how do I export a report," doesn't need the same scrutiny. A wrong call there costs a few extra minutes, not a security incident.
The lesson. A quick read after a change tells you whether a tool still feels okay to someone who already knows what to expect. It doesn't tell you whether the same tool still gets a security ticket right, because nobody was checking that on purpose. A check nobody else could repeat isn't a program. It's a mood.
Now here is the same thing as a story
Read the short version above for the two-minute answer. Read this when you want to feel why skimming twenty tickets felt like enough, right up until it wasn't.
Iosif Dulac can read a support ticket in about ten seconds and know exactly which queue it belongs in. He'd been doing it long enough, four years on Corriehill's support desk before he moved into running ops, that new hires still asked him to double-check their calls.
When Deskcompass launched fourteen months ago, the good months were good. Tickets that used to take four minutes to sort landed in the right queue in under a second. Iosif's team went from six people doing nothing but sorting mail to two, freed up to actually answer tickets instead of filing them.
At first, every change got a careful read. Iosif and two others would sit down together, pull fifty recent tickets, and argue over every one that looked off. After a few clean releases in a row, that group review became one person's job, usually Iosif, reading through whatever tickets happened to be recent that morning. By the tenth or eleventh change, the read-through had shrunk to twenty tickets, skimmed between meetings, and if nothing jumped out, the change shipped.
The trigger was small. A new hire, checking her own work against Iosif's, asked a plain question: "How do we know this change is actually better, not just different?" Iosif didn't have a real answer.
Three weeks after that question, a ticket came in describing repeated failed logins from a country the customer had never logged in from before. Deskcompass filed it in the Account queue, priority two, standard response time. It sat there for fourteen hours, behind eleven other tickets, until a senior engineer scanning the queue for something unrelated noticed the words "failed login" and "unfamiliar location" and pulled it out himself.
The anchor, close up. Not a rubric. A fixed list, weighted, agreed on twice, built to grow.
The ticket didn't cost Corriehill a breach. It cost something quieter: proof that eighteen changes had shipped since launch, and not one of them had been checked against anything more solid than a person's morning read.
It was never really about that one ticket. Iosif had a stack of clean-looking releases and zero way to say, with a straight face, that any of them were actually better than the one before it.
Go back to the meeting where Deskcompass first shipped. The team agreed that a person skimming the last batch of tickets after each change was enough proof it worked, since back then it only sorted two queues and a few hundred tickets a week, and nothing about it felt like it could hurt anyone. Nobody wrote down what "enough proof" would mean once the queue count doubled and a security category got added.
Run the same three weeks again with the golden set built. A hundred and fifty real tickets, labeled twice, weighted toward Security and Billing. The change that would have caused the near miss gets scored against that set before it ships. It drops from ninety-six percent correct priority to eighty-one percent, specifically on security-flagged language, and gets caught in an afternoon, before a single real ticket ever sees it.
Two weeks later, Corriehill launched single sign-on. Zero tickets like it existed in the set yet. This is what the anchor is built to survive.
One design trusted a good morning to speak for every change that came after it. The other one asks every change to beat the same fixed answer, every single time, before it ever reaches a real ticket.
What I'd tell myself, back in that first launch meeting: the day you decide a quick look is proof enough, ask what "enough" means once the tool is doing three times the work. Nobody asked. That one's on the room, not on Iosif.
SPARK, when there is nothing built yet
Five letters, run forward instead of backward, since nothing here is being changed, it's being built for the first time.
SSituation. How does the job get done today, without an eval program?
Deskcompass ships prompt and model changes every few weeks. The only check is a person skimming twenty or so recent tickets and deciding, by feel, whether the routing "looks right." No fixed set, no repeatable comparison, nothing a new hire could learn to run themselves.
Ground it in the real habit today, or the anchor ends up solving a problem nobody actually has.
PPayoff. What habit do you want this to build?
Before shipping any change to Deskcompass, check it against the same fixed set of real tickets every time, and only ship if it does at least as well as the version it's replacing. What the team stops doing: trusting "it looked fine when I skimmed it" as proof anything actually got better.
Name the habit, not the tool. A dashboard nobody has to look at doesn't build a habit.
AAnchor. The one design decision everything else hangs on.
A golden set of a hundred and fifty real, already-resolved tickets, weighted toward Security and Billing, with the right queue and right priority agreed by two senior agents and a tie-breaking third. Score every model or prompt change against this same set before it ships, using a plain rubric: right queue, yes or no; right priority, yes or no. Nothing fuzzier than that on day one.
This is the answer to the question. Everything else in this recap either grounds it or protects it.
RRisk. What breaks the first time you're wrong?
A ticket type Corriehill has never seen before, say a wave of tickets from a brand-new feature launch, and the golden set has zero examples of it. A change that dips badly on exactly that new category still scores fine against the golden set, because the set never learned it exists.
The anchor has to survive being wrong in a spot it never covered, not just be right where it was already tested.
KKeep out. What you will not build on day one.
Not day one: a full automated pipeline grading the tone, empathy, or writing quality of the eventual agent reply. Not day one: dedicated examples for every one of Corriehill's forty-some ticket subcategories. Just the two things that decide whether a ticket lands somewhere dangerous: right queue, right priority.
Depth and breadth come later. The floor comes first, or the program never ships anything real.
Why the anchor survives the risk
Check it against the near miss and the new-category risk both. Does scoring against a fixed set catch the security regression? Yes, that's the whole point of the anchor. Does it also survive a ticket type the set never saw, like the single sign-on tickets two weeks later? Only because the anchor was never designed as a one-time build. Every real miss found in production gets added within a week, so the set keeps growing exactly where it's weakest, not just where it started.
The rejected alternative is worth naming directly: building a full automated grading pipeline first, scoring a dozen quality dimensions with an LLM judge before shipping anything. It lost because it would have taken months to build, and every one of those months, Corriehill would still be flying on a morning skim, exactly the gap that let the near miss through. The AI-specific failure worth naming by name is a cold start blind spot: a fixed golden set can score perfectly clean on a brand-new ticket type it has never seen, because a model can sound just as confident routing something new as something it actually knows well. The guardrail is the continuous-addition rule plus a low-confidence fallback: any ticket Deskcompass isn't sure about gets kicked to a human review queue instead of auto-assigned. And the trade being accepted on purpose is real: that fallback runs on about four percent of Corriehill's ticket volume, something like ninety tickets a week, each one taking longer to resolve than an auto-routed one would. That's slower service on a small slice of tickets, traded against catching a misrouted security ticket before it becomes a real incident instead of a lucky catch.
The five, in one line each: S: today, a person skims and decides by feel, with nothing fixed to check against. P: the habit worth building is checking every change the same repeatable way, every time. A: a hundred and fifty real, agreed-on tickets, weighted toward the categories that hurt most, scored before every change ships. R: a ticket type the set never saw can still score clean, so the set has to keep growing. K: not grading tone or covering every subcategory yet. Right queue, right priority, first.
Same five moves, a dispatch board instead of a helpdesk
Different desk, different stakes, same shape of gap: a team judging an AI tool by a quick look instead of a repeatable check.
S. Duskgate runs home heating and plumbing service calls across three counties. Routeglass listens to the incoming call, decides whether it's an emergency (no heat, gas smell, flooding) or routine, and assigns the nearest available technician. Today, without any eval program, Bastiaan Perreault, who runs dispatch operations, spot-checks a handful of calls each morning and decides if Routeglass still "feels right." P. The habit worth building: check every model change against a fixed set of real calls with agreed labels, not a morning skim of whatever came in that day. A. A golden set of eighty real, transcribed dispatch calls, labeled by two senior dispatchers, weighted toward gas-smell and no-heat-in-winter calls, since those are the two kinds of miss that put someone's family in danger, not just a technician's schedule off by a day. R. A new call type the set never saw. Duskgate added smart-thermostat installs, and a customer calling in about their thermostat going offline sounds routine on the surface, but for one household it meant their whole heating system had silently shut down overnight. The golden set had zero examples of that call shape, so a change that mishandled it still scored clean. K. Not day one: predicting exact repair time or which parts a technician needs to bring. Just two questions: emergency or routine, and is the assigned technician actually in the right service zone.
Calls logged as routine that were actually urgent, by call type, the quarter before Duskgate built a golden set
Gas smellNo heat, winterGeneral maintenance
A blended, company-wide misfile rate would have looked small. Broken out by call type, gas-smell calls were misfiled as routine sixteen times more often than general maintenance calls, exactly why the golden set weights those categories instead of spreading eighty examples evenly.
Same method, different shape. A misrouted software ticket and a misfiled gas-smell call look nothing alike, but both are a team trusting a quick look to stand in for a repeatable check, on the two categories where being wrong costs the most.
Swap the trigger and it still runs
Speed: if Corriehill needed changes to ship the same day, the golden set doesn't get skipped, it gets smaller. Fifty labeled tickets scored in ten minutes still beats zero.
Cost: if there's no budget for two senior agents to sit down together, one person labels the first hundred, and a second only checks the security and billing ones. Smaller pass, same discipline.
The model got better, for real: say the new version is measurably more accurate on average. That's exactly when the golden set matters most, because a model that's usually right can still be confidently wrong in one narrow spot, and a better-looking average makes that spot easier to miss, not harder.
Where people run it wrong
Treating the golden set as a one-time build instead of something that grows every time a real miss gets found in production.
Letting the same people who wrote the routing prompts also be the only ones labeling the golden set, so it only tests what they already expected to fail at.
Trying to grade tone, empathy, and writing quality on day one, so the two questions that actually matter, right queue and right priority, never get a clean answer either.
How to use it live
If you're asked this cold, ask what the team currently uses to say a change is better. If the honest answer is a feeling, that's the whole design problem in one sentence, and it buys you a beat to build the real answer from.
Flashcards (tap any card to flip it)
1 · THE SITUATION
What's the situation, before this eval program existed?
Tap to flip
ANSWER
Deskcompass shipped changes every few weeks, checked only by a person skimming twenty recent tickets and deciding by feel whether the routing looked right. No fixed set, no repeatable comparison.
2 · THE PAYOFF
What's the real habit this program is trying to build?
Tap to flip
ANSWER
Checking every change against the same fixed answer, every time, instead of trusting "it looked fine when I skimmed it" as proof anything actually got better.
3 · THE ANCHOR
What's the one concrete decision, the anchor?
Tap to flip
ANSWER
A hundred and fifty real, hand-labeled tickets, weighted toward Security and Billing, labeled by two senior agents with a tie-breaking third. Every change gets scored against this same set before it ships.
4 · THE RISK
What breaks the first time the anchor is wrong?
Tap to flip
ANSWER
A ticket type the golden set has never seen, like Corriehill's new single sign-on tickets, can still score clean, because the set never learned that category exists yet.
5 · THE PROOF
What almost got missed, in the real story?
Tap to flip
ANSWER
A ticket about repeated failed logins from an unfamiliar country got filed as a normal-priority Account ticket instead of top-priority Security, and sat for fourteen hours before a senior engineer noticed it by chance.
6 · THE NUMBER
Fill in the blank: the near-miss ticket sat for ___ hours, and it was the ___th change shipped with nothing checked against a repeatable set.
Tap to flip
ANSWER
14 hours; the 18th change. Once scored against the golden set, that same change is caught dropping from 96 percent to 81 percent correct priority on security-flagged language, in an afternoon.
7 · THE REPLAY
Same three weeks, golden set built. What changes?
Tap to flip
ANSWER
The change that would have caused the near miss is scored against the golden set first, drops to 81 percent correct priority on security tickets, and gets caught and fixed in an afternoon, before a single real ticket sees it.
8 · CROSS-PRODUCT
Section 4 runs SPARK again on a different product. Which one, and what does its anchor test?
Tap to flip
ANSWER
Routeglass, the call-triage and dispatch tool at Duskgate, a home heating and plumbing service company. Its golden set weights gas-smell and no-heat-in-winter calls, the two miss types that put a household at real risk.
Check yourself Score: 0 / 0
Fill in the blank
1. Fill in the blank: Iosif's team built a golden set of ___ real tickets, weighted toward the ___ and ___ queues.
Show hint
The number and the two queue names both show up in the anchor step of the walkthrough, and again in the SPARK recap.
Show answer
150 tickets, weighted toward Security and Billing. Those are the two queues where a wrong call costs the most, so they get more of the golden set's attention than a queue like general questions.
Multiple choice
2. What is the one concrete decision, the anchor, this answer argues for?
A. Hire more support agents to double-check every ticket by hand.
B. Build a small, hand-labeled set of real tickets and score every Deskcompass change against it before it ships.
C. Wait until the team has a full year of production data before building anything.
D. Automatically grade the tone and empathy of every agent's reply.
Show hint
Look at stage 4 of the walkthrough, and the A step of the SPARK recap.
Show answer
B. A adds review work instead of a repeatable check. C means shipping blind for a year. D is explicitly kept out for day one, since it doesn't touch the failure that actually happened.
True or false
3. True or false: because the golden set would have caught the security near miss, it also guarantees the team catches every new kind of ticket Corriehill ever creates.
True
False
Show hint
Look at the R step of the SPARK recap and the single sign-on example.
Show answer
False. A fixed set only knows what's already in it. A brand-new ticket type, like Corriehill's new single sign-on tickets, can score clean on a set that never saw one, which is exactly why the anchor's design adds every real miss within a week instead of stopping at the first build.
Short answer, name the reversal
4. What old decision does this answer take back, and why did it make sense when Deskcompass first shipped?
Show hint
Look at the block-key box titled "The choice that mattered" in Section 1.
Show answer
Model answer: The team decided a person skimming twenty recent tickets after each change was proof enough that it still worked. That made sense when Deskcompass only touched two queues and a few hundred tickets a week. It stopped making sense once it handled four queues, twenty two hundred tickets a week, and one of those queues was security.
Short answer, apply it yourself
5. Pick a process at your own work that gets judged by a quick look instead of a fixed, repeatable check. What would the first twenty examples of a real golden set for it look like?
Show hint
Look for something someone signs off on by skimming it, then think about what "right" would even mean if you had to write it down and check it the same way twice.
Show answer
Model answer: "I approve marketing emails by reading them once before send. My first twenty golden set examples would be past emails that clearly worked and clearly didn't, each labeled with the one or two things that actually made the difference, like a subject line that named a real number instead of a vague benefit."
Multiple choice
6. In the quarter before Duskgate built a golden set for Routeglass, about 16 percent of real gas-smell calls got logged as routine instead of emergency, while general maintenance misfiles ran near 1 percent. What does that gap tell you about how to build the golden set?
A. Spend the same number of examples on every call type, since averages hide nothing.
B. Weight the golden set toward gas-smell and no-heat calls, since that's where being wrong costs the most.
C. Skip building any golden set examples for general maintenance calls at all.
D. The 16 percent gap proves the model should never be trusted for dispatch.
Show hint
Compare the size of the gap between call types, not just the average across all of them.
Show answer
B. A blended, company-wide number would have looked small and steady. Broken out by call type, gas-smell misfiles ran sixteen times higher than general maintenance, which is exactly why the golden set weights those categories instead of spreading examples evenly.
Before you close the answer
Why this works
Tests whether you can go from zero to a trustworthy floor fast, without either shipping blind on a feeling or stalling for months trying to build the perfect eval system before checking anything real. Most candidates describe eval best practices in general instead of naming the one thing to build first.
Follow-up traps
"Isn't a hundred and fifty tickets too small to mean anything?" Response: size isn't the point, agreement and weighting are. A hundred and fifty tickets that two people agree on and that lean toward the categories that hurt most catches more real regressions than a much bigger set nobody has actually reviewed by hand.
"What stops the golden set itself from just being wrong?" Response: two senior agents label independently, with a third breaking ties, so no single person's blind spot becomes the whole team's standard. And because every production miss gets folded back in within a week, a bad early label gets challenged by real evidence fast.
If pressed
The pass bar was never "match the golden set exactly." It's "match or beat the version being replaced," scored separately on the security and billing slice and on the rest, since a change that trades three points of accuracy on general questions for zero drop on security tickets is usually a fine trade, and a change that does the reverse never should ship, even if the blended average looks the same either way.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
