CaseAdvancedEval-Driven Specification / Golden datasets and test set ownership / #11
Explain the relationship between the golden set and customer-specific quality complaints.
The direct answer
A golden set and customer-specific complaints can point in opposite directions at once, because the set is a fixed snapshot and the seller base is not. When a healthy, climbing golden-set score sits next to a rising pile of complaints from one kind of customer, check whether that customer's product type is even in the set. If it barely is, resample the set to match who is actually using the product today, and score every release against a live pull of real complaint photos from that segment, not the blended average alone.
Do this, in order
Resample the golden set's mix to match today's real seller base before trusting any release's score.Why: 8 of 200 examples were reflective or transparent products, while those sellers now make up 22 percent of the customer base.
Rule out a broken grader before blaming the set's coverage.Why: if the automatic edge-match score disagreed with a human check, the whole segment gap would be a measurement bug, not a real one.
Recut complaints by whether the seller's product resembles anything in the golden set, not one blended rate.Why: 22 percent of sellers were generating 76 percent of all quality tickets, hidden inside a blended complaint rate that barely moved.
Tag every golden-set example by surface type, and refresh the set whenever the seller mix shifts, not on a fixed review calendar.Why: the reflective and transparent seller share tripled in four months and nobody touched the set.
Route real complaint photos back to the eval team as a standing pipeline, not a one-time pull.Why: support closed hundreds of these tickets with credits and never once forwarded an image to the team that owns the set.
Leave the opaque and matte segment alone.Why: it already holds a 1.2 percent complaint rate, unchanged since launch, so more review there just slows down what already works.
How to answer this, stage by stage
Seven moves. The hardest part of this question is refusing to let a climbing golden-set score answer it by itself, so two full stages are spent on the gap between what the set says and what customers are actually reporting.
1
Pin it to one real product and number
Say it like this
"Let me put a number on this. Say there's a tool called Clearcut. A seller uploads a product photo, and it cuts the background out clean, ready to list. Elodie Kanters runs eval there, and two years ago she built a golden set of two hundred real product photos, checked by hand against a clean cutout, so every release has to beat the last one's score against that same set before it ships."
Why this works
Naming the tool, the person, and the exact process gives every later step something concrete to check, instead of a general worry about old test sets.
2
Say exactly what "customer-specific complaint" means here
Say it like this
"A customer-specific complaint isn't the golden score. It's a support ticket from one named seller account, saying the cutout left a smear or a ring of the old background behind. That's a different number, tracked in a different system, and nobody was putting the two side by side."
Why this works
Separating the internal score from the real-world signal is the whole question. Blur them together and there's nothing left to explain.
3
State the paradox plainly, then explain it
Say it like this
"Here's the actual relationship. The golden set climbed from 79 to 97 over four releases. Complaints climbed too, just not where the set was looking. It's not that the model got worse. The set only ever had eight glass and jewelry photos out of two hundred, so a whole surface type could fail badly and barely touch the average."
Why this works
Naming the mechanism, a fixed mix grading a moving customer base, beats saying "the set is outdated" and leaving it vague.
4
Give the fix up front
Say it like this
"So here's what I'd do. Tag every golden-set photo by surface type: matte, glossy, reflective, transparent. Check that mix against the real seller base every quarter. Pull real complaint photos into the set as they come in, and score every release against both the golden set and a fresh sample of real complaints, before anyone calls a release a win."
Why this works
Matches the direct answer. Naming the fix, not just the flaw, shows you can repair the relationship, not only describe it.
5
Show the timeline: when complaints outpaced the set's promise
Say it like this
"Here's the timeline. The golden set went in at month one, built almost entirely from matte fabric and shoes. In month five, Clearcut's bulk-import tool onboarded a wave of jewelry and glassware sellers. Nobody touched the golden set. By month nine, the score was at 97 and reflective-seller complaints had gone from 5 percent to 42 percent."
Why this works
Naming the exact month the seller base shifted, against the month nobody updated the set, turns the drift into a fact on a calendar.
6
Rule out a broken grader, then recut by segment
Say it like this
"Before blaming the set's coverage, I'd check whether the scoring itself is even right. Two people hand-checked twenty reflective examples against what the automatic grader said, and they agreed nineteen times out of twenty. So the grading was fine. Then split complaints by seller type: opaque sellers sit at 1.2 percent, steady since launch. Reflective and transparent sellers are 22 percent of the base and 76 percent of every complaint ticket."
Why this works
Skipping the grader check is the easy mistake. Rule it out first, then the segment split becomes proof instead of a guess.
7
Run the evidence test, then say what you'd leave alone
Say it like this
"Here's the test that settles it. Take forty real photos that actually triggered a complaint, score them the same way as the golden set. The golden set says 97. Those forty real photos score 51. That's the whole gap in one number. And I wouldn't touch the matte and opaque segment, it's most of the business and it already works."
Why this works
Ending on a checkable number, then naming what stays untouched, shows judgment instead of blanket distrust of the whole set.
Let's learn
Clearcut is a tool a seller drops a product photo into, and it hands back the same photo with a clean, empty background, ready to list.
Knowledge spark: what's an edge-match score?
A way of grading a cutout by comparing it, pixel by pixel, to a version a person cut out by hand. Close to the human line, high score. Ragged, smudged, or missing a corner, lower score. It only means something if the photos it's tested on look like the photos being cut out today.
Before any tuning, Clearcut's edge-match score against Elodie's two hundred golden-set photos was 79. Nobody trusted the model much yet. That felt about right.
Nine months and four releases later, that same score sits at 97. Every release beat the one before it. Every release replaced the model already live.
Two lines, four releases over nine months
What the dashboard showed: golden-set edge-match score
Quiet signal: reflective and transparent seller complaint rate
m1m3m5, wave of new sellersm7m9
The golden-set score climbed eighteen points in nine months. The complaint rate among reflective and transparent sellers climbed thirty-seven points over the same stretch, starting right around the month the seller base shifted. Nothing about that second line ever touched the first.
Here is the turn. Those eighteen extra points on the golden score never had a chance to see the problem. The real story is a kind of seller the set was never built to represent: reflective and transparent products, growing from a sliver of the business to almost a quarter of it, quietly filing more and more complaints while the score kept climbing.
The model did not get worse at cutting out fabric. It just never got asked to cut out glass.
At its worst, this costs Clearcut named sellers with hundreds of listings stuck behind a halo the model can't see. Solenne & Fitch, a jewelry seller with nine hundred live SKUs, pulled its entire new-listing queue offline after finding that roughly four in ten of its own uploads still showed a faint reflection ring around the stone.
The gap between a climbing score and the surface it was never shown
The choice I would take back. Building the golden set almost entirely from matte fabric and shoes, two years ago, was the sensible call. Reflective and transparent sellers were a small slice of the business then. I would take back leaving the set frozen there with no trigger tied to seller-mix shifts. I would tie a golden-set refresh to any month the seller mix moves by more than a few points, the same way a launch gets a rollout plan.
The decision that mattered
Score every release against a set that matches today's real seller mix, not one frozen at launch. Not because the old set is worthless. Because a set that never learned glass and glossy ceramic will always agree with itself, right up until a seller pulls her whole catalog offline.
What I would leave alone. The matte and opaque segment does not need touching. It holds a 1.2 percent complaint rate, unchanged since launch, because that is exactly the surface type the golden set was built to test. More review there would only slow down the majority of sellers who already have a working product.
The lesson. A golden set is a photo album of the customers you had on the day you built it. The day your customer base changes shape, that album does not know it is out of date, and it keeps handing out a passing grade unless someone tells it otherwise.
The week Solenne & Fitch stopped trusting the upload button
Read the short version above if you're short on time. This is the long version, for when you want to feel exactly where the nine months went.
Elodie Kanters has run eval for Clearcut for two years. She can look at a week's edge-match score and tell, inside a minute, whether a release earned it or got lucky on an easy batch of photos.
She built the golden set herself, with a freelance photo retoucher, the month Clearcut first launched. Two hundred real seller photos, cut out by hand, checked twice. Mostly clothing on hangers, shoes, ceramic mugs. Fast to build. Easy to trust, because the team had looked at every one of those images with their own eyes.
For the first four months, watching the score climb felt like the model getting sharper. Seventy-nine. Eighty-five. Ninety. Every point felt earned. Somewhere in there, without anyone deciding it on purpose, the support team quietly stopped forwarding sample complaint photos to Elodie's inbox the way they used to when the tool was new. The golden score was climbing on its own. Forwarding photos started to feel like double work for a problem that already looked solved.
Then Clearcut's growth team closed a deal with a jewelry supply wholesaler, and a wave of glass, chrome, and rhinestone sellers signed up inside six weeks, using the new bulk-import tool.
Nobody touched the golden set. Why would they. It kept scoring in the nineties.
Three weeks before a big seasonal sale, Solenne Achebe, who ran Solenne & Fitch out of a rented studio with her sister, noticed a faint ring of gray still sitting around the stones in a batch of new listings. She thought it was one bad photo. Then she checked ten more. Six of the ten had the same ring.
She pulled all nine hundred live listings and started recutting them by hand, one at a time, the way she did before she ever heard of Clearcut.
We did not lose one ring around one photo. We lost a jeweler's trust in every photo she'd ever uploaded.
I want to say the model got worse at cutting things out. It did not get worse. It just never got asked, not once in two hundred examples, to cut a clean line around glass, chrome, or a faceted stone.
So here is the decision I would take back.
Two years earlier, building the set almost entirely from fabric and matte surfaces was the sensible call. Reflective and transparent sellers barely existed on the platform yet. I would put a refreshed, tagged sample back alongside it: real reflective and transparent photos, checked every time the seller mix shifts, not just at launch. Not because the fabric set is wrong. Because of what it would have shown, months earlier: 94 percent clean cutouts for the surface it was built to test, and a fraction of that for the surface nobody had written a single example of.
That is the whole difference. One process trusts a score that only ever answered for the surfaces it already knew. The other checks that score against the surfaces sellers are actually uploading tonight.
And the part I would want to tell myself, if I could go back: we built a test that only ever answered for the customers we had on day one. Nobody decided that on purpose. We just never rebuilt it when the customers changed.
What the recut actually showed
Before blaming the golden set's coverage, Elodie's team checked whether the score was even being read right. Two people hand-checked twenty reflective examples against what the golden set's own rubric said they should score. They agreed on nineteen of twenty. The grading was not the problem. That left the coverage.
Share of sellers versus share of complaint tickets
78%
24%
22%
76%
Opaque and matte sellers Fabric, wood, matte ceramic
Reflective and transparent sellers Glass, chrome, jewelry, sheer plastic
Share of active sellers
Share of all complaint tickets
Reflective and transparent sellers are 22 percent of everyone using Clearcut, and the golden set had almost none of them: eight of its two hundred examples, about 4 percent. That same 22 percent of sellers is generating 76 percent of every quality complaint filed.
Shipping on the golden score alone
97 on the golden set, nine months of tuning
4 percent of the golden set's own 200 examples were reflective or transparent, the same surface behind 76 percent of real complaint tickets
Requiring the complaint fold-in check to agree too
Adopted the week Solenne & Fitch went offline
51 percent match rate on 40 real complaint photos, scored the same way as the golden set
Three ways a healthy golden score can sit next to rising complaints
Not because anyone cut a corner on purpose. A set built entirely from the old customer mix can be beaten honestly, release after release, and still stop meaning anything the day the customers change.
Three separate, checkable ways a set can pass and still miss the real complaints
Way 1
Frozen at the old mix. The set predates the sellers filing the complaints.
All but eight of the golden set's two hundred examples are matte fabric, wood, or ceramic. The set was built before reflective and transparent sellers were a meaningful share of the business, and nobody added examples once they were.
How you'd check it: count how many golden-set examples are tagged reflective or transparent, and compare that share to the real share of the seller base in that category. Four percent against twenty-two is a real gap.
Way 2
One blended rate hides a small segment failing hard.
The overall complaint rate barely moved, from 1.3 to 3.4 percent, because reflective and transparent sellers were still a minority of the base for most of the nine months. A blended number can look calm while one slice of it is on fire.
How you'd check it: split the complaint rate by segment before reporting it as one number. The blended 3.4 hid a segment sitting at 42.
Way 3
Complaint tickets never loop back to eval.
Support closed each ticket with a store credit and moved on. Nobody's job was to forward the actual photo that triggered the complaint to the team that owns the golden set. The set never heard about a single one.
How you'd check it: ask whether any support workflow includes "send the disputed image to eval." At Clearcut, it did not, until after Solenne & Fitch went offline.
Reading TRACE off a complaint that never touched the golden set
This reads like a question about two numbers, but the real job is diagnosis: work out why a set the team trusted for nine months quietly stopped predicting real complaints. GUARD would fit if the harm were one group losing a fair appeal; here the harm is a whole surface type the set never learned existed.
T, timeline. The golden set went in at month one, built almost entirely from matte and opaque products. The seller mix shifted at month five, when a wholesale deal brought in a wave of reflective and transparent sellers. Nobody touched the set at month five, or at any release after it. The reflective-segment complaint rate started climbing the same month, from 5 to 42 percent by month nine, while the golden score kept climbing from 79 to 97 over the same stretch.
R, recut. Split complaints by seller surface type. Opaque and matte, 78 percent of the base: 1.2 percent complaint rate, steady since launch. Reflective and transparent, 22 percent of the base: 76 percent of every complaint ticket. A blended 3.4 percent hid a segment that was quietly failing four in ten sellers.
A, assume nothing. Before blaming the golden set's coverage, two people hand-checked twenty reflective examples against what the set's own rubric said they should score, and agreed nineteen times. The scoring itself was fine. The gap was real.
C, cause candidates. Three, named and separate: the golden set is frozen at the old seller mix, with almost no reflective or transparent examples in it; one blended complaint rate hides a small, fast-growing segment failing hard, since averaging with a much larger healthy segment buries it; and no process routes a real complaint photo back to the eval team, so the set never learns from the failures piling up against it.
E, evidence test. Pull forty real photos that actually triggered a complaint, and score the current model against them the same way as the golden set. Golden set: 97. Real complaint photos: 51. The golden set can't grade a surface it was never shown.
Why the evidence test is the hard step
Anyone can suspect a golden set has fallen behind a growing customer base. A test earns its place by turning that suspicion into a number: how the set's own score compares to the score on the exact photos that caused real complaints. Do that comparison and you've checked something real. Call a golden set "probably stale" without it, and you've only said the same worry in a more confident voice.
Same shape, a claims tool that only ever saw a sedan in daylight
Oakmere Mutual Insurance runs ClaimSight, a tool that scores whether a submitted damage photo supports an auto claim before an adjuster reviews it. Petronella Vance, the quality lead, built the golden set eighteen months ago: a hundred and fifty claim photos, nearly all sedans, shot in daylight, scored against a benchmark called ClaimCheck. Every release has had to beat the last version's score on that set before it replaces the one live.
T. ClaimCheck's score climbed from 74 to 92 over five releases. The reopen rate on truck, motorcycle, and night-photo claims, customers disputing the first assessment, never moved off roughly one in three, the whole time. R. Split by claim type. Sedan and daylight, seventy percent of volume: reopen rate 3 percent. Truck, motorcycle, or night photo, thirty percent of volume: 46 percent, after a mobile app launch made night and phone-flash submissions common. A. Before blaming coverage, two adjusters blind-rescore twenty-five disputed claims against ClaimSight's own output. They agree on twenty-three of twenty-five. The scoring checks out. C. Three candidates: the golden set's hundred and fifty examples are almost all sedans in daylight, none at night; the dispute team never forwards a reopened claim's photo to the team that owns the golden set; and mobile submissions, mostly phone-flash night photos, only became common after a launch that happened after the set was built. E. Score the current model against thirty-five real disputed photos. Golden set: 92. Disputed photos: 48. ClaimSight barely tests the exact conditions behind almost half its reopened claims.
Swap the trigger and it still runs
Speed: instead of a slow five-month drift, the trigger is a same-week flash sale that floods Clearcut with jewelry sellers overnight. TRACE still starts with what the golden set was ever built to represent, not with how fast the sellers arrived.
Cost: the team shrinks the golden set from 200 to 80 examples to save review time, on the idea a smaller set is close enough. The recut still has to show which surface types got cut, not just how many examples remain.
The model really did get better: the case on this page. Clearcut genuinely got sharper at matte fabric. The golden set just never had a category that could tell that real gain apart from a fake one on glass and jewelry.
Where people run it wrong
Treating "we beat the last release's score" as proof the model works for every kind of customer, instead of asking which customers the golden set was ever built from.
Reading nine months of a climbing score as improvement everywhere, when a fixed set, tuned against long enough, can climb for reasons that have nothing to do with the harder surface.
Writing the golden set once at launch and never rebuilding it when the customer base changes shape, the way nobody redraws a map once new roads get built.
How to use it live
Buy yourself ten seconds by naming the split out loud. "So there's the score against the team's own examples, and there's whatever the real customer base looks like today, which might have grown since that set was built. A set frozen at the old mix can hide an entire kind of customer completely. Let me say how I'd check whether that's happening here." That's not stalling. That's where the real diagnosis starts.
Flashcards (click a card to flip it)
This is a diagnosis question about two numbers pointing different ways, not a habit changing, so these eight test the TRACE moves and the real figures behind them.
1 · THE FRAMEWORK
Which framework fits "explain the relationship between the golden set and customer complaints," and why?
Tap to flip
ANSWER
TRACE. The real task is a diagnosis wearing a definition question's clothes: why a set that looked healthy kept passing while one kind of customer kept complaining. LEAD would fit a question about which metric to watch; this is about why two existing metrics disagree.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Elodie Kanters, eval lead for Clearcut, a background-removal tool for e-commerce product photos. She built the 200-photo golden set herself, four months before the seller wave that broke it arrived.
3 · THE HABIT
What did the support team stop doing once the golden score kept climbing?
Tap to flip
ANSWER
Forwarding sample complaint photos to Elodie's inbox. It started to feel like double work once the golden score alone looked convincing, so the loop between real complaints and the eval team quietly closed.
4 · THE THREE WAYS
Name the three ways a healthy golden score can sit next to rising complaints.
Tap to flip
ANSWER
Frozen at the old customer mix (almost no examples of the new surface type), one blended rate hiding a small segment failing hard, and complaint tickets never routed back to the team that owns the set.
5 · THE NUMBER
The golden set scored 97, but reflective and transparent sellers were generating ______ of all complaint tickets.
Tap to flip
ANSWER
76 percent, from just 22 percent of the seller base. That same segment was only 4 percent of the golden set's own examples.
6 · THE CHECK
Name the one test that turned the suspicion into a number.
Tap to flip
ANSWER
Scoring 40 real photos that actually triggered a complaint the same way as the golden set. The golden set said 97. The real complaint photos scored 51.
7 · THE FIX
What does the fixed process require that the old one didn't?
Tap to flip
ANSWER
A golden set tagged by surface type, resampled whenever the seller mix shifts, scored alongside a live pull of real complaint photos, plus a standing pipeline routing disputed photos back to eval.
8 · CROSS-PRODUCT TRANSFER
Section 4 runs TRACE on a different product. Which one, and what's the number?
Tap to flip
ANSWER
Oakmere Mutual's ClaimSight. The golden set scored 92 overall, but scored only 48 on real disputed photos from truck, motorcycle, and night-time claims, a segment behind 46 percent of reopened claims.
Check yourself Score: 0 / 0
Multiple choice
1. Elodie's team requires every release to beat the last version's score against the same 200-photo golden set. What's the actual problem with that process, as written?
A. Two hundred examples is too small a set for a background-removal tool to be judged against.
B. It never checks whether the set's mix still matches the sellers actually using Clearcut today, so a growing surface type can hide inside a passing score.
C. The golden set should be thrown out and replaced with a completely different benchmark.
D. The score should be lowered until every reflective photo gets a manual review.
Show hint
Look at what the process checks, and what it never checks, about which sellers the set represents.
Show answer
B. The process names a score with no check on whether the examples cover the sellers Clearcut is actually serving now. That's the real gap, not the count of examples.
True or false
2. True or false: once the two-person check confirmed the automatic grader agreed with a human on reflective examples, that proved the golden set covered reflective and transparent products well.
True
False
Show hint
Confirming the scoring agrees with itself only rules out one kind of problem.
Show answer
False. That check (the A step) only ruled out a broken grader. It took the segment recut and the fold-in evidence test (R and E) to show the set was missing the surface type behind most real complaints.
Short answer
3. Name a place in Clearcut's process where you'd leave the current golden set exactly as it is, and say why.
Show hint
Think about the segment where the golden set and real complaints already agree.
Show answer
Model answer: "Keep it exactly as is for matte and opaque sellers, 78 percent of the base. There, the golden set already reflects the real mix, and the complaint rate sits at 1.2 percent, unchanged since launch. A second layer of review there would just slow down the majority of sellers who already have a working product."
Fill in the blank
4. Only about ______ of the golden set's 200 examples were reflective or transparent products, compared with ______ percent of the active seller base by month nine.
Show hint
Look at the segment chart's note under the grouped bars.
Show answer
4 percent (8 examples), 22 percent. Nearly every real reflective or transparent seller had almost no counterpart anywhere in the set.
Multiple choice
5. A teammate says the real fix is simpler: just write twenty more examples from matte fabric photos, since that's most of the volume anyway. Why doesn't that fix what's wrong?
A. Because writing more examples would take too many weeks.
B. Because more examples from the same surface type the team already knows would still skip reflective and transparent products, since nobody would think to write one from a category they haven't built for yet.
C. Because the golden set's scoring already disagreed with the manual check, so no amount of examples fixes it.
D. Because sellers would stop trusting the upload button entirely.
Show hint
One answer treats the count as the problem. The rest of the answer says the kind of example is the problem.
Show answer
B. More examples from the same old surface type repeat the same blind spot. The count grows; the kind of product covered doesn't.
Short answer, apply it yourself
6. Think of a test or checklist you rely on somewhere, a hiring rubric, a certification exam, a code review checklist, that was built for one version of the people or cases it judges. What's one way the real population could shift past it while the test kept passing?
Show hint
Look for a case where who or what the test judges changed shape after the test was written, and nobody rebuilt it.
Show answer
Model answer: "A code review checklist built for one codebase can keep passing pull requests cleanly even after the team adopts a second language nobody wrote a checklist item for, so every review looks thorough on paper while a whole class of real bugs slips through unchecked." Any honest answer works if it names a real case where the population judged grew and the test never grew with it.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
