ConceptAdvancedEval-Driven Specification / Writing an eval spec / #10
How do you keep an eval set from being gamed by prompt tuning?
The direct answer
Keep a second eval set nobody tuning the prompt has ever seen, and only trust a prompt change if it also scores better on that hidden set. A gain that only shows up on the set you tune against is the prompt learning the test, not the model getting better at the job.
Do this, in order
Keep a held-out set nobody tuning the prompt has ever seen, and require a real gain there before trusting a gain on the visible set.Why: a score that only moves on the set you tune against is the prompt learning the test, not learning the job.
Rule out a broken or unfair eval before you call it gaming.Why: if the held-out set is a harder mix, or two graders can't agree on the same output, the gap is measurement noise, not proof of gaming.
Recut the pass rate by visible-set versus held-out before you trust one blended number.Why: a 92 percent average can be a real 92 on the set you tune against sitting next to a real 61 on the set you don't.
Name the exact way a prompt could be gaming the score, not a vague worry about overfitting.Why: copying visible wording, cherry-picked examples, and a gamed rubric are three different bugs with three different fixes.
Refresh or rotate the held-out set on a schedule, so it can't quietly become known too.Why: a hidden set that never changes eventually gets memorized by the same loop it was built to catch.
Leave the visible set alone for day-to-day tuning.Why: it's still the fastest way to catch an obvious break mid-sprint; the fix is what you believe from it, not whether you use it.
How to answer this, stage by stage
Eight moves. This is a diagnosis question about a process, not a person's habit, so most of the work is proving the gain is real before you're allowed to call it gaming.
1
Pin it to one product and one eval set
Say it like this
"Let me put a number on this. Say we've got an AI tool that drafts sections of a grant proposal for small nonprofits: the need statement, the budget narrative, the outcomes section. We built a 150-example eval set a year ago, scored on a rubric, does every claim trace back to something real, does it match how this funder actually talks. Every sprint, we tune the prompt and re-run it against those same 150."
Why this works
A vague question about "gaming" invites a vague answer. Naming the exact set, and how often it gets reused, gives every later step something to point at.
2
Say what "gamed" actually means here
Say it like this
"Before I answer, I want to say plainly what gaming looks like in practice. It's not the model cheating on purpose. It's a prompt that gets rewritten, sprint after sprint, against the exact same 150 examples, until it's very good at those 150 specifically, and nobody can tell that apart from it getting better at the actual job."
Why this works
This turns "how do people cheat" into "how does an honest process quietly produce the same result." That's the mechanism worth answering.
3
Give the one decision up front
Say it like this
"So here's what I'd do. Pull out a second set, maybe 50 examples, that nobody tuning the prompt ever sees. Keep tuning against the visible 150 all you want, that's fine, it's fast. But I'd only sign off on a prompt change once it also scores better on the 50 it's never met. If it only helped on the visible set, I'd call that no real improvement at all."
Why this works
This matches deliverable zero. Saying the concrete rule up front stops the interviewer from wondering where you're headed.
4
Rule out a broken eval before you call it gaming
Say it like this
"Before I blame the prompt, I'd check my own test. I'd make sure the 50 held-out examples come from the same mix of funder types and section types as the 150, not accidentally harder. And I'd get two people to re-score the same 30 outputs blind and check they agree. If the rubric itself is shaky, a gap between the two sets doesn't prove gaming, it just proves the ruler is bent."
Why this works
Skipping this is the most common mistake. Call it gaming before checking your own test is fair, and you fix the wrong thing.
5
Show the exact case where the visible score lied
Say it like this
"Here's the case that made it real for us. The visible-set score climbed from 71 to 92 percent over ten weeks. Meanwhile the share of drafts nonprofit staff still had to heavily rewrite before sending to a funder barely moved, still about 4 in 10. Then one drafted outcomes section said sessions were up 40 percent after a new curriculum. Nobody's real data said that. A funder's program officer asked where the number came from, and the nonprofit couldn't answer."
Why this works
One real failure is worth more than a warning. It's the story the interviewer remembers, and it shows the gap wasn't theoretical.
6
Name the three ways it could be happening
Say it like this
"I'd want to know which of three things was going on. One, the prompt's been rewritten around the specific wording of the 150 examples that used to fail. Two, someone quietly swapped the hardest examples out of the set over time, so it got easier without anyone deciding that on purpose. Three, the rubric itself has a quirk, it rewards any sentence with a percentage in it, so the prompt learned to write percentage-shaped sentences instead of true ones."
Why this works
A vague "it might be overfitting" doesn't lead anywhere. Three named, checkable mechanisms do.
7
Name the check that tells real gains from gamed ones
Say it like this
"The test I'd run: take this quarter's prompt and last quarter's prompt, and blind-score both against a brand new set of 50 examples nobody has tuned toward. On the visible set, this quarter's prompt beat last quarter's by 21 points. On the fresh set, it only won by 3. That gap is the whole answer. Most of the gain was aimed at the test."
Why this works
This is the strongest move in TRACE. It turns "I suspect we're gaming ourselves" into a number nobody in the room can argue with.
8
Say what to leave alone, then close
Say it like this
"I wouldn't throw out the 150-example set, it's still the fastest way to catch an obvious break mid-sprint. I'd just stop treating a good score on it as proof of anything by itself. So: tune against the visible set all day, but only believe the number that also holds up on the set the prompt has never seen."
Why this works
Ending on what you'd keep, not just what you'd change, shows judgment instead of blanket suspicion of the whole eval process.
If you remember one thing
Stage 4 and stage 7 are what's being graded. Rule out your own test first. Then run one check that grades a prompt against examples it has never met. A score that only ever gets checked against the set that built it will always look healthy, right up until it isn't.
Let's learn
GrantForge is an AI writing assistant that drafts sections of a grant proposal for small nonprofits: the need statement, the budget narrative, the outcomes section, the parts a funder actually reads before writing a check.
Nino Beridze has run the eval process for GrantForge for a little over a year. She built the original 150-example eval set herself, with the company's program staff, pulling real need statements and budget narratives from past submissions and scoring each one against a rubric: does every claim trace back to something real, does the tone match how this funder actually talks, does the ask add up. Every sprint since, the prompt team tunes the wording and re-runs it against those same 150 examples. Ten months ago the set scored 71 percent. Nobody thought much of that number. It was just where they started.
The loop the score lived in, and the one thing it never touched
Here is the turn. Ten weeks of steady tuning took that score from 71 to 92 percent. Twenty-one points. Nobody thought that was suspicious either. It looked like a team doing its job well. The problem isn't that the score went up. The problem is what it was actually measuring the whole time: not whether GrantForge writes a better grant section, but whether it can pass 150 specific examples it sees, unchanged, every single week.
Knowledge spark: what a held-out set actually is
A slice of examples kept completely separate from the ones used to build or tune something. Nobody writing the prompt ever sees it. Its only job is to answer one question honestly: does this still work on something new?
The cost showed up in a place the dashboard wasn't watching. A caseworker at a small literacy nonprofit asked GrantForge to draft an outcomes section for a renewal application. The draft said program sessions had increased 40 percent since a new curriculum launched. Nobody's real numbers said that. It read well, it had a percentage in it, it sounded exactly like the kind of sentence a rubric checking for "quantifies outcomes" would reward. The nonprofit's program officer at the funder emailed asking for the source of the 40 percent figure. The nonprofit couldn't produce one. They filed a hurried correction two weeks before the renewal decision, and the funder's contact stopped taking the rest of the application at face value.
We didn't tune the prompt to write better grant sections. We tuned it to remember 150 of them.
The choice I would take back. Building one 150-example set and tuning against it every sprint was the sensible call when the set was new and the score was low; there was plenty of real room to improve, and nobody was close to the edge of it yet. I would take it back and split it from day one: keep the 150 for fast, everyday tuning, and add a second, smaller set that the prompt team never sees at all.
What I would build instead
Two sets, not one. The 150 stays visible, for speed. A separate 50, drawn from the same mix of funder types and section types, stays hidden from anyone tuning the prompt. A change only counts as an improvement once it moves both.
What I would leave alone. The visible 150 doesn't get thrown out. It's still the fastest way to catch an obvious break in the middle of a sprint, a section that's suddenly incoherent, a rubric category that collapses to zero. Losing that speed to chase a slower, more careful process for every single tuning pass would be its own kind of mistake.
The lesson. A score that keeps climbing isn't proof of anything by itself. I built a set to measure the model. I never asked how many times the same set could be looked at before it stopped measuring anything except itself.
The weeks the dashboard didn't show
The gap didn't open on one bad day. It built across the whole ten weeks of tuning, in two lines that should have moved together and didn't.
Two lines, one sprint cycle, ten weeks
What the dashboard showed: visible-set pass rate
Quiet signal: share of drafts staff still heavily rewrite before sending
wk 1wk 2wk 3wk 4wk 5wk 6wk 7wk 8wk 9wk 10
The visible-set score climbed steadily for ten straight weeks. The share of drafts staff still had to heavily rewrite before sending to a funder never moved off roughly 4 in 10. A score that measures the job should track a number like that. This one didn't, because it was never built to.
Nobody connected the two lines until a new hire did. Callum Otieno joined the eval team as a junior engineer and sat in on his first sprint retro in week nine. Nino's team was celebrating the 91 percent mark. Callum asked, plainly, "Are we always tuning against the same 150 examples? How do we know it's not just gotten good at looking good on those?" Nobody in the room had a ready answer.
One score, two very different truths
Before answering Callum's question, Nino's team ruled out the easy explanation first: that the held-out set was simply a harder mix. It wasn't. The 50 fresh examples were stratified to match the same funder types and section types as the visible 150, and two graders blind-rescored 30 of the same outputs to check the rubric wasn't the problem. Agreement between the two graders came back above 90 percent. The test was fair. That left one explanation standing.
Pass rate by prompt version, visible set vs. a fresh holdout nobody tuned against
71%
58%
92%
61%
Old prompt, week 1 before ten weeks of tuning
New prompt, week 10 after ten weeks of tuning
Visible-set pass rate (150)
Fresh-holdout pass rate (50, never seen)
On the visible set, ten weeks of tuning bought 21 points. On a fresh set drawn from the same mix, that same ten weeks bought 3. Most of what looked like progress never left the set it was tuned on.
Three ways a prompt learns to pass, not to write
Not because anyone cut a corner on purpose. An honest tuning process, run against the same fixed set for long enough, quietly stops improving the thing it was supposed to.
Three separate, checkable ways to hit a score without meeting its point
Way 1
Overfitting. The prompt copies the visible wording.
Sprint after sprint, the prompt team added instructions and examples pulled straight from the 150 items that had been failing. Each fix targeted the exact phrasing of a specific example instead of the underlying skill it was supposed to represent.
How you'd check it: read the prompt's own instructions and see how many quote or closely echo specific eval examples by name or phrase. Nino's team found dozens.
Way 2
Cherry-picking. The hardest cases got quietly swapped out.
Whenever an example proved persistently hard, like a budget narrative that needed real arithmetic to check, someone would swap it for an easier one "to keep the set current." Nobody decided to make the set easier. It just got easier, one swap at a time.
How you'd check it: compare today's 150 examples against a year-old copy of the set. Count how many of the original hard cases are still in it.
Way 3
Rubric gaming. The score rewards the shape of a number.
The rubric's "quantifies outcomes" check fired on any sentence that paired a percentage with a time comparison. It never checked whether the percentage was real. The prompt learned to write percentage-shaped sentences, whether or not there was a real number behind them.
How you'd check it: pull every output the rubric scored highest on "quantifies outcomes," and check how many of those numbers trace back to something the nonprofit actually reported.
Checking the prompt's own version history against the visible examples is what separated a guess from a finding:
Trusting the visible score alone
92 percent on the 150 examples everyone tunes against, every sprint
3 real points gained on a fresh, never-seen set of 50
Requiring both sets to move
A prompt change only ships once the held-out score improves too
0 prompt changes have shipped on a gamed score since
TRACE, run backward from a score that looked too good
This is a diagnosis question about a metric wearing a "how would you prevent it" coat, so TRACE is the framework, not a story about a person's habit changing. A "design the feedback mechanism" question would reach for SPARK instead.
T, timeline. The gap didn't open on one day. Across ten weeks of tuning, the visible-set score climbed from 71 to 92 percent while the share of drafts staff still heavily rewrote stayed near 4 in 10 the whole time. The two lines should have moved together. Only one of them had a dashboard.
R, recut. Split by whether an example sat inside the 150 everyone tunes against or in a fresh 50 nobody had seen. Visible set: 92 percent. Same-week fresh set, matched to the same mix of funder and section types: 61 percent.
A, assume nothing. Before calling it gaming, Nino's team checked their own test: the held-out set matched the visible set's distribution, and two graders blind-rescored 30 outputs and agreed above 90 percent of the time. The eval itself was fair. The gap was real.
C, cause candidates. Three, named and separate: the prompt copying exact phrasing from examples it used to fail, the eval set quietly losing its hardest cases as people swapped them out over months, and a rubric that rewards any sentence shaped like a real number, whether or not the number was true.
E, evidence test. Blind-score last quarter's prompt and this quarter's prompt against a brand new batch of 50 examples nobody had tuned toward. This quarter's prompt won by 21 points on the visible set. It won by 3 points on the fresh one. Most of the gain never left the set it was tuned on.
Why E is the hard step
Anyone can suspect a score looks too good. A test earns its place by grading a prompt against examples it has never met, not against another pass through the same 150. Score a prompt against a set it already learned, and you haven't checked anything, you've just asked the same question twice.
Same trick, a newsroom that grades its own drafts
BeatDraft is an AI tool that drafts summaries of town council meetings for a small local news wire. Its standing eval set is 80 past meetings from 12 towns. Its launch bar: match a human editor's summary closely enough that the editor doesn't need to rewrite it. Over six weeks of prompt tuning, Aurelio Bautista, the wire's editorial ops lead, watches the score to make sure the tool is ready to cover two more towns.
T. The visible-set score climbs from 74 to 95 percent over six weeks. The share of live drafts editors send back for a rewrite never moves off roughly 1 in 4, the whole time. R. Split by whether the meeting comes from one of the 12 towns in the eval set. Visible-town accuracy holds at 95 percent. A sample from two new towns, added this quarter, scores 68 percent. A. Before blaming the model, Aurelio checks whether the new towns' meetings are just harder: longer, more contested. They aren't, average length and agenda size are close to the original 12. He also has a second editor blind-check 20 summaries against the first editor's scores. They agree closely. The eval itself is sound. C. Three candidates: the prompt was tuned to match the exact named officials and phrasing that recur across the 12 eval towns, the standing set had quietly dropped its most contested zoning-fight meetings after they kept failing, and the rubric's "names at least two speakers" check got satisfied by naming any two attendees, relevant or not. E. Pull 25 fresh transcripts from towns and dates nobody has tuned against, and blind-score the current prompt against last month's, before either one covers a new town alone.
Swap the trigger and it still runs
Speed: instead of steady prompt tuning, the trigger is a cheaper model swapped in under a deadline. TRACE still starts with what the eval score actually measures, not with how fast the swap got approved.
Cost: the team shrinks the eval set to save review time, on the theory that a smaller set is close enough. The recut still has to show what got dropped in the shrink, not just what got saved.
The model really did get better: the case on this page. GrantForge's tone and clarity genuinely improved. The eval just never checked whether the facts inside the sentences were real, so it couldn't tell the two kinds of "better" apart.
Where people run it wrong
Treating a fixed eval set as neutral forever, instead of asking who has seen it and how many times.
Reading "the score is up" as "the product is better," when a score can move for reasons that have nothing to do with the product getting better.
Building a held-out set once and then letting the same few people quietly memorize that one too, so it stops being held-out at all.
If you are asked this cold
Buy yourself ten seconds by naming the gap out loud. "So there's the set the score gets measured on, and there's the ability I actually want. Tuning against a fixed set for long enough can move one without touching the other. Let me say how I'd check which one actually moved." That's not stalling. That's where the real answer starts.
Flashcards (click a card to flip it)
This is a diagnosis question about a metric, not a flip story, so these eight test the TRACE moves and the real numbers instead of a habit changing.
1 · THE FRAMEWORK
Which framework fits "keep an eval set from being gamed by prompt tuning," and why?
Tap to flip
ANSWER
TRACE. The real question is a diagnosis: why can a score climb honestly, sprint after sprint, without the product actually getting better. FLIPS is for "what if something changed," and here nothing about the model changed at all.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Nino Beridze, the eval lead for GrantForge, an AI writing assistant that drafts grant proposal sections for small nonprofits. She built the original 150-example eval set and rubric herself, a year before the gap surfaced.
3 · RULING IT OUT
What did Nino's team check about the eval itself before blaming gaming?
Tap to flip
ANSWER
Whether the held-out set was just a harder mix, and whether the rubric itself was consistent. The 50 held-out examples matched the visible set's mix of funder and section types, and two graders blind-rescoring 30 outputs agreed above 90 percent of the time.
4 · THE THREE WAYS
Name the three ways a prompt can learn to pass an eval without getting better at the job.
Tap to flip
ANSWER
Overfitting to the visible examples' exact wording, cherry-picking which hard examples stay in the set over time, and gaming the scoring rubric's own quirks, like rewarding any sentence shaped like a real number.
5 · THE NUMBER
On the fresh, never-seen holdout, this quarter's prompt beat last quarter's by only ______ points, versus a 21-point gap on the visible set.
Tap to flip
ANSWER
3 points, 58 percent to 61 percent. Against a 21-point jump on the visible set, 71 percent to 92 percent, over the same ten weeks of tuning.
6 · THE CHECK
Name the one test that turned a guess into a number, and what it holds still.
Tap to flip
ANSWER
Blind-score two prompt versions against a brand new, never-seen batch of examples. It grades the prompt against something it has never met, not against another pass through the set it was tuned on.
7 · THE FIX
What does Nino actually change about how GrantForge's eval works?
Tap to flip
ANSWER
Splits it in two. The visible 150 stays for fast, everyday tuning. A separate 50-example set, matched to the same distribution, stays hidden from the prompt team, and a change only counts once it improves both scores.
8 · CROSS-PRODUCT TRANSFER
Section 4 runs TRACE on a different product with the same shape of problem. Which one, and what's different about the cause?
Tap to flip
ANSWER
BeatDraft, an AI tool drafting town council meeting summaries for a local news wire. The cause shifts to a prompt tuned to specific recurring officials and towns, plus a standing eval set that had quietly lost its most contested meetings.
Check yourself Score: 0 / 0
Multiple choice
1. Why doesn't a jump from 71 to 92 percent on GrantForge's visible eval set prove the tool actually got better at writing grant sections?
A. It does prove it, since the eval set was built from real past submissions.
B. The gain only showed up on the same 150 examples tuned against every sprint; a fresh, never-seen set barely moved.
C. 92 percent is too close to 100 percent to mean anything either way.
D. The rubric can't be trusted to score writing at all.
Show hint
Look at what happened on the 50-example holdout that nobody tuned toward, next to the 21-point climb on the visible set.
Show answer
B. The visible set climbed 21 points over ten weeks. The fresh holdout, matched to the same distribution, only climbed 3. Most of the "improvement" was aimed at the test, not the skill.
True or false
2. True or false: once Nino's team confirmed the held-out set matched the visible set's mix of funder types, that alone proved the visible-set gains were gamed.
True
False
Show hint
Ruling out an unfair test only tells you the comparison is fair. It doesn't by itself tell you what the comparison found.
Show answer
False. Matching the two sets' distributions (the A step) just confirmed the test was fair. It still took the fresh-holdout evidence test to actually show that most of the visible-set gain didn't transfer.
Fill in the blank
3. On the brand new 50-example holdout nobody had tuned toward, this quarter's prompt beat last quarter's prompt by only ______ points, versus a 21-point gap on the visible set.
Show hint
Compare the two bars in the "New prompt, week 10" group of the grouped-bar chart.
Show answer
3 points. 58 percent to 61 percent on the fresh holdout, against 71 percent to 92 percent, a 21-point gap, on the visible set over the same ten weeks.
Short answer
4. Name a place in GrantForge's eval process where you'd leave the visible 150-example set exactly as it is, and say why.
Show hint
Think about what the visible set is still good for, even after you stop trusting it alone.
Show answer
Model answer: "Keep using it for daily prompt iteration between sprints. It's fast, and it still catches an obvious break in the model's output. The fix isn't to stop using it, it's to stop treating a good score on it, by itself, as proof of anything."
Short answer, apply it yourself
5. Think of a test or review process you rely on where the same fixed set of examples, or the same reviewer, gets used over and over. What's one way it could quietly get "passed" without the underlying thing actually getting better?
Show hint
Look for a case where whoever is doing the work has learned the reviewer's habits, not the underlying standard.
Show answer
Model answer: "A team always tests a new linter rule against the same five sample pull requests before shipping it. The rule looks great because it was tuned to those five PRs specifically, not because it reliably catches real problems in new code nobody has tested it against." Any honest answer works if it names a real case where a fixed test could be learned instead of the real skill.
Multiple choice
6. A teammate says the real fix is simpler: just ask the prompt engineers not to look at the eval examples so closely while tuning. Why isn't that the same as the decision in this answer?
A. It is basically the same fix, both stop people from seeing the examples.
B. Asking someone to tune "less closely" isn't something you can rely on or check. A held-out set nobody can see enforces the same thing every sprint, whether or not anyone remembers to hold back.
C. Prompt engineers can't improve a prompt at all without looking closely at examples.
D. Building a held-out set would take less time than asking people to be careful.
Show hint
One of these is a request that depends on someone's memory. The other is a decision that runs the same way every time, on its own.
Show answer
B. "Look less closely" is a request, not a design; it depends on people remembering to hold back, sprint after sprint. A held-out set nobody can see enforces the same check automatically, with no one needing to remember anything.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
