ConceptIntermediateEval-Driven Specification / Writing an eval spec / #21
Describe the cadence at which evals should be re-run and by whom.
The direct answer
Put a standing weekly re-run on the calendar, owned by one named eval owner, independent of whether any code shipped. Gate every prompt or model change with a fast automated check before it can merge, but treat that as the floor, not the whole cadence. A deploy-triggered check only catches what you broke. The standing weekly one is the only check watching for what the job market broke.
Rank the cadence by what breaks first if skipped
Put a standing weekly re-run on the calendar, owned by one named eval owner, independent of deploys.Why: it's the only check watching for drift the code never caused, and skipping it lets quiet drift run for months instead of days.
Gate every prompt or model change through the fast automated golden-set check before merge.Why: the cheapest, fastest catch there is for an obvious break, skip it and a bad deploy reaches real recruiters the same day.
Refresh the golden set itself every quarter with real examples from new job categories.Why: a golden set that never grows can stay green forever while the product it's grading goes stale underneath it.
Run the quarterly calibration against real interview and offer outcomes.Why: it's the only check confirming the whole pipeline produces hires, not just plausible-looking numbers.
Trigger an off-cycle re-run the moment a new job category or posting taxonomy ships.Why: waiting for the next scheduled check on a change you already know about wastes the one moment you already knew to look.
Leave the deterministic format checks on deploy-only gating.Why: whether the score renders or the call comes back empty never drifts with the job market, so a calendar slot for it adds noise, not safety.
How to answer this, stage by stage
Seven moves, in the order I'd actually say them out loud. This question tempts you to describe one number and one schedule. Don't. There are two different kinds of change hiding inside "how often," and most answers only ever mention one.
1
Put a real number and a real owner in the room
Say it like this
"Let me make this concrete. Say it's a resume-to-job match score at a recruiting platform, and there's one eval owner, not a rotating on-call. Cadence questions fall apart fast when there's nobody named to actually run the check."
Why this works
An interviewer can't grade "regularly." Naming a feature and an owner earns you the room to give a real answer.
2
Split the question into two different kinds of change
Say it like this
"There are two totally different things that can break a match score. My own code, a new model, a new prompt. And the world underneath it, new job titles, new industries, that show up whether or not I ship anything. Those need two different schedules, not one."
Why this works
This reframe is the whole answer. Most candidates stop at the first kind and never mention the second.
3
Give the fast, automated layer first
Say it like this
"Every prompt or model change runs against a golden set, a few hundred real resume-job pairs with a right answer already agreed on, before it can merge. Owned by whoever's shipping the change. Cheap, fast, and it should never be optional."
Why this works
Shows you understand dependency: the cheap, fast check has to exist before a slower one is worth trusting.
4
Give the standing layer, the one most answers skip
Say it like this
"Separately, on a calendar, not tied to any deploy: a named eval owner pulls a live sample every week and checks it by hand. Not because code changed. Because the job market did, and nothing in a deploy-triggered check will ever notice a job title on its own."
Why this works
This is the actual answer to the question. It's also the line that separates this candidate from the rest.
5
Prove it with the failure
Say it like this
"This is close to something I've seen happen. A golden-set score sat at 93 percent for eleven straight weeks, because the golden set never changed. Meanwhile interview rates on new job categories fell from 74 to 52 percent, and the first person to notice was a client, not the eval."
Why this works
A concrete number the interviewer can picture beats an abstract claim about drift.
6
Say what you'd refresh, and what you'd leave alone
Say it like this
"Every quarter, I'd add real examples from new categories into the golden set itself, so it stops testing a year-old world. But the deterministic checks, does the score render, does the call come back empty, stay deploy-only. They don't drift with the market, so a weekly slot for them is just noise."
Why this works
Shows judgment instead of blanket paranoia, which is exactly what "by whom" is testing for.
7
Close on the one line
Say it like this
"So: automated regression on every change, a standing weekly check owned by one named person no matter what shipped, and a quarterly refresh of the golden set itself. If you remember one line: a deploy-triggered eval only watches for what you broke. Somebody has to watch for what the market broke, on a calendar, whether you touched the code or not."
Why this works
Ends on the actual decision, in a sentence a reader could repeat back cold.
Let's learn
How long can a check stay green while the thing under it quietly goes wrong? That's really what "how often should an eval run" is asking.
Say a recruiting platform builds a score. It reads a resume and a job posting, and it hands a recruiter one number, out of 100, for how good the match is. Rolecast calls it Match Score.
Knowledge spark: what a golden set is
A stack of real examples with a right answer a person already agreed on. Not a guess at what "good" means. Rolecast's golden set is 400 real resume-job pairs, each one already marked by a recruiter as a good match or not.
When Match Score first launched, the eval owner wired a bot to re-run that golden set every time the model or the prompt changed. For eight months nothing else touched it. The score held steady around 93 percent the whole time.
Then, without a single line of code changing, the number that actually mattered started to slide. Across eleven weeks, the share of top matches that actually landed an interview fell from 74 percent to 52 percent. Not on every posting. On three job titles that didn't exist when those 400 pairs were written: AI safety analyst, prompt engineer, applied ML engineer.
Here is the turn. The extra bad matches were never the real problem. A dozen or so weaker matches a week, spread across a big platform, is a rounding error nobody staffs a fire drill for. The real problem is what nobody did. Nobody re-ran anything, because nothing had shipped, and "nothing shipped" had quietly become the same thing as "nothing to check."
We didn't need a broken deploy. We needed eleven quiet weeks.
The fix that actually holds has four layers, and each one only means something once the one before it can be trusted: a fast automated check on every change, a standing weekly check that runs whether or not anything shipped, a quarterly refresh of the golden set itself, and a quarterly look at whether the top matches actually turned into interviews.
Each layer only means something once the one before it can be trusted
Interview rate on new job titles, eleven quiet weeks
The golden-set score never moved off 93 percent the entire time, because the golden set itself never changed. It was reading the same 400 pairs it read at launch, so it had nothing new to be wrong about.
At its worst, this nearly cost Rolecast a client. Bramwell Logistics, a warehousing chain that had filled roles through the platform for two years, raised it at a quarterly review: their offer rate on Rolecast candidates had dropped a third since spring, and nobody on the account team had an answer. That's the number that actually threatened the business. Not the 52 percent. The one meeting where nobody could explain it.
The choice I would take back
When the eval bot was first built, the question in that fifteen-minute setup meeting was how to keep a forty-row report off four people's desks every Monday. The fix was to run the check only when a prompt or model changed, and post one pass-or-fail line to Slack. On a two-person team trying to ship, that made sense. I would take it back and run a second check, on a calendar, that has nothing to do with whether anyone shipped anything.
What I would leave alone. Match Score also runs a handful of pure format checks: does the number render between 0 and 100, does the API call ever come back empty. Those never drift on their own. A model that's perfectly stable can still time out and return nothing, and that only ever shows up right after a change. Those stay deploy-triggered. A calendar slot for them would just be more noise on the same Monday.
The lesson. A check that only runs when you ship something is quietly betting that you're the only thing that can break the product. The job market doesn't wait for your release calendar, and it doesn't file a ticket when it changes.
Now here is the same thing as a story
You don't need this to answer the question. Read it slower, when you want to feel why the second layer matters and not just recite that it exists.
Every Monday at 7am, a bot dropped one message into Lusine Grigoryan's Slack: a green check and a single number.
Lusine had been Rolecast's only eval owner for nine months, since she inherited the golden set from the data scientist who built Match Score and left three weeks after launch. The 400 pairs he'd left her were a mess of raw scores nobody outside data science could read. She rewrote the whole rubric so a recruiter, not a data scientist, could look at a failure and know exactly what broke. That rubric was the reason anyone trusted the number at all.
For months, she opened the full report behind that Monday message every single week: ninety-something rows, one per job category, spelled out. It always cleared 90 percent. Around month five, she started reading only the summary line at the top. By month seven, she wasn't opening the report at all. Just the green check, then straight to her actual inbox.
Nothing about that felt careless. The check had never once turned red.
There was no single Tuesday where it went wrong. It built the way a slow leak does, the kind nobody notices because nobody's watching the gauge. It surfaced instead at a quarterly business review, when Bramwell Logistics' head of talent asked a question nobody at Rolecast had an answer for: why had their offer rate on Rolecast candidates dropped a third since spring?
The account manager didn't know. She pinged Lusine that afternoon.
Lusine pulled the full report for the first time in what she realized, checking the timestamps, was eleven weeks. The top line was still green, 93 percent, same as always, because the golden set behind it hadn't changed since the day she inherited it. But broken out by category, for the first time in months, three rows told a different story. AI safety analyst. Prompt engineer. Applied ML engineer. Titles that didn't exist when those 400 pairs were written. On those three, the share of top matches actually landing an interview had fallen from 74 percent to 52.
We didn't lose candidates that week. We nearly lost the account that took eight months to sign.
The model was never the door that mattered
Which model scored the queue, Lusine could swap that any week she wanted, and had, twice. But the eleven weeks of shortlists Rolecast had already sent to recruiters at Bramwell and a dozen other clients were gone. Nobody could quietly go back and re-rank who got seen in March.
She never had a number in her head either, if she's honest about it. She had a feeling with two settings: green, don't open it. Not-green, open it. Eleven straight green checks, and the feeling never once told her to look, because the check that turned green every Monday was reading the same 400 pairs it had read since launch. It had nothing new to be wrong about.
The choice that made sense at the time got made in a fifteen-minute meeting, back when the eval bot was first built. The question on the table was how to keep a forty-row report from landing on four people's desks every single Monday. The answer was to post one line to Slack and file the rest somewhere nobody had to open unless they went looking. On a two-person team trying to ship, that was the right call.
I would take that back. Not the golden set, not the rubric. Just the wire between "did we ship something" and "do we look." Those never should have been the same switch.
Run the same eleven weeks with a standing Monday check that isn't tied to any deploy, and by week two the number on those three categories has already dipped from 74 to 68. That's still a number worth a Slack message, not a client meeting nobody can answer. Bramwell's account manager gets an email about a fix already in progress, instead of a question she can't answer out loud.
One design hands Lusine a checkmark. The other hands her a chart she actually has to look at.
The thing I'd tell myself, looking back: I built a test that could only ever tell me if I'd broken something. I never built one that could tell me if the world had.
ORDER, timed instead of dated
This is a scheduling and ownership question wearing an eval question's clothes. TRACE would go looking for a mystery to diagnose, and there isn't one, the cause is plain once you look. LEAD would go looking for an early metric, but the metric already exists, the question is who watches it and how often. What's actually being ranked is a set of checks by how badly it hurts to skip each one, so the framework is ORDER.
ORDER, worked against a schedule instead of a table of contents
O, outcome. Every layer of this cadence competes to protect one thing: a recruiter sees a good match, even for a job title that didn't exist when the golden set was built. Not "the check passed." Whether the right person got seen.
R, reversibility. A bad deploy is the easy mistake to undo. Roll the model back, and by the next morning the wrong matches stop going out. Eleven quiet weeks of drift with nothing rolled back is the hard one: the shortlists already reached recruiters, and some of those decisions about who got seen already happened.
Knowledge spark: what a stale golden set actually costs you
Nothing, until the world changes underneath it. A golden set built at launch grades the product against the job titles that existed at launch. A new title is invisible to it, not wrong to it. It just never gets asked the question, so it keeps answering "fine" forever.
D, dependency. The quarterly calibration against real hire outcomes means nothing if the golden set underneath it is a year out of date, so the golden set has to get refreshed first. And the golden set can't refresh itself with categories nobody's found yet, so the standing weekly check, independent of any deploy, has to exist before the refresh does. Neither of those is a judgment call.
E, evidence. Pulling fifty live matches from the newest job categories and reading them by hand costs one eval owner a single afternoon, and it would have shown the gap at week two, before a client ever had to ask a question nobody could answer.
Weeks until the drift got caught, old cadence vs. new
Same drift, same product. The only thing that changes is whether somebody was already scheduled to look before a client had to ask.
R, rank. The fast automated check runs on every prompt or model change, gating the merge, owned by whoever's shipping it. The standing weekly check runs every Monday no matter what shipped, owned by one named eval owner. The golden set gets refreshed every quarter with real examples from whatever categories that weekly check has flagged. The outcome calibration against real hires runs quarterly too, last, because hire decisions take weeks to happen and it's the slowest signal in the cadence.
The check that makes ORDER honest
Swap what's actually at stake and the order moves. If Match Score only ever showed a recruiter a private draft ranking, never sent anywhere until she'd reviewed it herself, a missed category could sit for a month before it cost anyone anything, and the standing weekly check could slide down the list. It isn't the eleven weeks themselves that earn that check the top spot under O. It's that a shortlist already reached a recruiter before anyone at Rolecast got a second look.
Run it where the queue is a permit stack, not a mailbox
The City of Ashcombe's building department uses a tool that scores permit applications for completeness and code risk before a plans examiner ever opens the file, so the riskiest ones get a same-day look instead of sitting in a three-week queue.
O. Every check here protects one thing: an application with a real fire-code or structural risk gets flagged for a same-day human look. Not "the score ran." Whether a risky wall gets caught before it's built.
R. A wrongly queued application that's already under construction on a bad approval is the hardest thing here to undo, you can't unbuild a wall. Which model scores the queue, the city can swap that any month.
D. The quarterly comparison against real inspection outcomes means nothing until the standing weekly check confirms the score still reads current permit types, like the new backyard accessory-unit category the state added this year. Reality forces that order, not preference.
E. A senior plans examiner reads forty applications filed this month by hand, one morning's work, and that alone would show a new category slipping through weeks before a state audit ever could.
R. Automated regression on every rule or model change first. A standing weekly sample read by a plans examiner second, independent of deploys. A quarterly refresh of the labeled set with new permit types third. A quarterly comparison against real inspection results last.
Swap the trigger and it still runs
The state adds a whole new permit category mid-year, for backyard accessory units. The order doesn't move. The standing weekly check is what catches a new category, not a golden set that was only ever built once, at launch.
The scoring vendor doubles its price right before renewal. Same order. The automated check and the weekly sample still run first, now to prove a cheaper model clears the same bar, not because the calendar got tighter.
Early testing shows the new model scoring noticeably better. Doesn't move the standing check up or down. Better is a claim until a live sample confirms it, and a missed structural risk costs the same either way.
Where people run it wrong
Treating a golden-set score that's stayed green for months as proof nothing's wrong, when the golden set itself hasn't grown in just as long.
Only re-running an eval when someone ships a change, so a live product can drift for months with nobody watching.
Handing the weekly check to whoever's free that week instead of one named owner, so it quietly stops happening and nobody notices for a quarter.
If you're asked this cold
Say the outcome out loud before you name a single interval. "Every check in this cadence is protecting one thing: someone gets seen even in a category the eval was never built to expect." Ten seconds, and every interval you name after that has a reason attached to it instead of sounding like a cron schedule.
Flashcards (click a card to flip it)
1 · THE FRAMEWORK
Which framework fits "how often should evals re-run, and by whom," and why not TRACE or LEAD?
Tap to flip
ANSWER
ORDER, for ranking a set of checks by how badly it hurts to skip each one. TRACE diagnoses a mystery, and this cause isn't a mystery. LEAD finds a new early signal, and the metric already exists here, the question is only who watches it and how often.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Lusine Grigoryan, Rolecast's only eval owner for nine months, who rewrote the golden-set rubric so a recruiter could read a failure, not just a data scientist.
3 · THE HABIT SHE STOPPED
What did Lusine stop doing because the report always came back green?
Tap to flip
ANSWER
She stopped opening the full per-category breakdown every Monday. First she read only the summary line, then she stopped opening the report at all, just glancing at the bot's green check.
4 · HARDEST TO UNDO
Which is harder to recover from: a bad deploy, or a missed standing check? Why?
Tap to flip
ANSWER
A missed standing check. A bad deploy is loud and gets rolled back the next morning. Weeks of silent drift mean real shortlists already reached real recruiters, decisions that already happened and can't be replayed.
5 · THE OLD DECISION
What scheduling choice would you take back, and why did it make sense at the time?
Tap to flip
ANSWER
Tying the eval cadence entirely to deploys, so "nothing shipped" quietly meant "nothing to check." It made sense on a two-person team trying to keep a forty-row report off everyone's desk every Monday.
6 · THE NUMBER
Interview rate on the three new job categories fell from ______ to ______ over eleven weeks, with zero deploys to the model.
Tap to flip
ANSWER
74 percent to 52 percent. The golden-set score, meanwhile, stayed flat at 93 percent the entire time, because it was still reading the same 400 pairs it read at launch.
7 · THE REPLAY
Same eleven weeks, cadence fixed: what changes?
Tap to flip
ANSWER
The standing weekly check catches the dip at week two, when interview rate first falls from 74 to 68 percent, not week eleven at 52. Bramwell's account manager gets an email about a fix already underway instead of a question she can't answer.
8 · THE TRANSFER
Section 4 runs ORDER again on a different product. Which one, and what plays the role of the standing weekly check there?
Tap to flip
ANSWER
A city building department's permit-risk scoring tool. The equivalent is a plans examiner reading a weekly sample of filed applications by hand, independent of any code change, to catch a new permit category the score was never built to expect.
Check yourself Score: 0 / 0
True or false
1. True or false: since the matching model itself never changed during those eleven weeks, skipping the weekly re-run was the right call. Say why.
True
False
Show hint
Ask what changed even though the code didn't.
Show answer
False. The model didn't change, but the job market did. Three new job titles appeared that the golden set had never seen, and no code change was ever going to trigger a check for that.
Multiple choice
2. Which of these has to be trustworthy before the quarterly hire-outcome calibration means anything?
A. The launch announcement email
B. The standing weekly check confirming the golden set still reflects current job categories
C. The account manager's quarterly review deck
D. The color of the Match Score badge in the UI
Show hint
If the golden set is stale, a bad quarterly number could mean the test is wrong, not the product.
Show answer
B. Calibrating against real hires only tells you something if the score you're calibrating is already reading a current world. That's the dependency the D in ORDER is testing.
Fill in the blank
3. Rolecast's golden set had ______ labeled resume-job pairs, built before job titles like "AI safety analyst" existed.
Show hint
It's the number Lusine inherited from the data scientist who left after launch.
Show answer
400. That number never changed for the entire eleven weeks, which is exactly why the golden-set score stayed flat at 93 percent while real interview rates on new titles were falling.
Short answer
4. Name a part of Rolecast's eval suite where a standing weekly re-run genuinely would not matter, and say why.
Show hint
Look for a check that's about the code itself, not about the job market.
Show answer
Model answer: "The deterministic format checks, does Match Score render as a number between 0 and 100, does the API call ever come back empty. Those don't drift with the job market. If they ever break, it's right after a code change, so deploy-only gating catches them fine. A weekly slot for them buys nothing."
Short answer, apply it yourself
5. Pick an AI feature you use or are building. What changes in the world around it, not in its code, that a deploy-triggered eval schedule would completely miss?
Show hint
Look for something that shifts on its own calendar, not yours: a season, a regulation, a new category of user.
Show answer
Model answer: "A tax-prep tool's document classifier. Nobody redeploys it in January, but a new tax form or a new gig-work category shows up every filing season, and a check that only runs on code changes would never notice until refund season was already underway."
Short answer, the number question
6. If interview rate had drifted from 74 percent to only 68 percent over the same eleven weeks, instead of 52, would the cadence recommendation change? Say what moves and what doesn't.
Show hint
Reversibility is about what happens on the day it's caught, not about how far it's already fallen.
Show answer
Model answer: "The cadence stays the same. The standing weekly check still runs on its own calendar, owned by the same named person, because a smaller drift doesn't make it any easier to undo once shortlists have already gone out. What changes is urgency: a smaller drop might not need an off-cycle client conversation, but it still needs to be caught on schedule, not stumbled into."
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
