U2xAI
…
Artifact critiqueAdvancedEval-Driven Specification / Writing an eval spec / #25

Write the section that defines what constitutes a regression.

The direct answer
Write the regression as one sentence with three parts: the metric, how far it has to fall below its own rolling average, and how many nights that has to hold before it counts as real, not one. Do not try to write that same sentence for every metric the eval reports on day one. Write it for the one metric that decides whether real bugs get missed, ship that, and expand later.
Do this, in order
  1. Write the regression rule as one sentence: a metric, a threshold below its rolling average, and a number of nights it has to hold.Why: without that sentence, the same drop reads as "ignore it" to one person and "call an incident" to another, and the argument itself becomes the cost.
  2. Require the drop to hold for two nights running, not one.Why: a single bad night happens by chance about once a week; two in a row almost never does, and that's what actually filters noise, not the size of the number.
  3. Set the threshold and window against what a real drop looks like, not against normal night-to-night noise.Why: too loose and the team gets paged for nothing until they stop trusting the page; too strict and a real regression sits live for a week before anyone official calls it one.
  4. Name what the section will not cover yet: only the one metric that decides whether real bugs get missed.Why: a rule trying to define regression for every metric at once never ships, and the old arguing keeps costing real days while it's being drafted.
  5. Log every single-night dip even when it doesn't trigger the rule.Why: you need a record of what normal noise actually looks like before you can defend the threshold to anyone who asks where the number came from.
  6. Hold the section to one number: how many nights a real regression can sit unnoticed before the rule catches it.Why: a definition nobody can test against a real timeline is just an opinion with a due date.

How to answer this, stage by stage

Seven moves. Ground it in one metric that already runs itself every night, before touching the definition.

1
Scope it to one metric on one real product
Say it like this
"Say I'm the PM for Notary, an AI code-review bot at Northlane Software that comments on a pull request before a person ever opens it. I'll write the regression section against catch rate, the number that says whether Notary actually finds the bugs it's supposed to, not against every score the eval reports."
Why this works
One real number that already moves on its own makes "what counts as broken" concrete instead of philosophical.
2
Say your structure out loud
Say it like this
"Here's how I'll walk through it: what arguing about a drop used to cost us, what the section actually has to settle, the rule I'd write, the two ways to get that rule wrong, and what I'm leaving for later on purpose."
Why this works
Two seconds of structure tells the interviewer you have a plan, so they're following your answer instead of guessing where it's headed.
3
Reframe what the section actually has to settle
Say it like this
"Most people hear 'define a regression' and go looking for the right number. The real gap is that without a written rule, the exact same drop means 'ignore it' to one person and 'call an incident' to another, and both of them are being reasonable."
Why this works
This is the actual insight being tested. Skip it and you've written a threshold, not a definition.
4
Give the anchor, both parts named
Say it like this
"So the section says one thing: catch rate falling more than 3 points below its 7-day rolling average counts as a regression, but only if it holds for two nights running. One bad night happens by chance about once a week. Two in a row almost never does."
Why this works
A number, a window, and a reason for the window is something an interviewer can picture as an actual spec line, not a promise to "watch it closely."
5
Prove it with a near miss, in four sentences
Say it like this
"Here's what happens without that second half. Catch rate drops to 66 the first night after a model swap, which looks exactly like a dip we'd already seen bounce back, so nobody names it. It's still down at 62 the next morning, and it stays there. Eight nights later, a customer's production outage is the thing that finally tells us."
Why this works
A specific near miss with a real count does more work than "this could go wrong" ever will.
6
Say what you would leave out on day one
Say it like this
"I wouldn't write this same rule for every number Notary's eval reports on day one, false-alarm rate, how long it takes to comment, how polite the comments read. Just catch rate, the one number that decides whether real bugs get missed. A rule covering everything never ships, and the old arguing keeps costing real days while it's being drafted."
Why this works
Shows judgment instead of a wish list, and answers the obvious follow-up: "what about the other metrics?"
7
Close on the one line
Say it like this
"So: a threshold tied to the rolling average, held for two nights, not one, one metric covered first, and a number I can defend, how many nights a real regression sits unnoticed. Right now that's eight. With this rule, it's two."
Why this works
Interviewers remember the last line most, and this one hands them something they can check, not just a mood.

Let's learn

Here is what happens when a number drops and nobody ever wrote down how much of a drop counts as real.

Say we build Notary, a tool that reads a pull request and leaves comments on it, the way a second reviewer would, before a person even opens the file. It runs against a set of 240 old pull requests every single night, each one already marked up by hand with the bugs a good reviewer should catch. The score it gets is called catch rate.

Knowledge spark: what is a golden set? A batch of past examples where someone has already written down the right answer by hand. A tool gets tested against it automatically every night, instead of a person checking its work by hand each time.

Most nights, catch rate sits around 71 out of 100. One night, after a routine change meant to make Notary cheaper to run, it reads 66. Nobody blinks. Notary has dipped to numbers like that before and bounced back the next night. The morning after, it reads 62. It stays down near there for over a week.

A hand-sketch of a desk with a laptop showing an eval dashboard and several sticky notes with conflicting scribbles: its noise, its real, check tomorrow, no written rule card in sight
What arguing about a drop used to look like, before this section existed

Here's the part that's easy to miss. Those extra points were not the real problem. The real problem was that nobody could say, in one sentence, whether ten points even counted as broken. One engineer called it noise. Another said compare it to last week, it's real. The PM had no rule to point to, so the argument just ran until someone got tired of having it.

Catch rate, sixteen nights, threshold marked
68 change ships night 9: new rule fires night 16: customer ticket (old way) night 1 night 16
normal week, averaging 71 after the change, holding low where the two-night rule would fire
Night 4 dipped to 67, alone, and bounced back the next night. That's why a single bad night, night 8's read of 66, still got waved off. Night 9 is different: it's the second night in a row under 68. The old process had no rule for that, so nothing fired until a customer's outage ticket found it, eight nights after the change shipped.
We didn't lose ten points. We lost the one sentence that would have told us it mattered.

At its worst, this costs a customer's team a bug in production that Notary was supposed to catch and didn't, because for eight nights it was quietly worse at its job and nobody had agreed on what "worse" would even look like on a dashboard.

The decision that mattered Write one sentence: catch rate falling more than 3 points below its 7-day rolling average counts as a regression, but only when it holds for two nights running, not one.

The choice I would take back. Months before any of this, while the eval was still being built, someone asked whether we should write down an exact regression threshold. We decided not to. We had no production history yet, no sense of how much the score normally moved on its own, and picking a number out of thin air felt riskier than picking none. The plan became "we'll know it when we see it." That worked fine while everyone watching the dashboard had watched it every night since launch. It stopped working the day it had to work for someone who hadn't.

What I would leave alone. The average length of Notary's comments drifts up and down a little every week, and nobody has ever changed anything because of it. If it moved five words in either direction tomorrow, no decision downstream would change. Only write a regression rule for a number someone would actually act on.

The lesson. A dropped number by itself isn't a decision. If the same drop can mean "ignore it" to one person and "call an incident" to another, we didn't write a rule. We wrote an opinion with a due date.

Now here is the same thing as a story

Read this one when you've got three minutes, since a Tuesday morning says more than any spec section ever could.

Yusra Kader can tell within about ten seconds of opening the eval dashboard whether last night's run means anything, or nothing at all.

She's the only PM who owns Notary's eval suite, inherited whole from the engineer who built it and left the year before. Notary comments on a pull request the way a second reviewer would, before anyone at a customer's company opens the file themselves, on 80-odd pull requests a night across every team using it. For most of a year, the dashboard was routine. Catch rate sat near 71. Yusra read the full report every morning at first, then just the headline number, then, most weeks, only if someone pinged her about it.

Close hand-sketch of an index card spec: catch rate more than 3 points below 7 day average, held two nights running, with a stamped regression check and a stamped noise check
The anchor: one written rule, not a feeling about the dashboard

On a routine Tuesday in October, an engineer shipped a change meant to shave cost off Notary's model calls, nothing dramatic, the kind of change that ships every couple of weeks. That night, catch rate read 66.

Yusra had seen readings like that before. Days earlier, in that same baseline week, the number had dipped to 67, alone, and bounced right back the next night. So a 66 looked like weather, not news. She noted it and moved on.

The next morning it read 62. The morning after, 61. It never bounced back.

Two panels: left, old process, a dip is dismissed and stays low for eight nights until a customer ticket arrives; right, new process, the second low night trips the written rule and the change is reverted the same morning
The day it was wrong, with and without the anchor in place

Nobody named it, because there was nothing to name it with. One engineer, in the thread that finally opened four days in, called it noise, the model's just having a bad stretch. Another said compare it to last week, obviously it's real. Yusra had no sentence to end the argument with, so it ran for two days while catch rate sat in the low 60s the entire time.

On night eight after the change shipped, a support ticket came in from Ashgrove Systems, a customer whose engineers use Notary on every pull request. A null-pointer bug had reached production, the exact kind of bug in Notary's own golden set, the exact kind it was supposed to reliably catch. It hadn't. The ticket read, more or less, your review bot said this was fine.

We didn't lose ten points. We lost the one sentence that would have told us it mattered.

Yusra pulled the original eval design doc that night. The line about regressions read: "the team will monitor the dashboard and flag anything that looks off." Written before the eval had even run once, before anyone had a number to compare a bad night against. Nothing had gone wrong yet, and writing a precise rule for a problem that hadn't happened felt like guessing. Nobody argued with it. Why would they.

Run the same Tuesday forward with the rule Yusra wrote after. Night one still reads 66, still logged, still not flagged, because one bad night on its own has always happened by chance about once a week. Night two reads 62, the second night in a row under the 68 floor, and the rule fires that morning, automatically, before anyone has opened a laptop. The change gets reverted before that night's run. Eighty customer pull requests went through Notary at the degraded rate under the old process for every one of eight nights, six hundred forty in total. Under the new rule, exposure stops at two nights, a hundred sixty.

One design waits for someone's gut to raise a hand. The other runs the same check every single night, whether anyone is watching or not.

And the thing I'd tell myself, back when we designed the alerting: we spent a whole meeting deciding what Notary should say about someone else's code. We never spent five minutes deciding what would make us say something about our own.

SPARK, aimed at the argument instead of the interface

This question asks for a spec section, not a screen, but it's still a design decision about the exact moment a number has to mean something to two different people at once, so SPARK still fits. A question asking how to measure Notary's quality in the first place would reach for LEAD instead.

SPARK laid out as five stacked rows, situation, payoff, anchor, risk, keep out, each with a short phrase about the regression definition section
SPARK, aimed at a definition instead of a screen
S, situation. Any eval that scores a product overnight, before this section existed: a number drops, and whoever notices argues about it in a thread. Nobody had ever agreed, in writing, what counts as broken.
P, payoff. Not "fewer bugs missed." The habit worth building: whoever opens the dashboard in the morning can say regression or noise in about ten seconds, without a meeting, because the rule already made the call.
A, anchor. One written sentence. Catch rate falling more than 3 points below its 7-day rolling average counts as a regression, but only when it holds across two nights running, not one.
R, risk. Set the window too short, one night instead of two, and normal dips like night four's 67 wrongly trip it, paging someone for nothing until the team stops trusting the page. Set it too long, or the threshold too forgiving, and a real regression sits live for over a week, the way this one did, before a customer's outage finds it instead.
K, keep out. No attempt yet to write this same rule for every number the eval reports, false-alarm rate, response time, how the comments read. Just catch rate, the one metric that decides whether real bugs get missed.
Why the anchor has to survive the risk Check it against the Tuesday: does the rule still hold when night one looks exactly like a dip that bounced back before? Only because it needs a second night, not because the threshold is clever. An anchor that fires on any single bad reading hasn't solved the real problem, it's just moved the argument from "is this real" to "why did it page me for nothing."
A hand-sketch of the same rule card, catch rate drawn solid and current, with false alarm rate, response time and comment tone drawn dotted and greyed out labeled not day one
What day one covers, and what it leaves out on purpose

And if you want to be sure it really works, try it somewhere else

A demand-forecasting model for a bakery-supply distributor is a different business entirely, and the same gap between "a number moved" and "we agreed on what that means" shows up there too, just wearing a different truck.

A hand-sketch of a warehouse desk with a forecast-accuracy dashboard and a similar written rule card for a demand forecast regression, with a delivery truck visible outside a window
Same framework, a different warehouse, a different number

S. Wietske Fenna is the only demand planner covering forecast accuracy at Rye Hollow Distributors, which predicts how much flour and yeast each bakery customer needs before the order even comes in. Today, without a written rule, a jump in forecast error gets argued over: was it a holiday, a promotion, or the model actually getting worse.
P. The habit worth building: whoever opens the Monday dashboard can call it regression or noise in the time it takes to read one line, instead of checking three other dashboards first to guess.
A. Same shape, different content. Forecast error rising more than 4 points above its 7-day average, held for three days running, not one, because bakery ordering is naturally spikier than a nightly code eval, and a single odd Tuesday means less here.
R. The first time it's needed, someone new to the team calls a single bad Tuesday during a known holiday week a regression. Trucks get rerouted on the strength of that call, and two warehouses run short that same week on products that were never actually mispredicted.
K. No attempt yet to write this rule separately for flour, yeast, and packaging. Just the one blended forecast-error number, the one operations actually acts on when a truck gets rerouted.

It took a Friday call from a store manager asking where her flour order went for anyone to check whether the alert had fired for a real reason, or for a holiday week that happens every single year.

Swap the trigger and it still runs

  • Speed: even if the golden set scored itself in real time, checking one night instead of two is still checking the wrong number of nights. Speed doesn't fix a rule aimed at the wrong shape of evidence.
  • Cost: if running the nightly eval got free tomorrow, that still wouldn't tell anyone how many bad nights in a row count as real. Cheap evaluation isn't the same as a working definition.
  • The model gets better: if catch rate climbed to 90, a genuine regression would still show up as roughly the same shape of drop, just against a higher average. A better model doesn't remove the need to define what falling back down looks like.

Where people run it wrong

  • Writing a rule that only covers the number moving down, with no rule for how many bad readings in a row it takes before anyone acts.
  • Treating "we have a dashboard for it" as the same thing as "we have a definition for it," without ever agreeing what number, on what window, actually counts.
  • Reaching straight for one rule that covers every metric the eval reports, and never shipping any written definition at all because the full version felt like the "real" one.

How to use it live

If you're asked this cold, ask what the team currently does when a number drops: does a person have to notice it, or does something notice it for them. Then ask how many bad readings in a row it takes before anyone actually acts. That second question is where the real definition lives.

Flashcards (click a card to flip it)

1 · THE SITUATION
Before this section existed, what happened when Notary's catch rate dropped overnight?
Tap to flip
ANSWER
Whoever noticed argued about it in a thread for days, because nobody had ever written down how much of a drop, held for how long, actually counted as broken.
2 · THE PAYOFF
What's the actual habit a written regression definition is trying to build?
Tap to flip
ANSWER
Whoever opens the dashboard can call it regression or noise in about ten seconds, without a meeting, because the rule already made the call.
3 · THE ANCHOR
State the regression rule this answer argues for, both parts.
Tap to flip
ANSWER
Catch rate falling more than 3 points below its 7-day rolling average, on two nights running, not one.
4 · THE RISK
What went wrong the one time this rule didn't exist yet?
Tap to flip
ANSWER
A routine change dropped catch rate from 71 to the low 60s. The first night's reading looked like a normal dip that had bounced back before, so nobody flagged it. It stayed low for eight nights before a customer's outage ticket surfaced it.
5 · THE PROOF
What actually happened to Ashgrove Systems during those eight nights?
Tap to flip
ANSWER
Notary missed a null-pointer bug in one of Ashgrove's pull requests, exactly the kind of bug it was supposed to reliably catch, and it reached production and caused an outage.
6 · THE NUMBER
___ customer pull requests went through Notary at the degraded catch rate before the old process noticed, versus ___ under the new two-night rule.
Tap to flip
ANSWER
640; 160. Eighty pull requests a night, times eight nights under the old process, versus two nights under the new one.
7 · THE REPLAY
Same regression, new rule in place. What changes?
Tap to flip
ANSWER
Night one still reads 66, logged but not flagged. Night two reads 62, the second night below the floor, and the rule fires that morning. The change gets reverted before the next run. Exposure drops from 640 pull requests to 160.
8 · CROSS-PRODUCT
Section 4 runs SPARK again on a different product. Which one, and what does its anchor add?
Tap to flip
ANSWER
Rye Hollow Distributors' bakery-supply demand forecast. Its anchor keeps the same shape, a threshold held over multiple readings, but widens the window to three days, since bakery orders swing more than a nightly code eval.

Check yourself Score: 0 / 0

Fill in the blank
1. Under the old process, the regression went undetected for ___ nights, because the first night's reading, ___, looked the same as a dip the team had already watched bounce back.
Show hint
Look at the chart's night-four dip, and how many nights passed between the change shipping and the customer ticket landing.
Show answer
8 nights, and 66. Night four's dip to 67 had bounced back on its own, so night eight's reading of 66 got waved off the same way, right up until a customer's outage ticket surfaced it.
Multiple choice
2. Which pair of numbers matches the regression rule this answer argues for?
  • A. Catch rate below 71, checked once.
  • B. Catch rate more than 3 points below its 7-day average, held for two nights running.
  • C. Catch rate below 60, checked once a month.
  • D. Any drop at all, flagged the same night it happens.
Show hint
The rule needs both a threshold tied to the rolling average and a rule for how long the drop has to hold.
Show answer
B. A has no window at all. C's threshold is far looser than the real drop and would have missed it even longer. D is exactly the design that would have wrongly flagged night four's normal dip.
True or false
3. True or false: the safest way to write this section is to define a regression rule for every metric the eval reports, catch rate, false-alarm rate, response time, comment tone, before shipping any of them.
  • True
  • False
Show hint
Think about what a rule covering every metric at once costs while it's still being drafted.
Show answer
False. Covering every metric at once is the keep-out. Shipping the one rule that decides whether real bugs get missed beats a complete version that never ships while the old, undefined arguing keeps costing real days.
Short answer
4. What old decision does this answer take back, and why did it make sense when it was first made?
Show hint
Think about what made "we'll know it when we see it" feel like the safe choice while the eval was still new.
Show answer
Model answer: Early on, the team chose not to write a specific regression threshold, because they had no production history yet and picking a number out of thin air felt riskier than picking none, so the plan became "we'll know it when we see it." That worked while everyone watching the dashboard had watched it every night since launch. It stopped working the day it had to work for Yusra, who inherited the eval and didn't have months of gut feel to fall back on.
Short answer, apply it yourself
5. Pick a metric you track yourself, at work or anywhere. If it moved tomorrow, could you say in one sentence, with a number, whether it counts as a real problem? What would that sentence need to say?
Show hint
Look for whether you have a written threshold and a window, or just a feeling you check against.
Show answer
Model answer: "My team tracks weekly support tickets. Right now, if it jumps, we just talk about it in standup. A real rule would need a number: tickets rising more than 15 percent above the 4-week average, held for two weeks running, not one, since a single busy week already happens around every product launch."
Multiple choice
6. If the rule had been set to fire after one bad night instead of two, based on this answer's own numbers, what would have happened during the seven normal nights before the real regression?
  • A. Nothing, all seven nights would have passed quietly.
  • B. Night four's dip to 67 would have wrongly fired the rule, on evidence that turned out to be normal noise.
  • C. The rule would have caught the real regression one night earlier than the two-night version did.
  • D. The golden set would have needed to grow past 240 pull requests.
Show hint
Check night four's reading against the 68 floor, and remember it bounced back the very next night.
Show answer
B. Night four read 67, already under the 68 floor. A one-night rule would have fired on it, even though it bounced back the next night on its own, which is exactly the false-alarm risk a two-night window is built to avoid.
From U2xAI Academy

From answering questions to owning outcomes.

A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.

  • A live AI agent you actually shipped
  • A launch decision you can defend under pressure
  • An interview-ready portfolio, not more flashcards
Know more

More on Writing an eval spec