Describe what an eval is and how it differs from a unit test.

Switchyard reads every pull request at Millbrant Systems and posts a review comment before a human merges. Isolt Feyer owns the process that decides whether a change to it is good enough to ship. Three weeks after a prompt change made its comments less noisy, Switchyard approved a pull request that removed a null check. The unit tests had stayed green the entire time.

How to answer this, stage by stage

Nobody is grading whether you can recite two definitions. They're grading whether you can say, with a real number, which one would have caught the bug that got through.

Let's learn

What actually proves a review comment is good, and what only proves the code compiled?

Switchyard is Millbrant Systems' own pull request reviewer. It reads a diff before a human does, and posts a comment: looks fine, or here's what to check first. Isolt Feyer owns the process that decides whether a change to Switchyard is good enough to ship.

Every pull request passes through the same five steps. The middle one is the check nobody was watching.

Two gates sit in front of every Switchyard release. Only one of them was actually a hard stop.

For most of a year, Switchyard caught 42 of the 50 known bugs in Isolt's golden set, a stack of real past pull requests, each one carrying a bug someone actually shipped once. Eighty four percent. The 340 unit tests covering Switchyard's own code, the part that parses a diff and posts a comment through GitHub's API, stayed green the entire time, every single day.

Then Kaspar Enderby, an engineer on the Switchyard team, fixed a real complaint. Developers were tuning Switchyard out: 31 percent of its comments got marked "not useful." He rewrote the prompt to raise the bar before it spoke up. It worked. The "not useful" rate fell to 12 percent within a week.

Two teammates read three pull requests together and called it good. The real fifty-item set never got touched.

Two teammates checked three pull requests by hand, agreed the new comments read sharper, and shipped it. The unit tests never even needed a second look, they test the plumbing, not the judgment, and nothing about the plumbing had changed.

Isolt's golden set ran automatically every night, the whole time, and it told the truth to a dashboard nobody was reading that week. Three weeks after the change shipped, with catch rate already down to 63 percent, Switchyard reviewed a pull request from the payments team that quietly removed a null check from a refund webhook handler, a line someone had added two years earlier for a payload shape everyone agreed "never happens anymore." Switchyard wrote: nice cleanup, this simplifies the logic nicely. The engineer merged it the way everyone merged once Switchyard signed off, without a second read.

Eleven days later, a refund arrived in the exact shape everyone had agreed never happened anymore. Refund processing stalled for 41 minutes. Six hundred and forty orders needed a person to reconcile by hand, most of it after midnight. The bill came to about 19,400 dollars.

What I would leave alone: a comment-wording tweak, fixing a typo in Switchyard's own phrasing, doesn't touch judgment at all. Gating that behind a fifty-PR eval run would slow down harmless fixes for no reason. Only prompt and model changes that could move what Switchyard actually decides need the gate.

The lesson: green tests are a real signal. They just were not the signal anyone needed here. A test suite can stay green for the rest of a company's life while the thing it's guarding gets quietly worse at the one job it exists to do.

Now here is the same thing as a story

The short version above is what you actually say in the room. Read this one for the fourteen months it took to build, and nearly stop keeping, the habit that should have caught this in a day.

Isolt Feyer spent six years writing code before she ever managed a product. She built Switchyard's golden set herself, in her second month on the job: fifty real pull requests, pulled out of Millbrant's own history, each one hiding a bug that had actually shipped once, an off-by-one error, a race condition in the billing worker, a null check someone had removed on purpose to "simplify" a function. She graded Switchyard against that stack the way she used to grade her own code reviews: not is it fast, is it right.

For the first several months, checking the golden-set dashboard was the first thing she did every morning, before her second coffee. Five minutes, sometimes less. It sat at 84 percent, steady, and every week it stayed there felt like proof the whole idea worked.

Nobody decided to stop checking closely. It just got easier not to, one quiet morning at a time.

Around month three, the check got quicker. She'd learned the shape of a healthy morning, and a glance told her almost as much as a careful read used to. By month six, some weeks she skipped it outright, usually the weeks a launch was close and four other dashboards needed her more.

Kaspar Enderby had a real problem that quarter. Developers were ignoring Switchyard. Not arguing with it, just scrolling past it, the way you scroll past a coworker who corrects your comma placement in a group chat. Thirty one percent of its comments got a thumbs down for being not useful. He rewrote the prompt so Switchyard held its opinions to a higher bar before sharing them. Two of his teammates read through three pull requests together, agreed the new comments read sharper, and called it good. Nobody pulled up Isolt's fifty pull requests. Nobody thought to. The tests were green. It felt done.

The "not useful" rate dropped to 12 percent inside a week. Kaspar posted about it in the team channel with a small trophy emoji. People reacted.

Isolt's golden set kept running every night, same as always, and it kept telling a different story to a dashboard nobody was reading that week. Eighty four became seventy nine. Seventy nine became seventy one, nine days after the ship, past the point where she used to say out loud, this is when I'd start asking questions.

Nobody asked. That was the whole problem, and it wasn't really about the model.

Three weeks after the change, Switchyard reviewed a pull request from the payments team. It removed a null check from a refund webhook handler, a line someone had added two years earlier for a payload shape that "never happens anymore." Switchyard read the diff and wrote: nice cleanup, this simplifies the logic nicely. The engineer merged it the way everyone merged once Switchyard signed off, without a second read. Eleven days after that, a refund arrived in the exact shape everyone had agreed never happened anymore. Refund processing stalled for 41 minutes. Six hundred and forty orders needed someone to reconcile by hand, most of it after midnight.

I want to say the problem was Kaspar's prompt. It was, a little. But that's not really the story either. Isolt never had a single number burned into her memory that told her the loop was healthy. She had a morning habit, and the habit only still worked on the mornings she actually kept it.

The decision she'd take back sits in a meeting from Switchyard's first month, when three people wrote the release rule in about twenty minutes flat: ship once the tests are green. Someone floated making the golden-set run a hard gate too. Someone else said that felt like overbuilding a check nobody needed yet, for a tool with one engineer and one prompt. They were right, for that week. Nobody ever came back to update the rule once the team grew and the ship cadence went from monthly to weekly.

Run the same three weeks again, with the gate Isolt built afterward already wired in: any prompt or model change has to clear 75 percent on the golden set, on a small canary, before it can reach every engineer's pull requests. Kaspar's change lands on the canary, scores 71, and the pipeline blocks it automatically, no meeting required. He gets a message instead of a trophy emoji: catch rate dropped six points, here's the pull request it's now missing. He tightens the prompt, reruns it, clears 82, and ships to everyone three days later than he wanted to. The payments team's null check never reaches Switchyard's desk with a passing grade it didn't earn.

The old design put a good number in a dashboard and trusted a habit to keep reading it. The new one puts the same number in a gate, and a habit doesn't get a vote anymore.

What I'd tell myself, back in that twenty-minute meeting: a release rule sized for one engineer and one prompt a month is not a release rule. It's a snapshot of that week wearing a release rule's clothes, and it will keep answering questions honestly right up until the day the team it was built for stops existing.

LEAD, for the score CI will never turn red

Not a way to prove Isolt is smarter than Kaspar. LEAD is what forces you to say which check actually catches a quality regression, and to catch it before a production incident does the talking for you.

Fast and honest beats fast and empty. The eval is the only signal that is both.

The recap, one line per letter: link the eval to what a bug reaching production actually costs, not to whether Switchyard's own code behaved. The early signal is the golden-set score itself, because a complaint and an incident both arrive weeks late. Name both ways the term gets abused, a hand-eyeballed example wearing the name, and a green test suite mistaken for coverage it never had. And the decision step is what makes it real: a different move at each threshold, not one blanket rule.

Two things worth saying plainly, since the real judgment sits here. Isolt considered a simpler fix first: tell Kaspar's team to eyeball ten pull requests by hand instead of three, next time. She rejected it, because a bigger hand-check is still the same mistake at a bigger sample, not a structural fix, and it doesn't scale once more than one team ships prompt changes in a week. The AI-specific failure worth naming by name is silent judgment drift: a model's review quality can slide for weeks with nothing crashing, nothing erroring, no log line anywhere that says "I got worse," because a language model doesn't fail the way code fails. The fix that actually catches it is the golden-set gate itself, wired into the merge pipeline so a passing score has to be earned fresh on every change, instead of trusted from the last time someone happened to check. And the trade-off was real and worth naming: gating every prompt change costs Switchyard's team a canary run and a day of extra wait on ships that used to go out the same day. That's a real cost, paid on purpose, in exchange for catching a 19,400 dollar mistake before it reaches a customer's refund instead of after.

And if you want to be sure it really works, try it somewhere else

Same four letters, a legal translation reviewer instead of a code reviewer, and this time the gap hides inside a contract instead of a diff.

Thistlecombe Legal Translations runs Norvellis, a tool that machine-translates contracts and hands a lawyer a draft to review before signing. Fennella Wrensworth runs product for it, and hit a quieter version of Isolt's exact gap four months after launch.

Same shape as Switchyard's two gates. A contract instead of a diff, a lawyer instead of a developer.

Norvellis had two kinds of check from day one. A format check: every placeholder filled, section numbers intact, no leftover English sitting in a translated paragraph, page count within a few percent of the source. All deterministic, all pass or fail, the same kind of thing a unit test is. And a quality eval: forty real past contracts, scored by senior bilingual lawyers on one question, does the obligation still bind the same way, does "shall indemnify" still mean shall indemnify.

Reviewing lawyers complained the translations ran long and dense. Norvellis's team shortened the prompt to tighten the phrasing. Format checks stayed 100 percent green, nothing about document structure had moved. The quality eval, run once a quarter rather than on every change, wasn't touched before shipping.

Eleven days after the shortened prompt shipped, the quarterly eval finally ran. Score fell from 91 to 68. One contract, already signed by a client, had translated "shall indemnify" with a softer, conditional verb, quietly weakening a liability clause nobody had meant to touch. Thistlecombe caught it before a dispute ever tested it, and paid for a costly amendment instead of a lawsuit.

Mapped onto LEAD, the shape holds. The link is the same: what actually matters is whether a signed contract still means what the original meant, not whether the document's shape looks right. The early signal is again the eval score itself, this time run too rarely to lead anything; the fix wasn't a smarter model, it was running the same eval on every prompt change instead of every quarter. The abuse Fennella found wasn't a hand-checked example, it was a calendar standing in for a gate. And her decision matched Isolt's: format checks stay automatic and instant, the quality eval runs on every prompt or model change, and nothing ships past a small batch of contracts until both clear.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: a unit test proves the code runs, an eval proves the model's judgment is still good, and a PM gates every release on both, because green tests alone have already let one real bug through.
Cost: no budget to build an automated golden-set gate this quarter. Whoever owns the feature runs the real fifty-item sample by hand before every ship, on a spreadsheet if that's what it takes, a slow real eval still beats a fast fake one.
The model got better, for real: say a new model raises catch rate to 92 percent out of the box. The gate stays, because a stronger model can still regress on the one narrow bug class the golden set was built to catch, and a higher starting score is not the same thing as a floor that holds.

Where people run it wrong.
They treat a green test suite as proof the model's output is fine, when the tests were never built to check that.
They call one hand-checked example "the eval," because it feels like due diligence and takes three minutes instead of an afternoon.
They build a golden-set eval once, at launch, and never add a new case to it, so it keeps measuring last year's bugs while this year's regression walks straight past it.

How to use it live. When an interviewer asks you to define two technical terms, buy yourself a breath by restating the real question first: "the real question is what each check can actually catch, and what it can't." That line is true, it buys you a second to think, and it previews the whole answer before you've said a single definition.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0