How often should evals run in a CI pipeline versus on demand?

Run the cheap slice on every pull request and the expensive suite on its own clock, because the moment the expensive one becomes the gate, people stop waiting for it.

How to answer this, stage by stage

Nobody is grading whether you know the word "nightly." They are grading whether you can say, out loud, which mistake you would let sit for a day and which one you never would.

Let's learn

Every pull request to Whittle used to sit in a queue for most of an afternoon before it was even allowed to merge. That was the whole gate: one eval suite, six thousand held-out coding tasks across twelve languages, each one actually run against real unit tests, forty two minutes, about thirty five dollars in compute, every single time.

When Quillhouse shipped a handful of pull requests a week, forty two minutes was nothing, closer to a coffee break than a cost. As Whittle grew past forty thousand developers using it inside their IDE every day, the team shipping it grew too, and pull requests climbed to more than thirty a day. The queue for that one gate stretched past three hours on a busy afternoon.

Nobody decided to stop trusting the gate. It happened in three small steps. First, engineers started batching several small changes into one bigger pull request, to make the wait worth it. Then, on a genuinely small change, someone would merge against an old passing check from earlier that day instead of waiting on a fresh one, "it's basically the same diff." Then that became the normal move for anything that looked minor, and nobody was really watching which pull requests had a fresh check behind them anymore.

A new model version shipped, meant to make completions faster across the board. Company wide, the blended pass rate barely moved, seventy one percent down to sixty nine. But the model had picked up a bad habit in Rust: it started suggesting .unwrap() inside functions that were supposed to hand an error back to the caller, code that compiles fine and then panics the first time something actually goes wrong. Rust's own held-out pass rate fell from seventy four percent to forty one percent, and the blended number never showed it, because Rust was a small slice of everything Whittle suggested that week.

Senna Vysotsky found this out because a new hire asked her a plain question: didn't the eval suite catch this? The honest answer was that the pull request carrying the model swap had queued for over three hours, and the engineer shipping it had re-merged against an earlier green check rather than wait for a fresh run against the new model. The gate existed. It had simply stopped being the thing that actually ran before code shipped.

By the time the new hire's question surfaced it, roughly eighteen hundred real Rust completions had already been accepted by working developers with the panic-prone pattern inside them. At its worst, an autocomplete tool that quietly hands out code that crashes in production is worse than no autocomplete at all, because a developer trusts a suggestion more than a blank line.

What I'd leave alone: the check that scans every suggested completion for a hardcoded secret, an API key or a password pasted into code. That one stays full and blocking on every single pull request, because it's already small and fast on its own, checking a pattern in text costs nothing like running six thousand real coding tasks. The split is a fix for a check that is both big and slow. Not every check is.

The lesson: a fast, cheap eval slice is not a smaller version of safety, it's a different tool for a different job. The mistake wasn't building a fast slice. It was letting the one slow, thorough suite quietly become the only thing standing between a bad model swap and forty thousand developers, on a clock too slow for anyone to actually wait for.

Now here is the same thing as a story

Read the long version below when you want to feel why a three hour queue, not a bad model, was the actual failure. The short version is above.

Senna Vysotsky joined Quillhouse when Whittle shipped one pull request every couple of days, most of them from a team of six. She built the eval suite herself: six thousand tasks, real code, run against real tests, and she was proud that nothing merged without it. Forty two minutes felt like nothing next to the confidence it bought.

For most of a year, that held up fine. The team grew, Whittle got picked up by bigger companies, and pull requests climbed from a handful a week to more than thirty a day. The eval suite didn't get any faster. It just got queued behind more and more requests to run it.

Nobody announced a change in how people merged code. It crept in over a few sprints. An engineer with three small fixes started bundling them into one pull request instead of three, so the wait only happened once. Someone waiting on a queued run for a genuinely tiny typo fix noticed an earlier run from that morning had passed on nearly the same diff, and merged against that instead of waiting again. It felt harmless. It happened again the next week, on something slightly less tiny.

The model swap that caused the actual damage looked, on paper, like a routine upgrade. It shipped on a Wednesday. The pull request carrying it queued for three hours and twenty minutes. The engineer shipping it, watching the clock, merged against a check that had passed against the previous day's code, reasoning the model change itself was "just a config flip," and moved on to the next thing on their plate.

Four days passed. A new hire, three weeks into the job, was reviewing Whittle's Rust suggestions for a training doc and noticed something odd: a suggested completion inside a function that clearly returned a Result was ending in .unwrap(), code that would panic instead of handing the error back up. She asked Senna, plainly, whether the eval suite had caught this. It was a fair question. Senna didn't have a fast answer, and that bothered her more than the bug did.

She pulled the merge history for the model-swap pull request and found the stale check. Then she pulled Whittle's own weekly numbers and split them by language instead of trusting the blended line. Rust's held-out pass rate had fallen from seventy four percent to forty one percent, the day the model shipped, and stayed there. The blended number, sitting at sixty nine, had never once dipped below seventy on the dashboard anyone actually looked at.

The decision Senna would take back went back to the day she built the suite. She'd made one eval suite answer two different questions: is this pull request safe to merge, and is Whittle, as a whole, ready to ship to everyone using it. Those questions don't need the same amount of certainty on the same clock. Bundling them made total sense when a PR came along every couple of days. It quietly became the whole problem once PRs came every twenty minutes.

Run the same Wednesday again with the new design. A small, three-hundred-task slice, weighted toward Rust since that's what the model-swap diff touched, runs in under two minutes and blocks the merge either way. But this pull request carries a model version change, so policy requires the full six-thousand-task suite, in full, before it's allowed to merge at all, no stale-check shortcut available. The full run finishes in forty two minutes, flags Rust's pass rate falling to forty one percent, and the merge is blocked before a single developer's IDE ever offers that panic-prone suggestion.

One design trusted a single, slow gate to still mean something even after the team outgrew the wait it demanded. The other design keeps the fast promise fast, and reserves the slow, thorough one for exactly the changes that need it.

What I'd tell myself, back on the day I built that first suite: the moment a check gets slow enough that people start finding a way around it, they will find one, quietly, without ever deciding to cut a corner on purpose. That's not a discipline problem. That's a design problem, and it was mine.

PICK, said out loud before the reasons

Four letters, not five. PICK commits first, then shows you exactly why one kind of miss is cheap and the other one is not.

One miss is small and gets caught by morning. The other one is quiet, and it never gets caught, because the team stops trusting the thing meant to catch it.

Three things worth stating plainly, since this is where the real judgment sits. The rejected alternative was throwing more compute at the single full-suite gate instead of splitting it, parallelising the six thousand tasks across dozens of runners to shrink forty two minutes down. It lost for two reasons: several of those tasks spin up a real language runtime and a real test framework per task, so the wall clock has a floor near fifteen to eighteen minutes even maxed out, still too slow against thirty plus pull requests a day, and running that much compute on every single pull request, forever, costs roughly forty times what running it once a night does. The AI specific failure worth naming by name is a silent, per-category model regression: a base model swap that looks fine on a blended score while one language quietly falls apart underneath it. The guardrail is recutting the nightly pass rate by language, never trusting the blended number alone, plus the on-demand rule that any model swap earns the full suite before merge, not after. And the trade being accepted on purpose, out loud: the fast slice will occasionally let a real regression through for up to a day, in exchange for a gate fast enough that nobody is ever tempted to route around it. That's a latency and completeness trade, taken deliberately, not a hope that both could be free.

And if you want to be sure it really works, try it somewhere else

Same four letters, an insurance claims tool this time, and the pick flips: here the full suite is the everyday gate, on purpose.

Backstitch is a tool adjusters use at Bellcastle Insurance to draft a first-pass summary of a claim, coverage, damages, and any exclusions that might apply, before a human adjuster signs off. Adalet Doyen runs quality on it.

Backstitch's own eval suite holds two thousand held-out claim scenarios, checked against real determinations a senior adjuster already made. Adalet's team ships maybe three or four pull requests a week, not thirty a day. So the full suite, all two thousand scenarios, runs in full, blocking, on every single pull request, twenty five minutes, every time. A fast, small smoke check exists too, but it's only for engineers iterating on a branch before they've even opened a pull request, to get quick feedback without waiting on the real gate.

Same method, opposite result: at Bellcastle, the cost asymmetry runs the other way. A slow gate barely costs anything, since pull requests are rare and nobody's ever stuck in a real queue behind one. A missed regression costs a lot, and it can't wait a day. So the full, expensive suite stays the everyday blocking check, and the fast check is only for private iteration, never the thing that actually gates a merge.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Say the split straight: "fast slice gates the merge, full suite watches the main branch overnight, and any model change always earns the full suite before it merges." That line alone answers the question.
Cost: there's no budget this quarter for a nightly server just to run the full suite on a schedule. At minimum, keep the full suite required on demand for the two riskiest moments, a model swap and a release cut, since those are exactly what a diff-scoped slice is built to miss.
The model got better, for real: say Whittle's completion model gets upgraded to a stronger version. A higher blended pass rate isn't proof every language improved together. Rerun the full suite, recut by language, before trusting the new blended number at all.

Where people run it wrong.
They treat "run evals in CI" as meaning the full, expensive suite has to clear on every commit, and either it becomes too slow to be a real gate, or the team quietly stops enforcing it and nobody notices for months.
They let a nightly cadence stand in for every kind of change, including the rare, high-risk ones, a model swap, a prompt rewrite, that genuinely need the full check before merge, not the next morning.
They watch only the blended pass rate, and never recut it by whatever a real regression is most likely hiding behind, language, claim type, region.

How to use it live. Say the split out loud before you give a number: "before I answer with a cadence, let me ask what a missed regression actually costs here, in hours, because that decides whether the slow check runs before the merge or after it." That buys a beat to think, and it shows the interviewer you're pricing the mistake, not reciting a schedule.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0