How do you write an eval for a feature where there is no single right answer?

How to answer this, stage by stage

Seven moves. Most of the weight sits in stage five, the paragraph where a good release almost gets blocked by the wrong kind of test. Every stage has the actual words to say.

Let's learn

Here's what happens when a test built for one right answer meets a feature that has fifty of them.

Penumbra is a fiction writing app. One of its features writes a scene starter prompt on demand: a few sentences to get a stuck writer going again, a character, a want, a problem in their way.

A year before scene starters existed, Penumbra shipped a smaller feature: chapter titles, generated on request. Titles are short, and there's really only a narrow band of good shapes for five words. So the eval the team built for it was simple: save one strong title per test case, and score a new title by how close its wording sat to that one. Nightly, in under a minute, without anyone opening a single one of the 300 test titles by hand, it decided if a release was safe. For a year, it worked. It caught every real regression that shipped, and its pass rate held around 91%.

Then scene starters launched. They're forty to eighty words, wide open, dozens of different angles are equally good for the same character. The same eval ran on them anyway, because it already existed and rewriting it wasn't on anyone's list. As scene starter prompts took over more and more of the nightly test batch, week by week, the pass rate quietly drifted down.

Here is the turn. Those extra failed cases were not the real problem. The real problem was what Carys almost did next.

Carys Pentreath runs that nightly eval. She'd been at Penumbra two years, and six years copyediting genre fiction before that. She can tell a lazy prompt from a strong one in one read. For two years, when the score dropped, she trusted it completely and pulled the release, because on chapter titles the score had never once lied to her.

So when the pass rate hit 58% on a Thursday night, her first move was the one that had always been right before: she started drafting an email to block Friday's push.

At its worst, this costs the exact thing the eval was built to protect. A release gets pulled for nothing, once, twice, and eventually the team stops trusting the eval at all, on the one feature where the eval is the only thing checking the model's work.

What I would leave alone. Penumbra's scene break formatting, converting a manuscript's break marks into the app's internal markup, has exactly one correct output for every input. The old exact match eval is still completely right there. A checklist would only slow it down for nothing.

The lesson. An eval that only knows one right shape doesn't get more careful as a feature gets more open. It gets wrong in a way that looks exactly like the feature getting worse, and nobody notices until someone reads the actual output by hand.

Now here is the same thing as a story

Pull this one out when there's room to sit with it, not just tick it off a list.

Carys Pentreath spent six years copyediting genre fiction for a small press before Penumbra hired her, and she can still tell a cliche opening line from a real one before she's finished reading it. Two years ago she built the nightly eval for the app's chapter title feature: pick one strong title per test manuscript, score every new title by how close its wording sits to that one saved answer, fail anything too far off.

For the first few months she read every single flagged failure herself, all of them, line by line, before she trusted the number attached to it.

By month four she was sampling ten or so a night, and they always matched what the score said. By month eight she'd stopped opening them at all. She'd glance at the pass rate over coffee, and if it held above 90%, she shipped. It had never once been wrong.

Scene starters shipped in March, a much bigger feature: forty to eighty word prompts, a character, a want, a problem in their way, dozens of good ways to write any one of them. Nobody built a new eval for it. Carys's reference eval just ran on the new test cases too, because it was already there and Friday releases don't wait for a rewrite nobody had asked for.

For six weeks the scene starter share of the nightly batch crept up quietly, a tenth, then a quarter, then most of it, as older chapter title test cases got retired. The pass rate drifted down a little each week. Nothing alarming. Nothing anyone flagged.

Then on a Thursday in late April, scene starters became the entire nightly batch for the first time. The pass rate fell to 58%.

Carys did what two years of a trustworthy score had taught her to do. She opened a new message to the release channel: "Blocking tomorrow's push. Regression in prompt quality. Will investigate Monday."

She had the cursor on send when she stopped, for no better reason than the number felt too big to be true. So instead of clicking it, she opened fifty of the failed prompts and actually read them, the way she used to.

Forty six of them were good. Genuinely good. A prompt about a lighthouse keeper who'd stopped trusting the fog, failed, because it didn't share a single sentence with the one saved "ideal" prompt about a lighthouse keeper the eval had on file. A different one, about a locksmith who could open anything except her own front door, failed the exact same way.

Four of the fifty really were bad, a want with no obstacle in it, one that barely made sense at all. Those four were real. The other forty six were the eval punishing a writer's idea for the crime of being a different good idea.

Here's the call I'd take back. A year earlier, when two engineers built the eval for chapter titles, they picked one strong title per test case and scored new titles against it. Nobody in that meeting was wrong. Titles are five words long, there genuinely isn't much room to be good in a different direction, and the eval caught real regressions for a year straight. Nobody ever came back to ask whether that same idea would survive a feature with fifty good shapes instead of two.

People, and evals, are switches, not dials

I'd take that back and replace it with a written checklist: does the prompt name a real want, does it put a real obstacle in the way, does it match the genre the writer asked for, is it under the length cap. Any prompt that clears all four passes, whether or not it sounds anything like Carys's saved example.

Run the same Thursday again, checklist in place instead of one saved answer. The same 300 prompts get scored again. This time 267 pass, 89%. Carys spot checks a fresh sample of the 33 flagged fails, and this time nearly all of them, roughly thirty, are real: a want with no obstacle, the wrong genre, something over the length cap. By nine that evening, before the on call meeting even starts, she sends a different message: "Cleared for Friday." The release ships on schedule.

And there's a second thing the checklist gave back that the score never showed on any dashboard. For six weeks before that Thursday, the team writing Penumbra's few shot examples, the sample prompts fed to the model to steer its style, had quietly been nudging them closer to the one saved answer per category, because that's what kept the score up. Under the checklist, they stopped. The tool's actual range in later reader testing widened back out. Testers stopped seeing the same three cliche openings over and over.

If I'm honest, building the eval the fast way a year earlier wasn't the mistake. Anyone would have. The mistake was never asking, the day scene starters shipped, whether "one saved answer" was still the right idea for a feature this open.

The switch inside the eval, spelled out

The letters matter less than which one breaks first. Here's the same five steps, mapped onto Carys's Thursday.

FLIPS, five rows

A small, slow drift. One hard snap in what she did about it.

"She got less careful about checking" is a diagnosis anyone can offer after the fact. The harder part is naming the exact thing that had to stop first, reading the raw text, and showing there was no smaller version of it left once the score had been right for two straight years.

And if you want to be sure it really works, try it somewhere else

Trailmark Coaching drafts personalized weekly workout plans with the same kind of tool, and keeps its own eval to certify a new model version before real clients see a plan. Same question, a fitness app instead of a fiction app, and a flip that isn't a quiet redesign this time. It's people walking away.

F. Bram Castillo, senior coach who oversees eight junior coaches piloting the plan drafting tool inside Trailmark's coaching team.
L. For his first few weeks, every junior coach flagged an AI plan they disagreed with straight into the review dashboard, every time, no exceptions.
I. A different flip. Nobody hides a bad plan or builds a workaround. They stop flagging anything at all. Files a dashboard flag the moment a plan looks off, or quietly rewrites it by hand and never opens the tool again for that client, with nothing in between once flagging stopped changing anything.
P. The pilot's eval scored a new plan against one saved "ideal" plan per client type, built when the pilot only covered marathon training. A plan that used a different, equally valid approach to build endurance got flagged as broken every single time, so every flag a junior coach filed came back "as designed," week after week.
S. Swap in a checklist, respects the stated injury limits, keeps weekly volume in a safe range, actually builds week to week, instead of matching one saved plan. The same kind of alternate plan stops getting flagged. Coaches start opening the tool daily again inside two weeks.

Flashcards (click a card to flip it)

Eight fixed slots, pulled straight from the answer above.

Check yourself Score: 0 / 0