How do you write criteria that hold across a model upgrade?

How to answer this, stage by stage

Six moves. The trap in this question is treating "the score went up" as proof the criteria still work, so two of the six stages are spent forcing that assumption to earn its keep.

Let's learn

Hooklines is a feature inside Brightloop, a marketing platform. A client picks an email campaign, and Hooklines writes the subject line, so the client doesn't have to.

When Hooklines launched, Astrid wrote five rules a subject line had to pass: a length limit, no shouting in all caps, no more than one emoji, a correctly formatted name tag, and no spammy punctuation. Fourteen months later, after a model upgrade, the pass rate on those five rules climbed from 92 to 98 percent.

Here is the turn. Those six extra points on the rubric never had a chance to see the problem. The real story is a kind of subject line the rubric was never built to check: one that states a real number, growing from a small slice of everything Hooklines wrote to more than a third of it, and increasingly making that number up.

At its worst, this costs Brightloop named clients with campaigns already sent. Wick and Ember, a candle brand with two hundred units of a bestseller in stock, sent a subject line reading "Only 6 left, don't miss it," and spent the next two days answering angry replies from customers who'd tried to buy a candle that was never actually running out.

The gap between a climbing pass rate and the mistake it was never built to see

The choice I would take back. Writing the five rules around the old model's actual mistakes, fourteen months ago, was the sensible call. That model almost never wrote a specific number, so nobody had a reason to write a rule for one. I would take back leaving those five rules as the whole rubric, with no standing check tied to "does this claim exist in real data." I would write that sixth rule the day the tool launched, even while it rarely fired, so it would already be there the day it started to matter.

What I would leave alone. The length, caps, and emoji rules don't need touching. They still catch what they always caught, at close to the same rate, because those are exactly the mistakes the new model still occasionally makes. Rewriting rules that already work spends time the real gap needs.

The lesson. A rubric is a list of the mistakes you've already seen. The day the model changes, that list does not know it's incomplete, and it keeps handing out a passing grade unless someone goes looking for the mistake it was never written to catch.

The two days Wick and Ember stopped trusting the send button

Read the short version above if you're short on time. This is the long version, for when you want to feel exactly where the sixteen months went.

Astrid Fenmark has run eval for Hooklines since it launched. She can read a week's pass rate and tell, inside a minute, whether a release earned it or got lucky on an easy batch of campaigns.

She wrote the original five rules herself, with two copywriters, the month Hooklines first shipped. Under sixty characters. No shouting in caps. At most one emoji. A correctly formatted name tag. No spammy punctuation. All five came straight from watching that early model's actual mistakes, week after week, and writing a rule for each one.

For the first several months, watching the pass rate climb felt like the model getting sharper. Eighty-eight. Ninety. Ninety-two. Every point felt earned. Somewhere in there, without anyone deciding it on purpose, the review team quietly stopped double-checking every flagged subject line by hand. When only one in eight lines mentioned a number, checking each one against the real campaign was easy. Nobody had a reason to worry about capacity.

Then, at month fourteen, Brightloop rolled out a model upgrade across the whole platform.

Nobody rewrote the rubric. Why would they. It kept scoring in the high nineties.

Two months later, three weeks before a seasonal sale, Ramona Vetch, who ran Wick and Ember out of a small workshop with her brother, noticed a subject line claiming her bestselling candle had six left. She had two hundred in the back room. She thought it was one strange line. Then she pulled the last ten campaigns. Six of the ten had a number that wasn't real.

She paused every scheduled send and started writing her own subject lines by hand, the way she did before she ever used Hooklines.

I want to say the model got worse at writing subject lines. It did not get worse. It got fluent enough to sound certain about a number it never checked, and none of the five rules had ever been built to notice.

So here is the decision I would take back.

Fourteen months earlier, writing five rules straight from the old model's real mistakes was the sensible call. That model almost never stated a specific number, so nobody wrote a rule asking whether one was true. I would put a sixth rule in from day one: any subject line naming a price, a percent, a count, or a date gets checked against the campaign's real data before it ships, not skimmed by a person for tone. Not because the five old rules were wrong. Because of what it would have shown, two months earlier: the new model stating a real number three times as often as before, and getting more than half of them wrong.

That is the whole difference. One rubric trusts a score that only ever answered for the mistakes it already knew. The other checks that score against the mistake the model just started making.

And the part I would want to tell myself, if I could go back: we wrote a test that only ever answered for the model we had on day one. Nobody decided that on purpose. We just never rebuilt it when the model changed.

What the recut actually showed

Before blaming the rubric's coverage, Astrid's team checked whether the scoring itself was even right. Two people hand-checked 20 flagged subject lines against what the automatic rubric said they should score. They agreed on 19 of 20. The grading was not the problem. That left the coverage.

Three ways a passing rubric can miss a brand-new mistake

Not because anyone cut a corner on purpose. A rubric built entirely from the old model's habits can keep passing, release after release, and still stop meaning anything the day the model starts making a different kind of mistake.

Three separate, checkable ways a rubric can pass and still miss the real mistake

Reading TRACE off a rubric that never learned to doubt a number

This reads like a question about writing good rules, but the real job is diagnosis: work out why a rubric the team trusted for fourteen months quietly stopped predicting real client complaints. GUARD would fit if the harm landed unevenly across a protected group; here the harm is a whole kind of mistake the rubric never learned existed.

Same shape, a maintenance reply that promised a technician nobody dispatched

Fallowmere Property Group runs an AI tool that drafts the first reply to a tenant's maintenance request, for a human property manager to lightly edit and send. Declan Whitfield, the operations lead, wrote the rubric ten months before a model upgrade: does the reply name the specific issue, does it give a timeframe, does it avoid promising a technician the office hasn't actually scheduled.

T. The rubric's overall pass rate climbed from 90 to 97 percent across three releases, including the upgrade. Tenant complaints about a broken promise, someone showing up to wait for a technician who never came, held steady around one in thirty replies for months, then started climbing six weeks after the upgrade shipped.
R. Split by whether the reply names a specific date or time. Vague replies, 69 percent of volume: complaint rate barely moved. Specific replies, 31 percent of volume, up from 8 percent before the upgrade: 44 percent of them turned out to promise a visit nobody had dispatched, up from 3 percent.
A. Before blaming coverage, staff hand-check 15 flagged replies against the rubric's own verdict. They agree on 14 of 15. The scoring checks out.
C. Three candidates: the rubric was worded around the old model's habit of writing vague, generic replies, with no rule checking whether a stated appointment is real; the review queue was sized for when only 8 percent of replies included a date, and got swamped once that share nearly quadrupled; and reviewers were trained to skim for a warm, reassuring tone, never to check a stated time against the dispatch system, because the old model rarely gave them a specific time to check.
E. Pull 40 real replies with a specific date or time, sent after the upgrade, and check each against the dispatch system. Rubric pass rate: 95 percent. Real match to an actual scheduled visit: 52 percent.

Flashcards (click a card to flip it)

This is a diagnosis question about a rubric quietly going blind, not a habit changing, so these eight test the TRACE moves and the real figures behind them.

Check yourself Score: 0 / 0