Describe a metric that would have caught a quality regression before users complained.

How to answer this, stage by stage

Nobody is grading whether you can name a metric. They are grading whether you can find the day a regression actually started, before you go looking for the number that would have caught it. Eight moves get you there.

Let's learn

What does a quality regression look like three weeks before anyone complains about it?

Marginalia is a bot built by Kestrelwood Systems. It reads a pull request and leaves comments right on the lines that matter, sorted into four kinds: style, correctness, security, and performance.

Through February, Marginalia's performance comments (the ones that flag a slow query or a loop that runs longer than it needs to) scored 89 out of 100 on Kestrelwood's golden set, a fixed stack of 150 real past pull requests with known issues already marked. Developers dismissed about 13 percent of those comments as unhelpful, roughly the same rate as the other three categories.

On March 3rd, the team shipped a small prompt edit meant to make every comment type sound less preachy. Nobody flagged it as risky. It was a tone pass, not a rewrite. Three weeks later, the same golden set scored performance comments at 61. The dismiss rate for that one category had climbed to 44 percent. Nothing else moved.

At its worst, this ran for 40 days before anyone found it: about 1,900 vague performance comments landed on real pull requests at customer companies, each one a small, silent reason to stop trusting that category of comment. Only two developers ever said anything out loud about it.

What I would leave alone: the one line comment Marginalia posts once, on a repo's very first pull request ("Marginalia is now reviewing this repo"). It is not written by the model. It cannot drift, so it does not need the same daily watch as everything else.

The lesson: a number that blends four different jobs into one score will always look calm. Calm is not the same thing as healthy. It just means you built one number where you needed four.

Now here is the same thing as a story

The short version is above. Below is the week this almost went unnoticed for good, told in order.

Every Friday afternoon, Edda Falk pulls the week's quality digest before anyone else at Kestrelwood sees it. She has run product for Marginalia for two years, since before it had a name, and she built the golden set herself: 150 real pull requests, each with the issues a good reviewer should catch already marked. For eighteen months the digest read the same story every week, one blended score, usually 89 to 92, a green check next to it, and Edda moved on to the next thing on her list.

Early March, the tone edit for less preachy comments shipped. The Friday digest that week read 90 percent. Edda glanced at it for maybe four seconds and closed the tab. Two weeks later, still holding near 88. Why would she reread it. It was close to where it had lived for over a year. By the last week of March, the digest read 86, four points under where it had sat for months. She noted it, decided four points was nothing, and moved on to the next thing.

On March 26th, a senior engineer at a customer company posted one line in the shared support channel: "Marginalia's perf comments have gotten vague lately. Feels like it's just guessing now." Support saw five tickets that week against a normal week of three, filed it as within range, and closed it.

Edda almost didn't see it either. A second, sharper complaint came in on April 2nd. Two data points is still easy to explain away at a company with hundreds of customers. What actually made her stop was smaller than either complaint: a habit she had built years earlier, never trust a summary number without reading five real examples underneath it. On April 7th, instead of glancing at 86 percent again, she pulled ten real performance comments Marginalia had left that week and read them next to ten from January.

It was never about the two tickets. It was about the number that had stayed calm for five straight weeks while the thing underneath it got worse. Three suspects made sense once Edda actually went looking: the model could have quietly changed under Kestrelwood's feet, the March 3rd prompt edit could have changed what the performance template asked for, or an unrelated change to how large diffs get chunked before hitting the model could have hurt every category at once.

The model version pin ruled out the first suspect in minutes. The other two needed a real check, not a guess, before either one could be crossed off.

The model version log had not changed since January, so that suspect was out fast. Between the prompt edit and the chunking change, Edda pulled ten real performance comments from before and after March 3rd and read them next to correctness and security comments from those same large files. Correctness and security still cited real numbers. Only performance had gone vague. That ruled out chunking (which would have hurt every category on a big file, not just one) and pointed straight at the prompt edit. The template's own git history showed exactly what changed: the line that used to read "cite a specific complexity class or an estimated latency or throughput number" was gone from the performance template, cut in the March 3rd tone pass, still present in all three of the others.

Two years earlier, when the golden set first shipped, the design meeting lasted maybe twenty minutes. Someone asked whether the score should be split by category or reported as one number. One number was simpler, and at the time all four categories moved together anyway, so splitting it felt like solving a problem nobody had yet. Nobody in that room was picturing one category quietly failing while the other three propped up the average. Why would they. It had not happened yet.

Edda reverted the missing line on April 12th, at 2:14 in the afternoon, forty days after it shipped. Ten weeks later, in June, a routine dependency upgrade nudged the security category's golden set score from 92 to 84 overnight. The new daily, per category alert caught it by 6:40 the next morning. Edda reverted the change by 11 a.m. Zero complaints, zero tickets, and a fix that took under five hours instead of forty days.

What I would tell myself, back in that twenty minute meeting: a score that blends four jobs into one number will always look calmer than the truth underneath it. You do not find out which job broke until you build a place where each one can fail on its own.

TRACE, in the order I actually ran it

This is a diagnosis question read backward from a symptom to a cause, so TRACE fits. Not a forward-looking metric picked with no incident behind it.

Two things are worth naming straight, since this is where the AI-specific judgment actually sits. The team considered just raising the alert on total complaint volume instead, watching for five tickets in a week instead of three. That was rejected on purpose: complaint count is a lagging, blended signal, and it sat inside its normal weekly range for all twenty three days the regression was live, because most developers who lose trust in a comment type just stop reading it instead of writing in about it. The failure mode worth naming by name is silent degradation, a prompt or model change that makes output less specific without ever becoming wrong in a way an existing check would catch, and the guardrail is the daily, per category golden set score plus a per category dismiss rate alert, not a person spot reading the weekly digest. The bar for that score is calibrated, not absolute: a category's gate holds at or above 85 on a rolling three day average, not any single day's read, because one bad day happens on its own and a rollback triggered on noise wastes an afternoon for nothing. The trade off is real too: a comment that cites a real number or a complexity class runs longer and costs more tokens to make than the shorter, friendlier version that shipped by mistake. That is the cost of a comment a developer can actually check, instead of one that just sounds confident.

And if you want to be sure it really works, try it somewhere else

Same five letters, a county permitting office instead of a code review bot, so the method proves itself instead of repeating a story I happened to prepare.

Dellmar County's planning department runs Curbline, a tool that pre-screens residential permit applications and flags anything missing or wrong before a human reviewer opens the file. Ambrose Brennick supervises the review team.

T, timeline. A vendor pushed a routine update to the document reading model behind Curbline on a Tuesday in October. Nobody at the county controls that update or gets a warning before it ships.
R, recut. Split the correct flag rate by permit type instead of reading one office wide number. New construction permits, mostly typed digital plans, stayed steady. Renovation permits, mostly scanned pages with a contractor's handwriting on them, fell hard.
A, assume nothing. Checked whether fewer renovation permits simply came in that month. Volume was flat, about 40 a week, the whole time.
C, cause candidates. The vendor's document reading update, a queue change that had started routing renovation permits to a newer reviewer pool, or turnover on the scanning team changing how pages got prepared.
E, evidence test. Ambrose reran the exact same batch of scanned renovation applications through Curbline twice, once against the model version from before the update and once against the current one. Same files, same scans, a different score. That isolated the vendor's update as the cause in under an hour.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to the recut: whatever the blended number says, split it by the group most likely to be hurt, before you trust it either way.
Cost: engineering says a daily, per category eval costs too much compute to run every day. Do not fall back to a blended weekly number as a stopgap. Run it daily for the categories most likely to drift and weekly for the rest, instead of daily for none.
The model got better, for real: say the underlying model gets meaningfully better at writing comments generally. That is still not proof one category, tuned by one prompt template, is safe. A better model finds a new way to satisfy a vague instruction just as easily as an old one did.

Where people run it wrong.
They read a flat blended number as proof nothing changed, instead of asking whether four different things are being averaged into it.
They treat the first complaint as the start of the problem, instead of the day it finally got loud enough to hear.
They chase every possible cause at once instead of naming three real suspects and running the one check that tells them apart.

How to use it live. Say the split before naming a single cause: "the number I would trust least right now is any average blending more than one kind of output, because an average is exactly where a real regression hides." That buys you room to actually trace it, instead of reciting "we would monitor closely" on reflex.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0