Describe the failure mode where evals improve but users do not notice.

A rising eval score can be completely honest and still be measuring the wrong population's tracks. Here is how to catch the gap before a musician does.

How to answer this, stage by stage

Nobody is grading whether you know what a golden set is. They are grading whether you would trust a number just because it is going up.

Let's learn

Lacquer is a mastering suggestion tool built by Hollowbody Audio. A musician uploads a rough mix of a song, and in under twenty seconds Lacquer suggests three EQ and loudness moves to make it sound closer to a finished, professional master.

Before Lacquer, an independent musician without a label paid a mastering engineer somewhere between one hundred fifty and three hundred dollars a song, then waited three to five days to get it back. Or they mastered it themselves with a forty dollar plugin and guessed.

Lacquer answers in under twenty seconds, free on the basic tier. Checked every week against a golden set of two hundred forty professionally mastered reference tracks, its reference match score climbed from seventy one to eighty to eighty nine over three releases, about twelve weeks.

Every release, the team called the climbing score a win. But the share of musicians who actually clicked apply on Lacquer's suggestion barely moved: thirty four percent, then thirty five, then thirty seven. A survey asking whether the suggestion made the track sound better held near sixty percent, then sixty one, then sixty three. Eighteen points of score. Two or three points of trust.

The golden set behind that score was built from two hundred forty reference masters Hollowbody Audio could license cheaply when Lacquer first launched. One hundred ninety seven of them, eighty two percent, were pop, rock, or country records mastered for a major label. By the time the score hit eighty nine, sixty one percent of what real musicians were uploading to Lacquer was something else entirely: bedroom pop, lo-fi hip-hop, home recorded folk, ambient tracks cut on a laptop mic. The golden set had never heard most of what Lacquer was actually being asked to master.

The team's first guess, when a musician's comment about the tool showed up in Lacquer's Discord, was that one person had a bad ear, or a bad day. That's a fair thing to check before you build a whole investigation on one comment. Signe Munck, who runs quality on Lacquer, pulled the actual audio instead of arguing about it, and then built something bigger: a blind panel where working audio engineers rated real suggestions with no idea what score each one had gotten.

At its worst, Lacquer was quietly making some musicians' tracks worse while the one number everyone was watching said it was getting better. A demo that used to sound like a warm bedroom recording came back brighter and louder, a bad imitation of a Spotify pop single, with none of the warmth the artist actually wanted.

What I'd leave alone: the pop, rock, and country suggestions. That's exactly the sound the golden set was built from, and the apply rate there really did rise, thirty six percent to forty four. Spending audit time re-checking a genre where the score and real usage already agree would waste time better spent on the genres the golden set never saw.

The lesson: an eval score can be computed honestly and still lie, if it only tells you about the population it was built to represent. Eighty nine percent told the team Lacquer was getting better everywhere. It never told them the eighty nine was ninety one on tracks the golden set knew and eighty eight on tracks it had never once heard, and that the second number was a mechanical trick, not a real gain.

Now here is the same thing as a story

Read the long version below when you want to feel why a climbing score fooled a whole team for three releases, not just be told which segment it was hiding in.

Before Hollowbody Audio, Signe Munck spent six years mastering records in a small studio above a record shop, one song at a time, by ear. She knew the difference between a mix that needed brightening and a mix that needed to stay exactly as warm and quiet as the artist made it.

She built Lacquer's scoring pipeline herself, in the company's first year. For months, every Friday release note read the same way: the score is up again. Seventy one, then seventy four, then seventy eight. The team celebrated each one in the same ten minute Friday meeting. Signe liked the meeting. It meant the thing she'd built was working.

Slowly, the meeting changed shape. First, the apply rate number moved from the top of the slide to a small line near the bottom. Then someone stopped updating that line for a release or two, because nobody asked about it. By the time the score hit eighty nine, the Friday meeting only had one number in it.

The trigger was small. A musician posted in Lacquer's Discord: "the new version made my demo sound like a Spotify ad, not what I sent it." One line, from one person, easy to read past.

Signe didn't read past it. She pulled the actual track, before and after, and put on headphones. The demo had been a deliberately quiet, warm home recording. Lacquer's suggestion pushed the top end brighter and the loudness up, the exact direction that would help a thin pop mix and the exact wrong direction for this one.

She spent a Saturday building the blind panel: forty working engineers, fifty of the highest scoring suggestions from tracks in genres the golden set barely covered, rated with no score attached, against the original mix. Twenty two of fifty came back rated no better or worse than doing nothing at all. She ran the same test on fifty suggestions from genres the golden set did cover. Only six of fifty came back that way.

The decision that opened the door went back to a meeting from Lacquer's first month, when the team picked which reference masters to license. Two hundred forty tracks, cheap to get the rights to, almost all pop, rock, and country. Nobody in that room was doing anything wrong. Nearly every early user of Lacquer was, in fact, a pop musician. The set fit the product it was built for. Nobody ever came back to ask whether it still fit the product a year later, because the score kept saying yes.

Run the same Friday meeting again with one change: a blind panel of ten tracks, rated by two working engineers, runs automatically the Tuesday before any release ships, split by golden set coverage. The forum comment never has to be the thing that catches it. Two days after the model change that would have caused it, the panel flags a drop in the uncovered group, and the release gets held before a single musician hears a brighter, louder version of a quiet song they never asked to be loud.

One design let a single company wide score decide when to celebrate. The other design makes a person listen to what the number can't hear before it gets the last word.

What I'd tell myself, back in the room where the golden set got licensed: ask who isn't in these two hundred forty tracks, not just who is, because the users you don't have data on yet are exactly the ones a rising score will quietly get worse for.

TRACE, the method for a number that would not stop climbing

Not five guesses about why users don't care. TRACE rules candidates out on purpose, until one blind test actually separates a real gain from a shortcut.

The golden set was locked before any release shipped. Every release after that widened the same gap.

Two of the three suspects are real. Only one of them is the actual cause of the gap.

Three things worth stating directly, since this is where the real judgment sits. The rejected alternative was trusting the still climbing score and assuming the golden set gap would get fixed on its own once a future data project widened it. It lost because it meant shipping suggestions that made real tracks audibly worse for months, on a fix with no date attached, while the one number everyone watched kept saying things were fine. The AI specific failure worth naming by name is metric gaming through a domain gap: the model didn't lie or hallucinate, it found the cheapest way to satisfy a similarity score, pushing loudness and brightness toward the reference masters, and that shortcut only ever gets caught by checking against people, not against the same metric that got gamed. The guardrail is two part and it has a real cost attached. The full golden set eval runs automatically on every commit, four minutes, next to free, and catches nothing about this kind of gap on its own. The blind panel is expensive, forty engineers, a few hours, real money, so it can't run on every commit. The trade accepted on purpose: cheap and constant for the score, expensive and occasional for the only check that can catch it lying, gated before every release ships rather than reviewed after.

And if you want to be sure it really works, try it somewhere else

Same five letters, a different newsroom, and this time the shortcut isn't a loudness push. It's copy and paste.

Gist is a summarization tool The Cutbank Bulletin, a small town newspaper, uses to draft a short version of every story before an editor reads it. Eryk Bretz, the paper's managing editor, runs quality on it.

Gist's eval score, how closely its summary's wording overlaps a golden set of professionally written summaries, climbed from sixty eight to eighty four over two releases. The share of Gist's summaries an editor used with only light edits held flat near twenty eight percent the whole time.

Eryk recut the failing summaries and pulled the twenty highest scoring ones by hand, line by line, against the source article. Most of each "summary" turned out to be one or two sentences lifted straight from the article and stitched together, not a real compression of anything. That raises an overlap score mechanically. It doesn't save an editor a single minute of reading, because the summary is still, word for word, most of the article.

A different shortcut than Lacquer's, and a different cause behind it. Same method to find it.

Same method, a different shortcut: stop rewarding any shared wording with the source, and score a summary down when a single lifted span makes up most of it. Two metrics can both be an "overlap score" and still be measuring completely different things, one rewarding real compression, one rewarding a well disguised copy and paste.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to the one test, run a blind check with no score attached, and see if a person rates it better. That single line answers the question on its own.
Cost: there's no budget this quarter for a forty person blind panel every release. Run a small version instead, ten samples, two reviewers, on whichever segment the golden set covers worst. It catches the same shortcut for a fraction of the price.
The model got better, for real: say the base model itself gets upgraded to a stronger version. That's not proof the golden set gap closed. A stronger model can find the same shortcut faster, so rerun the blind panel before trusting the new score.

Where people run it wrong.
They treat a climbing eval score as proof of quality and never check who the golden set actually represents.
They read "no support tickets" as "nothing's wrong," when the real signal, apply rate, satisfaction, edit rate, already went flat and nobody put it on the same dashboard as the score.
They fix a proxy gap by raising the score's pass bar, instead of testing whether the metric measures the right thing at all.

How to use it live. Say the split out loud before trusting the number: "before I say this got better, let me ask who the eval was actually built to represent, and whether that's still who's using it." That buys a beat to think instead of taking a rising score at face value in front of the interviewer.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0