What is the difference between an offline eval and an online eval in the spec?

The four moves for splitting offline from online

This is a yes-or-no about where one specific criterion runs, not a rulebook for every check in the spec, so PICK does the work here.

One more thing before the walkthrough moves on: this pick covers one criterion, not the whole spec. Most candidates hear "offline vs online" and answer like it's a company-wide policy. Say which specific check needs which kind of test, and you've shown real judgment instead of reciting a definition.

Let's learn

Coilwave is a music app. Feed it a track, and a tool called Moodline tags its mood: chill, hype, melancholy, and a dozen more. Those tags decide which playlists a track lands in, like a "Sunday Chill" mix or a "Morning Hype" one.

Before Moodline, five editors tagged every new track by hand. They listened to the first thirty seconds, picked a mood, and moved on, about two minutes a track. With 3,500 new tracks landing on Coilwave every week, they were always behind.

Now Moodline tags a track in under a second. An editor still checks its work, but only against a fixed set of 900 tracks pulled from the older, already-established catalog. That check runs once a year, in about six minutes.

Here is the turn. Moodline's mistakes on that old, fixed set are not the real problem. The real problem is a brand new track, uploaded this morning, that the fixed set has no way to test, because by definition it isn't in the set yet. If Moodline gets a brand new track's mood wrong, nothing catches it before it ships.

At its worst, that looks like this: Moodline tags a brand new single "coffee-shop chill" when it's really built for a workout mix. It lands in slow, quiet playlists. Chill listeners skip it fifteen seconds in, because the tempo is wrong for a coffee shop. Workout listeners never see it at all, because it was never placed anywhere they'd look.

What I would leave alone. The fixed-set check, exactly as it is. It's still the fastest way to catch a bad model update across the whole catalog before a single listener hears it, and nothing about the new-release problem changes that.

The lesson. We built one eval and pointed it at every kind of question Moodline answers. Some of those questions only have a real answer once real people hear the track. We never separated the two.

Three emails, six weeks apart

You don't need this to answer the question. Read it if you want to feel why the split has to happen before the next release, not after.

Callan Bracewell has owned Moodline's eval spec for three years, since before Moodline tagged a single track on its own.

Before Moodline, the five editors caught everything, slow as they were. Nothing shipped without a human ear on it. When Moodline took over the tagging, Callan kept doing what the editors used to do, just faster: every Friday, she pulled twenty random tracks from that week's uploads, played the first minute of each, and checked the tag herself.

They always matched. For a year, Moodline was better at this than the editors had ever had time to be.

So she stopped pulling twenty. She started pulling five.

Then she stopped pulling any at all, and just watched the yearly fixed-set pass rate on her dashboard. Green meant fine.

Then, over one quarter, three separate messages landed, six weeks apart, from three different corners of the business. A label partner asked why a new single's first week looked flat. An artist-relations rep passed along the same question from someone else. A support ticket got forwarded from a listener who couldn't work out why a track "wasn't showing up anywhere."

None of the three, on their own, looked like anything. A single week can be flat for a hundred reasons.

Same tool, two very different ways to miss a tag

It was the third message that made Callan actually pull the tracks. All three were under two weeks old at release. All three had been tagged once by Moodline and never checked again, because the yearly pass only tests the fixed 900-track set, and none of these three were anywhere near two years old.

She pulled up the skip data. All three had a first-hour skip rate above 55%, more than double the 24% a correctly tagged track usually gets. All three had been tagged "chill" or "ambient." All three were built to move, not to sit still.

By the time Callan found the pattern, the earliest of the three was six days past release, out of the ten-day window Coilwave gives every new track in its New Music placements. Four days left, and a track that's already fallen out of favor with the algorithm rarely climbs back in.

I want to say the problem is that Moodline got these three wrong. It did get them wrong. But the real problem is that nothing was ever built to catch a wrong tag on a track this new. The yearly check looked at 900 tracks that already existed when the check was written. It was never going to see a single one of these three.

Months earlier, when the eval spec was first written, the team spent an afternoon arguing over how often the fixed-set check should run. Weekly felt like overkill for a check that rarely turned anything up. They settled on once a year, and moved on to bigger questions.

Callan split the eval the week after the third message. The fixed set stays exactly where it was, still catching the kind of regression it was built for. But now, every track's skip data gets watched for its first ten days, and if the skip rate crosses 50% inside the first day of real listening, someone looks at the tag by hand before the track's window runs out.

One design waits a year to ask if a tag is wrong. The other one watches every new track for the ten days that actually matter to it.

And the thing I'd tell myself, if I could go back to the afternoon we picked "once a year": we asked how often the model changes. We never asked how often a brand new track shows up with nothing in its history to check it against. On an old catalog, that's never. On a new release, it's every single time.

PICK, walked through on one mood tag

This is a tradeoff about where one specific check should run, not a rule for every line in the eval spec, so PICK is the tool.

Run PICK again, on a city's pothole tagger

A county public-works department runs Curbwatch, a tool that reads a citizen's photo of road damage and tags how urgent it is, so repair crews get sent to the worst ones first. It has its own fixed set of 600 hand-labeled photos, checked once a quarter.

P. Run the urgency tag for a newly reported pavement failure type online, against how crews actually rate it after a real dispatch. Keep the quarterly fixed-set check for the road-damage patterns it already knows.
I. A fixed-set miss on a familiar pothole type gets caught before the next model ships, an engineer sees a red check, nobody outside the team notices. A new failure type the fixed set has never seen, like a sinkhole starting under a crosswalk, can get tagged low priority and sit for weeks, until a driver hits it or a crew stumbles on it by chance.
C. The familiar-type miss is loud and cheap, caught in a routine check before it ever reaches a crew's queue. The new-type miss is quiet until someone gets hurt or a car gets damaged, and by then the county is answering for weeks nobody was watching.
K. Once enough of a new failure type has real crew-confirmed labels behind it, fold it into the fixed set. That's the same line Coilwave's team would use once enough history piles up.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0