What is the cost of your eval and monitoring infrastructure relative to serving?

Reedline's eval and monitoring budget was never supposed to track serving dollar for dollar. It was supposed to shrink on purpose. The one time it shrank the wrong way, a guest's company name came back wrong in three sponsors' compliance reports before anyone caught the pattern.

How to answer this, stage by stage

Nobody is grading whether you can name a percentage. They're grading whether you can say why that percentage should be different this year than it was last year, and different again next year.

Let's learn

Every week, a host at Brackenfen finishes recording, and by the time they've poured a coffee, Reedline has already handed back a transcript and a full draft of the show notes: guest names, timestamps, and the three quotes worth pulling for a clip.

Before Reedline, a producer wrote show notes by hand after every episode, about fifty minutes each, checking the recording back for exact quotes and spellings. That only happened for the episodes a producer had time for, so about half of Brackenfen's back catalog never got notes at all.

Reedline drafts a full set of show notes in under three minutes, for every episode, published or not. Brackenfen went from a 240-episode pilot across three shows to 14,000 episodes a month across forty networks in about nine months.

The turn: the extra mistakes were never really about Reedline getting worse. Serving cost barely moved; the model ran the same version the whole quarter. The turn is that Briarcask's review policy said sample one in five episodes by hand, a rule written back when every host on the platform had been recording for years. It stayed at one in five as new hosts started joining faster than the model had time to get used to them, and a flat sample kept missing the exact episodes most likely to have a name wrong.

Here's the build-up behind the floor Reedline eventually landed on. By month twenty two, four thousand five hundred and ninety dollars a month was going into eval and monitoring, and almost none of it was a person reviewing an episode nobody needed to worry about.

By month nine, the ratio had already fallen to about one point two times serving cost, and on a spreadsheet that looked like the plan working exactly as intended. It was, mostly. The part the ratio couldn't show was which episodes that shrinking review budget was actually landing on.

Serving capacity is a dial you can turn back any week. A sponsor's trust, once a wrong name reaches their compliance report, isn't. That's the whole reason eval gets ranked first even on the month it isn't the biggest number.

At its worst, a review policy nobody revisits keeps sampling the wrong slice every month, sponsors keep catching what the audit should have caught first, and a network stops trusting the auto-published notes enough to ask for full human review again, more expensive than never automating in the first place.

What I'd leave alone: established-host review at about one percent genuinely doesn't need to grow. Those episodes were running under one percent errors the whole time this was happening, and spending more there wouldn't have caught the pattern any faster.

The lesson: a ratio that's falling exactly the way you planned can still be falling for the wrong reason underneath it. Watch what the shrinking budget is actually still covering, not just how small the number got.

Now here is the same thing as a story

Read the long version below when you want to feel why a ratio falling exactly on schedule still missed something, not just be told that it did.

Esben Brandsma can listen to the first ninety seconds of a podcast episode and tell you whether Reedline is going to have an easy time with it or a hard one. He'd built eval harnesses for a speech recognition team for four years before Briarcask hired him to own Reedline's cost and quality together.

The pilot months were good, genuinely good. Three shows, 240 episodes a month, and Esben's team reviewed every single one by hand while they built the first golden set. By month five, Brackenfen wanted every show on the platform, not just the pilot three, and volume started climbing fast.

It faded in three beats, and none of them looked like a mistake at the time. Beat one, back at pilot launch, the review policy said sample one in five episodes by hand, chosen because it caught almost every error when every host had been recording for years and the model already knew their voices well. Beat two, as volume climbed past a thousand episodes a month, Esben's team built an automated judge to run on all of them, and folded the old one-in-five human sample in underneath it, unchanged, as the safety net for whatever the judge might miss. Beat three, new hosts started joining Brackenfen and the other networks faster than before, syndication deals, guest-hosted spinoffs, and each one arrived with names and terms the model had never heard, spread thin across a sample that still treated every episode the same.

It surfaced on an ordinary Tuesday, in a routine review nobody expected to find anything. Every quarter, Briarcask's compliance lead pulls twenty episodes at random and reads them cold, mostly to keep the automated judge honest. This time she also asked for the same twenty split by how long the host had been recording, since a sponsor had emailed the week before about a guest's company name being spelled wrong in an ad-verification report.

Esben pulled the numbers that afternoon. The blended error rate on the dashboard had read a steady two point two percent all quarter, healthy, nothing to flag. Split by host tenure, established-host episodes were running at zero point six percent. New-host episodes, about one in eight of everything Reedline touched, were running at thirteen point four percent, more than twenty times higher, and the blended average had been hiding it the entire time.

The dollars were smaller than the pattern. Three sponsors across the quarter had already caught a wrong company name in their ad-verification reports and asked for a make-good, about four thousand two hundred dollars in credits Brackenfen had to eat before anyone traced where the errors were actually coming from.

The decision that opened the door went back to the very first review meeting, before Brackenfen had signed a single network beyond the pilot three. Someone asked whether the sample should account for how new a host was. The honest answer at the time was no, every host on the platform had been recording for years, there was no such thing as a new one yet. Nobody wrote that assumption down as an assumption. It just became the rule, and the rule outlived the reason for it.

Run that quarter again with one change: any host in their first two weeks gets full review, human and automated both, everyone else gets the same one percent spot check as before. New-host episodes take about forty minutes to clear instead of three, but there are only about seventeen hundred of them a month out of fourteen thousand. The thirteen point four percent error rate on that slice would have been caught before publish instead of after a sponsor's email, and the quarter's make-good bill drops from about four thousand two hundred dollars to close to nothing.

One design assumed the mix of hosts would stay the same as the day the policy was written. The other design checks who is actually in the sample this month, and spends the review budget there instead of wherever the rule happened to point five years ago.

What I'd tell myself, back in that first review meeting: write down the assumption behind the sample, not just the sample. A rule with no stated assumption can't tell you when it's quietly stopped being true.

ORDER, and the five calls that set the ratio

Not a story wearing a framework's clothes. This is a budget-priority problem, and ORDER is what stops "which line looks bigger this month" from quietly standing in for "which one can't be undone."

Three things worth stating directly, since this is where the real judgment sits. The alternative Esben's team considered first, and dropped, was doubling the flat sample from one in five episodes to one in two, across every show. It lost fast: roughly eighty eight percent of that added review would have landed on established-host episodes that were already running under one percent errors, nearly doubling review cost while barely touching the slice that was actually failing. The AI-specific failure worth naming by name is cold start on an unfamiliar voice: a new host arrives with names, company terms, and a way of talking the model hasn't calibrated to yet, so hallucinated or misheard names cluster on exactly those episodes and disappear inside a blended average that mostly describes hosts the model already knows well. The guardrail is cohort-based monitoring, splitting the error rate by host tenure instead of trusting one platform-wide number, paired with an automatic elevated review tier for any host's first two weeks. That guardrail isn't free. New-host episodes now take about forty minutes to clear instead of three, a real latency cost Briarcask accepted because those episodes are a small share of volume and sponsors' turnaround windows run twenty four to forty eight hours, long enough that the extra wait never actually threatens a deadline.

And if you want to be sure it really works, try it somewhere else

Same five moves, a farming cooperative's leaf photos instead of a podcast feed, and this time the lever was a camera, not a calendar.

CropScan is Grovemont AgriTech's AI tool. A field scout photographs a crop leaf on their phone, and CropScan flags disease risk back in under ten seconds, so nobody has to wait for an agronomist to drive out and look. Rasha Odede runs cost and quality on it, and Marrowvale Growers Cooperative is one of the co-ops running it across its member farms.

The build-up: CropScan processes about 5,000 scans a month early on, at roughly five cents a scan. Eval and monitoring ran about two thousand one hundred dollars against two hundred fifty dollars of serving cost, eight point four times over, while Grovemont built its golden set of confirmed disease photos and had an agronomist review every flagged scan by hand. By month twenty, with 300,000 scans a month across the co-op's full membership, serving cost had grown to fifteen thousand dollars and eval and monitoring had settled near two thousand eight hundred fifty dollars, about zero point one nine times serving, close to the same floor Reedline found.

The fix Grovemont tried first wasn't the same one Briarcask reached for. They considered replacing every scout's phone with one standard, higher-end model, removing the image-quality variable entirely. It lost on time, not logic: fitting two full growing seasons to re-equip every scout across the co-op, while a monitoring change that flagged results by device model shipped in a week and cost almost nothing.

Same rank, different lever: for Reedline the lever was a calendar, how long a host had been recording. For CropScan it's a camera, which phone model actually took the photo. A newly donated low-cost device, sent to volunteer scouts in one district, produced images CropScan read correctly less than half the time it caught the same blight on an established phone. The blended accuracy number never dropped enough to notice, because that district was a small share of total scans, the same way new-host episodes were a small share of Reedline's.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: spend more on eval than serving while small, shrink it only by automating the harness, and never let it hit zero, because the thing you can't undo is what sets the floor.
Cost: there's no budget this quarter for both a bigger golden set and more serving capacity. Fund the golden set. A cheap eval harness on a small pile of traffic beats a fast pipeline nobody has checked against a device or a host it hasn't seen yet.
The model got better, for real: say Reedline's underlying transcription model gets meaningfully more accurate on unfamiliar voices. That's real, and it should let the review tier for a new host's first two weeks shrink over time. It doesn't mean the tier goes away, a genuinely new voice is still a genuinely new voice to whatever model comes next.

Where people run it wrong.
They treat the eval-to-serving ratio as one fixed percentage instead of something that's supposed to fall as the product matures.
They let a sampling rule that was right on day one keep running unexamined long after the mix of what it's sampling has changed.
They cut monitoring the moment its dollar figure looks big next to serving, without asking which of the two is actually the one they can't undo.

How to use it live. Say the real question out loud before naming a percentage: is the population this rule samples still the same population it was written for. That buys a beat to check instead of defending a number nobody's re-derived in months.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0