Explain the difference between automating a task and augmenting a person doing it.

Auristem places the ads you hear inside podcast episodes: it finds the natural break, picks an ad that fits the topic, and checks whether the segment around it is safe for that advertiser's name. Melaine Osgerby owns the brand-safety and placement model. Hendrina Rutherfoord runs partner growth, and needed the long tail of smaller shows publishing ads as fast as a rival network that had just started shipping with no review at all. Melaine had to decide which parts of that job a model could own outright, and which parts still needed a person standing between the model's answer and an advertiser's name.

How to answer this, stage by stage

Nobody in the room is grading whether you can define "automate" and "augment" like a dictionary. They're grading whether you can point at the one call in a real job that still needs a person, and defend it with a number.

Let's learn

Ten weeks ago, nothing published on Auristem's long tail without a number attached to it. A brand-safety score, and if that score dropped under ninety five percent, a person looked before the ad went live.

Four steps for every episode, and only the last one used to have anywhere to send a number nobody trusted yet.

Auristem listens to a podcast episode, picks an ad slot, matches an ad to the topic, and scores whether the segment around it is safe for that advertiser's name. Across the whole network, that's about forty thousand episodes a week.

The long tail, about nineteen hundred smaller shows and fourteen thousand episodes a week, used to work like this: anything the model scored ninety five percent confident or higher published straight away, in under two minutes. Anything under that line went into a queue, and a person looked before it aired, sometimes the same hour, sometimes not until the next morning if the queue backed up.

Then a rival ad network started shipping with no review anywhere, live in under two minutes on every episode, no exceptions. Three of Auristem's long-tail partner shows switched over inside a month, chasing the speed. Hendrina set a goal: match that on the whole long tail, this quarter. The fix that shipped didn't just speed up the queue. It removed the queue. Every long-tail episode now publishes in about ninety seconds, whatever the score says.

Same model either way. The only thing that changed is whether its "not sure" has anywhere left to go.

Here's the turn. The extra mistakes were never really the problem. Most episodes score well above ninety five percent, and the model is right about almost all of them. The real problem is what happens on the ones it isn't sure about, now that "not sure" doesn't send anything anywhere.

Night Ledger runs mostly as a straight true-crime interview show, tagged family safe for its regular episodes. A few times a year it puts out a bonus episode: darker, looser, with explicit language and blunt descriptions the regular show never uses. One of those bonus episodes uploaded on a Tuesday morning. The model scored it eighty nine percent confident clean, which isn't very sure. Under the old rule, that would have landed in someone's queue that morning. Under the new one, it just went out, with a Pinwheel Kids toy ad sitting at the top of it.

Two of these were always safe to automate. Two of them were exactly the ones that needed a person left in the room.

What I would leave alone: Auristem's ad relevance model, which of two already brand-safe ads best fits a listener, was fully automated on the long tail long before any of this, and it should stay that way. A wrong pick there just means a slightly less relevant ad played. It shows up in next week's numbers and gets swapped. Nobody needs to watch it happen.

The lesson: a model's confidence number can be honest on the cases it's seen a thousand times, and quietly miscalibrated on the rare ones it's barely seen at all. That's exactly why a threshold that sends "not sure" answers to a person exists in the first place. Auristem kept computing the number. It just stopped answering it.

Now here is the same thing as a story

The short version above is what you'd actually say out loud. Read this one for what it cost Auristem to learn it the slow way.

Every Monday morning, Melaine Osgerby pulled the same report: how many long-tail episodes had landed under the ninety five percent line the week before, and how many of those a reviewer had actually caught something on. Most weeks it was small, a handful out of thousands. She still read every row, because the handful was the whole point of the number.

She'd owned the brand-safety model for a year and a half, long enough to know which advertisers cared how much, and which shows ran hot on language without ever meaning any harm by it.

The rival showed up in a trade newsletter first, a small item about an ad network shipping podcast spots "instantly, no manual review, ever." Within a month, three of Auristem's smaller partner shows had quietly moved their inventory over, chasing the speed. Hendrina Rutherfoord, who ran partner growth, brought it to Melaine's team with a number of her own: match under two minutes, network wide, by the end of the quarter.

Two weeks between the rival's first mention and the day the queue disappeared for good. What actually broke arrived ten days after that.

The team built it fast, and mostly it built the right thing. Ad slotting got quicker. Matching got quicker. The one place where the plan and the calendar disagreed was the review queue, since a separate fast path and slow path inside the same new low-latency pipeline looked, on a whiteboard, like nearly double the work, for a check that had flagged a real problem on maybe one show in fifty over the past year. So the queue came out. Not on purpose, exactly. It just stopped being anyone's job to rebuild it.

For the first nine days after the switch, nothing happened. Every long-tail episode went live in under two minutes. Ad revenue on the long tail ticked up. Nobody complained.

Then Night Ledger uploaded a bonus episode at six in the morning, the kind it ran three or four times a year: looser, funnier in a way that could turn dark fast, with language and content warnings its regular episodes never carried. The model scored the whole episode eighty nine percent confident clean. Under the old rule, eighty nine would have landed in a queue by breakfast. Under the new one, there was no queue to land in. It published at six oh four, with a Pinwheel Kids ad, bright and cheerful, sitting right at the top.

Neither of them was wrong about what they were holding. The target was real. So was the number. The question was which one still had somewhere to go.

By the time a parent posted a screenshot, three days had passed and about forty one thousand people had downloaded that episode. The screenshot showed the ad, mid roll, right before a blunt, ten-second content warning the show itself had recorded. It found its way into a parenting forum, then a couple of larger accounts, tagging Pinwheel Kids directly. Pinwheel Kids pulled its entire quarterly campaign that same week, not just from Night Ledger, from the whole network, about a hundred and eighty thousand dollars of spend. Auristem spent the next several weeks hand-reclassifying every episode in the long-tail backlog, just to be able to tell its remaining advertisers nothing else had slipped through.

I want to say the problem was that the model got something wrong. It did. But that's not really the story. Nobody at Auristem decided, on purpose, that unsure outputs should ship anyway. They decided that building two paths through one pipeline wasn't worth the sprint. The number kept getting computed. It just stopped landing anywhere.

Months earlier, in the actual planning meeting, an engineer raised the two-path idea and someone above him said, reasonably, that six percent didn't feel worth slowing the whole quarter's roadmap for. Nobody in that room was being careless. They were weighing a real cost against a risk that hadn't happened yet, which is exactly the kind of bet that looks fine right up until it doesn't.

Here's the replay. Same six weeks, same rival, same deadline. This time Melaine keeps the ninety five percent gate inside the fast pipeline, and pairs it with a small change: flagged episodes go to a reviewer's phone as a push alert instead of an email digest, so the wait drops from up to a day to under ten minutes on a normal morning. Ninety four percent of episodes still publish in ninety seconds, automatically. The Night Ledger bonus episode scores eighty nine percent again, gets flagged again, and this time a reviewer sees it in six minutes, catches the mismatch between a kids' toy ad and an explicit warning, and reroutes the ad before a single download happens.

One version of this story trades a queue for a network-wide incident. The other trades it for six minutes on one Tuesday morning nobody outside the review team ever hears about. Same model, same deadline pressure, same rival. The only thing that changed was whether "not sure" still had somewhere to go.

What I'd tell myself, watching that queue quietly disappear in a planning meeting nobody thought was a big decision: the sprint estimate for keeping a safety gate is not the same question as whether you can afford to lose it. Nobody ever asked the second one out loud.

PICK: who's still watching when the model is wrong

Not a rule about trusting AI less. PICK only earns its keep when it names the one decision that actually needs a person, and lets the model run free on everything else.

One path costs a scheduled hundred and forty hours. The other already sent Auristem its real bill, in a single week, from a single advertiser.

Four branches, one question: if nobody's watching and it's wrong, how hard is that to undo.

The trade worth saying out loud: keeping a person in that loop costs real minutes on six percent of the long tail, every single week, forever, not just during a rollout. That's slower, and it's worth paying, because the alternative, a confident-sounding model shipping to nobody's supervision on a case it's barely seen, only shows its real cost after thousands of people have already heard it.

And if you want to be sure it really works, try it somewhere else

Same four letters, a mail-order pharmacy instead of a podcast network, and the missing check is a self-reported pill, not a bonus episode.

Cobbleworth Pharmacy Group runs a refill-approval model across its mail-order arm: check the new refill against the patient's history, flag anything with a possible drug interaction, and clear the rest. After a staffing shortage left refills backing up for days, someone proposed auto-approving any refill the model scored as a "routine, no new interaction" case, no pharmacist look, to clear the backlog fast.

Position: automating removes the pharmacist from approving a refill entirely, once it clears the "routine" bar. Augmenting keeps the pharmacist approving every refill, with the interaction check already run and attached, so they're deciding, not searching. Impact: automate a refill that actually needed a look, say a patient who self-reported a new over-the-counter drug that weakly interacts with what they're already on, a case the model had barely seen labeled either way, and a real interaction ships with nobody watching. Augment a refill that's identical to last month's, same drug, same dose, nothing new reported, and you keep a pharmacist doing work a model could safely own, which is exactly the backlog they were trying to clear. Cost asymmetry: keeping a pharmacist's look on anything with a new self-reported drug costs a few extra minutes per case, cheap, the kind of minutes a pharmacy budgets for anyway. An interaction that ships unreviewed costs a real adverse event, and there's no version of that bill that's small. Kill criteria: does a wrong approval, unreviewed, risk real harm that's hard to undo. If the refill has anything new attached to it, a pharmacist stays in the loop. If it's identical to last month's, the model clears it alone.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: name the one test, unreviewed and hard to undo, before anything else.
Cost: no time to build a full fast path before a real deadline. Fine, but keep the threshold gate on the riskiest slice first, brand safety or new drug interactions, not a faster path for the easy cases that were never the risk.
The model got better, for real: say a new version scores near perfect on average. Still don't drop the gate. A good average can hide one rare case nobody separated out.

Where people run it wrong.
They automate a whole category because most of it is safe, instead of automating the safe part and keeping the risky sliver augmented.
They keep a person on everything out of habit, long after the model has earned the easy calls.
They cut the safety gate to hit a speed target without ever deciding, out loud, that they were cutting it.

How to use it live. If you're ever asked whether something should run on its own, buy yourself a second by asking one plain question out loud: "if this is wrong and nobody looks, how hard is that to undo." That question is the whole method, asked instead of stated.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0