Describe how you would run a planning session when effort estimates are genuinely unknowable.

Tapecourt transcribes a podcast episode and pulls out speaker-labeled quotes for the show notes. Nerissa Eichhorn is the product manager for its diarization pipeline, and Vidar Isaksson is the research engineer who owns it. Cobblecast, an independent podcast network, runs Tapecourt across its whole slate, and Gudrun Moreland, Cobblecast's head of production, wants a date for fixing the wrong-guest quotes coming out of The Roundhouse, its five-guest flagship show, after one nearly went out under the wrong name.

How to answer this, stage by stage

Nobody is grading whether you can say "diarization error rate" correctly. They're grading whether you can turn "we genuinely don't know how long this takes" into a real plan, not a guessed date and not a shrug.

Let's learn

Tapecourt listens to a podcast episode, works out who's talking, and turns that into a written summary with quotes pulled out and labeled by speaker.

Five boxes, one pipeline. The second one, working out who's talking, is where a checkpoint actually needs to look.

Before Tapecourt, a Cobblecast producer spent about 90 minutes after each two-guest interview episode, listening back and typing up who said which quote for the show notes. With Tapecourt running, that dropped to about ten minutes of skimming an already-labeled draft. On those shows, the model's diarization error rate, how often it mixes up who's speaking, sits around 5 percent. Close enough to trustworthy that a producer barely has to check it.

Then Cobblecast pushed Tapecourt onto The Roundhouse, its flagship weekly panel show, five guests and a host, all talking over each other by the second half of most episodes. The model still runs. It still drafts quotes. But its diarization error rate on Roundhouse audio is 24 percent, not 5. Close to one word in four gets attached to the wrong speaker somewhere in a segment.

Same model, two shows. The gap between them is exactly what a single average diarization score would hide.

Here's the turn. Those extra mistakes were never really the problem. The problem is what a Roundhouse producer does next: she stops trusting the auto-labeled quotes and goes back to checking almost every one by hand, on the one show where the audience, and the sponsors, are paying the closest attention.

What it costs at its worst: three weeks before Gudrun called Nerissa asking for a fix date, a wrongly labeled quote nearly went out under a guest's name he never said it under, on a topic that would have embarrassed him publicly. A Cobblecast editor caught it in a final read minutes before the show notes were due to publish. Nobody outside that room ever saw it. Nerissa still calls it the closest her team has come to a real correction.

What I would leave alone: Tapecourt's two-guest interview shows, still running at 5 percent diarization error, still trusted, still saving a producer most of ninety minutes a week. Nothing about the Roundhouse fix should slow those shows down or change how they ship.

The lesson: an estimate you pull out of someone under status-meeting pressure was never a real estimate. It's a placation, and everyone downstream ends up planning against a number that was never actually meant as a promise.

Now here is the same thing as a story

The short version above is what you'd actually say in the room. Read this one for what almost went out under the wrong guest's name.

Nerissa Eichhorn has run product for Tapecourt's diarization pipeline for two years, ever since Cobblecast became its first real network client. For most of that time, the routine planning question was the same one every team asks: how long will this take. And for most of that time, it was a fine question, because the work behind it had a knowable shape. A new show format to onboard. A latency fix. A change to the show-notes editor. Vidar could look at work like that and give a date that held.

Diarization on The Roundhouse was never that kind of work.

Five guests, a host, and a habit of talking over each other by the second half of most episodes. Tapecourt's model, tuned mostly on two-guest interviews, hit a 24 percent diarization error rate there against 5 percent everywhere else. Nerissa asked Vidar the usual question anyway: how long to fix it. He said six weeks, the way you say a number when a room is waiting for one and admitting "I don't actually know" feels like failing at your job.

Six weeks came and went. The model improved, some. It was nowhere near good enough to publish without a person checking almost every quote, which meant the fix had, functionally, not shipped.

One date is a guess with a deadline attached. One checkpoint is a guess with a price tag and a return date.

Nobody blamed Vidar for missing the date. He'd done the work the date implied. The problem was the date itself. Nobody had ever asked what it was actually built from, and the honest answer was: nothing. It was a guess, offered under the exact kind of pressure that makes guesses sound like plans.

Three weeks after that six-week date quietly slipped, a segment producer flagged something during final review of a Roundhouse episode. A sharp, borderline-defamatory line about a rival podcast host had been auto-labeled as coming from the wrong guest. The actual guest who said it never got flagged at all. The show notes were forty minutes from publishing. A human editor caught it because she happened to remember the moment from listening live. If she hadn't, it would have gone out under Gudrun Moreland's name on the network's biggest show, attributed to a guest who never said it.

Gudrun called that afternoon, not angry, but rattled in a way that mattered more. She didn't ask for six more weeks. She asked when this would actually be fixed, and Nerissa realized she was about to give the same kind of answer that had already failed once.

She didn't. Instead she asked Vidar for a different meeting: not "how long," but "how many experiments can we run before we know more than we do today, and what does each one cost." Vidar had never been asked that question by a PM before. It took him one afternoon to price it. One tuning cycle: about three engineer days, around 260 dollars of compute. History with similar tuning problems put the likely range at four to eight cycles before diarization error would clear a 9 percent bar, the point past which a Roundhouse quote was trustworthy enough to auto-publish again.

Nerissa took that to Gudrun instead of a date. A checkpoint in three weeks, four iterations in, showing exactly where the error rate stood. A second checkpoint at six weeks if the first one wasn't enough. A real number, either way, not a promise made just to end a meeting.

Run it forward. Checkpoint one lands at week three: four iterations, diarization error down to 13 percent. Better, not there yet, but the trend line said something worth funding another round on. Checkpoint two lands at week six: eight iterations total, 28,480 dollars spent, diarization error at 8.6 percent. Under the bar. The confidence floor that had been catching shaky quotes since the near miss stays on, for good, for exactly the audio that's still hardest.

What I'd tell myself, back in that first six-week meeting: a date nobody can defend is not a plan. It's a hope wearing a plan's clothes, and it costs exactly as much as a real miss does, just later, and with less warning.

BOUND, for the planning session where nobody can honestly promise a date

Not a way to sound careful about uncertainty. BOUND is what turns "we genuinely don't know how long this takes" into a number Gudrun can actually plan a sponsor launch around, instead of a guess dressed up as a date.

Four weeks of slack sit between the worst-case checkpoint and Cobblecast's actual deadline. Nobody had to invent that room, it was already there.

Four things, not one date. This is what Gudrun actually left the room holding.

Three honest endings to a checkpoint, not one hopeful one. Only the middle branch buys another three weeks.

One alternative Nerissa considered and rejected: putting a second research engineer on diarization tuning in parallel, to compress the calendar. It lost, because the bottleneck was never idle hands, it was Vidar's evolving read on exactly where the model was failing on Roundhouse audio specifically, and a second engineer would have needed to relearn that same read from scratch before contributing an independent iteration, doubling the cost per cycle without shortening the number of cycles needed. The AI-specific failure sitting underneath all of this: a wrongly labeled quote reads exactly as confident as a correctly labeled one, nothing in the summary itself signals doubt. The guardrail is the per-segment confidence floor, any quote the model itself isn't sure about gets routed to a person before it publishes, live from the day of the near miss, not held back until the checkpoints finish. And the trade-off is real and accepted on purpose: the team is spending real calendar time and up to 28,480 dollars instead of shipping fast and cheap on a guessed date, in exchange for actually knowing, at each checkpoint, whether the fix is real or just later.

And if you want to be sure it really works, try it somewhere else

Same five letters, a shipping terminal instead of a podcast network, and this time the old habit being replaced isn't the question, it's an assumption baked into how every experiment gets priced.

Hullwatch scans photos of shipping containers at the gate and flags damage before a container gets loaded, run by Eberhard Fallowmead, a computer vision researcher. Ironbridge Terminal runs it across every inbound container, and Xantippe Quaid, the terminal's operations lead, wants to know when Hullwatch will reliably catch hairline stress fractures on refrigerated containers, a damage type the model currently catches only about 55 percent of the time, the same way it already catches surface dents and scrapes about 97 percent of the time.

Run BOUND on it. Break it down: the exact number of experiments needed to get the crack classifier above a usable catch rate is unknowable the same way Tapecourt's was, but the cost of one experiment and the bar for "good enough" aren't. Own the numbers: one vision iteration on Hullwatch's setup costs about 2,900 dollars, cheaper than Tapecourt's because the compute-heavy part is shorter and needs less of Eberhard's own time per cycle, sourced the same way, off his last several runs. Use a range: three to six iterations before reassessing, a shorter range than Tapecourt's because the labeled crack data Ironbridge already had on hand was richer than what Cobblecast could offer at the same stage.

Different terminal, same shape of session. The number that changed wasn't the range, it was what one iteration used to assume about itself.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: separate what's knowable, the cost and the bar, from what isn't, the iteration count, and turn the unknowable part into a range with a checkpoint, never a single date.
Cost: no budget for a second checkpoint this quarter. Shrink the range instead of hiding it, tell the room honestly that four iterations is all the budget allows, and what "good enough" has to mean if the model isn't there by then.
The model got better, for real: say a newer base model needs half as many tuning cycles to hit the same bar. The discipline barely changes, because it was never really about the iteration count, it was about pricing one iteration honestly before promising anything built on top of it.

Where people run it wrong.
They let "how long will this take" stand as the question, instead of splitting it into the parts that are actually knowable.
They give a range with no priced ends, which is just a wider guess, not an estimate.
They treat "we don't know yet" as the end of the planning session instead of the reason to schedule the next checkpoint.

How to use it live. Before answering, ask yourself one plain question out loud: "what's the one number in this problem I could actually defend right now." Whatever answer comes first is usually where your knowable pile starts.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0