Explain when a manual Wizard of Oz test beats a technical spike.

Mossgrove Audio is a small independent audiobook publisher. Saoirse Mullen runs editorial there, and had to decide whether to build a technical spike for an AI chapter-title generator, or fake the feature by hand first and see if anyone could even tell the difference.

How to answer this, stage by stage

Nobody is scoring whether you've heard the phrase "Wizard of Oz." They're scoring whether you can tell, on the spot, which kind of doubt actually needs a model built to resolve it, and which kind doesn't.

Let's learn

Say a listener finishes a chapter of an audiobook and sees a title pop up for the next one: something short, a little intriguing, never giving away what happens.

Before any of this, an editor at Mossgrove Audio picked every chapter title by hand, reading the full manuscript first, taking maybe fifteen minutes per book across forty chapters. Listeners rarely commented on titles at all, which the editorial team read, correctly, as a sign the titles were doing their job invisibly.

What the human faking the feature actually does, and the one part, writing down why, that most teams skip.

Here's the turn: the pitch for an AI title generator wasn't wrong. The doubt wasn't about whether a model could technically produce a plausible-sounding chapter title fast. It could, easily. The real doubt was whether it could reproduce the specific, hard-to-name judgment an editor makes about how much to reveal, and building a full technical pipeline to test that judgment call is expensive and slow compared to just faking it by hand first.

At its worst, engineering spends five weeks building a full pipeline, complete with retrieval and formatting logic, only to learn in week six that the underlying editorial judgment was never actually captured, because nobody faked it by hand first to find out what that judgment even was.

What I would leave alone: I wouldn't run a Wizard of Oz test for a feature where the real question is raw speed or scale, like whether a model can generate captions live during a broadcast. A human faking real-time behavior at scale doesn't tell you anything true about whether a model can actually do it.

The lesson: the cost of the wrong test was never really about hours. It was about which kind of wrong you'd rather find out about: a bad afternoon, or a bad month.

Now here is the same thing as a story

The short version above is what you'd say defending the plan in a five-minute editorial meeting. Read this one for what it felt like the week someone else's failure landed on Saoirse's desk two days before her own kickoff.

Here is what happens when a tool works fine in a demo and nobody has yet asked the question that actually matters: does it make the same call a good editor would make, for a reason nobody wrote down.

Saoirse's laptop has a cracked hinge held together with tape, three years of manuscript notes still open in half its tabs. She had scoped the AI title generator project for two weeks, ready to greenlight a technical spike, when a colleague from a sibling imprint mentioned, almost in passing, that their own title-generator pilot had just been quietly shelved.

The cost that decided everything: one wrong bet is loud and cheap. The other is quiet and expensive.

The sibling imprint had built the full pipeline first: real-time generation, a formatting layer, a review dashboard. It scored well against an internal rubric the engineering team had written themselves. Then it shipped to real listeners, and the titles it produced kept doing one specific thing: giving away the emotional turn of the chapter before the listener got there.

Nobody on the engineering team had done anything wrong, technically. The model produced fluent, plausible titles quickly and reliably. What nobody had ever tested was the actual editorial judgment: how much to reveal, and how much to hold back, a call a human editor makes by feel and had genuinely never written down.

The one question that would have caught the sibling imprint's mistake before it shipped.

Saoirse changed her own plan that afternoon. Instead of scoping a technical spike, she asked one editor to fake the feature by hand: read each of the next twenty books' final drafts, pick a chapter title the old way, and this time write down, in one sentence, why that title and not a more revealing one.

Five steps, and the fourth one, logging why, is the part a straight-to-engineering plan always skips.

Across twenty books, a real pattern emerged, one the editor could finally name out loud: titles that named a place or an object worked better than titles that named a feeling. "The Locked Drawer" over "A Moment of Doubt," every time, because the object teased without revealing the emotional turn.

Three things that make the fake version actually useful, instead of just a slower way to do the same job by hand forever.

Only once that pattern held for twenty real books did engineering start a technical spike, this time scoped narrowly around one instruction: favor concrete nouns over named emotions. The spike took nine days instead of five weeks, because it already knew exactly what to test for.

What I'd tell myself, hearing how close Mossgrove came to repeating the sibling imprint's exact mistake: the technical question was never the hard one. The hard one was a judgment call an editor had been making silently for years, and no model was ever going to guess it without someone faking it by hand first.

PICK, in one screenNot a script for faking every feature by hand forever. PICK is what tells you exactly which kind of doubt a human can resolve cheaper than a model can.

The recap, one line per letter: position is fake it by hand first for judgment-quality doubts, impact is a wasted afternoon against weeks of misdirected engineering, cost asymmetry favors the visible, cheap mistake over the hidden, expensive one, and kill criteria is a proven pattern across twenty real cases, which is exactly what let the eventual technical spike run in nine days instead of five weeks.

And if you want to be sure it really works, try it somewhere elseSame four letters, a real-time interpreter-matching tool instead of a chapter title. Different flip family entirely, the same discipline about which doubt to test by hand.

Oskar Lindqvist coordinates bookings at Cindermere Language Partners, weighing whether to spike an AI tool that matches incoming interpretation requests to the best available interpreter. Mapped onto PICK: position is fake the matching decision by hand first, since the doubt is about judgment, matching a client's specific dialect and context to the right interpreter, not raw model speed. Impact is a coordinator wasting an afternoon manually testing match quality, against engineers building a live matching pipeline around criteria nobody confirmed actually mattered to clients. The underlying flip here differs from Mossgrove's substitution flip: it's an abandonment flip, coordinators who get one bad automated match quietly stop trusting the tool for anything but the easiest, most obvious bookings, and nobody notices the tool going unused for the hard cases. Cost asymmetry still favors the cheap, visible test. Kill criteria is the same shape: a human coordinator's manual matching logic proven consistent across twenty real requests before any pipeline gets built.

The same question, asked of a completely different booking desk: can a human fake this specific judgment call before anyone builds a pipeline around it.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "fake the judgment call by hand first, build the spike only once the pattern holds," and stop.
Cost: there's no editor available to run even a short Wizard of Oz test. Say so honestly, and scope the smallest real version, one editor, five books, rather than skipping straight to engineering.
The model demos surprisingly well, for real: even a strong early demo is worth a Wizard of Oz check first, since a model can sound fluent while still missing the specific, unwritten judgment call that actually matters to real listeners.

Where people run it wrong.
They treat "can the model technically produce this" as the same question as "is this the right thing to automate."
They skip logging the human's reasoning, so the Wizard of Oz test produces a good decision but nothing engineering can actually build from.
They let a good Wizard of Oz result skip the technical spike entirely, instead of confirming the pattern holds at real scale.

How to use it live. The moment an interviewer asks this, buy two seconds by asking yourself: is the actual doubt here about speed and scale, or about a judgment call a person makes without ever writing down why? The second one is always the Wizard of Oz answer.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0