Explain when a Wizard of Oz prototype beats a real model.

How to answer this, stage by stage

This is a yes-or-no about which tool to build first, not a rulebook for every prototype, so PICK does the work here.

One more thing before the walkthrough moves on: this pick is about order and scope, not a ban on real models. Most candidates hear "Wizard of Oz vs. a real model" and answer like it's forever. Say which specific question needs a human first, and you've shown judgment instead of reciting a definition.

Let's learn

Hearthline is a chat window on Corrandale Home Insurance's website. Instead of filling in a form, a customer types what happened, in their own words, and the window asks a few follow-up questions until a claim gets started.

Before Hearthline, starting a claim meant a fourteen-field form. Customers took about nine minutes to finish it, and 22 percent gave up partway through.

Behind Hearthline, at the very start, there was no model at all. A real claims specialist read what a customer typed and wrote every reply by hand, pretending to be the bot. Customers finished describing their claim in about three minutes, and the number who gave up fell to 6 percent.

Here is the turn. Those fast, low-drop numbers are not really the finding. They only prove that people will talk to a chat window about a claim. They say nothing about whether a real model could hold that same conversation safely, because there still wasn't one.

At its worst, a model gets built straight off that early success, skips the trust question, and answers a real customer's coverage question with total confidence, and gets it wrong. Nothing was ever built to catch that until a claim is denied and the customer asks why the chat told them otherwise.

What I would leave alone. The claims-math system, exactly as it is. Working out what a payout should actually be needs the real system. No specialist typing behind a curtain can safely fake that number. Wizard of Oz answers the trust question. It was never built to answer the arithmetic one.

The lesson. A prototype's job depends on the question. A Wizard of Oz shell tests whether people want the conversation. A real model tests whether it can hold one safely. Build the second before answering the first, and real engineering weeks get spent on a question nobody had actually asked yet.

What the customer sees, and what is actually there

The same five answers, said again and again

You don't need this to answer the question. Read it if you want to feel why the split has to happen on real evidence, not on a date picked before the pilot even started.

Thora Wexley has run product for Corrandale's digital claims team for four years, since before Hearthline was a chat window or anything else.

When the Hearthline pilot went live, three claims specialists sat behind it, invisible to the customer, writing a fresh reply to whatever someone typed. Thora read every single transcript herself each night for the first three weeks, about forty minutes a night, checking that the replies actually matched what a trained adjuster would say.

They always matched. The specialists were good at their job, and for a while Hearthline read like the best version of the old phone line, just typed.

So she stopped reading every transcript. She sampled ten a night. Then five. By week six she was skimming a daily summary a specialist wrote for her.

Then, in week eight, a new specialist joined the desk and asked the group chat a question. Why did everyone keep pasting almost the same four sentences for a water-damage claim? Wasn't the whole point to write something fresh, the way a real adjuster would?

Nobody had a good answer. So Thora pulled the real numbers instead of a feeling.

By week six, 78 percent of every conversation had already been answered using one of five saved snippets the specialists had quietly built for themselves: one for water damage, one for a cracked window, one for a late payment question, one for a burst pipe, one for storm roof damage. Nobody had told them to do this. It just turned out that most claims, most days, needed almost the same thing said back.

Same product, two very different ways to get it wrong

I want to say the problem here is that the specialists got lazy. They didn't. The real problem is that nobody had set up a way to notice this on purpose. The pilot was scoped to run twelve weeks, then Thora's team would decide, model or no model, on the calendar date they'd picked before the pilot ever started.

Months earlier, in the meeting where they scoped the pilot, someone asked how they'd know it was time to build the real thing. The answer that won the room was "let's just run it for a quarter and see." Twelve weeks felt long enough to be safe and short enough to keep momentum. Nobody wrote down a number that would tell them sooner.

Once Thora had the real percentages in front of her, the pattern was obvious: 22, 38, 50, 60, 68, 78 percent, week after week, then holding above 75 for two weeks running by week seven. With a kill line marked at 75 percent held for two weeks, Thora's team would have had their answer by week seven instead of week twelve, five weeks earlier, and they'd have known exactly which five answers were safe to hand to a model first, with everything else still going to a person.

One design waits for a date on the calendar. The other watches for the moment the conversation actually stops surprising anyone.

And the thing I'd tell myself, back in that scoping meeting: we asked how long we'd run the pilot. We never asked what evidence would tell us we could stop early. On a fixed date, that's never. On real transcripts, it's usually sitting right there by week six or seven, if anyone's counting.

PICK, applied to the Hearthline pilot

This is a yes-or-no about which tool to build first, not a rule for every prototype Corrandale ever makes, so PICK is the tool.

Run PICK again, on a veterinary chain's symptom chat

Larkhollow Veterinary Group, a chain of eleven clinics, wants pet owners to describe a symptom in a chat window and get told whether it's urgent or can wait for a normal appointment. Before building a triage model, they put real vet technicians behind a chat called Vetline, typing every answer themselves.

P. Run the trust question, whether owners will describe a symptom in chat and act on what Vetline tells them, through a Wizard of Oz shell first. Keep a real triage model out until the technicians' own answers settle.
I. A technician's slow reply is felt by the technician, caught the same shift, fixed with a note. A model's confident wrong answer, telling an owner a limping dog can wait until Monday when it actually needs same-day care, is felt by the dog and the owner, and nobody at Larkhollow knows until the pet arrives worse than it should have.
C. The Wizard of Oz miss is loud and cheap: a supervisor hears about it before the shift ends. The real-model miss is quiet and expensive: an owner trusts a confident wrong answer, and the cost shows up as a harder case in the waiting room, not as a support ticket anyone reads in time.
K. Once the technicians' answers settle into a small, stable set across enough seasons and species, fold those into a scoped model, with a hard rule that anything outside the set still goes to a person. Same kind of line Corrandale's team would use.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0