Describe the difference between a demo prototype and a learning prototype.

How to answer this, stage by stage

Six moves. Most of the weight sits in stage five, the four sentences where a real number finally shows up. Every stage has the actual words to say.

Let's learn

Here's what happens when a prototype gets great at one thing, being watched, and never gets tested at the other thing it's actually for.

Ledgerfox is a fintech startup. It's building a feature called the categorizer: a tool that reads one line on an expense report, a receipt, a vendor name, an amount, and guesses which spending category it belongs to. Travel. Meals. Software. Client entertainment. Guess right, and an employee never has to touch the dropdown.

For one quarter, the categorizer went through three demo cycles. Every single one ran on the same fifteen receipts, picked because they always worked. Every single one got a great reaction. Nobody in the room ever saw it get anything wrong.

Then a new engineer, three weeks into the job, asked Simone a plain question after the third demo. "So what did we actually learn building this?" Simone opened her mouth to answer and found she had exactly one sentence. People like it.

Here is the turn. The fifteen perfect receipts were never the real problem. The real problem was what Simone's team had quietly stopped doing every single time they got ready for a room full of people: trying to find the receipt that would break it.

At its worst, this costs exactly what the categorizer was built to save. Ship it without ever finding the hard cases, and the first time it guesses wrong on a real client dinner, someone in Finance has to notice, unwind it, and recode it by hand anyway, which is slower than if the categorizer had never existed at all.

What I would leave alone. The very first demo Simone ever ran, the one that got Ledgerfox's leadership to approve three more months of budget for the categorizer, that one is fine being built purely to excite. Nobody was going to ship code off the back of it. They were deciding whether to keep paying for it.

The lesson. A prototype that only ever meets the cases you already know it can handle isn't rough, it's decorated. Rough is fine. Untested is not the same thing as rough, and it's easy to mix the two up when the room keeps clapping.

Now here is the same thing as a story

Pull this one out when there's more time and you want the room to feel it, not just note it down.

Simone Aldridge spent four years coding expense reports by hand before Ledgerfox hired her onto the categorizer team. She can tell a personal Uber ride from a client dinner ride most days, just from the time stamp and the fare. She built the first version of the categorizer's demo herself, on a Sunday, from fifteen receipts pulled out of her own old expense history, because she already knew exactly how each one should be coded.

The first demo, in January, was small. Four people in a conference room, fifteen receipts, fifteen correct guesses. Somebody said it felt like magic. Simone went home early that day for the first time in months.

By the second demo, in February, the room was bigger, two board observers this time, and Simone added one flashy touch: a receipt in Japanese yen, converted and categorized correctly on screen. She'd tested that one receipt six separate times that week to be sure it would land. It landed. A board observer asked when they could pilot it with real employees.

By the third demo, in March, they let the room submit a live receipt on the spot. It felt spontaneous. It wasn't, quite. An engineer had quietly scanned every submitted photo an hour early and swapped out two that looked risky for backups pulled from a folder of twenty receipts they already knew worked.

Nobody set out to build a folder of receipts that couldn't fail. It just kept being the fastest way to have a good Thursday.

After the third demo, the new engineer, three weeks into the job, caught Simone in the hallway. "That was great," she said. "What did we actually learn today?"

Simone started to answer and stopped. She had reactions. Nods, a couple of laughs at the yen receipt, one board member asking about pricing. She did not have one single fact about the categorizer that she hadn't already known back in January.

"People like it," she said, and heard how small it sounded the second it left her mouth.

Here's the decision I'd take back. Back in December, when the team was setting up how demos would work, somebody suggested running a dry run the night before each one, just to make sure nothing embarrassing happened live. That made sense. Nobody wants a bad moment in front of the board. But the dry run and the actual test of the categorizer became the same meeting. Whatever survived the dry run was, by definition, whatever got shown. There was never a separate pass whose only job was trying to make the thing fail.

A prototype is a switch, not a dial

I'd take that back, and split it in two. Keep the dry run, for the room's sake. Add a second pass, a week earlier, whose only goal is finding the ugliest real case in the pile.

Run the fourth demo cycle again with that fixed. This time Simone doesn't start from her own fifteen easy receipts. She starts from one real question: can the categorizer tell a personal Uber ride from a client dinner Uber ride, when the receipt alone doesn't say which one it is? She pulls forty real receipts, already sitting in Ledgerfox's own expense system, that a human had to sit and puzzle over before coding.

Cold, before any fix, the categorizer gets twenty-six of the forty right. Not fifteen of fifteen. Twenty-six of forty.

That's the ugly number nobody wanted in a demo. It's also the first true thing the team has learned about the categorizer since January.

The fix turns out to be small: when the model isn't sure, don't guess silently, ask the employee one tap, personal or client? Run the same forty receipts again. Thirty-seven right, and the three still wrong are flagged, not hidden.

The fourth demo ends at nine at night with a number on the screen instead of applause. Thirty-seven out of forty, three known misses, a plan for each one. Nobody claps. Simone doesn't mind. For the first time since January, she knows something about the categorizer that she didn't already know before she built it.

If I'm honest, picking fifteen easy receipts for that first demo wasn't the mistake. Anybody would have, in January, with four people in a room and nothing built yet. The mistake was never asking, three demos later, whether easy was still the point.

The five letters, applied to a demo that never got tested

The letters matter less than which one breaks first. Here's the same five steps, mapped onto Simone's hallway conversation.

FLIPS, five rows

The folder barely changed. What she did about it snapped in one move.

"She got less curious about the hard cases" is a diagnosis anyone can offer after the fact. The harder part is naming the exact thing that had to stop first, asking a question nobody had answered yet, and showing there was no smaller version of it left once three demos in a row had already gone well.

And if you want to be sure it really works, try it somewhere else

Suncrest is a residential solar startup, nowhere near fintech. Its prototype is a roof screener: a tool that looks at one photo of a roof and predicts whether it's a good candidate for panels. Same question, a different flip this time. Nobody stops checking because the news is good. The eval photos themselves quietly get groomed.

F. Anouk Delvaux, ops lead at Suncrest, six years estimating roofing jobs before this, who can guess a roof's pitch from the sidewalk.
L. Every quarterly review reused the exact same folder of thirty roof photos, all from Suncrest's own flat, sunny, tree-free service area, because collecting new ones ate a whole afternoon nobody had.
I. A different flip from Simone's. Nobody rides good news into skipping a check. The eval set itself quietly narrows. Tests the roof screener on the same friendly thirty photos every time, or tests it on the specific roofs the team is quietest about, the shaded ones, the steep ones, the ones with an awkward chimney, with nothing in between once a sales rep needs a real answer standing in front of a customer.
P. When the eval folder was first built, an engineer picked thirty clean, representative photos to keep quarterly reviews quick. The folder just never got refreshed after that, because building a new one wasn't anyone's job.
S. Pull 25 real, hard roof photos that field reps had flagged over the past year. Cold, the screener is only right on 9 of those 25. Add a required manual check for anything shaded or steep instead of a silent guess. Re-run: 22 of 25 called correctly. The next investor review runs on the hard 25, not the easy 30, and ends with a real, slightly imperfect number instead of a clean sweep.

Flashcards (click a card to flip it)

Eight fixed slots, pulled straight from the answer above.

Check yourself Score: 0 / 0