What does a prototype tell you about cost, and what does it not?

How to answer this, stage by stage

Six moves. Name the boundary, what a prototype's cost number can and can't tell you, before the story, or the answer sounds like a warning instead of a decision.

Let's learn

Sablewood Fulfillment built a tool that reads a shipping label before it goes on the truck, and flags the ones likely to bounce, a wrong zip code, a weight that doesn't match the box, a name pasted onto the wrong address.

Before Kwabena Sarfo had any real number, the plan for the label checker was written on faith. The team guessed it would run "a few cents a label, tops," and moved on, because nobody had run it against anything real yet. Then Kwabena built a small prototype himself. He ran it on fifty labels pulled straight off that morning's printer, all clean, one line, all domestic. It priced out at about four tenths of a cent a label. Multiply by the twenty thousand labels the warehouse ships a day, and the number on his slide read twenty four hundred dollars a month. Finance signed it in the same meeting.

Here's the turn. The mistake was never the model. The mistake was the fifty labels.

Nobody re-ran the number once the pilot moved off Kwabena's laptop and onto a shared queue fed by whatever the warehouse actually printed that hour. The real labels looked nothing like the demo set, torn scans, international addresses, multi-line delivery notes, and about one in eight got flagged as unsure and sent back through the model a second time. Tracked honestly, week by week, the true cost climbed the entire time nobody was watching it.

At its worst, that gap doesn't just cost money. It costs the next number anyone brings finance. A budget approved off a demo that turns out to be five times short doesn't just blow a line item, it makes every future estimate from that team something finance now double-checks by hand, which is exactly the manual work the tool was supposed to remove.

Fifty clean labels priced the tool. Twenty thousand real ones paid for it.

What I'd leave alone. A prototype spun up in an afternoon, just to see whether the model can read a torn label at all, doesn't need this. Nobody is signing a budget off a spike like that. Save the real-mix cost tracking for the moment a number is about to go in front of someone who controls money.

The lesson. A cost number a prototype gives you is never really wrong. It's honest about the fifty calls you actually ran. The mistake is treating those fifty calls as if they were the twenty thousand you're about to run instead.

Now here is the same thing as a story

Use this version when you've got a few minutes. The short version is above. This is for when the five-times number needs to actually land.

Kwabena Sarfo can smell a bad estimate before the second slide even loads. Four years running product for Sablewood Fulfillment's label team taught him that. Every quarter someone brings a number that's rounded in a suspicious direction, and he's usually the one who catches it.

So when he built the first version of the label checker himself, on a laptop with a cracked hinge that never left the ops office cart, he did the responsible thing. He ran it on fifty labels pulled straight off the printer that morning, timed it, priced it out. Four tenths of a cent a label. Multiply by twenty thousand labels a day, and the number was clean: about twenty four hundred dollars a month. He put it on a slide, finance signed it in the same meeting, and for six weeks the project moved exactly the way a project is supposed to move.

The fading happened in three small steps, none of them looking wrong at the time. First, the pilot moved from Kwabena's laptop to a shared test queue, and the labels feeding it came from whatever the warehouse happened to print that hour, not from a set anyone chose on purpose. Second, the finance number stayed the one everyone quoted in meetings, because nobody had a reason to go back and recheck it once it was signed. Third, the ops team started routing the checker's flags straight to the reprint station, which meant it was quietly running on real volume weeks before anyone called it "live."

Then the first month's invoice landed on Kwabena's desk. Twelve thousand six hundred dollars. He read it twice, assuming a decimal was in the wrong place.

It wasn't. He pulled the logs and found the demo's fifty labels sitting right where he'd left them, clean, single line, all domestic, no torn scans, no rush stamps, none of the international addresses three of Sablewood's biggest clients had started shipping to that same quarter. The real labels the tool had actually been running on all along looked nothing like them.

The dollar gap wasn't even the real cost. The real cost was that finance now had to ask, on every number Kwabena brought them after that, whether it had been tested on the fifty easy labels or the real twenty thousand.

Two months before, in the meeting where they scoped the pilot, someone had asked whether fifty labels was really enough to estimate from. Kwabena remembers saying it was fine for a first pass, they'd widen the sample once it needed to be exact. Nobody ever circled back to say the number had already left the building and quietly become the number.

I would take that back. I'd track the real cost against the real input mix from the day the first prototype call went out, not after an invoice forced the question.

Here's the replay. Same fifty-label demo on day one, but this time Kwabena logs cost against a weekly-widening real sample from week one, the way he eventually did after the invoice. By week three the honest number already reads about one cent, not four tenths. By week six it's flat at the true two cents, three weeks before a single label goes anywhere near the reprint station. Finance signs a budget of eleven thousand, not twenty four hundred. The invoice, when it lands, is a formality.

One design hands Kwabena a number from a lab. The other hands him a number from the floor.

And the thing I'd tell myself, back in that scoping meeting: fifty easy labels were never a small sample of the real twenty thousand. They were a different population wearing the same units.

LEAD, when the number is a price tag, not a score

This sounds like a question about whether a model is cheap. Underneath it, it's still asking whether you can act on a number and be right. That's LEAD, run on a dollar figure instead of an accuracy score.

And if you want to be sure it really works, try it somewhere else

An AI scribe listens to a vet's exam-room dictation and drafts the visit note before the vet leaves the room, so nobody's typing up records at nine at night. Marek Dvorak runs product for Bramble Creek Veterinary, a chain of forty clinics, and priced the scribe the same way most teams do: run it on a handful of routine visits, get a number, move on.

One page priced the scribe. The ninety-minute visit is a separate question.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0