Explain what a token is and why a PM should care about it.

Quarrywatch reads every loan application Ledgerwell gets and tells a fraud analyst whether it is worth a second look. Fiacre Grissom owns it. Four days after a prompt change that made it smarter, a routine Monday check caught something the bill and the fraud queue would not have shown for weeks: the tool was now costing three and a half times what anyone had agreed to pay, for every single flag it produced.

How to answer this, stage by stage

Nobody is grading whether you can recite that a token is about four characters. They're grading whether you can say what changes, in dollars and seconds, the moment a prompt gets one line longer.

Let's learn

What does a model actually charge you for, every single time it answers?

Quarrywatch is the tool Ledgerwell, an online lender, points at every loan application before money moves. It reads the form, a summarized bank statement, and a call center note, and hands a fraud analyst one of three flags: clear, review, or escalate.

Every application passes through the same five steps. The middle box is the one nobody was watching.

For most of a year, Quarrywatch cost about six tenths of a cent and a second and a half to review one application. Nine thousand applications moved through it a day. The bill came to roughly sixteen hundred dollars a month, and nobody thought about it much, because it never changed.

Then an engineer fixed a real gap. A pattern of applications with slightly off employer details kept slipping past the flag. Feeding the model the whole raw bank statement instead of a shorter summary, plus a longer worked example, pushed recall on that pattern from seventy one percent to eighty nine in testing. It shipped to every application at once, on a Thursday afternoon.

At the new rate, Quarrywatch would cost about fifty seven hundred dollars a month and take about five seconds to answer instead of a second and a half. Nobody would have known any of that from the bill, which takes about a month to arrive, or from the fraud queue, which just looked like a slightly busier stretch for weeks before anyone called it a pattern.

One of these numbers is available the day a change ships. The other one is available on a billing cycle.

The same blind spot showed up somewhere else, too. Ledgerwell's regional language loan notes had been priced the same way as English ones, by counting words. The real cost of a script that tokenizes far less efficiently than English ran about four times higher than budgeted, and a word count alone would never have shown it.

A word count and a token count agree on English text. They stop agreeing the moment the script changes.

What I would leave alone: Quarrywatch also sends compliance a digest of every flagged application overnight, a batch job nobody is waiting on at two in the morning. A tight token cap there would save a few dollars a month and cost nothing in return, because nobody's day depends on it arriving in three seconds instead of thirty.

The lesson: a model can get better at exactly the thing you asked it to get better at, and still cost three times more to run, and the accuracy number alone will never tell you that. Tokens are the one number that would have said so in four days instead of a month.

Now here is the same thing as a story

The short version above is what you actually say in the room. Read this one for the fourteen months it actually took to build, and nearly stop keeping, the habit that caught this in four days.

Every Monday morning, before her coffee finished brewing, Fiacre Grissom opened the same dashboard. Average tokens per call, by feature, seven days back. She had built the habit herself, eighteen months into running product for Quarrywatch, after a much smaller cost surprise taught her that nobody else was going to watch this number for her.

For most of that year, the habit paid for itself the boring way. The line sat flat, somewhere between eleven and twelve hundred tokens a call, month after month. Quarrywatch read a loan application, a summarized version of the last three months of bank statements, and two short examples showing the model how to reason about a flag, then handed a fraud analyst one of three verdicts. Nine thousand applications a day moved through it, at a little under a cent a call.

Around month four, the Monday check stopped taking ten minutes and started taking three, since nothing ever needed the extra time. By month seven it was a glance while the coffee finished, maybe ninety seconds. By month ten she sometimes checked the total and skipped the four smaller features underneath it, because none of them had moved in so long that a careful look felt like checking a smoke detector that had never once gone off.

Grethe Fenneck, who owned Quarrywatch's fraud model, had a real problem to solve that spring. A pattern of synthetic identity applications with slightly off employer verification kept slipping past the flag, three or four a week, small enough not to trigger an audit but real enough that Grethe could name every one from memory. Her fix worked. Feeding the model the whole raw bank statement instead of a summary, plus a longer worked example showing the exact reasoning chain for that fraud pattern, took recall on it from seventy one percent to eighty nine in her offline test. She shipped it to everyone on a Thursday afternoon, pleased, and said so in the team channel.

Nobody flagged it as a token question, because nobody had ever needed to before. The offline test only measured whether the flags were right. It had nothing to say about what getting them right now cost.

Nobody decided to stop watching closely. It just got easier not to, one quiet Monday at a time.

The following Monday, four days after the prompt went out, Fiacre's one minute glance landed on a line that had broken its own year long band and kept climbing. Not a wobble. A line that had gone from flat around eleven hundred to sitting steady near forty seven hundred, in four days, with no dip back down.

I want to say the problem was that Grethe's prompt got expensive. It did get expensive. But that is not really the story. Fiacre never had a single number in her head that told her the loop was healthy. She had a Monday habit, and the habit only still worked because she had never actually let herself skip it, even on the weeks it felt like checking something that would never go off.

While she traced where the new tokens were coming from, raw statements instead of summaries, six examples instead of two, she pulled up something else almost by accident: the cost model Ledgerwell had built for its regional language applications, launched two months earlier so borrowers could submit KYC notes in Hindi instead of English. The budget for those notes had been sized the same way Quarrywatch's whole cost model was first sized, fourteen months earlier, at the company's launch: count the words, multiply by three quarters of a token, call it done. Nobody had ever gone back and actually run a Hindi note through the real tokenizer to check. When Fiacre finally did, a normal two hundred and twenty word note came back at seven hundred and twenty six tokens, not the one hundred and sixty five anyone had budgeted for. Every regional language application had been running about four times over budget, quietly, since launch.

The decision she would take back sits in a meeting fourteen months earlier, when three people built Quarrywatch's original cost model in four days flat to hit a launch date. Someone suggested actually measuring tokens on a real sample. Someone else pointed out that Ledgerwell only took English applications at the time, and a word count was close enough, and building a proper tokenizer harness for a pilot felt like solving a problem they did not have yet. They were right, for the company that existed that week. Nobody ever came back to check whether it was still true once the prompt grew and the applicant base stopped being English only.

Run the same four days again, with the alert Fiacre built afterward already wired into the deploy pipeline: any change that moves average tokens per call by more than a quarter blocks a full rollout and pages her directly. Three months later, a different engineer's prompt change trips that same alert. It never reaches more than the five percent canary. Fiacre's phone buzzes four minutes after the deploy, not four days after a Monday she almost rushed through.

The old design put the number in a dashboard and hoped someone kept looking. The new one puts the number in a gate, and nothing has to hope.

What I would tell myself, back in that four day meeting: a cost model built for the applications you have this week is not a cost model. It is a snapshot wearing a cost model's clothes, and the day your applicants or your prompts change, it stops telling the truth without ever announcing that it has.

LEAD, counted in the unit the model actually bills

Not a way to prove Fiacre is smarter than Grethe. LEAD is what forces you to say which number actually predicts a cost problem, and to catch it before the bill does the talking for you.

The recap, one line per letter: link the token count to what Ledgerwell actually pays and waits for, not to the model's own sense of itself. The early signal is the token count itself, because the bill and the queue both arrive too late to help. Name both ways it gets gamed, a word based guess and an uncapped accuracy win. And the decision step is what makes it real: a different move at each threshold, not one blanket rule.

Two things worth saying plainly, since this is where the real judgment sits. Fiacre considered a simpler fix first: ask finance to approve the tripled monthly cost outright, since the accuracy gain on that fraud pattern was real and arguably worth it. She rejected that, because nobody had actually chosen the trade-off on purpose. It was a side effect of one prompt edit, with nothing stopping the next "just one more example" from tripling the number again. The AI specific failure worth naming by name is silent context truncation: Quarrywatch's model caps out at eight thousand tokens a call, and a long enough raw bank statement combined with an uncapped prompt could quietly push past that limit, with the model dropping whatever did not fit, possibly the one transaction that would have flagged the fraud, with no error and no warning to anyone. The guardrail is the token cap itself, set to truncate on purpose and flag it for manual review, instead of leaving that decision to whatever the model's own reading limit happens to cut. And the trade-off was real and taken on purpose: capping tokens and bringing back a summary step for long statements gives up some of the raw detail that helped Grethe's prompt hit eighty nine percent recall, in exchange for a cost and speed the team actually chose instead of one that happened to them.

And if you want to be sure it really works, try it somewhere else

Same four letters, a veterinary telehealth app instead of a lender, and this time the token count is padded by a photo instead of a prompt.

Brindlecoat runs veterinary telehealth. Before a video call, pet owners upload a photo of their pet's paper vaccination record along with a typed description of the symptom, and a tool drafts a short triage summary for the vet to read before the call starts. Ivenna Rowantree runs product for it, and hit a smaller version of Fiacre's exact problem five months after launch.

Same shape as Quarrywatch's pipeline, a different artifact feeding it: a photographed paper record instead of a bank statement.

For its first five months, Brindlecoat's triage tool cost almost nothing to run, about six hundred tokens and well under a second per visit, structured fields only: species, age, one typed symptom line. Then the team added a real improvement. Instead of asking owners to retype their pet's vaccination history, the app would photograph the paper record and read it automatically. The raw text that OCR pulled off a photo of a handwritten vet record, misreads, stray marks, and all, went straight into the same prompt. Average tokens per visit jumped from about six hundred to about twenty nine hundred. The triage draft that used to appear in under a second now took three and a half.

Mapped onto LEAD, the shape holds. The link is the same: token count has to track what a vet actually gets, a fast, accurate triage draft before the call, not the model's own read of a scanned page. The early signal is again the token count itself, watched daily; the call slot complaints took three weeks to become a pattern anyone would call a trend. The abuse Ivenna found was the character based guess, not a deadline ask this time, just an assumption that held for clean text and broke on garbled scans. And her decision step matched Fiacre's: a hard token cap, sized off a real count, with structured fields plus a short OCR excerpt standing in for the full raw transcript.

Four different situations, four different moves. None of them is "add a bigger server."

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: a token is what the model actually reads, writes, and gets billed for, and a PM watches tokens per call as its own leading number, capped on purpose.
Cost: no budget to add token monitoring this quarter. Whoever already owns the feature manually pulls a token count on any prompt change before it ships past a canary, even a spreadsheet count beats no count.
The model got better, for real: say the new model gets twice as accurate per token. The reading limit and the price per token rarely improve at the same rate the accuracy does, so the token budget still needs its own check, not a victory lap.

Where people run it wrong.
They watch accuracy or an offline test and call a prompt change shipped the moment the score improves, without ever checking what happened to the token count that got them there.
They budget cost or reading room in words or characters because that number is easy to eyeball, and it quietly breaks the moment real text, a different script, a garbled scan, does not tokenize the way clean English prose does.
They let the model's own reading limit do the cutting for them, so when a call runs over budget, the model silently drops whatever did not fit, instead of the product deciding on purpose what to cut.

How to use it live. When an interviewer asks you to explain a technical term, ask yourself one question before answering out loud: "what does a PM actually do differently once they understand this number." If the honest answer is nothing, you have not found why it belongs in a PM interview yet.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0