Model the monthly cost of a feature used by 50,000 users averaging 12 interactions each.

Fifty thousand users, twelve checks a month each, adds up to a small, boring number. The real risk was never the total. It was how unevenly that average was actually spread across users.

How to answer this, stage by stage

Nobody's grading whether you can say the word "tokens." They're grading whether you can show the arithmetic behind one number and say which part of it you trust least.

Let's learn

What does 50,000 users averaging 12 checks a month actually cost to run, once you stop treating "12" as a fact about every single one of them?

Clausewatch is a browser extension from Verrick that reads what you type into any text box on the web and flags grammar and style problems before you hit send.

Before anything like Clausewatch, a careful writer read their own message twice before sending it, catching maybe 7 out of 10 real problems, an awkward phrase, a wrong word, a run-on sentence, and missing the rest because nobody proofreads their own writing well at 11pm.

Clausewatch flags an average of 3 issues on a typical paragraph, in under two seconds, and Verrick's own numbers say it catches something close to 9 out of 10 of the same problems a careful human editor would flag. The base pass is fast and cheap on purpose, and it only hands a check to a slower, pricier model when the fast model's own confidence is low, trading a little latency and cost for catching the fixes a user would actually notice being wrong.

The turn: Clausewatch didn't get worse, and the model didn't get expensive. Fifty thousand users averaging 12 checks a month is a small, boring number, 600,000 checks, and on paper the monthly bill for that comes to well under a thousand dollars. The real risk was never the total. It's that "12 average" is not what any real user actually does.

The honest answer was never one number. It ran from about $545 to $839, with a best estimate near $629, well under what escalating every check instead of just the unsure ones would have cost.

At its worst, a cost model built off one flat average can look perfectly healthy for months, then get blindsided the one quarter a small slice of users, students during finals, a marketing team going all in on a launch, ramps up the expensive path hard enough to blow past a budget nobody thought needed watching.

What I'd leave alone: the base check line doesn't need this kind of scrutiny. It's flat, cheap, and dominated by small-model pricing that barely moves no matter how usage shifts, so watching it closely every week would spend attention where there's no real risk to find.

The lesson: an average can be completely honest and still be the wrong number to plan a budget around, the moment the thing it's multiplying doesn't spread evenly across the people doing it.

Now here is the same thing as a story

Read the long version below when you want to feel why a total that kept agreeing with itself still went wrong, not just be told that it did.

Idunn Osterlind edited copy for six years before she ever wrote a line of product spec. She can read a paragraph once and tell you exactly which sentence a reader will trip on.

She joined Verrick the year Clausewatch launched, and for the first year she built its cost model the way almost anyone would: one number. Multiply the interactions by the price per interaction, round up, done. At 50,000 users and 12 checks a month, that number came out under a thousand dollars, and every month the finance dashboard showed a green light next to it.

For a while, green was the right color. The team shipped a deep rewrite feature, a button that rewrites an awkward paragraph in a clearer voice, and it landed well. Usage climbed steadily, and the monthly total climbed with it, exactly as predicted, exactly on the line the model had drawn.

Idunn stopped checking the model against anything except itself sometime in the second year. Not on purpose. The total kept landing close to what the spreadsheet predicted, month after month, and a number that keeps agreeing with itself stops feeling like something you need to open the hood on. At first she pulled a sample of real usage logs every month, just to eyeball whether the spread looked sane. By month nine that had thinned to once a quarter. By month sixteen, she wasn't pulling logs at all. The one number on the dashboard was doing the whole job the sample used to do.

It came back during an ordinary quarterly review. Nadege Winterburn, the finance partner who signed off on Clausewatch's compute budget, pulled the raw usage logs herself this time, not for Clausewatch specifically, just a routine pass across every AI feature the company ran. She noticed something odd in one column before she noticed anything in the total: a small slice of user IDs showing up on almost every page of the deep rewrite log.

Idunn's first instinct was the sensible one. Maybe a handful of accounts were bots, or a QA script left running. She'd seen that before. But she pulled the real distribution instead of guessing, and the shape of it stopped her.

Forty nine thousand users were averaging 8 checks a month, quietly under the "12 average" the whole time. A thousand users, professional writers, marketing teams drafting five versions of the same email, a small cluster of university accounts during exam season, were averaging 208 checks each, about 26 times as many, and a much higher share of theirs were the expensive kind. The fleet average of 12 was real. It was also true of almost nobody.

The decision that opened the door went back to Clausewatch's very first budget meeting, before it had a thousand users, let alone fifty thousand. The model tracked one number because one number was all anyone needed to answer "can we afford this." Nobody decided, on purpose, that it would still be the only number two years and forty nine times the user base later. It just kept agreeing with itself, and a number that keeps agreeing with itself stops looking like a choice anyone made.

Run that quarterly review again with one change: a per-user running total, checked automatically every week, not just the fleet sum. The same thousand power users still show up, the same eighty percent share of the expensive calls still exists. But now it shows up in week three of month one, on a chart Idunn already has open, not as a coincidence Nadege happened to notice in a column she wasn't even looking for.

One design trusted a total that kept agreeing with itself. The other watches the shape underneath it, not just the number sitting on top.

What I'd tell myself, back in that very first budget meeting: a single total is a fine answer to "can we afford this today." It was never going to be a fine answer to "who is actually using this, and how much," and the day nobody's asking that second question is a day that doesn't announce itself on any dashboard.

BOUND: the arithmetic Idunn's one number was standing in for

Not a story question wearing a framework's clothes. This is an estimation problem, and BOUND is what keeps one comfortable total from hiding the shape underneath it.

Three things worth stating directly, since this is where the real judgment sits. The alternative Idunn's team considered, and rejected, was dropping the confidence-based escalation entirely and just running every check through the stronger model, one fewer moving part in the system. It lost because that alone would push the bill to about $1,014 a month, about 61 percent more, mostly paying the strong model's price on checks the small model was already getting right. The AI-specific failure worth naming by name is cost blindness to usage skew: a feature's own routing decision, which calls escalate to a pricier model, depends on the model's own uncertainty about that day's writing, not a fixed unit-economics number, so the trigger rate can drift with who's using the product even while total interaction count stays exactly on plan. The guardrail is a per-user cap: any single account whose deep rewrite usage crosses a set weekly ceiling falls back automatically to the cheap base check, no hard cutoff, so a power user's fix bar gets shorter, never blank. That guardrail isn't free. It trades a little of the deepest editing a heavy user might want against a bounded, predictable ceiling on what any one account can cost. And the bar Clausewatch holds itself to was never zero missed style issues across 600,000 checks a month, no probabilistic checker can promise that. It's a cost ceiling tied to actual usage shape, checked weekly per user, not one fleet-wide average standing in for 50,000 very different writing habits.

And if you want to be sure it really works, try it somewhere else

Same five letters, a library's rare-book scans instead of a browser extension's paragraphs, and this time the lever that swings the bill isn't a trigger rate at all.

Coverleaf is a cataloging tool the Loamcastle Library Consortium built. A cataloger photographs a book's cover and title page, and Coverleaf drafts the catalog description from the photo plus whatever metadata already exists. Petrona Bramante runs digitization for the consortium's forty branches.

The build-up: Coverleaf handles about 6,000 new items a month. A standard call, one photo at normal resolution plus the item's existing record, costs a fraction of a cent, about $18 a month across the whole consortium. An archival pass, three to five photos at four times the resolution for a rare or damaged item, costs more than ten times as much per item. At launch, Loamcastle modeled archival passes at under 1 in 20 items, based on its first year of mostly ordinary donations.

Two years and one large estate donation later, one branch alone was routing closer to 1 in 4 items through the archival path, and that branch's monthly bill had quietly grown past the whole consortium's original combined budget line, not because more books arrived, but because the mix of what arrived had shifted underneath a threshold nobody had revisited.

Same method, different lever: for Clausewatch, the trigger rate that mattered was how often a check needed the expensive rewrite pass. For Coverleaf, it's how often an item needs the expensive archival scan, and that share moves with what people donate, not with anything the engineering team controls.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: break the bill into real lines, size the biggest lever as a range, sanity check the total per unit of work, not just the fleet sum.
Cost: there's no budget this quarter for both a bigger review sample and a per-user cap. The cap wins. A ceiling on what any one account can cost beats a bigger sample that just re-measures the same skewed average more precisely.
The model got better, for real: say Clausewatch's base model gets upgraded and its baseline accuracy climbs. That's not proof the deep rewrite trigger rate improved with it. The easy, common checks could get even easier while the power-user tail keeps triggering the expensive path exactly as often as before.

Where people run it wrong.
They price the feature off the average user, and never ask how the usage behind that average is actually shaped.
They notice the total is small and stop watching it, instead of asking whether small and evenly spread are the same thing.
They fix a scary bill by throttling everyone a little, instead of finding the small slice of usage actually driving it.

How to use it live. Say the real question out loud before quoting a number: "before I give you a total, do you want the fleet average, or do you want to know if that average is true of anyone." That buys a beat to think instead of repeating a comfortable number nobody has actually checked against the real spread.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0