How do you model cost when a small share of users generate most of the usage?

Askly's average cost per user looked calm for the better part of a year. A small slice of power users was already driving the real bill, and the metric built to catch that sat one click away, unused.

How to answer this, stage by stage

Nobody's grading whether you know the words precision or recall here. They're grading whether you'll trust a number because it's calm, or ask what shape it's supposed to be describing.

Let's learn

What does it cost to answer a business question in plain English, when almost nobody asks the expensive kind?

Askly is the tool Northgait Analytics built so a business user can type something like "what were our returns in the Northeast last quarter" and get an answer, no SQL, no waiting on an analyst. Before Askly, a question like that went to a data analyst and came back in about two and a half days, and Northgait's early customers said their analysts spent close to fifteen hours a week just working through requests exactly like it.

With Askly, the same question comes back in under twenty seconds, most of the time. Across the roughly 9,000 people who use Askly every month, average AI cost per user sat near $4.10 for nine straight months. It looked like a cheap, sustainable habit.

The turn isn't that Askly got worse at answering questions. It's that averaging away a handful of very expensive users is exactly what an average is built to do, and nobody had a number that refused to be averaged.

Here's what was actually driving that line. Askly's model doesn't just write one query and stop. If the first attempt gets a join wrong across the schema, it rewrites the SQL and tries again, up to a few attempts, and every attempt resends the same forty-table schema for the model to work with. A simple question, "what were sales last month," costs about two cents. A question that spans several fiscal years across a dozen tables, and needs two or three tries to get right, can cost well over a dollar.

At its worst, a cost metric that stays calm while the real bill grows is worse than no metric at all, because everyone upstream believes the product is healthy. Northgait kept selling flat-fee plans on the assumption that AI cost would always stay small next to what an account paid, right up until it didn't, on an account nobody was watching for exactly this.

What I'd leave alone: the 8,550 or so ordinary users asking two or three simple questions a week don't need any of this. Average cost per user is a perfectly honest number for them, and building threshold checks or retry caps around their usage would spend engineering time on a group that was never the problem.

The lesson: a calm average isn't proof a product is healthy. It's proof the cost is being spread out thin enough that nobody can see where it actually lives. The fix isn't a smarter average. It's a number shaped like the truth, one that respects that a distribution has a shape, and only a metric built for that shape will warn you before the shape breaks something.

Now here is the same thing as a story

Read the long version below when you want to feel why a calm number went so wrong, not just be told that it did.

Every Monday morning, before anyone else at Northgait Analytics reached their desk, Ilya Vahter pulled up the same three numbers: new signups, churn, and average AI cost per user. He'd built Askly's cost dashboard himself, back when the whole company was twelve people and every number on it still meant something he could picture.

Askly launched to real customers about fourteen months ago. In the first weeks, Ilya did more than watch the dashboard. He picked three or four accounts a week and read through their actual query logs by hand, checking that the pattern matched what the average implied.

It matched, for months. Most accounts looked alike: a handful of straightforward questions a week, mostly one table, mostly answered right the first time. Average cost per user was a fair number because the usage really was fairly even. Ilya's spot checks kept confirming the same thing, week after week, and confirming the same thing feels like proof. It's really just proof of the one week you happened to look.

By month seven, the spot checks had thinned to one account every couple of weeks. By month eleven, Ilya still opened the dashboard every Monday, saw average cost per user sitting near $4.10 like it always did, and moved on to whatever was actually on fire that day. Nothing was on fire. That was the problem.

It came back during Alina Salvesen's quarterly margin review, the one she runs every close, pulling gross margin apart by account instead of trusting the company-wide number. She noticed Windermist Foods first: a Company-plan account, flat $1,200 a month, whose AI compute cost that same month had come to $2,380. Northgait was paying to serve one of its own customers.

Alina pulled the account's actual usage. It wasn't fraud, and it wasn't a bug. Windermist's finance team had three analysts who lived inside Askly, asking things like "compare vendor spend by region across the last three fiscal years, broken out by category," the kind of question that touches a dozen tables across different schema versions. Askly's model wrote a first attempt, got the join wrong, rewrote it, got closer, rewrote it again. Three or four tries wasn't unusual for questions like that, and each try sent the model the same forty-table schema to work with again.

Ilya went looking for how far this went beyond one account. Company-wide, out of about 9,000 monthly active users, the top 5 percent, roughly 450 people, were generating 61 percent of all AI compute cost. Average cost per user had never had a reason to move, because 8,550 ordinary users asking two or three simple questions a week were exactly cheap enough to keep it looking calm.

It was never really about whether $4.10 was a healthy average. There was no single number for a distribution shaped like this one. Averaging a distribution with a long tail doesn't describe it, it erases the tail and reports what's left.

The decision that opened the door went back to that very first product review, before Askly had a single customer. The team picked average AI cost per user as the number to watch, because it was simple, and because in the earliest pilot every account really did look the same. Nobody chose that number carelessly. It was the right number the week it was chosen, for exactly the accounts that existed that week.

Run that quarter again with one change: p95 cost per active user, tracked weekly from week one, not average cost, checked once a month. By week six, the line has already climbed past the $20 line Ilya's team agreed to treat as a real signal, not noise. The complex, multi-table questions from accounts like Windermist get capped at two self-correction retries instead of an open-ended number, and past the cap, Askly offers a cheaper guided path instead of guessing a third time. Windermist's own finance team gets a plain note that their usage has crossed into a tier a different plan is built for. None of it waits for a quarterly margin review to notice.

One design let a calm-looking average speak for a distribution it was never built to describe. The other watches the shape of the thing, not just its center, and it would have rung six weeks earlier.

What I'd tell myself, back at that first product review: the average was never wrong, exactly. It was just answering a question nobody was asking anymore, the moment the first power user showed up, and nobody had given that moment a name.

LEAD, the four letters behind the $34.60

This isn't a story wearing a metric's clothes. It's a metric question, and LEAD is what stops a calm average from standing in for a shape it can't actually describe.

Three things worth stating directly, since this is where the real judgment sits. The alternative Ilya's team considered first, and dropped, was a flat cap on the number of queries any one user could run per day. It lost because it punishes exactly the users who prove the product works, the ones asking the hardest, most valuable questions, and it doesn't touch the actual cost driver, which is retries on complex queries, not query count. The AI-specific failure worth naming by name is unbounded agentic self-correction: a model that keeps rewriting its own SQL and resending the full schema after a failed attempt, with no ceiling on how many times it tries. The guardrail is a hard cap, two retries, then a cheaper guided path, deterministic filters instead of another free-form guess. That guardrail isn't free: capping retries at two instead of leaving them open drops first-try success on the hardest multi-table questions from about 91 percent to about 84 percent, a real quality-cost trade worth naming, not wishing away. And the bar Askly holds itself to was never zero cost variance across 9,000 users; no flat-priced product can promise that. It's a threshold-specific bar, weekly p95 held under $20 for the typical week, checked every week against the real distribution, not one calm-looking average standing in for a shape it was never built to show.

And if you want to be sure it really works, try it somewhere else

Same four letters, a veterinary telehealth chat instead of a business analytics tool, and this time it's chronic-condition pet owners driving the tail, not finance analysts.

Pawline is an AI triage chat Kellwick Veterinary Network offers pet owners: describe a symptom, and Pawline asks follow-up questions before saying watch it at home, book a video visit, or go to emergency care now. Frida Sohal runs product on it.

The build-up: Kellwick's "Unlimited chat" plan is flat too, one monthly fee per household, however much they use it. Average AI cost per active pet owner sat near $1.35 a month for most of a year. The top 4 percent of users, households managing a chronic condition like a diabetic cat or an epileptic dog, generate 58 percent of total chat cost, long, multi-turn conversations with real follow-up over weeks, not one symptom check and gone.

p95 weekly conversation cost per active user climbed from $3.10 to $11.40 over ten weeks before Kellwick's quarterly plan-margin review caught gross margin on "Unlimited chat" sliding from 68 to 49 percent, the same gap between a leading weekly number and a lagging quarterly one that Askly had.

Two clocks, same underlying problem. The weekly signal would have rung six to ten weeks before the quarterly one ever did.

Same rank as before: watch the distribution, not the average, and act before the lagging number confirms it. The fix is the same shape too: cap how many follow-up turns a single symptom thread gets before Pawline hands a chronic-repeat household to a lower-cost scripted check-in flow instead of full free-form chat, and build a chronic-care add-on tier instead of leaving the plan flat-only.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: track the tail, not the average, pick a distribution-aware metric, set two real thresholds, and act at each one.
Cost: there's no budget this quarter for both a pricing overhaul and the retry cap. The retry cap wins, since it's the cheaper fix actually pointed at what's driving the cost, not just a symptom of it.
The model got better, for real: say Askly's SQL-writing model gets a lot more accurate on the first try. That's not proof the tail shrinks. Power users don't get more accurate, their questions get more ambitious to match, so retries can keep climbing even as first-pass accuracy improves everywhere else.

Where people run it wrong.
They watch a single blended average because it's the number every other dashboard already tracks, and never check whether their own usage has a long tail.
They notice the tail, and "fix" it by throttling or hard-capping their heaviest users, punishing the accounts most likely to be their best evidence the product works.
They wait for the lagging business number, margin, churn, to move before acting, when a distribution-aware usage metric would have told the same story weeks or months earlier.

How to use it live. Say the real question out loud before quoting a metric: "is this asking about the typical user, or the user actually driving the cost?" That buys a beat to think instead of reciting whatever number the dashboard already shows.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0