CaseAdvancedQuality, Cost & Token Economics / Cost modeling and unit economics / #7
Describe how you would find the most expensive one percent of your traffic.
TRACE · cost & traffic
Quaywell's blended cost per pricing call barely moved for two months. The first time anyone sorted requests by their own cost instead of the average, the top one percent of traffic turned out to be paying for most of the bill, and three separate causes were hiding inside it.
The direct answer
Stop trusting the average cost per call. Sort every request by its own cost, then pull the top one percent and recut it by client, endpoint, and cache status. Rule out a billing bug before blaming behavior, then run the one query that tells a runaway context chain apart from a cache dodging scraper and a stuck retry loop, because all three hide inside the same healthy looking mean.
Do this, in order
Sort every request by its own cost and pull the top one percent, not the daily average.Why: the average moved only a few hundredths of a cent while the top slice grew eleven times over eight weeks.
Rule out a billing or logging bug before blaming traffic.Why: a retry that gets billed twice looks exactly like real cost growth on a dashboard.
Recut the expensive slice by client, endpoint, and cache status.Why: this is what split one retailer's bundle feature from a scraper from a stuck retry loop, three separate fixes.
Run the one query that tells the three causes apart.Why: fixing the wrong one leaves the bill exactly where it was.
Cap context depth and cache by catalog state instead of raw session id.Why: fixes the bundle chain and the scraper in the same change, without punishing real shoppers.
Never make blanket rate limiting the first move.Why: it would have throttled Cindertrail's own wholesale buyers while the scraper just rotated its session id around the limit.
How to answer this, stage by stage
Nobody is grading whether you know the words precision or latency here. They are grading whether you would trust a number that has not moved in two months, or go sort the raw traffic yourself.
1
Scope it to one concrete product before answering in the abstract
Say it like this
"Let's ground this in one product. Quaywell is the AI pricing engine Kesterlin sells to online retailers. It reads a shopper's page and recommends a real time price. Renu Dransfield owns the infrastructure cost behind it, and Cindertrail Outdoors is one of the retailers running on it."
Why this works
An abstract "find the expensive traffic" question turns into a shrug fast. One product turns it into a real number problem.
2
Say the method out loud before touching a single number
Say it like this
"I'm going to build a timeline of when the cost actually started concentrating, recut the traffic to find where it lives, rule out a logging bug before I blame behavior, name a short list of real causes, then run the one check that tells them apart."
Why this works
Tells the interviewer you have a method, not a hunch, before you've named a single cause.
3
Build the timeline first, including things that looked like wins
Say it like this
"Quaywell's blended average cost per call barely moved for eight weeks, about thirty eight hundredths of a cent the whole time. But the ninety ninth percentile climbed the entire time, and it started the same week Cindertrail shipped a feature called Complete the Look, which nobody flagged as a cost risk because it looked like a sales win."
Why this works
The flip almost always happens weeks before the number that finally gets someone's attention.
4
Recut the expensive slice instead of arguing about the average
Say it like this
"I'd split the top one percent by client, by endpoint, and by whether the request hit cache. Cindertrail's public browse quote endpoint, with almost no cache hits, is where nearly all of it lives. The company wide average hides that completely."
Why this works
A one percent tail spread evenly across every client would need a totally different fix than a tail that's really one client's one feature.
5
Rule out a billing bug before trusting the behavior story
Say it like this
"Before I chase a root cause, I'd check whether the cost pipeline is double billing retries, and check the raw model provider invoice against what our own metering says for the same hour. If those two numbers agree, the tail is real traffic, not a counting mistake."
Why this works
A tracking bug can look exactly like a usage change on a dashboard, and chasing the wrong one wastes a week.
6
Name three real cause candidates, not a list of everything possible
Say it like this
"Three things could do this. One, Complete the Look calling Quaywell once per extra item with a growing context. Two, a scraper hitting the public quote endpoint with a fresh session id every time, so it never hits cache. Three, a validation bug on one older client that retries the same call up to five times."
Why this works
Three named suspects with a reason each beats a shrug that says "probably a bot somewhere."
7
Run the one query that tells the three apart
Say it like this
"I'd group the expensive calls by session and look for three shapes. One session where the item count in context keeps growing, that's the bundle chain. Thousands of unique sessions with an identical short request and zero cache hits, that's the scraper. One session sending the exact same request id twice inside two seconds, that's the retry bug."
Why this works
This is the strongest move in the whole method. It turns three guesses into three separate, checkable facts.
8
Close on the fix, not just the diagnosis
Say it like this
"So: sixty one percent of the tail is the bundle chain, cap how many items its context carries. Twenty six percent is the scraper, cache by what's in the catalog instead of by session id. Thirteen percent is a bug, just fix it. Three causes, three separate fixes, not one blanket rate limit."
Why this works
Ending on the fix is what makes this sound like judgment instead of a diagnosis nobody acted on.
Let's learn
Every day, about five million shoppers ask Quaywell the same quiet question. What should this item cost right now?
Quaywell is the AI pricing engine Kesterlin built for online retailers. A shopper opens a product page, and Quaywell reads the catalog, the competitor prices, and the shopper's own cart, then recommends a price in under a second. Retailers like Cindertrail Outdoors just take the number and show it.
Before a tool like this, Cindertrail's own merchandising team repriced items once a week from a spreadsheet. That took a day and a half every Friday, and a price could sit stale for up to six days before anyone noticed a competitor had moved. Quaywell reprices every page view instantly, and blended across every client on the platform, the average cost of one pricing call sat at about thirty eight hundredths of a cent, so the monthly bill looked small and healthy.
Knowledge spark: what is the ninety ninth percentile?
Sort every call by its own cost, cheapest first. The ninety ninth percentile is the value ninety nine out of every hundred calls sit under. Watching it tells you what the worst calls cost, not just what the typical one costs.
The turn: the extra cost was never a mistake anyone could point to. It was quieter than that. Even while the average call cost held flat for eight straight weeks, one percent of the traffic was already paying fifty eight times more than everyone else, and blending that in with the other ninety nine percent made the whole thing disappear.
Cost per call, the middle versus the top: eight weeks
p50, the typical callp99, the worst one percent
The median cost per call never moved. The ninety ninth percentile grew eleven times, starting the week Complete the Look shipped, five weeks before anyone looked.
By week eight, Renu Dransfield finally sorted every call by its own cost instead of trusting the blended number. Five million calls a day, top one percent, fifty thousand calls. Those fifty thousand calls cost about eleven hundred sixty dollars a day, fifty eight percent of Quaywell's entire daily bill, coming from about one percent of the traffic.
What's driving the top one percent, by cause
Bundle chain, 61%Scraper, 26%Retry loop, 13%
One client's bundle feature was more than three times the size of the other two causes combined.
The evidence test split one suspicious slice into three separate, checkable causes, and confirmed which one carried most of the weight.
The average was never wrong. It had just stopped describing anyone.
The choice that mattered
Quaywell shipped its first billing dashboard eighteen months earlier with one number, the blended average cost per call, no view of the distribution underneath it. That was a fine call back then. Kesterlin had three clients, all about the same size, all with about the same traffic shape, so the average genuinely described everyone. It stopped being fine the day the clients stopped looking like each other.
At its worst, Kesterlin keeps pricing every new retailer contract off that blended average forever, and quietly loses money on any client whose traffic looks like Cindertrail's, without a single number on the dashboard ever turning red.
What I'd leave alone: the authenticated checkout path does not need this same scrutiny. It is always one item, one call, no recursion possible, and real money changes hands there, so it already runs fresh, uncached, every time. The runaway tail only had room to grow on the browse path, where nothing forces the request to stay small.
The lesson: a number that has not moved in two months can still be hiding a tail that grew eleven times, because an average only ever tells you about the middle. It never tells you about the edge.
Now here is the same thing as a story
Read the long version below when you want to feel why a number this steady still went wrong, not just be told that it did.
Renu Dransfield can look at a spend graph for four seconds and tell you whether the shape is a traffic increase or something stranger. She had spent three years running platform cost for a logistics routing tool before Kesterlin hired her to own Quaywell's infrastructure bill.
The first two months after Cindertrail Outdoors came on as a client were good, genuinely good. Quaywell's blended cost per pricing call held at about thirty eight hundredths of a cent, same as every other client on the platform, and the weekly finance review barely spent thirty seconds on that line before moving on.
Renu used to open the per client cost breakdown every Monday, just to eyeball it. By the second month she was down to a glance at the blended total, since the number never surprised her. By week seven she was not opening the per client view at all. The one number on the dashboard had never once needed her to.
It came back on an ordinary Wednesday, and not from an alert. Kesterlin's finance partner mentioned it in a hallway between meetings, almost as a joke, that Cindertrail's monthly infrastructure line had grown faster than any other client's, and asked, half laughing, whether Cindertrail had simply gotten more popular.
Renu's first instinct was the sensible one. More traffic costs more, that is not a mystery. But the comment sat with her, and instead of writing it off she pulled the raw request log and did what the dashboard had not made her do in months: sorted every call by its own cost instead of the average.
We were not chasing a broken number. We were chasing a healthy one that had quietly stopped meaning anything.
Fifty thousand calls a day, out of five million, were costing fifty eight times what an ordinary call cost. Nearly all of them belonged to Cindertrail's browse quote endpoint, and almost none of them hit cache. Renu spent that whole Wednesday afternoon grouping those calls by session, and three separate shapes fell out. One session where the item count kept growing, over and over, that was a bundle feature called Complete the Look, recommending a price for each new item using the whole growing list of items already recommended as context. Thousands of unique, one time sessions with an identical short request, that was a scraper, quietly pricing out Cindertrail's whole catalog with a fresh session id on every hit so it never once landed in cache. And one session sending the same request twice inside two seconds, a validation bug on an older client integration, retrying a rejected call it should never have rejected.
The decision that opened the door went back to Quaywell's very first billing meeting, eighteen months earlier. Back then Kesterlin had three clients, all about the same size, all with about the same traffic shape. Someone asked whether they needed cost broken out by client and by call type from day one. The honest answer was no, not yet, the blended number told the whole story because every client looked the same. Nobody decided, on purpose, that it would still be the whole story eighteen months and forty clients later. It just kept being the number on the dashboard, and a number that keeps looking fine stops feeling like a choice anyone made.
Run that Wednesday again with one change: a weekly job flags any client whose ninety ninth percentile cost per call climbs past five times the platform average, three days running, no person has to remember to look. Cindertrail's flag would have fired in week three, the week Complete the Look shipped, not week eight after a hallway comment. Renu would have found the bundle chain in about twenty minutes instead of a full afternoon, and Kesterlin would have shipped the fix before the tab ran about three thousand five hundred dollars past what it should have.
One design waited for a person to remember to look, and then waited a little longer for a colleague to happen to notice a spend line in passing. The other design watches the shape of the bill on its own and says something the moment one slice stops looking like the rest.
What I'd tell myself, back in that very first billing meeting: a blended average is a fine story right up until your clients stop looking like each other, and the day they stop is a day nobody ever puts on the calendar.
TRACE, the five moves behind the fifty eight times number
Not a story wearing a framework's clothes. This is a diagnosis problem, and TRACE is what keeps a flat looking average from hiding a tail that grew eleven times underneath it.
TTimeline. When did it actually start, not when someone finally noticed?
Cindertrail's blended average held at about thirty eight to forty hundredths of a cent for eight straight weeks. The ninety ninth percentile climbed the entire time, from about $0.0021 the week before Complete the Look shipped to $0.0232 by week eight, eleven times higher, and no dashboard watched that number at all.
Start the timeline before the number anyone actually tracks moves, or you will start weeks too late.
RRecut. Slice it until the average stops hiding anything.
Recut by client: Cindertrail alone. Recut by endpoint: the public browse quote path, not the authenticated checkout path. Recut by cache status: the entire expensive slice missed cache. Three cuts, one very small, very specific place the cost actually lives.
A one percent tail spread evenly across the platform needs a different fix than a one percent tail that is really one endpoint.
AAssume nothing. Rule out the ruler before you trust what it measured.
Before trusting any of it, Renu checked the metering pipeline against the raw model provider invoice for the same eight hour window. They matched within half a percent. The tail was real traffic, not a double counted retry sitting inside the billing code itself.
A logging bug looks exactly like a usage spike. Check the ruler before you trust what it is measuring.
CCause candidates. Name three, not a shrug that blames "bots."
One, Complete the Look, recursively calling Quaywell once per extra item with a growing price history in context. Two, an anonymous scraper hitting the public quote endpoint with a fresh session id every time, so it never once hit cache. Three, a validation bug on one older client integration, retrying the same call up to five times.
This is also where the rejected alternative sits, see below: a blanket rate limit instead of naming each cause on purpose.
EEvidence test. The one query that separates the top candidates.
Grouping the expensive calls by session told the three apart cleanly. Sessions with a steadily growing item count in context, the bundle chain, sixty one percent of the tail. Thousands of unique sessions, an identical short request, zero cache hits every time, the scraper, twenty six percent. One session, the same request id sent twice inside two seconds, the retry bug, thirteen percent.
The hardest step, and the one most answers skip. Three named causes with no test to separate them are still just three guesses.
Three things worth stating directly, since this is where the real judgment sits. The alternative Renu's team considered first was a blanket rate limit on anonymous traffic, capping requests per IP address. It lost fast: Cindertrail's own wholesale buyers sit behind one shared IP on a corporate network, so the cap would have throttled real, high value shoppers, while the scraper simply rotated its session id and slid under any per IP limit anyway. The AI specific failure worth naming by name is unbounded context growth inside a recursive recommendation chain, each extra item Complete the Look adds re sends the whole growing price history back into the model, so cost compounds quietly with cart size and no single call ever looks wrong on its own. The guardrail is a hard cap on how many items of history one chain can carry, five, with anything past that served from a rolling summary instead of the full raw list. And the fix accepts a real trade off: the public browse quote endpoint now caches its recommendation for ninety seconds, keyed to the catalog's own price state instead of the shopper's session, which can leave an anonymous browsing quote up to ninety seconds behind a competitor's price move. Checkout never uses that cache. The moment real money is about to change hands, Quaywell always asks the model fresh.
And if you want to be sure it really works, try it somewhere else
Same five moves, a city claims desk this time, nothing about pricing or e commerce anywhere in the room.
Halcourt is an AI claims triage assistant Palmquist Mutual built to read an incoming auto claim and recommend a fast track or a full human review. Kofi Msengi runs cost and eval on it.
The build-up: Halcourt reads about twelve thousand claims a day, and the average cost to read one holds steady around two cents, mostly a fender bender with two photos and a short police note. The top one percent, about a hundred twenty claims a day, are large multi vehicle pileups with dozens of attached photos and full police reports. Those claims cost about a dollar fifty four each, seventy seven times the average, and make up seventy seven percent of Halcourt's whole daily bill.
The decision Kofi would take back
Treating claim reading cost as roughly flat because most claims looked like a fender bender, and never separating multi vehicle claims with dozens of attachments out of the average before that traffic shape had grown into a real share of the docket.
Same method, different lever: for Quaywell, deploy of one feature was the moment the tail started. For Halcourt, nothing shipped at all. The lever is how many pages and photos ride along inside one claim, not anything the engineering team changed. A regional storm that triples multi vehicle claims in a single week would triple the tail without a single line of code moving.
Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: sort by the request's own cost, recut the top slice by segment, rule out a logging bug, then confirm the real cause with one query before fixing anything.
Cost: there is no budget this quarter for a full automated flagging job. Ship the cheaper version first, a manual weekly sort of the top one percent by cost, alerting a human instead of a system, until the automated version earns its build time.
The model got better, for real: say Quaywell's underlying model gets cheaper per token across the board. That is not proof the tail shrank with it. The easy, common calls could have gotten even cheaper while the bundle chain's context kept growing at exactly the same rate.
Where people run it wrong.
They watch the average, see it holding steady, and stop looking, the same trap the blended dashboard set here.
They find one expensive client and blame the whole account, instead of separating which specific mechanism inside that account is actually driving it.
They fix the cost by rate limiting everyone, which punishes ordinary high volume users right alongside whatever is actually broken.
How to use it live. Say the real question out loud before naming a cause: "before I guess why, can I sort by cost instead of trusting the average, that's usually where the real answer is hiding." That buys a beat to think instead of repeating whichever cause happens to come to mind first.
Flashcards (tap any card to flip it)
1 · THE FRAMEWORK
What framework is this, and what's its one job?
Tap to flip
ANSWER
TRACE: rule out, then narrow, until one query tells the real cause apart from the guesses. Built for diagnosis questions, not a story about a habit fading.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Renu Dransfield, the engineer who owns Quaywell's infrastructure cost at Kesterlin. Spent three years running platform cost for a logistics routing tool before this.
3 · THE OLD HABIT
What did Renu stop doing over seven weeks, without anyone deciding to stop?
Tap to flip
ANSWER
The weekly per client cost check. It shrank from a full Monday review, to a glance at the blended total, to nothing at all by week seven.
4 · THE RECUT
What did slicing by client, endpoint, and cache status actually reveal?
Tap to flip
ANSWER
Nearly all of the expensive one percent lived in one client's one endpoint, missing cache almost every time, not spread evenly across Quaywell's whole platform.
5 · THE OLD DECISION
What decision would Renu take back?
Tap to flip
ANSWER
Billing and monitoring Quaywell by one blended average cost per call, with no view of the distribution, decided back when all three of Kesterlin's clients had the same traffic shape.
6 · THE NUMBER
Fill in the blank: the top one percent of traffic cost ___ times the platform average, and made up ___ percent of the daily bill.
Tap to flip
ANSWER
58 times, and 58 percent. Fifty thousand calls a day, out of five million, were quietly paying for most of Quaywell's entire bill.
7 · THE REPLAY
Same Wednesday, new design, what changes?
Tap to flip
ANSWER
A weekly job flags any client whose p99 cost per call passes five times the platform average for three straight days. The flag fires in week three, not week eight, and the fix ships before the tab runs about $3,500 past where it should have.
8 · CROSS-PRODUCT TRANSFER
Section 4 answers this same question again for a different product. Which product, and what's the different lever there?
Tap to flip
ANSWER
Halcourt, a claims triage AI at Palmquist Mutual. There the lever is how many pages and photos ride along inside one claim, not anything the engineering team shipped.
Check yourself Score: 0 / 0
Multiple choice
1. Why did Quaywell's blended average cost per call stay flat for eight weeks, even while the real problem was already growing?
A. The pricing model got cheaper at the same rate the tail got more expensive.
B. The expensive one percent was small enough in count that averaging it with the other ninety nine percent hid the growth.
C. Kesterlin's finance team was rounding the number down each week.
D. Cindertrail's traffic volume dropped at the same time its cost per call rose.
Show hint
Look at the two lines in the first chart. One barely moves. Look at how small a slice one percent of five million calls actually is.
Show answer
B. Fifty thousand expensive calls a day, blended into five million total, barely nudges the average even while those fifty thousand grow eleven times more costly.
True or false
2. True or false: because the platform wide average cost per call barely moved, that means no client's traffic was quietly getting more expensive.
True
False
Show hint
Check what the p99 line was doing in the first chart while the p50 line sat flat.
Show answer
False. The ninety ninth percentile climbed eleven times over the same eight weeks the average held steady. A healthy average said nothing about what was happening at the edge.
Fill in the blank
3. The top one percent of Quaywell's traffic cost about ___ times the platform average, and made up about ___ percent of the daily bill.
Show hint
Check the number stated right after the second chart in Section 1.
Show answer
58 times, then 58 percent. Because share of cost divided by share of traffic gives the multiplier directly: 58 percent of the bill from 1 percent of the calls is the same fact stated two ways.
Short answer, name the rejected alternative
4. What did Renu's team consider first for cutting the runaway cost, and why did it lose?
Show hint
Look at the paragraph right after the five TRACE steps, where the rejected fix gets named.
Show answer
Model answer: A blanket rate limit on anonymous traffic, capping requests per IP address. It lost because Cindertrail's own wholesale buyers share one corporate IP, so the cap would have throttled real shoppers, while the scraper just rotated its session id and slid under any per IP limit anyway.
Short answer, apply it yourself
5. Pick an AI product you use yourself that runs a lot of small requests. Name one place its expensive one percent might be hiding, and how you'd go find it.
Show hint
Think of a product where most uses are short and simple, but a few uses could quietly carry a lot more content or context than usual.
Show answer
Model answer: A photo storage app's search feature might be cheap for most searches, one or two words. But someone who pastes a whole paragraph, or searches through a folder of thousands of photos at once, could cost far more per search. I'd sort search requests by their own processing cost, pull the top one percent, and check whether they share an unusually long query or an unusually large folder.
Multiple choice
6. Quaywell's total daily traffic doubles to ten million calls, but Cindertrail's expensive slice stays exactly fifty thousand calls a day. What happens to that slice's share of total traffic?
A. It stays the top one percent, since the calls themselves did not change.
B. It drops to about half a percent of traffic, so a flat one percent cutoff would no longer catch it.
C. It becomes exactly two percent of traffic.
D. Share of traffic does not depend on total volume.
Show hint
Fifty thousand out of five million is one percent. Work out fifty thousand out of ten million.
Show answer
B. Fifty thousand out of ten million is half a percent. A flat percentile cutoff has to move with total volume, or a slice that used to trip the alarm quietly slides underneath it.
Before you close the answer
Why this works
Tests whether you'll trust a blended average or go sort the raw distribution yourself, and whether you separate which client from which mechanism before proposing a fix. Most candidates say "look at the biggest client" and stop there.
Follow-up traps
"Couldn't you just cap spend per client and call it done?" Response: a cap would have throttled Cindertrail's real shoppers along with the bundle chain and the scraper, since all three traffic types share the same client id. The fix has to target the mechanism, not the account.
"Isn't fifty eight percent of the bill from one percent of traffic just what pricing engines always look like?" Response: no, it only looked that way because three unrelated mechanisms happened to land on one client in the same eight weeks. The evidence test is what proves that instead of assuming it.
If pressed
The automatic flag is not a hard yes or no rule. It fires when a client's ninety ninth percentile cost per call passes five times the platform average for three consecutive days, a calibrated bar tuned to catch a real drift without paging someone over ordinary daily noise.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
