Artifact critiqueIntermediateQuality, Cost & Token Economics / Cost modeling and unit economics / #19
Critique a cost model that ignores failed and retried requests.
A cost model that only prices the call that worked cannot see the calls that did not, and it goes blind at the exact moment a change starts making the model fail more, not less.
The direct answer
A cost model that divides spend by accepted outputs while ignoring failed and retried calls will always understate the real bill, and the gap grows fastest exactly when something makes the model fail more. Fix it by pricing every call that happens, successful or not, dividing by requests attempted rather than outputs accepted, and tracking retry rate and dead-fail rate as their own numbers next to it, not folded invisibly into an average that cannot see them.
Do this, in order
Cost every model call that happens, not just the one that worked.Why: a formula that only prices successful calls is structurally blind to retries and dead failures, so it cannot warn anyone when they spike.
Divide by requests attempted, not outputs accepted.Why: dividing by acceptances hides the exact moment a change makes the model fail more, which is when the true unit cost moves the most.
Put retry rate and dead-fail rate on the same dashboard as cost per output.Why: a flat cost line sitting next to a climbing retry line tells the truth, and only one of those two lines is watching the real bill.
Rule out a billing, attribution, or timeout bug before blaming the model for the retries.Why: a timeout set too tight for a slower model looks exactly like a genuine capability regression until someone checks the real response latency.
Tag every retry with a reason code, and recut by prompt or model variant before trusting the company wide average.Why: a blended retry rate can sit at a moderate number while one specific prompt variant is driving almost all of it.
Leave simple, non-model cost formulas alone where there is nothing probabilistic to retry.Why: adding retry accounting to a plain database write is complexity nobody needs.
How to answer this, stage by stage
Nobody is grading whether you know the word "retry." They are grading whether you can see that a cost model built around success is a cost model that cannot see the exact thing it most needs to catch.
1
Scope it to one real product before answering in the abstract
Say it like this
"Let's ground this in one product. CaptionMint is the AI feature inside Bellcrest, a social media scheduling tool, that writes a caption and five hashtags for a photo someone is about to post. Mila Brankovic is the PM who owns it."
Why this works
Keeps the critique attached to a real formula and real numbers instead of a general complaint about metrics.
2
Say your structure out loud
Say it like this
"I'll answer this in two parts. First, exactly what's wrong with a cost model that only counts the calls that worked. Then, what it actually cost Bellcrest the one time this really happened."
Why this works
Signals a plan before diving in, and tells the interviewer this will not be a vague complaint about metrics.
3
Reframe the question before answering it
Say it like this
"This isn't really a question about accounting precision. A cost model that ignores retries and failures isn't a little bit wrong. It's wrong in a way that gets worse exactly when you need it most, the moment something quietly makes the model fail more."
Why this works
Separates a sharp answer from a list of reasons the number might be "a bit off."
4
Give the one decision, plainly
Say it like this
"Price every call that happens, not just the one that worked. Divide by requests, not by accepted outputs. And put retry rate and dead-fail rate on the same dashboard as cost per output, so nobody can ship a change on a number that's gone blind to its own failures."
Why this works
This is the actual fix, said as something you would do, not a category of thing you would consider.
5
Prove it with the failure, cut to four sentences
Say it like this
"Here's what happened at Bellcrest. They shipped a prompt change requiring five trending hashtags on every caption, and it broke structured output often enough that retries jumped from 9 percent to 34 percent. The dashboard's cost per accepted caption never moved, it sat at $0.0058 the whole time, because the formula only ever counted the call that worked. The real bill for that month came in about $4,100 higher than the dashboard implied, and by the time finance's invoice reconciliation caught it, the change had already been live on 100 percent of traffic for three weeks."
Why this works
Shows the exact mechanism, not just the conclusion that costs were higher than reported.
6
Say what you would measure going forward
Say it like this
"I'd watch retry rate and dead-fail rate as their own numbers, tagged with a reason code, right next to cost per output. And I'd gate any prompt or model rollout on the retry-rate number, not just the quality number, before it gets past a small canary."
Why this works
Shows you're designing the fix to catch the next one, not just explaining the last one.
7
Say what you'd leave alone
Say it like this
"Not every cost model needs this. Bellcrest's post-scheduling feature just writes a row to a database, there's no model call in it to retry, so a plain formula there is completely fine."
Why this works
Shows judgment instead of retrofitting retry accounting onto every dashboard in the company.
8
Close on the decision, not the story
Say it like this
"So: a cost model that only prices success is lying by omission, and it lies hardest at the exact moment something starts breaking. Cost every call, divide by attempts, and watch the failures as their own number."
Why this works
Ends on the rule, which is what a candidate can actually reuse the next time this comes up.
Let's learn
Every month, Mila Brankovic watched one number on Bellcrest's finance dashboard: cost per accepted caption, six tenths of a cent. CaptionMint is the part of Bellcrest, a social media scheduling tool, that looks at a photo someone is about to post and writes a caption and five hashtags for it in about two seconds.
Before the trouble started, CaptionMint handled about 2.4 million caption requests a month. Out of every 1,000 requests, about 910 came back right the first time. Another 75 needed one retry, usually because the model returned a caption with broken formatting, and came back fine the second time. About 15 never came back at all, no matter how many times the app retried, and the request was abandoned. The cost dashboard's formula only ever priced the call that finally worked: price per call times the number of accepted captions. It reported $0.0058 per accepted caption. Nobody questioned it, because for a year it matched the vendor's invoice almost exactly.
Knowledge spark: what's a dead-fail?
A request the app tried and retried until it gave up, with no caption ever produced. It still cost money, three tries at the model's going rate, but no caption came out the other end, so a formula built around "cost per accepted caption" never counts it at all.
Then Bellcrest shipped a prompt change: every caption now had to include five trending, currently relevant hashtags, to help posts get more reach. On the structured-output path CaptionMint depended on, stuffing in the hashtag instructions broke the model's formatting often enough that retries jumped from 9 percent of requests to 34 percent. Dead-fails, the ones that never came back at all, climbed from 1.5 percent to 6 percent. Out of every 1,000 requests, only 660 now came back clean on the first try.
The retry rate climbed for three straight weeks before anyone outside the model logs could see it.
Cost per accepted caption, reported vs real, over five weeks
Reported cost per accepted captionReal cost per accepted caption
The reported number never moved off $0.0058, because the formula only ever priced the call that worked. The real number, every call divided by every accepted caption, climbed from $0.0065 to $0.0086 in the same five weeks, about 32 percent higher.
We didn't lose money on the captions that failed. We lost it on the ones that finally succeeded, on the second or third try, and only the last try ever showed up on a bill anyone was watching.
Where the spend hid, per 1,000 requests, before and after
First-try acceptedRetried, then acceptedRetried, never accepted
Before, the retried and dead-fail slices were small, about $1.13 of the $6.41 spent per 1,000 requests. After the prompt change, those two slices alone cost more than the first-try spend did before it, and none of it ever reached the reported number.
The choice that mattered
When CaptionMint's cost dashboard was first built, the team decided in a planning meeting to price only the call that produced the accepted caption, and to leave retries and dead-fails out of the formula entirely. At launch, retries sat under 10 percent and barely moved for a year, so the shortcut cost almost nothing to make. It stopped being safe the day the team started shipping prompt and model changes that could move the retry rate on their own.
What I'd leave alone: Bellcrest's post-scheduling feature, the part that actually publishes a post at the time someone picked, is just a database write with no model call in it. There's nothing probabilistic to retry, so its simple cost formula needs no fix at all.
The lesson: a cost model that only prices success will always look calm right up until the day it matters most. The retries were never the real risk. The real risk was building a number that could not see them coming, and trusting it anyway because it had been right before.
Now here is the same thing as a story
Read the short version above for the two minute answer. Read this for why letting that formula stand for two years felt like the reasonable thing to do the whole time.
Before CaptionMint had its own dedicated PM, Mila Brankovic ran unit economics for three different Bellcrest features out of one shared spreadsheet, and she was good at it. She could smell a cost anomaly in a spend report before finance ever flagged one. When CaptionMint launched, she built its cost model herself, deliberately simple: one price, times one count of accepted captions.
The good months were good. For over a year, the dashboard's number tracked the vendor's invoice within a rounding error every single time finance checked it. CaptionMint grew from a small beta to Bellcrest's most used feature, and every quarterly review opened with the same slide: cost per accepted caption, flat, healthy, nothing to discuss.
The habit thinned in three small beats. First, as Bellcrest scaled and CaptionMint got its own small team, Mila stopped personally reconciling the dashboard number against the raw vendor invoice line by line each month, and let a "variance under 3 percent, no action needed" auto-check do it instead. Then, as the team started running quarterly experiments to improve engagement, retry rate was never put on the same dashboard as cost per caption, so a change could double it and nothing watching cost would notice. Then Mila herself moved up to overseeing three AI features instead of one, and started skimming the monthly unit-economics summary instead of opening the query behind it.
The trigger was small and it wasn't hers. A finance analyst running the routine quarterly reconciliation noticed the vendor's invoice for CaptionMint's model API came in about $4,100 higher than the internal dashboard's own numbers implied it should. Nothing on the dashboard explained where the extra money had gone. Cost per accepted caption still read $0.0058, unchanged, same as every month before it.
The dashboard hadn't gotten worse. It had simply stopped being able to see the thing that had gone wrong.
Mila spent two days with an engineer pulling the raw call logs behind the dashboard, not the dashboard itself. Six weeks earlier, a prompt change requiring five trending hashtags on every caption had shipped to a small canary, looked fine on the quality metrics the launch review actually checked, and moved to 100 percent of traffic after two weeks. Retry rate had climbed from 9 percent to 34 percent over that stretch, and dead-fails from 1.5 percent to 6 percent, invisible to the one number everyone was watching, because that number only ever priced the calls that worked.
The decision that opened the door went back to that first planning meeting, two years earlier, when pricing only the successful call felt like the obvious way to keep a new dashboard simple. Nobody wrote a rule saying the formula would need revisiting the day retries stopped being rare, because at the time nobody could picture what would make that happen.
Run the six weeks again with one change. Retry rate and dead-fail rate, tagged by prompt variant, now sit on the same canary dashboard as cost per output, with a rule that any variant crossing 20 percent retries blocks its own rollout automatically. On the replay, the hashtag prompt trips that gate on day six of the canary, at 10 percent of traffic, before it ever reaches the other 90 percent of users or shows up on an invoice anyone has to explain later.
One design trusted a number that could only describe success. The other trusted a number built to notice failure the moment it started.
What I'd tell myself, back in that first planning meeting: the day pricing only the successful call feels like a harmless shortcut, ask what happens the first time someone ships a change meant to help the product that accidentally breaks the model's formatting. Nobody asked. That's on the formula, not on Mila.
TRACE, the five moves that turn a cost anomaly into a real cause
Not a checklist for a finance meeting. Five moves that build toward the one that actually separates a real cause from a guess: the evidence test.
TTimeline. When did it actually start, and what shipped near that date?
The hashtag prompt shipped to a 10 percent canary six weeks before the invoice mismatch was caught, and it cleared the launch review's quality bar cleanly. Retry rate started climbing the same week, quietly, three weeks before it reached every user.
A timeline that starts the day the invoice looked wrong starts too late. The real start is the release that looked like a normal, healthy ship.
RRecut. Slice it by outcome and by prompt variant, and see what the average was hiding.
Company wide, reported cost per accepted caption never moved off $0.0058. Recut by call outcome, first-try spend was flat, but retried-and-dead-fail spend per 1,000 requests grew from $1.13 to $4.66, more than tripling, entirely inside the hashtag-prompt traffic.
A metric that looks perfectly flat company wide is often hiding one slice that is doing all the moving.
AAssume nothing. Rule out a billing or timeout bug before blaming the model's judgment.
Before trusting the retry numbers, Mila's team checked whether the vendor had quietly changed per-token pricing, and whether the retry counter itself had started double-logging events after a recent deploy. Neither had happened; the price was unchanged and the counter matched the raw call logs exactly.
A pricing change or a logging bug can look exactly like a real behavior change on a dashboard. Ruling it out first saves the investigation's most valuable hours.
All three looked plausible on day one. Only one of them survived the evidence test.
CCause candidates. Name three, not everything possible.
One, the model had simply gotten worse at following format instructions under normal load. Two, the new hashtag instructions specifically broke structured output on this exact prompt shape. Three, the timeout threshold, tuned for the old prompt's shorter responses, was firing before the longer hashtag-heavy response finished, causing retries on calls that would have succeeded with one more second.
Three named suspects, not a shrug. This is the step most rushed cost investigations skip, jumping straight to whichever explanation someone guesses first.
EEvidence test. The one check that tells the suspects apart.
Mila's team pulled the reason code logged with every retry, parse error, timeout, or safety rejection, and matched it against which prompt template was live on that request. 78 percent of retries logged a JSON parse error, and every one of them came from the hashtag-heavy prompt variant, not spread evenly across CaptionMint's other traffic the way a general capability regression would be. Checking the timeout-tagged retries against real response latency showed genuine timeouts, not near misses.
This single check is the strongest move in the whole method. It turned three guesses into one confirmed cause in a single query.
Three things worth stating directly, since this is where the real judgment sits. The rejected alternative was banning retries outright the moment the mismatch was found, failing fast and showing the user an error instead. It lost because it would have collapsed the accepted-caption rate and pushed people back to typing captions by hand, worse for the product than a temporarily higher true cost per caption. The AI-specific failure worth naming by name is structured-output drift under prompt pressure: a model that formats cleanly under one instruction set can start breaking format the moment a new instruction, like forcing exactly five hashtags, competes for the same output budget, and this gets more likely, not less, as prompts accumulate more requirements over time. The guardrail is tagging every retry with a reason code and watching that number by prompt variant, not just the blended cost line. And the trade being accepted on purpose is real: gating every prompt or model rollout on a retry-rate threshold adds a few extra days to any canary before it can reach full traffic, and it will occasionally hold back a change that is genuinely fine, a real cost against a formula that had already proven it could hide a 32 percent real cost increase for six weeks.
The five, in one line each: T: the real start is the release that cleared review, not the day the invoice looked wrong. R: a flat blended average can sit right on top of one prompt variant quietly driving all the cost. A: rule out a pricing or logging bug before trusting that retries actually explain the gap. C: name three real suspects, never jump straight to the first guess. E: one evidence test, the reason code matched against the prompt variant, settles it.
Same five moves, a crop app instead of a caption tool
Not every hidden cost is a marketing feature chasing engagement. Sometimes the thing that got through is a camera struggling with the light.
Agronova runs LeafPulse, a tool that reads a photo of a crop leaf from a farmer's phone and returns a diagnosis, a disease, a pest, or a nutrient gap, plus a treatment step. Njideka Aliyu leads field operations on it. LeafPulse handles about 40,000 photos a week, and its cost dashboard, built the same way CaptionMint's was, reported a flat $0.014 per completed diagnosis for months.
Agronova then shipped a "crop and enhance" preprocessing step meant to improve accuracy in the field. It worked well in daylight testing. In real use, especially photos taken near dusk when many smallholder farmers finish their rounds, it pushed low-confidence images into a retry loop far more often. Retry rate climbed from 14 percent to 46 percent. The reported cost per diagnosis never moved off $0.014, because the formula still only priced the call that finally produced a diagnosis. The real cost per diagnosis, every call counted, climbed from about $0.016 to $0.021, roughly 31 percent higher.
The evidence test that settled it
Njideka's team pulled each retry's confidence score alongside its capture timestamp. 82 percent of retries clustered between 5:30 and 6:30 in the evening, dusk light, not spread evenly across the day the way a genuine model regression would be. The preprocessing step hadn't broken; it was simply working exactly as built on photos it was never tuned for.
Same method, different shape: a hashtag prompt breaking JSON formatting and a preprocessing step struggling with dusk light look nothing alike on the surface, but both are the same TRACE move: don't trust the blended average, recut by segment, then run one test that separates a real cause from a guess.
Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to the one line: any cost model that only counts successful calls will always undercount, and the gap grows exactly when a change increases retries, which is precisely the moment you most need the number to be honest.
Cost: there's no time this sprint for a full retry-aware rebuild. Do the two cheapest steps only, log a reason code on every retry and put retry rate next to cost per output on the same dashboard, and save the full rebuild for later.
The model got better, for real: say the new model actually is more reliable and retries fall from 9 percent to 2 percent. That's exactly when a flat-formula dashboard looks most trustworthy, right before the next change quietly breaks it again, since nothing about how the number is built ever got fixed, only its current input.
Where people run it wrong.
They watch the blended cost-per-output number and never separately watch retry rate or dead-fail rate, so the dashboard cannot show them the thing that is happening.
They fix the one bad prompt variant and never change the cost formula itself, so the identical blind spot reopens the next time anything else changes what makes the model retry.
They blame the model for "getting worse" the moment retries spike, without first ruling out a pricing, logging, or timeout bug that only looks like a real regression.
How to use it live. Say the two things a cost model has to get right before saying anything else: what counts as a unit, and what happens to spend on units that fail. That buys a beat of thinking time, and it tells the interviewer you know most broken cost models fail on the denominator, not the arithmetic.
Flashcards (tap any card to flip it)
1 · THE FRAMEWORK
What framework is this, and what's its one job?
Tap to flip
ANSWER
TRACE: timeline, recut, assume nothing, cause candidates, evidence test. Built for diagnosis questions, when a number is wrong and you have to find the real cause before you can fix it.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Mila Brankovic, the PM who owns CaptionMint, the AI caption and hashtag feature inside Bellcrest, a social media scheduling tool. She built its cost model herself when it launched.
3 · THE HABIT
What did the team stop doing as CaptionMint grew?
Tap to flip
ANSWER
They stopped putting retry rate on the same dashboard as cost per caption, and let a "variance under 3 percent" auto-check stand in for Mila's own line-by-line invoice reconciliation.
4 · THE TWO NUMBERS
What are the two cost numbers this answer turns on?
Tap to flip
ANSWER
Reported cost per accepted caption, which only prices the call that worked, versus real cost per accepted caption, which prices every call including retries and dead-fails.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Building the cost formula, at launch, to price only the successful call and leave retries and dead-fails out entirely, with no rule to revisit that choice once retries stopped being rare.
6 · THE NUMBER
Fill in the blank: retry rate climbed from 9 percent to ___ percent, while reported cost per caption stayed at $0.0058 the whole time.
Tap to flip
ANSWER
34 percent. Real cost per accepted caption climbed from $0.0065 to $0.0086 in the same stretch, about 32 percent higher, invisible to the reported number.
7 · THE REPLAY
Same six weeks, new design, what changes?
Tap to flip
ANSWER
A retry-rate gate on every canary trips on day six, at 10 percent of traffic, and blocks the hashtag prompt from ever reaching the other 90 percent of users or showing up on an invoice anyone has to explain later.
8 · CROSS PRODUCT TRANSFER
Section 4 answers this same question again for a different product. Which product, and what caused the hidden cost this time?
Tap to flip
ANSWER
LeafPulse, a crop-diagnosis app at Agronova. The cause was a new preprocessing step that pushed dusk-light photos into far more retries, invisible to a cost model that only priced completed diagnoses.
Check yourself Score: 0 / 0
True or false
1. True or false: because CaptionMint's reported cost per accepted caption never moved, that proves the hashtag prompt change did not really raise Bellcrest's real spend.
True
False
Show hint
Look at the line chart in Section 1. Compare what the reported line does against what the real line does over the same five weeks.
Show answer
False. The reported number stayed flat at $0.0058 by construction, since the formula only ever priced the calls that worked. The real cost per accepted caption climbed from $0.0065 to $0.0086 in the same window, about 32 percent higher, entirely invisible to the reported number.
Multiple choice
2. In the TRACE method, what is the point of the "assume nothing" step, applied to this cost investigation?
A. To double check the model's training data one more time.
B. To interview every engineer who touched the prompt template.
C. To rule out a pricing change or a logging bug before concluding retries actually explain the gap.
D. To ban retries immediately as a precaution while the investigation runs.
Show hint
Think about what a vendor pricing change or a double-logged retry counter would look like on a dashboard, compared to a real retry-rate increase.
Show answer
C. A pricing change or a logging bug can look exactly like a genuine retry increase on a dashboard. Ruling that out first saves the investigation's most valuable hours from being spent chasing the wrong cause.
Fill in the blank
3. Fill in the blank: the evidence test found that ___ percent of retries logged a JSON parse error, and every one of them came from the hashtag-heavy prompt variant.
Show hint
The number is in Section 3's evidence-test step and repeated in the hand-sketch diagram.
Show answer
78 percent. That concentration, all inside one prompt variant instead of spread across CaptionMint's other traffic, is what ruled out a general model regression and confirmed the hashtag prompt as the real cause.
Short answer, name the rejected alternative
4. What alternative fix did Mila's team consider and reject once they found the cost gap, and why did it lose?
Show hint
Look at the paragraph right after the evidence-test step in the TRACE recap section.
Show answer
Model answer: They considered banning retries outright, failing fast and showing users an error instead. It lost because it would have collapsed the accepted-caption rate and pushed people back to typing captions by hand, worse for the product than a temporarily higher true cost per caption.
Multiple choice
5. At LeafPulse, 82 percent of retries clustered between 5:30 and 6:30 in the evening. What does that pattern tell you?
A. Farmers who submit photos at dusk are simply worse at using the app.
B. A time-of-day cluster this strong points to a lighting cause, not a model capability regression.
C. The reported cost per diagnosis proves the preprocessing step is working correctly.
D. LeafPulse should stop accepting photos in the evening entirely, with no further check.
Show hint
Compare what a lighting-driven cause would look like against what a genuine model regression, spread evenly across the day, would look like.
Show answer
B. A genuine model regression would show up evenly across capture times. An 82 percent cluster in one narrow window is exactly the kind of recut TRACE's R step exists to find, and it traced back to dusk light, not a broken model.
Short answer, apply it yourself
6. Pick an AI product you use yourself that has some kind of usage-based cost or credit system. What's one thing that could be quietly failing and retrying behind the scenes, that its reported cost per use would never show you?
Show hint
Think about a product where a single visible action, like getting a finished result, could actually hide several unseen attempts behind it.
Show answer
Model answer: An AI photo-editing app that charges credits per finished edit. If the model sometimes returns a malformed or blank image and silently retries before showing the user anything, the credits charged per finished edit would never reveal how many attempts it actually took to get there, or how the real compute cost is drifting.
Before you close the answer
Why this works
Tests whether you assume "cost per successful output" is a safe metric, or whether you know it's structurally blind to its own failures. Most candidates suggest tracking retries too, without seeing that the existing metric can't even show you it needs fixing.
Follow-up traps
"Isn't logging a reason code on every retry just extra engineering work for a small edge case?" Response: retries weren't small once, they went from 9 percent to 34 percent in six weeks with zero warning from the number that was supposed to catch it. That's not an edge case, that's the exact scenario the model exists to defend against.
"Couldn't you just cap retries at one and call it done?" Response: capping retries doesn't fix the cost model, it only shrinks the size of the blind spot. The fix is making the formula count every call, not lowering how much it's allowed to miss.
If pressed
The evidence test that separated a real model regression from a prompt-specific bug was the retry reason code, cross-referenced against which prompt template was live. 78 percent of retries logged a JSON parse error, all traced to the hashtag-heavy variant, not spread evenly across traffic the way a genuine capability regression would be.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
