How does prompt caching change your unit economics, and when does it not help?

A prompt that matches the cached block everywhere except one added timestamp does not get a smaller discount. It gets none at all, and the bill quietly goes back to what it cost before caching ever shipped.

How to answer this, stage by stage

Nobody is grading whether you can define prompt caching. They are grading whether you know its discount is a switch, exact match or full price, and not a dial that just gets a little smaller.

Let's learn

Pellwick is the chat inside Kelmscott, a scheduling and invoicing tool for small service businesses. When a customer has a billing question, a wrong charge, a plan change, a refund, Pellwick answers it instead of the ticket sitting in a queue.

Before Pellwick, every billing question waited an average of six hours for a person to answer it. About one in five customers who filed a billing ticket cancelled their plan before anyone got back to them.

Pellwick answers most billing questions in under a minute now, and the support queue behind it shrank by more than half. But the part nobody outside engineering was watching was cost, because Pellwick's prompt carries a big shared block on every single call, six thousand tokens of plan tables, refund rules, and policy text, the same for every customer.

That drop is not the story here. The story is what happens the day the shared block stops matching, word for word, and nobody notices, because caching does not fail loud.

This is the whole answer to when it doesn't help. Close is not a discount. Close is full price.

At its worst, this costs more than never caching at all. Every call still pays a small fee to write a fresh cache entry, on top of full price for the block, and that fee only pays for itself if the exact same block gets read again before it expires. Miss that, and caching is a tax, not a discount.

In the review Meret Kranz, the finance partner who checks Kelmscott's vendor spend every quarter, ran that week: cost per resolved ticket sitting at one point one cents, almost exactly what it cost the year before caching ever existed, on a feature everyone still believed was cutting cost in half.

What I'd leave alone: Pellwick's rare conversations, the ones dragging a customer's whole account history into the prompt ahead of the cutoff. That block barely repeats from one customer to the next, so caching it would only add the write fee, never earn it back.

The lesson: caching doesn't wear down. It either matches or it doesn't, and the moment it stops matching, the bill goes straight back to what it cost before anyone turned caching on. Watching total spend told Kelmscott caching was probably fine, for eleven weeks, while it was already off.

Now here is the same thing as a story

Read the short version above when you're in the room. Read this one when you want to feel why Ilaria stopped opening a dashboard that was, by every number on it, still fine.

Ilaria Alderton built Kelmscott's model cost dashboard herself, three years before Pellwick ever existed, back when the company's first, simpler support bot only needed one chart to make sense of what it cost each month.

The week Pellwick's caching shipped, she opened the dashboard every single Monday morning, coffee in hand, and watched the cost per ticket line drop and hold. Ninety one percent of calls hit the cache. The line stayed flat and low for months, exactly the way a fix is supposed to look once it's actually working.

By month two, she'd stopped checking it first thing and checked it after lunch instead. By month four, she'd glance at it maybe every other Monday. By month six, she genuinely could not remember the last time she'd opened it on purpose, only because something else had made her look.

That April, a small tracing ticket came in, nothing to do with cost at all: could the support team tell which build of Pellwick had answered a given customer. An engineer added one line, a live timestamp, right at the top of the prompt, ahead of the shared block, so every log entry would carry it. It shipped on a Tuesday. Nobody flagged it for cost review, because it changed nothing Pellwick ever said out loud.

The habit didn't fade slowly. It just stopped, the week the line had held flat long enough that checking felt pointless.

Cache hit rate fell from ninety one percent to three percent within the hour, and stayed there. Every call that used to cost half a cent went back to costing more than a cent, plus a small fee for writing a cache entry that, because of that timestamp, would never be read again. Nobody saw it happen, because nobody was still opening the dashboard.

Pellwick kept answering billing questions exactly as well as it had the day before. Nothing a customer saw ever changed. The only thing that changed was a number on a dashboard nobody was opening, and a running total that kept climbing anyway, for a reason that had nothing to do with it.

Eight months earlier, in the meeting where Pellwick's alerts first got built, someone asked whether to alert on total daily spend or on cache hit rate itself. Total spend was already being logged for the monthly vendor report, so it was the free option. Hit rate would mean building a new chart from nothing, one more thing to maintain. Total spend won, reasonably, because back then Pellwick's ticket volume was flat, and any real problem would have shown up in the total within a day or two.

Ilaria didn't catch it. Meret Kranz did, three months after the timestamp shipped, during a routine quarterly review of vendor spend. She noticed cost per resolved ticket sitting at one point one cents, almost exactly what it cost the year before caching existed, and asked Ilaria why a feature that was supposed to be cutting cost in half wasn't showing up anywhere in the number.

Ilaria found the timestamp in about twenty minutes once she went looking. Run the same Tuesday again, with one change: a hit rate alert instead, set to page if the number drops below seventy percent for more than fifteen minutes. The same timestamp ships, the same silent break happens, and the page goes out twelve minutes later, before a single full day of full price billing has passed. Eleven weeks of paying full price becomes twelve minutes of it.

One design tracked the thing caching was supposed to save. The other tracks the thing caching actually does, whether the exact block still matches, and only one of those two designs can tell you the moment it stops.

What I would tell myself, back in that first alerts meeting: the moment a discount depends on an exact match, ask what a broken match looks like on a dashboard that only watches the total, because it looks like nothing at all, right up until someone outside engineering notices the bill didn't move the way it should have. Nobody asked. That's on the room, not on Ilaria.

FLIPS, and the one letter with no partial credit

Not five guesses about what could go wrong with a discount. FLIPS names the one habit that snapped, and asks which old choice made snapping the only option.

Five steps. Only the I step has no middle setting to fall back on.

Three things worth stating directly, since this is where the real judgment sits. The alternative the team could have tried first was tightening the total spend alert's threshold, paging on a smaller jump instead of building a second chart. It loses because a climbing spend line can mean two very different things, more tickets at the same price, or the same tickets paying full price instead of a cached one, and a dollar total alone can't tell those apart. The AI specific failure worth naming by name is silent cache invalidation: a code change put a different token ahead of the cache breakpoint, and nothing about Pellwick's answers to customers ever looked wrong, so the transcript level view stayed clean while the price underneath it quietly went back to full. The guardrail is the hit rate alert itself, plus one more rule: nothing gets added to the front of the prompt without a canary check on hit rate before it ships, not after. That guardrail isn't free. Building and maintaining a hit rate chart costs more engineering time than the one spend alert Kelmscott already had for the vendor report, a real trade accepted on purpose, because paying full price silently for eleven weeks costs far more than the extra dashboard ever would. And the bar it enforces was never a hundred percent hit rate, a cache going cold after its time limit expires is normal and expected. It's a probability bar, checked against a real drop: the alert pages when hit rate clears an unusual fall, below seventy percent for more than fifteen minutes, not a promise that every single call reuses the cache.

And if you want to be sure it really works, try it somewhere else

Same five letters, a company that will never see a support ticket, and this time the shared block isn't a paragraph of billing policy. It's a rate table stapled to every quote a technician sends.

Renhurst Mechanical is a heating and cooling company. Rafter is the assistant its technicians use from a tablet in the truck to draft written quotes on site. Sioned Sallow is the ops manager who used to type every quote by hand before Rafter, the standard labor rates, the licensing text, the warranty language, then the specific diagnosis for that one furnace.

The case for handing it over: for its first five months, Rafter cut the average time to send a quote from twenty two minutes to about four. Cost per quote fell fast too, because the two thousand one hundred token rate table and warranty block at the top of every quote was identical across every technician and every job, so it hit the cache almost every time.

The case against it: about one job in six is not routine. A furnace under an old manufacturer clause, or installed under a code Renhurst rarely sees anymore, needs a different line inserted into the middle of that same rate table, not added after it, because the clause changes which labor rate applies. Once a technician typed that clause in, the block stopped matching, word for word, for that job and every one after it that needed the same clause.

A technician quoting a forty year old furnace, still under a clause Renhurst rarely deals with anymore, spent nine minutes inserting the right language into the middle of the rate table. That one quote cost about six times what a routine quote cost, full price for the whole block, plus a cache write fee that was never going to be read again. Nobody thought it was a bug. It looked like a hard job costing more, which felt right.

Sioned's fix wasn't a smarter alert, since nothing here was hidden the way Pellwick's break was, everyone could see the quote taking longer. It was a scope decision: Rafter now runs on maintenance visits and standard tune ups, where the rate table never gets touched, and technicians go back to typing custom clauses by hand for older equipment. Routine quotes still cache near ninety percent. The custom ones cost what they cost, honestly, instead of quietly eating a write fee for a block that was never going to be read twice.

Same rank as before: cache the part that's genuinely identical every time, and know which jobs don't qualify before you run them through it. Rafter's overall program cost fell about forty percent once the scope narrowed, even with the same total number of jobs going through it.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to the fix, cache the exact block, put anything that changes after the cutoff, never inside it.
Cost: there's no budget this quarter to build a hit rate alert. Sample it, checking hit rate by hand on twenty calls a week beats reading a total spend line that can't tell a discount apart from more volume.
The model got better, for real: say Pellwick's underlying model gets upgraded to a newer version. That's not proof the cache is still matching. A better model can answer just as well on a full price call as a cached one, so it hides a broken cache even more completely.

Where people run it wrong.
They read a healthy total spend line as proof caching still works, and never check hit rate on its own.
They let something get inserted ahead of the cache breakpoint, a timestamp, a request id, a customer note, without checking whether it breaks the match.
They cache a block that's only ever going to be read once, and count the write fee as a mystery loss instead of noticing it was never going to pay for itself.

How to use it live. Say the split out loud before answering: "is this block genuinely identical every time, or does it just look that way." Naming the split buys a beat to work out which one caching actually needs, instead of guessing out loud in front of the interviewer.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0