ConceptIntermediateModel Fluency & the AI PM Role / The AI literacy baseline every PM needs / #15
What is prompt caching and what product decision does it enable?
ORDER · which caching-enabled feature Answerbind builds first for Nashira Trailware's support team
Glasswick Labs builds Answerbind, a tool that drafts customer-support email replies. Nashira Trailware, an outdoor-gear retailer, is a client. Escher Fitzwilliam runs Nashira's support team of 22 agents, who draft around 900 replies a day. The week Glasswick's model provider ships prompt caching, Escher asks for two things at once: the full 34-page policy playbook in every draft, and a tone profile for each of his agents. Bevaline Wrexley, Glasswick's Head of Product, has one engineering pod for the quarter to decide which one is actually ready.
The direct answer
Prompt caching lets a model reuse a big, unchanged block of a prompt across many requests, so it only has to fully read and think through that block once, not on every call. At Glasswick, that makes it affordable to put the whole 34-page policy playbook into every single draft, since the playbook barely changes and every ticket reads the identical copy. Build that first, cached as one stable block. Hold the per-agent tone profiles: they change too often and differ by agent, so caching gives them almost nothing, and placed carelessly, they can even break the playbook's own cache.
Do this, in order
Build full playbook grounding first, cached as one stable prefix.Why: it's the only one of the two features that clears caching's own bar today.
Hold the per-agent tone profiles.Why: 11 of 22 profiles had already changed within a month, so the shared, stable block caching needs isn't actually there.
Never merge the unstable tone block into the shared playbook prefix.Why: a changing block placed before a stable one breaks the cache for everything that comes after it, not just for itself.
Measure how much of a real prompt is actually static before committing to any caching-based feature.Why: it's the cheap check that gave Bevaline the real numbers before a line of code got written.
Build a way to notice when a cost-driven default has outlived its own excuse.Why: the trimmed playbook sat unused for two months after caching made it affordable, because nothing flagged that its reason for existing was gone.
Leave the weekly ticket-volume digest email alone.Why: it's a plain database query with no model call in it, so caching has nothing to do with it.
How to answer this, stage by stage
Nobody is grading whether you can define prompt caching in a sentence. They're grading whether you can name a real decision it unlocks, and defend which feature earns the quarter and which one only looks ready.
1
Scope it to one product and one real week
Say it like this
"Let's ground this in one real week. Glasswick Labs builds Answerbind, a tool that drafts support-email replies. Nashira Trailware is a client. Escher Fitzwilliam runs their support team, and he wants two things built the same week caching ships. I've got one pod for the quarter."
Why this works
A named product and a real ask stop the answer from floating in the abstract.
2
Answer the literal question first, before ranking anything
Say it like this
"Prompt caching means the model reuses a big block of a prompt it's already seen, so it doesn't have to read through that whole block again on the next call, it just picks up from where the cached part left off. That's what makes it newly cheap to put something huge and shared, like a full policy document, into every request."
Why this works
The interviewer asked what prompt caching is. Answer that in plain words before the framework starts, or the rest sounds like a dodge.
3
Say your structure out loud
Say it like this
"I'll run this as ORDER. Outcome, what we're actually protecting. Reversibility, which choice is hardest to undo once it's built. Dependency, what has to be true for caching to actually pay off. Evidence, what's cheap to check first. Rank, the real call, defended."
Why this works
Two seconds of structure shows a method before an opinion about two features.
4
Name the outcome, so the ranking isn't a guess
Say it like this
"Both features are chasing the same thing: whether the reply an agent sends is actually right, and whether it costs Glasswick less to draft than before. Not 'ship more AI.' A reply that gets the return policy right, or a bill that doesn't quietly creep up."
Why this works
Naming the outcome first is what stops the ranking from being whichever feature Escher asked for loudest.
5
Give the ranked call, committed
Say it like this
"So here's the call. Put the whole 34-page playbook into every draft, as one cached prefix, this quarter. Hold the per-agent tone profiles. They don't clear the bar caching actually needs."
Why this works
This is the direct answer, spoken plainly, before a single number gets in the way.
6
Prove it with the dependency check, and name what you rejected
Say it like this
"We measured it before writing any code. The playbook changes about once a quarter, and every one of Nashira's 900 daily tickets reads the exact same copy, that's what a cache actually needs. The tone profiles are the opposite: 22 different blocks, and 11 of the 22 had already been edited in the past four weeks. I also looked at merging both into one cached prefix, and rejected it, since a changing block placed early in a prompt breaks the cache for everything after it, including the playbook."
Why this works
A real measurement beats an argument about which feature feels more ready, and naming the rejected option before they ask shows real judgment.
7
Close on the one line anyone could check
Say it like this
"So: full playbook, cached, ships this quarter, and it should cost about what the old trimmed summary cost, just with nothing missing. Tone profiles wait until a version of them changes on a fixed schedule instead of whenever a manager edits one. You'll know it's working when the cache hit rate stays above 90 percent, not when someone says the drafts feel smarter."
Why this works
Ends on something an interviewer could actually go verify, not just a confident-sounding pick.
Let's learn
What actually changes when a model can remember most of a prompt instead of rereading it every time?
Answerbind is Glasswick Labs' tool for drafting customer-support email replies. An agent opens a ticket, Answerbind reads it, and hands back a drafted reply the agent edits and sends. To get that reply right, every draft is supposed to lean on Nashira Trailware's own support playbook: a 34-page document covering returns, warranties, shipping, and brand voice.
Knowledge spark: what does "caching a prefix" actually mean?
A prompt is just the text sent to the model. Caching lets the model store its own work on the first part of that text, the part that stays the same call after call, so it doesn't have to redo that work next time. It only has to think through whatever's new. The catch: the stored part has to match exactly, from the very start, or the cache doesn't apply at all.
Same document, sent again and again. Only the ticket text at the end is ever actually new.
Before caching existed, sending the full 34-page playbook on every single draft was too slow and too costly to run all day. So eight months ago, when Answerbind first launched at Nashira, Glasswick's engineers trimmed it down: a 3-page quick-reference summary, hand-written, covering the common cases. It ran fast and cheap. It also quietly dropped some of the real detail, including the rule that a discontinued item loses its free-return-shipping window six weeks after it's pulled from the catalog.
None of this was careless. It was the only version of the playbook that was ever fast enough to use on every ticket.
Here's the turn. The week Glasswick's model provider ships prompt caching, Escher asks for two things at once, and both look, to him, like the same kind of free. He wants the full playbook back in every draft. He also wants each of his 22 agents to get their own tone profile, a short writing-style sample, so replies sound like the agent who supposedly wrote them. "Big prompts are cheap now," he tells Bevaline. "So let's do both."
Cost per drafted reply, three prompt designs
Cached, the complete playbook costs almost the same as the old trimmed summary always did. It's the uncached middle bar that was never affordable to run on every ticket.
What it costs at its worst: ship both blindly, and the tone-profile idea doesn't just fail to save money, it can quietly cost more than never caching anything at all. Worse, if an agent's tone block sits ahead of the playbook in the prompt, a common way to build it since "it's about who's writing," it breaks the cache for the playbook too, so every draft for that agent goes back to the slow, expensive, uncached path without a single error message telling anyone.
Caching a prompt does not make everything in it cheap. It makes the part that stays the same cheap, and charges full price for the part that never does.
The choice I would take back
Eight months ago, before caching existed, Glasswick's engineers trimmed the playbook to 3 pages and hardcoded it as Answerbind's default, to hit a cost target that mattered then. That was the right call at the time. Caching shipped platform-wide two months before Escher's near miss, and nobody built any way to notice that the trimmed default had already outlived the reason it existed.
What I would leave alone: the weekly digest email that sends Escher a summary of ticket volume by category doesn't touch the model at all. It's a plain database query. Caching, prompt design, none of it applies there.
The lesson: "big prompts got cheap" is not the same claim as "every big prompt got cheap." Caching has its own requirement, that the shared part is genuinely stable and genuinely shared, and that has to be checked per feature. It doesn't come free just because the platform shipped it once.
Now here is the same thing as a story
The short version above is what you actually say in the room. Read this one for the two months a stale default sat quietly on top of an answer that had already changed.
Escher Fitzwilliam has run support at Nashira Trailware for four years, and he can tell a genuine warranty complaint from a customer fishing for a discount before he's finished the first line. His newest hire is quicker than that, but she's careful, the kind of careful that comes from being six weeks into a job where every wrong reply has her name on it.
Answerbind arrived at Nashira eight months ago. For most of that time it did the small, unglamorous thing well: drafts came back in about two seconds, built off a 3-page quick-reference summary, and an agent could edit one in under a minute instead of writing a reply from scratch. Nobody complained. The trimmed summary covered the cases that came up every day, and the ones it didn't cover were rare enough that an experienced agent caught them from memory.
Two months ago, without much fanfare, Glasswick's model provider turned on prompt caching. Bevaline's team quietly started using it for a few internal tools. Nobody went back and asked whether Answerbind's own trimmed default, built for a world where big prompts were slow and expensive, still made sense in a world where it wasn't either of those things anymore. The playbook stayed trimmed. Nobody had a reason to look.
Then, on a Tuesday, the newest hire pulled up a return request for a jacket. Answerbind's draft looked right, the kind of reply she'd approved a dozen times that week: apologize, confirm the return, note the free shipping label. She was one click from sending it when something nagged at her. The order date looked old. Older than most returns she saw.
Nothing in the draft, or the 3-page sheet behind it, told her to stop. She stopped anyway, and got lucky that she did.
She checked the product page by hand. The jacket had been marked final sale, no returns, six weeks earlier, when Nashira discontinued the color. The 3-page quick-reference summary, unchanged since launch, had never been updated to say so. Answerbind's draft, built entirely from that summary, had no way to know either.
She caught it. Nothing shipped wrong that day. But she told Escher, and Escher called Bevaline the same afternoon, not angry, just done trusting a tool that had almost cost Nashira a returned jacket and a policy exception nobody approved.
"Fix it," Escher said. "Put the whole playbook in. And while you're at it, give my agents their own tone profiles too. You told me big prompts are cheap now."
We didn't almost lose one jacket's return shipping. We almost proved that a fast, cheap answer and a right answer had quietly stopped being the same thing, eight months ago, and nobody had checked back since.
Bevaline didn't say yes to both. On a call two days later, she walked Escher through what her team had actually measured. The full playbook, the real one, changes about once a quarter, when legal or ops updates a policy. Every one of Nashira's 900 daily tickets would read the exact same copy of it. That's the shape caching wants: one big, boring, unchanging block, reused constantly.
The tone profiles were a different shape entirely. Twenty-two of them, one per agent, and when Bevaline's team pulled a month of edit history, 11 of the 22 had already changed, sometimes because a coaching session flagged something, sometimes because an agent just wanted to sound less formal. A block that changes every couple of weeks, read by only one agent's forty-odd tickets a day, doesn't behave like the playbook. It behaves like something being rewritten out from under the cache before the cache ever gets to pay for itself.
Here's the decision Bevaline would take back, and it isn't the trim itself. Eight months ago, with no caching and a real cost problem, trimming the playbook to 3 pages was the sensible call. What nobody built was a way to notice, later, that the reason for the trim had quietly disappeared. The default just kept running, unexamined, for two months after it stopped needing to be a default at all.
Run that Tuesday again, with the full playbook already cached as the default. The newest hire's draft mentions the reclassification by name, because the model actually read the real policy this time, not a 3-page summary six months out of date. She still pauses at the order date, because that's good instinct and it should stay. This time the draft agrees with her instead of leaving her to catch a mistake it never should have made.
The two designs, side by side: one hands every agent a fast answer that quietly stopped being a complete one, and hopes an experienced eye catches the gap. The other makes the complete answer just as fast, so nobody has to be the last line of defense against a document nobody remembered was outdated.
What I'd tell myself, watching that first trim get shipped eight months ago: a cost-saving shortcut that's right on the day you build it doesn't stay right forever, and if nothing is watching for the day it stops being right, you find out from a near miss instead of a dashboard.
ORDER, for spending the one caching-shaped quarter you get
PICK would fit if this were only two options and an even trade. But Escher wants both, eventually, and the real question is which one actually earns the cached prefix first. That's ORDER's job.
OOutcome. What the rank actually has to protect.
Both features are competing for the same thing: whether a reply Nashira's agents send is actually right, and whether it costs Glasswick less to produce than it did before caching existed. Not "ship more AI." A reply that gets a reclassified item right, or a monthly bill that goes down instead of quietly creeping up.
Name the outcome before ranking either feature. Skip this and the ranking is just whichever feature Escher asked for first.
The second box is the one the tone profiles never actually reach.
RReversibility. Which choice is hardest to undo.
A small, targeted pilot of per-agent tone profiles, tested on two or three agents, is easy to cut. Nobody's replies stop working, they just go back to the shared default voice. Full playbook grounding is different. Once it's the default prefix every draft is built around, and agents get used to a fully grounded reply, ripping it back out means redesigning how every single draft gets built.
This is why the order matters, not just a preference. One mistake resets for two agents. The other one is load-bearing for the whole product the moment it ships.
Reversibility isn't a reason to avoid the harder one. It's a reason to be sure of it first.
DDependency. What has to already be true.
Caching's discount only applies to a block that's genuinely identical across many requests, read from the very start of the prompt. The playbook clears this: one document, changed about once a quarter, read the same way by all 900 of Nashira's daily tickets. The tone profiles don't. 22 separate blocks, 11 of 22 already edited within four weeks, each reused only by one agent's own tickets. Caching a block that keeps changing doesn't discount it, it just adds the extra cost of rebuilding the cache entry over and over.
This is why the order isn't a guess about which feature Escher wants more. One candidate's dependency is already true. The other one's isn't, no matter how badly anyone wants it to be.
EEvidence. What's cheap to check first.
Before committing the quarter, Bevaline's team pulled a week of real Answerbind requests and measured them token by token: about 13,000 of them are the identical playbook text, and about 600 are the ticket itself, always new. Then they pulled a month of tone-profile edit history and found 11 of 22 already rewritten. Both checks together, done before a line of code, told them exactly which feature actually qualifies.
Cheap, and it's the number that actually settled it, not whichever feature Escher wanted shipped by Friday.
Cache hit rate on Answerbind's full-playbook prefix, first two weeks live
Percent of drafts that hit the cached prefix
Day one is slow because the cache is still warming up. By day ten it's doing what it's supposed to, on the one feature whose dependency actually held.
RRank. The actual call, defended.
Build full playbook grounding this quarter, as a single cached prefix, ticket text appended strictly after it. Hold the per-agent tone profiles until a version of them changes on a fixed, predictable schedule instead of whenever a manager edits one, closer to the playbook's own quarterly cadence.
If this rank would be identical with a different outcome in the O step, say "ship whatever Escher asks for fastest," it was picked by pressure, not judgment. Change the outcome to "grow the account fastest, cost be damned," and the rank still holds, because a feature that can quietly cost more than no caching at all doesn't grow anything. That's how you know the rank is real.
Three things worth stating directly, since the real judgment sits here. The alternative worth naming and rejecting is merging both blocks into one combined cached prefix, playbook and tone profile together, so "everything gets cached at once." It loses because a changing block placed ahead of a stable one breaks the cache for both, turning a feature that should have saved money into one that reliably wastes it. The AI-specific failure worth naming is a silent cache miss: a prefix quietly stops matching because something upstream changed by even a few tokens, and every draft reverts to the slow, expensive, uncached path without a single error firing. The guardrail is watching the cache hit rate itself as a real metric, the same line this chart shows, so a regression shows up as a number dropping, not as a bill that just feels a little higher this month. And the trade-off is accepted on purpose: the very first ticket of each day pays close to the full uncached price and takes close to the full uncached time to build the cache fresh, in exchange for every other ticket that day costing a fraction of that and coming back in about a second.
And if you want to be sure it really works, try it somewhere else
Same five letters, a planning office instead of a support inbox, and this time the thing that isn't actually stable is one inspector's own shorthand.
Sparrowmere Civic Systems runs OrdinanceLine for the Town of Cricklewell's planning office, a tool that drafts answers to permit-application questions, grounded in the town's full zoning code. Ianthe Pennoyer, a senior permit reviewer, wants two things built on the same caching win: the complete zoning code in every drafted answer, instead of the trimmed one-page cheat sheet reviewers currently work from, and a personal-notes feature that folds each inspector's own annotation style into their drafts.
Different office, different document, the same shape of mistake waiting to happen.
Same steps, mapped onto Cricklewell. Outcome: protect whether a permit answer actually matches the real code, and whether it costs the town less to produce than the reviewers' current manual lookups. Reversibility: a small pilot of personal-notes injection for two reviewers is easy to cut; the full zoning code as the default prefix, once every drafted answer depends on it, is not. Dependency: the zoning code changes a handful of times a year and every request reads the identical text, a strong caching candidate; each inspector's personal notes change whenever they annotate a new case, sometimes daily, so the shared block caching needs was never really there. Evidence: a week of real requests shows the code is about 90 percent of every prompt's tokens and never varies within that week, while personal notes vary on almost every single request. Rank: cache the full zoning code as the default this quarter, hold personal-notes injection until it's rebuilt as something closer to a fixed reference than a live scratchpad.
Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: cache the block that's actually the same across requests, hold the one that changes per person, because a shared dependency doesn't care how fast anyone wants to move.
Cost: no budget to properly stabilize the changing block this quarter. Ship a visible note saying it isn't cached yet, instead of quietly caching something that won't discount anything.
The model got better, for real: say the underlying model gets meaningfully more accurate. Keep the same caching gate anyway. A better model can still be handed an unstable prefix, and better was never the same claim as cacheable.
Where people run it wrong.
They cache the feature that got asked for loudest, not the one whose prefix is actually stable.
They assume "caching shipped" means every big prompt in the product is now cheap, instead of checking each one.
They watch the monthly bill instead of the cache hit rate, so a silent regression shows up as a vague cost creep nobody can trace.
How to use it live. Before ranking anything, ask out loud: "which of these prompts is actually the same, word for word, across many requests, and which one changes under us?" Whichever answer comes first is usually the order an interviewer is actually listening for.
Flashcards (tap any card to flip it)
1 · THE FRAMEWORK
Which framework fits deciding which caching-enabled feature Answerbind should build first?
Tap to flip
ANSWER
ORDER: outcome, reversibility, dependency, evidence, rank. Built for ranking real candidates by what's hardest to undo and what's actually ready, which is exactly what "which feature first" needs.
2 · THE CAST
Who holds each role in this story, and where do they work?
Tap to flip
ANSWER
Bevaline Wrexley is Head of Product at Glasswick Labs, which builds Answerbind. Escher Fitzwilliam runs support for Nashira Trailware, the client, with 22 agents drafting about 900 replies a day.
3 · THE OUTCOME
What does the rank actually have to protect?
Tap to flip
ANSWER
Whether a reply is actually right, and whether it costs less to draft than before, not just whether the platform "has caching." A reply that gets a reclassified item right, or a bill that goes down instead of creeping up.
4 · THE DEPENDENCY
What has to already be true before a shared prefix is worth caching?
Tap to flip
ANSWER
The block has to be genuinely stable and read identically by many requests. The playbook clears this (quarterly changes, 900 identical daily reads). The tone profiles don't: 22 separate blocks, 11 of 22 edited within a month.
5 · THE OLD DECISION
What decision would you take back, and why did it make sense at the time?
Tap to flip
ANSWER
Trimming the playbook to 3 pages and hardcoding it as the default, eight months ago, to hit a cost target before caching existed. Right then; nobody built a way to notice it had outlived that excuse two months after caching shipped.
6 · THE NUMBER
Fill in the blank: cached, the full playbook costs about $___ a draft. Left uncached, it costs about $___ a draft.
Tap to flip
ANSWER
About $0.0057 cached, about $0.041 uncached, roughly seven times more before caching applied.
7 · THE RANK
State the final call, defended in one line.
Tap to flip
ANSWER
Full playbook grounding ships this quarter, cached as one stable prefix. Tone profiles wait until they change on a fixed schedule instead of whenever a manager edits one, because an unstable block can cost more than no caching at all.
8 · CROSS-PRODUCT TRANSFER
Section 4 runs ORDER again on a different product. Which one, and what plays the role of the feature that fails the dependency check there?
Tap to flip
ANSWER
OrdinanceLine, Sparrowmere Civic Systems' permit-drafting tool for Cricklewell's planning office. Per-inspector personal notes play the failing role, changing too often and varying per person the same way the tone profiles did.
Check yourself Score: 0 / 0
True or false
1. True or false: the fix for the tone-profile idea is simply to turn caching on for it too, the same way the playbook was cached.
True
False
Show hint
Check the Dependency (D) step in the ORDER recap.
Show answer
False. Caching only discounts a block that's genuinely stable and shared. Turning caching on for a block that keeps changing doesn't fix that, it can add the cost of rebuilding the cache entry over and over.
Fill in the blank
2. Cached, the full playbook costs about $___ a draft. Left uncached, the same full playbook costs about $___ a draft, roughly seven times more.
Show hint
Check the bar chart under Section 1.
Show answer
$0.0057 and $0.041. Same document, same model, the only thing that changed is whether the model had to reprocess it from scratch.
Multiple choice
3. Why can't Escher get both the full playbook grounding and the per-agent tone profiles at the same low, cached price?
A. Glasswick's model provider only allows one cached prefix per company account.
B. Only one of the two has a prefix that's actually stable and shared across many requests, which is what caching's discount depends on.
C. Nashira's support team is too small to justify building two features at once.
D. Per-agent personalization is technically impossible with a shared model.
Show hint
Check the Dependency (D) step and the 11-of-22 number.
Show answer
B. The playbook is one stable block read identically by 900 tickets a day. The tone profiles are 22 different blocks, most edited within a month, so the shared, stable prefix caching needs isn't there.
Short answer, name the old decision
4. What old decision would Bevaline take back, and why did it make sense when it was made?
Show hint
Look at the "choice I would take back" key point in Section 1.
Show answer
Model answer: Trimming the playbook to a 3-page quick reference and hardcoding it as Answerbind's default, eight months ago, to hit a cost target before caching existed. It made sense then; nobody built a way to notice it had outlived that excuse once caching shipped two months before the near miss.
Short answer, apply it yourself
5. Think of an AI product you use or have built where a big, shared block of context could be cached. What's the block, and is it actually stable enough to benefit?
Show hint
Look for the block that's identical across many requests, versus one that gets rebuilt per request.
Show answer
Model answer: A coding assistant that includes a whole style guide in every request. That's a good caching candidate if the guide only changes occasionally and every request in a session reads the identical copy. It's a bad one if it gets rebuilt per request from files that just changed.
Short answer, work the number
6. If Nashira's ticket volume tripled to about 2,700 a day, would the same ranking, full playbook first, tone profiles held, still make sense? Why or why not?
Show hint
Ask which candidate's dependency actually changes when volume changes.
Show answer
Model answer: yes, and it gets stronger. The playbook's dependency doesn't change with volume, it's still one stable prefix, now reused by even more requests, which only helps its economics. The tone profiles still fail their own dependency regardless of ticket count, since the problem is how often each profile changes, not how many tickets read it.
Before you close the answer
Why this works
Tests whether you understand caching as a mechanism with its own real requirements, not just a general claim that AI got cheaper. Most candidates can say caching saves money. Naming which feature actually qualifies, and which one only looks like it does, is the part that shows real judgment.
Follow-up traps
"Why not just shorten the tone profiles so they're cheap to reprocess every time, cached or not?" Response: that's a fair idea, but it answers a different problem. The issue isn't the profile's size, it's that a short, changing block placed before the playbook still breaks the playbook's own cache. The fix is ordering, variable content after the stable prefix, not shrinking.
"What if the model provider changes its caching discount later, doesn't that break the whole plan?" Response: yes, and that's exactly why the reversibility step matters here. The shared prefix stays one clean, swappable block instead of something merged into per-agent logic, so if pricing changes, only that one block's economics need rechecking, not 22 separate ones.
If pressed
Glasswick's cache entries expire automatically after about five minutes of no use. That's exactly why the first ticket after a quiet stretch, the start of a shift, the return from lunch, always pays the full cache-write price again, no matter how stable the playbook itself is.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
