ConceptFoundationalAI Opportunity & Model Strategy / Build vs buy vs fine-tune decisions / #1
Lay out the decision tree for build, buy, prompt, retrieve or fine-tune.
ORDERthe cheapest, most reversible option got tested first, and the model everyone assumed they needed never got built
VetBridge sells practice software to veterinary clinics. PawNote is meant to turn a vet's rushed shorthand notes into a clear after-visit summary a pet owner can read on their phone. Marisol Kwiek is the AI PM who has to decide how to build it, and Denholm Ayeni is the engineer whose one question in a planning meeting reordered the whole decision.
The direct answer
Climb the ladder in order, cheapest and most reversible first. Try prompting a general model against a real eval set. If it clears the bar, ship it. If it is close but missing your own facts, add retrieval before anything heavier. Only fine-tune once retrieval still cannot hold steady and real data actually exists. Buy when the capability is a commodity and your data is not the edge. Build from scratch last, only when the capability is genuinely core to what makes you different and nothing above it clears the bar.
Do this, in order
Test plain prompting first, against a real eval set, before anything else.Why: it is the cheapest, fastest, and most reversible way to find out if the problem even needs more.
Add retrieval over your own documents only once prompting alone shows a real gap.Why: retrieval fixes a missing-facts problem without committing to a model you cannot easily undo.
Confirm real, usable proprietary data exists before fine-tuning becomes an option at all.Why: fine-tuning without enough clean examples just bakes the same gap in for good.
Buy when the capability is a commodity, unless your own data is the actual edge.Why: paying a vendor for a solved problem beats months rebuilding it, though it can lock you into their terms.
Build custom only when the capability is genuinely core, and nothing above it clears the bar.Why: it is the most expensive and hardest option to walk back, so it earns the last slot, not the first instinct.
Rank every option by how expensive it is to reverse, not by how impressive it sounds in a roadmap review.Why: this is the actual reasoning the whole tree runs on, the part that survives a follow-up question.
How to answer this, stage by stage
Nobody is scoring whether you can name five technologies. They're scoring whether you have a real order to test them in, instead of picking whichever one sounds most advanced in the room.
Stage 1
Scope it to one real feature, not "AI strategy" in the abstract
Say it like this
"Let me ground this in one real case. VetBridge sells practice software to vet clinics. PawNote is supposed to turn a vet's rushed shorthand notes into a clear after-visit summary a pet owner can actually read. Before anyone picks a technology for it, I run the same five-option ladder every time."
Why this works
Keeps the answer from turning into a general lecture on build versus buy with no real feature behind it.
Stage 2
Say your structure out loud before any content
Say it like this
"I'll run this as ORDER. Outcome, what every option is actually competing to move. Reversibility, which one is cheapest to undo if it's wrong. Dependency, what has to happen before the next option even makes sense. Evidence, what I can learn cheaply before real budget gets spent. Rank, the actual order, and why."
Why this works
Signals a repeatable method for choosing a build path, not a one-off gut call dressed up as strategy.
Stage 3
Reframe the question: this isn't "which is best"
Say it like this
"This isn't a question of which of the five is the best technology. It's a question of which one you're allowed to try first, and what has to actually fail before you earn the right to reach for the next, more expensive one."
Why this works
This is where a strong answer separates from a list of five bullet points with no real order behind them.
Stage 4
Give the one decision: the actual tree, in order
Say it like this
"Here's the order I'd actually run. Prompt a general model against real cases first. If it clears the bar, stop there. If it's close but missing our own facts, add retrieval. Only fine-tune once we've confirmed real data exists and retrieval still can't hold steady. Buy if it's a solved problem elsewhere and our data isn't the edge. Build from scratch last, only if this is genuinely core to what makes us different."
Why this works
This is the direct answer, stated as an actual sequence you could run this afternoon, not a static diagram.
Stage 5
Prove it with the compressed evidence
Say it like this
"On a hundred and twenty real visit notes, plain prompting alone was fully correct sixty one percent of the time. Adding retrieval over our own dosing guide pushed that to eighty four. Fine-tuning on our own fourteen thousand historical notes, on top of retrieval, reached ninety six, which cleared the bar we'd set for anything touching medication instructions."
Why this works
This is where the story lives, compressed to the three real numbers that actually justified each step up the ladder.
Stage 6
Name the AI-specific reasoning and the trade-off being accepted
Say it like this
"The honest reason retrieval alone wasn't enough is that PawNote has to hold the same tone and format across thousands of visits, not just get one fact right once. A wrong dosage frequency in a summary is a hallucination a pet owner could act on at home, so we accepted a slower release and real data-curation cost, in exchange for a summary that stayed consistent instead of merely plausible."
Why this works
This is the load-bearing judgment. It wouldn't make sense to ask this about a feature with no model in it, since the risk is specifically a model sounding confident while getting a real fact wrong.
Stage 7
Say what skips the whole ladder, then close on one line
Say it like this
"I wouldn't run this whole ladder for something like suggesting appointment reminder times. That's a commodity problem any scheduling vendor already solves well, so it goes straight to buy. For PawNote, the order held all the way through: prompt, then retrieve, then fine-tune. Buy and build never even got a real turn."
Why this works
Closes with real judgment about when to skip the ladder entirely, and restates the direct answer in one breath.
Let's learn
PawNote is a feature inside VetBridge, software vet clinics already use to run their practice, that turns a vet's rushed shorthand notes into a clear after-visit summary a pet owner gets by text.
Before the ladder existed, a feature's technology got picked the same way every time: by whichever pitch sounded most impressive that week.
Before this ladder existed, VetBridge picked a technology for each AI feature by whichever option sounded most advanced in the room, usually whatever a vendor had just demoed or a conference talk had just praised. Nobody had a real number for what plain prompting alone could already do.
The actual tree, in the order it gets tested, not the order the five words usually get listed in.
Accuracy on the 120-case eval set, by approach
Prompt only+ Retrieval+ Fine-tuneBuy, unmodified
Each rung only got tested once the one before it fell short of the 95 percent bar. Fine-tuning wasn't picked because it sounded serious, it was picked because it was the first one to actually clear the line.
Here's the turn: the mistake was never picking the "wrong" one of the five. The mistake was committing to one of them, usually the most expensive one, before the cheaper ones had ever been tested, so a bad bet only got discovered after months of sunk cost instead of after a few days.
We weren't choosing the best technology. We were choosing which one had earned the right to be tried first.
Where PawNote actually sat mattered more than how advanced fine-tuning sounded. A generic, commodity problem never gets this far up the ladder at all.
Knowledge spark: why doesn't a better prompt just fix this on its own?
A prompt can tell a model what to say. It can't hand the model facts it was never shown, like a specific clinic's own dosing guide. Retrieval hands it those facts at the moment it answers. Fine-tuning goes further and reshapes how the model behaves by default, useful when the facts alone aren't the whole gap, the tone and format need to hold steady too.
Cumulative cost to reach a working result, actual path versus a straight-to-build path
Actual path, climbed the ladderIf we'd defaulted straight to build
The straight-to-build path costs more by week one than the real path ever spent in total, and still hadn't shipped by month six.
At its worst, skipping the ladder costs exactly what the dashed line shows: real months and real budget spent building the most expensive option, only to discover afterward that a cheaper rung would have cleared the bar just as well.
The choice I would take back
Letting earlier AI features default to whichever option sounded most serious, usually fine-tuning or a full custom build, without first testing plain prompting. That made sense when the team assumed the more advanced-sounding option was automatically the safer bet. It stopped making sense the moment a two-day prompting test could have answered the same question for a fraction of the cost.
What I would leave alone: a generic capability like call transcription or appointment scheduling doesn't need this ladder run in full. It's a solved, common problem with no real data advantage on VetBridge's side, so it goes straight to buy without testing the cheaper rungs first.
The lesson: the five options aren't five equally likely answers, they're five rungs on a ladder, ordered by how expensive each one is to walk back. Test the cheap, reversible ones first and let the real evidence decide how far up you actually need to climb.
Now here is the same thing as a story
The short version above is what you'd say out loud in the room. Read this one for what it actually felt like to almost commit six months and real money before anyone tested the cheapest option.
Marisol Kwiek had been VetBridge's AI PM for two years, long enough to have watched a few features get scoped around whichever technology sounded most advanced in a planning meeting, rather than whichever one had actually been tested. PawNote's kickoff meeting was headed the same way.
"Vets spend real time on this," Marisol said, opening the meeting. "About nine minutes a visit, times fourteen visits a day, just writing up a summary the owner can actually understand." Heads nodded. Someone on the team, fresh off a conference talk, suggested fine-tuning a model on VetBridge's own historical notes right away. It sounded like the serious, grown-up choice.
Denholm's question, drawn as the order it actually forced on the room.
Denholm Ayeni, an engineer two seats down, asked the question that reordered everything: "Have we even tried just prompting it first?" Nobody had. There was no eval set, no baseline, nothing to measure fine-tuning against except a feeling that it was the more impressive option.
Marisol asked for two days before committing to anything. The team pulled a hundred and twenty real, already-reviewed visit notes and ran them straight through a general model with a careful prompt, nothing else. Sixty one percent came back fully correct. Not nothing, genuinely surprising for zero extra engineering, but well short of what a medication instruction needed to be trusted unsupervised.
The real reason prompting went first was never that it was the best option. It was that it cost nothing to be wrong about.
We weren't testing whether Marisol's team could build a good model. We were testing whether they even needed to yet.
Adding retrieval, pulling in VetBridge's own dosing guide and a handful of past correct summaries as grounding, took three more weeks and pushed the same test to eighty four percent. Closer, but the team had set ninety five percent as the real bar for anything touching medication instructions, and eighty four wasn't it.
Marisol never had a fixed rule for exactly when a gap justified fine-tuning versus living with retrieval alone. It came down to a feeling with two settings: the gap is about missing facts, which retrieval fixes, or the gap is about consistency itself, which only fine-tuning reliably closes. Eleven percent short, and the errors weren't random facts, they were the model's tone drifting on the exact visits that needed the most precision. That was the second setting.
The shipped summary was never one technology's answer. It was retrieval and fine-tuning, each doing the specific part it was actually good at.
Back when the fine-tuning suggestion first came up in that kickoff meeting, reaching for the most advanced-sounding option wasn't an unreasonable instinct, the team genuinely believed a bigger commitment meant a safer bet. It stopped being reasonable the moment a two-day prompting test could answer the same question for almost nothing.
Here's the replay: fourteen thousand historical notes, already sitting in VetBridge's own systems, got curated and used to fine-tune a smaller model on top of the same retrieval setup. The retest hit ninety six percent, six weeks after the original kickoff, not six months. Buy and build never needed a real evaluation, because the ladder answered the question before either one got a serious look.
One version of this story spends six months and four hundred thousand dollars custom-building a model, then discovers retrieval alone would have gotten most of the way there. The other spends two days on a prompt, three weeks on retrieval, and six weeks on a fine-tune that was only reached for once the cheaper rungs had genuinely run out of runway.
What I'd tell myself, hearing Denholm's question land in that first meeting: the order you test things in is its own decision, and skipping straight to the option that sounds the most serious is usually just skipping the part where you find out you didn't need it.
ORDER, run on a decision everyone wanted to skip straight to the expensive end ofNot a script for always picking the cheapest option. ORDER is what makes sure the expensive one only gets picked once it's actually earned.
O
Outcome. What is every option actually competing to move?
A PawNote summary a pet owner can trust for medication timing, at a cost VetBridge can actually justify. Not "which technology is newest," a real, measurable bar: 95 percent correct on a real eval set.
Without a named outcome, ranking five options is just opinion dressed up as strategy.
R
Reversibility. Which option is cheapest to walk back if it's wrong?
Prompting: a new prompt in an afternoon. Retrieval: swap a document, no retraining. Fine-tuning and build: real weeks of committed engineering and data work. Buy: a real contract, harder to unwind once integrated.
This is the actual reasoning behind the whole tree, cheap-to-undo options get tried before expensive-to-undo ones, every time.
D
Dependency. What has to be true before the next rung even makes sense?
Fine-tuning only becomes real once two things exist: a measured gap from testing retrieval first, and enough real, clean proprietary data, VetBridge's fourteen thousand notes, to actually close it.
Skipping a dependency doesn't just risk failure, it means you can't even tell if the expensive option was worth it.
E
Evidence. What could the team learn cheaply before committing real budget?
A two-day prompting test against 120 real cases, then a three-week retrieval test against the same cases. Both ran before a single dollar went toward fine-tuning.
Cheap evidence, gathered before the expensive commitment, is what turned a guess into a real, ranked decision.
R
Rank. State the order, and defend the top pick in one line.
Prompt, then retrieve, then fine-tune. Buy and build stayed available the whole time but never got picked, because nothing earlier in the ladder failed badly enough to need them.
This is the direct answer, stated as the order actually run, not a diagram nobody tested.
The recap, one line per letter: outcome is the real 95 percent bar the whole tree is aimed at, reversibility is why cheap-to-undo options go first, dependency is what has to exist before fine-tuning is even allowed to be considered, evidence is the cheap tests run before real budget got spent, and rank is the order that actually shipped: prompt, retrieve, fine-tune, with buy and build never needed.
And if you want to be sure it really works, try it somewhere elseSame five letters, a municipal permitting company instead of a vet-clinic tool. This time the ladder stops one rung earlier, at buy.
Thaddeus Okonkwo is the AI PM at Permitly, which sells software cities use to process building-permit applications. A reviewer team asks for a tool that pre-checks a new application for missing required documents before a human ever opens it. Mapped onto ORDER: outcome is a pre-check that catches real missing documents without wrongly rejecting complete applications. Reversibility ranks the same way, prompting first, retrieval next, fine-tuning and build last. Dependency is different here: Permitly quickly finds that missing-document rules are nearly identical across most cities, a genuinely common, well-documented problem, not a proprietary one. Evidence comes from a two-week test where a plain prompt plus retrieval over each city's own checklist already hits 93 percent, and a specialist vendor already selling exactly this check at a fair price matches it at 94 percent, with none of the ongoing maintenance. Rank ends one rung earlier than PawNote's: buy, because the problem is common enough that Permitly's own data was never going to be the edge, and fine-tuning would have spent real money closing a gap that barely existed.
Same ladder, same order tested, but Permitly's climb stops two rungs earlier than VetBridge's did.
Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Skip straight to "prompt first against a real eval set, then retrieve, then fine-tune, buy or build only once those genuinely fall short," and stop.
Cost: no time to test retrieval before a decision is due. Say so honestly, and commit to the two-day prompting test as the very next step, not a guess at which rung to skip to.
The model got better, for real: say a future general model clears 95 percent on prompting alone, with no retrieval or fine-tuning at all. Rerun the ladder anyway, a better model just means the climb stops one rung earlier, it doesn't change the order.
Where people run it wrong.
They pick the option that sounds most advanced in the room, instead of the cheapest one that hasn't been tested yet.
They fine-tune before confirming real, clean proprietary data actually exists to fine-tune on.
They treat buy and build as opposites instead of two ends of the same reversibility scale, both usually belong later in the order, not first.
How to use it live. The moment an interviewer asks you to choose between build, buy, prompt, retrieve, or fine-tune, ask yourself first: which of these five is cheapest to undo if I'm wrong? That question alone buys real thinking time, and it's usually exactly where the honest order starts.
Flashcards (tap any card to flip it)
1 · THE FRAMEWORK
What framework fits laying out a build, buy, prompt, retrieve, or fine-tune decision?
Tap to flip
ANSWER
ORDER: outcome, reversibility, dependency, evidence, rank. It ranks the five options by what's cheapest to test and hardest to undo, instead of picking by which sounds most advanced.
2 · THE PEOPLE
Who is this answer about?
Tap to flip
ANSWER
Marisol Kwiek, the AI PM who runs PawNote's build decision at VetBridge. Denholm Ayeni is the engineer whose question, "have we even tried just prompting it," reorders the whole meeting.
3 · THE ORDER
What's the actual ranked order, cheapest and most reversible first?
Tap to flip
ANSWER
Prompt, then retrieval, then fine-tune, with buy and build weighed in parallel whenever the capability might be a commodity or a genuine core edge.
4 · THE DEPENDENCY
What has to be true before fine-tuning is even a real option?
Tap to flip
ANSWER
A measured gap from testing retrieval first, plus enough real, clean proprietary data to actually close it. Without both, fine-tuning is a guess, not a decision.
5 · THE OLD DECISION
What old decision does this answer take back?
Tap to flip
ANSWER
Letting earlier features default to whichever option sounded most serious, usually fine-tuning or a full build, without testing plain prompting first. Reasonable when a bigger commitment felt safer. Wrong once a two-day test could answer it for almost nothing.
6 · THE NUMBER
Fill in the blank: plain prompting scored ___ percent on the eval set, retrieval pushed it to ___ percent, and fine-tuning reached ___ percent.
Tap to flip
ANSWER
61 percent, then 84 percent, then 96 percent. Each rung only got tried once the one before it fell short of the 95 percent bar.
7 · THE REPLAY
Same request, tested in order instead of skipped straight to the expensive end. What changes?
Tap to flip
ANSWER
PawNote ships in six weeks for under 100,000 dollars, having climbed prompt, retrieval, and fine-tune in order, instead of six months and 400,000 dollars spent building a model nobody had confirmed was needed.
8 · CROSS PRODUCT TRANSFER
Section 4 runs ORDER again on a different product. Which one, and how does the outcome differ?
Tap to flip
ANSWER
Permitly's permit-application pre-check. Unlike PawNote, the climb stops one rung earlier, at buy, because the problem turns out to be common across cities rather than a place where Permitly's own data was the real edge.
Check yourself Score: 0 / 0
True or false
1. True or false: VetBridge fine-tuned a model before ever testing plain prompting.
True
False
Show hint
Look at Denholm's question in the story, and what happened in the two days after it.
Show answer
False. A general model was tested against 120 real cases first, scoring 61 percent, well before retrieval or fine-tuning were even tried.
Multiple choice
2. Why did the team add retrieval before ever considering fine-tuning?
A. Retrieval is cheaper to build and fully reversible if it doesn't close the gap, unlike a trained model.
B. Fine-tuning wasn't technically possible on VetBridge's own systems.
C. VetBridge's legal team required it before any model work could start.
D. Retrieval is always more accurate than fine-tuning, in every case.
Show hint
Look at the Reversibility step in the ORDER recap.
Show answer
A. Retrieval costs far less to build and undo than a trained model, so it earns a turn before anything that commits real weeks and real data curation.
Fill in the blank
3. Fill in the blank: plain prompting alone scored ___ percent on the 120-case eval set, and retrieval pushed that to ___ percent.
Show hint
Look at the bar chart in "Let's learn."
Show answer
61 percent, then 84 percent. Both real, measured numbers, and both still short of the 95 percent bar the team had set for medication-related content.
Short answer, where it wouldn't matter
4. Name a feature at VetBridge where this whole ladder would NOT need to be run in full, and say why not.
Show hint
Look at "what I would leave alone."
Show answer
Model answer: Appointment scheduling or call transcription. Both are common, well-solved problems elsewhere, and VetBridge's own data isn't the real edge on either one, so they can go straight to buy.
Short answer, apply it yourself
5. Think of an AI feature idea you've heard pitched. Run it through the first two rungs: would plain prompting likely clear the bar, or would it need retrieval on top?
Show hint
Think about whether the feature needs facts specific to one company, or just general reasoning a public model already knows.
Show answer
Model answer: A pitch for an AI tool that answers employee HR questions found plain prompting handled general policy questions fine, but needed retrieval over the company's own actual handbook to get specific numbers like vacation-day counts right.
Short answer, work the number
6. If fine-tuning had only reached 90 percent instead of 96, would VetBridge's decision to stop climbing the ladder still make sense?
Show hint
Compare 90 percent against the 95 percent bar the team had actually set for anything touching medication instructions.
Show answer
Model answer: No, not on its own. 90 percent still misses the stated bar, so the honest next move would be gathering more real training data before shipping, not settling for a number below the line just because it's the highest rung tried so far.
Before you close the answer
Why this works
Tests whether you have a real, repeatable order for choosing an implementation path, ranked by cost and reversibility, instead of picking whichever option sounds the most technically impressive in the room.
Follow-up traps
"Isn't testing every rung just slower than picking the right one up front?" Response: the whole ladder, prompt through fine-tune, took six weeks and under 100,000 dollars, versus a projected six months and 400,000 dollars for defaulting straight to a custom build, testing in order was the faster path, not the slower one.
"What if the model provider changes its pricing or gets deprecated?" Response: that's exactly why prompting and retrieval sit at the reversible end of the ladder, swapping providers there costs an afternoon, while a fine-tuned model or a full custom build carries real migration cost either way.
If pressed
The fine-tuned model still runs retrieval underneath it, not instead of it. Fine-tuning fixed tone and format consistency, but the actual dosing numbers still come from the live formulary lookup, so a drug-guidance update takes effect immediately instead of waiting on the next retraining cycle.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
