ConceptIntermediateAI Opportunity & Model Strategy / Build vs buy vs fine-tune decisions / #17
Describe how vendor lock-in shows up in an AI stack specifically.
SPARKthe model was always swappable. what was never swappable was everything underneath it
Halyard Capital lends to small businesses. MemoDraft is meant to draft a credit memo from a loan file, pulling numbers a loan officer currently copies by hand. Ottavia Renwick is the AI PM designing it, and Frobisher Callendar is the engineer whose story from a previous job made the real cost of lock-in impossible to ignore.
The direct answer
Vendor lock-in in an AI stack shows up in four specific places: your fine-tuned model weights, which usually only run on that vendor's own serving infrastructure; your embeddings, generated by one vendor's model and unreadable by another's; your prompts and eval harness, hand-tuned to one model's exact quirks; and your raw training data, once it's uploaded into a vendor's fine-tuning tool with no clean way to export it. Keep your own copy of the data, the evals, and the extraction logic outside any single vendor's walls, so only the model sitting on top is ever the hard part to swap.
Do this, in order
Own your extracted data and eval set outside any vendor's platform.Why: this is the one layer that, once locked in, is the most expensive to ever get back.
Export your embeddings regularly, or regenerate them from source on your own schedule.Why: embeddings tied to one vendor's model become unreadable the moment you switch, unless the source text is still yours to re-embed.
Keep prompts and eval cases documented as your own artifact, not buried inside a vendor's console.Why: a prompt tuned to one model's quirks has to be rewritten anyway when you switch, but only if you can find and read it first.
Treat fine-tuned weights as the one piece you accept is locked in.Why: retraining is real work regardless of ownership. The goal isn't zero lock-in, it's containing it to the one layer that's genuinely hard to avoid.
Don't build a full multi-vendor switching system on day one.Why: that's real engineering effort spent on a problem you don't have yet. Portability of data and evals is the cheap insurance; a live failover system is not.
How to answer this, stage by stage
Nobody is scoring whether you can recite "avoid vendor lock-in." They're scoring whether you can name the four specific places it actually lives.
Stage 1
Scope it to one real build, with a real precedent behind it
Say it like this
"Let's ground this. Halyard Capital is building MemoDraft, an AI credit-memo drafter. Ottavia owns the design. Two years ago, Halyard's earlier AI tool, PulseWatch, took eighteen weeks and two hundred ten thousand dollars just to leave one vendor. That's the real cost we're designing against."
Why this works
Grounds "vendor lock-in" in a real, numbered event instead of a general warning nobody can act on.
Stage 2
Say your structure out loud before any content
Say it like this
"I'll run this as SPARK. Situation, how the work gets done today. Payoff, the habit we want the feature to build. Anchor, the one design decision everything hangs on. Risk, what breaks if that anchor's wrong. Keep out, what we deliberately don't build yet."
Why this works
Signals a forward design method, not a retrospective complaint about a vendor that burned you once.
Stage 3
Reframe the question: lock-in isn't one thing, it's four
Say it like this
"People usually talk about 'vendor lock-in' like it's one risk. It's actually four separate ones, your weights, your embeddings, your prompts and evals, and your raw data, and each one gets locked in a different way, for a different reason."
Why this works
This is where the answer separates from a vague warning into something you could actually design against.
Stage 4
Give the one decision: the anchor
Say it like this
"Here's the anchor. MemoDraft's extracted loan data and its eval set of real memos both live in Halyard's own database, in a plain format, completely outside any vendor's platform. Whichever model drafts the actual memo text sits on top of that layer, and it's the only part we ever accept might be hard to swap."
Why this works
This is the direct answer, made inspectable as a real design decision instead of a stated principle.
Stage 5
Prove it with the compressed evidence
Say it like this
"PulseWatch's migration broke down like this: a hundred forty thousand dollars retraining weights that couldn't be exported, forty thousand reprocessing six hundred thousand documents' worth of embeddings, twenty thousand rewriting prompts tuned to the old model, ten thousand in contract overhead. MemoDraft's design, with the data and evals already portable, projects the same kind of switch at about twenty five thousand dollars and three weeks."
Why this works
Gives the interviewer a real cost breakdown, and a real projected number for the alternative design.
Stage 6
Name the AI-specific reasoning and the trade-off being accepted
Say it like this
"The honest reason this is specifically an AI-stack problem is that a model's fine-tuned weights, its embeddings, and its tuned prompts are all artifacts of training and interacting with one specific model, they don't transfer to another one by definition. We're accepting a bit of extra engineering now, keeping our own copies and our own eval harness, in exchange for the model tier being the only genuinely hard-to-swap piece later, instead of everything being hard to swap."
Why this works
This is the load-bearing judgment. It only makes sense because embeddings and fine-tuned weights are inherently tied to the model that produced them, unlike a generic API integration you could always rebuild from documentation.
Stage 7
Say what's deliberately not built yet, then close on one line
Say it like this
"I wouldn't build a live, automatic multi-vendor failover system on day one, that's real complexity for a problem MemoDraft doesn't have yet. What we build now is just the portable data and eval layer. Vendor lock-in in an AI stack always comes down to those same four places, own the data and the evals, and accept the model itself as the one piece you'll eventually retrain."
Why this works
Closes with real judgment about what's deliberately deferred, and restates the direct answer in one breath.
Let's learn
MemoDraft is meant to draft a credit memo, the document a loan officer prepares before a small-business loan gets approved, by pulling numbers from a loan file instead of a person copying them out by hand.
Ninety minutes of copying numbers between three systems, for every single memo, before MemoDraft existed.
Two years before MemoDraft, Halyard built PulseWatch, a churn-prediction tool, entirely inside one vendor's fine-tuning platform. It worked well. Nobody thought much about what would happen if Halyard ever needed to leave.
Four separate locks, not one. PulseWatch had all four, and nobody had checked any of them until it was time to leave.
Knowledge spark: why can't embeddings just move to a new vendor?
An embedding is a specific model's own internal representation of a piece of text, a long list of numbers that only means something relative to that same model's other embeddings. A different model produces a completely different, incompatible set of numbers for the exact same text, so switching vendors usually means regenerating every embedding from the original source documents, not just copying a file over.
PulseWatch's real migration cost, broken down
WeightsEmbeddingsPrompts and evalsContract overhead
Two thirds of the cost was retraining weights that were never designed to be exportable in the first place.
The anchor isn't a policy. It's a specific, inspectable decision about where the data actually lives.
The model was always swappable. What was never swappable was everything Halyard had let a vendor quietly own underneath it.
Here's the turn: the real cost of PulseWatch's lock-in wasn't the vendor being bad. The vendor was fine. The cost was that nobody had ever asked, while things were going well, what would happen the day Halyard needed to leave.
Cumulative migration cost over time, PulseWatch's design versus MemoDraft's
PulseWatch's actual pathMemoDraft's projected path
The gap between these two lines is entirely the cost of where the data and evals were allowed to live.
The choice I would take back
Building PulseWatch entirely inside one vendor's fine-tuning platform, with no local copy of the training data, the embeddings, or the eval set. That made sense at the time, since moving fast mattered more and nobody expected to need an exit. It stopped making sense the moment a price increase actually forced the question.
What I would leave alone: Halyard's plain rules-based fraud-flagging checklist, no model involved at all, needs none of this thinking. There's nothing to fine-tune, embed, or export.
The lesson: vendor lock-in in an AI stack is never one risk, it's four specific ones, and three of the four are avoidable with almost no extra cost if you decide to own them from day one instead of after a bill forces the question.
Now here is the same thing as a story
The short version above is what you'd say out loud in the room. Read this one for what it actually felt like designing against a cost Halyard had already paid once.
Ottavia Renwick had been at Halyard through PulseWatch's entire migration, eighteen weeks of retraining, reprocessing, and rewriting, just to leave a vendor whose only real offense was raising prices. She remembered exactly how reasonable the original decision had felt: get the churn model shipped fast, worry about portability later, if ever.
Each milestone was its own small, ugly surprise. Together, they added up to a number nobody had budgeted for.
Frobisher Callendar, an engineer who'd joined Halyard from a different fintech, brought up his old company's version of the same story in MemoDraft's kickoff meeting: they'd built a document classifier the same way, all-in on one vendor, and hit an almost identical wall when that vendor changed its pricing tiers eighteen months in. "It wasn't even the same vendor," Frobisher said. "It was the same mistake."
Either way this bet goes, the cost stays small. That's what made it an easy anchor to commit to.
Two different companies, two different vendors, the exact same four locks, discovered the exact same way.
Ottavia never had a fixed rule for exactly how much extra engineering effort was worth spending against a risk that might never materialize. It came down to a feeling with two settings: either the cost of staying portable was real and ongoing, or it was a small, one-time setup that paid for itself the moment it was ever needed. Sketching out MemoDraft's data layer took an extra week and a half. PulseWatch's actual migration had taken eighteen. That wasn't a close call.
Not every kind of portability was worth building immediately. The data layer was. A live failover system wasn't, yet.
Back when PulseWatch was first built, skipping any thought of portability wasn't an unreasonable call, the team was small, the deadline was real, and nobody had a specific reason yet to expect a vendor problem. It stopped being reasonable the moment a price increase turned "we could leave if we wanted to" into "we need to leave, and can't."
Here's the replay: MemoDraft ships with its extracted loan data and its eval set of real, reviewed memos sitting in Halyard's own database from day one. The model drafting the memo text is the only piece anyone expects to eventually swap. When Frobisher's old worry inevitably gets tested, a future vendor change, a price hike, a shutdown, MemoDraft's version of that migration is projected at three weeks and twenty five thousand dollars, not eighteen weeks and two hundred ten thousand.
One version of this story builds fast, trusts the vendor completely, and discovers the real cost only the day it needs to leave. The other spends a week and a half up front making sure that day, whenever it comes, is a minor inconvenience instead of a crisis.
What I'd tell myself, sitting through PulseWatch's actual migration: the vendor being good was never the question. The question was always what happens to your data and your evals the day you need them back, and that question is cheapest to answer before you've ever needed the answer.
SPARK, in one screenNot a script for distrusting every vendor relationship. SPARK is what makes sure the exit, if it's ever needed, is small and boring instead of an 18-week emergency.
S
Situation. How does the work get done today, without this feature?
A loan officer spends about 90 minutes per memo, manually copying numbers from three separate systems, the origination platform, bank statement PDFs, and a credit bureau report.
Grounding the design in one real task, not an abstract "AI strategy," is what keeps the anchor concrete later.
P
Payoff. What habit do we want this feature to build?
Loan officers stop manually re-typing extracted numbers into a memo template, and start reviewing and adjusting an auto-drafted one instead.
Naming the habit, not just the time saved, is what the whole design is actually built around.
A
Anchor. What's the one design decision everything else hangs on?
Own the extracted data and the eval set outside any vendor's platform, in Halyard's own database, with only the model tier itself sitting on top and treated as swappable.
This is the hardest, most load-bearing step, and the direct answer to the whole question.
R
Risk. What breaks the first time this bet is wrong?
If Halyard never actually needs to switch vendors, the anchor cost about a week and a half of extra setup, small and fully absorbable, compared to PulseWatch's actual 18-week, $210,000 migration.
The anchor survives being unnecessary cheaply, which is what makes it a safe bet either way.
K
Keep out. What do we deliberately not build yet?
A full, automatic multi-vendor failover system that can silently switch models. That's real complexity for a problem MemoDraft doesn't have until a second vendor is actually chosen.
Naming what's deferred shows judgment, not a wish list of every possible protection at once.
The recap, one line per letter: situation is the 90 minutes a loan officer spends copying numbers by hand, payoff is trading manual re-typing for reviewing an auto-draft, anchor is owning the data and eval layer outside any vendor's walls, risk is that the anchor costs little even if it's never needed, and keep out is skipping a full failover system until it's actually justified.
And if you want to be sure it really works, try it somewhere elseSame five letters, a farming co-op instead of a lender. This time the anchor protects photo labels, not financial data.
Rosalind Kettleworth is the AI PM at Cloverun Farms Co-op, designing BlightScan, a tool that flags likely crop disease from a photo a field worker takes on a phone. Mapped onto SPARK: situation is a field worker currently texting a photo to an agronomist and waiting up to two days for a diagnosis. Payoff is the field worker getting a same-day flag instead of waiting, with the agronomist only reviewing the uncertain cases. Anchor is similar in shape to MemoDraft's but protects a different asset: Cloverun keeps using a convenient third-party labeling tool day to day, but insists on a nightly export of every raw label into its own storage, so years of accumulated ground truth never live only inside the labeling vendor's platform. Risk is the same low-cost bet, the nightly export costs almost nothing to run whether or not Cloverun ever changes labeling tools. Keep out is different here: Cloverun doesn't build its own labeling interface from scratch on day one, that would be solving a problem the convenient vendor tool already handles well; the anchor only protects the data underneath it.
The nightly export costs almost nothing to run. Skipping it would have quietly erased two years of real ground truth, roughly 42,000 labeled photos, the moment the labeling vendor's export feature ever disappeared or changed hands.
Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Skip straight to "own your data, your evals, and your prompts outside the vendor's walls, the model itself is the only piece you accept might be hard to swap," and stop.
Cost: no engineering time to build a portable layer before launch. Say so honestly, and commit to at minimum a scheduled raw-data export as the cheapest version of the same protection.
The model got better, for real: say the vendor's next model is dramatically more accurate. Keep the anchor anyway, a better model today says nothing about that vendor's pricing or terms two years from now.
Where people run it wrong.
They treat "avoid lock-in" as a single, vague principle instead of naming the four specific places it actually shows up.
They build a full multi-vendor abstraction layer on day one, spending real engineering effort on a problem they don't have yet.
They assume a good vendor relationship today means portability was never worth the setup cost, right up until a price change or a shutdown proves otherwise.
How to use it live. The moment an interviewer asks about vendor lock-in in an AI stack, don't reach for a general warning. Name the four places it actually lives, weights, embeddings, prompts and evals, and raw data, and say which one you'd accept as the cost of doing business.
Flashcards (tap any card to flip it)
1 · THE FRAMEWORK
What framework fits designing an AI stack against vendor lock-in?
Tap to flip
ANSWER
SPARK: situation, payoff, anchor, risk, keep out. It's the design framework, run forward, since this is fundamentally a "design against a known failure" question, not a perturbation with a FLIPS flip family.
2 · THE PEOPLE
Who is this answer about?
Tap to flip
ANSWER
Ottavia Renwick, the AI PM designing MemoDraft at Halyard Capital. Frobisher Callendar is the engineer whose story from a previous job confirms the lock-in pattern isn't a one-off.
3 · THE ANCHOR
What's the one design decision everything else hangs on?
Tap to flip
ANSWER
Own the extracted data and the eval set outside any vendor's platform, in Halyard's own database, treating only the model tier itself as swappable.
4 · THE FOUR LOCKS
What are the four specific places vendor lock-in shows up in an AI stack?
Tap to flip
ANSWER
Fine-tuned weights, embeddings, prompts and eval harnesses, and raw training data uploaded with no export path. Each locks in for a different reason.
5 · THE OLD DECISION
What old decision does this answer take back?
Tap to flip
ANSWER
Building PulseWatch entirely inside one vendor's fine-tuning platform with no local copy of data, embeddings, or evals. Reasonable when speed mattered most. Wrong once a price hike forced an exit.
6 · THE NUMBER
Fill in the blank: PulseWatch's real migration took ___ weeks and cost $___.
Tap to flip
ANSWER
18 weeks and $210,000. Two thirds of that cost was retraining weights that were never designed to be exportable.
7 · THE REPLAY
Same kind of vendor change, hitting MemoDraft's portable design instead of PulseWatch's. What changes?
Tap to flip
ANSWER
The projected migration drops to about 3 weeks and $25,000, since only the model tier needs swapping, the data and eval layer never has to move at all.
8 · CROSS PRODUCT TRANSFER
Section 4 runs SPARK again on a different product. Which one, and what does the anchor protect there instead?
Tap to flip
ANSWER
Cloverun Farms Co-op's BlightScan. There, the anchor protects a labeled photo archive with a nightly export, rather than financial data, but the same low-cost-bet logic applies.
Check yourself Score: 0 / 0
True or false
1. True or false: the main cost of PulseWatch's migration came from the model producing worse predictions after the switch.
True
False
Show hint
Look at the stacked bar chart breaking down the migration cost.
Show answer
False. The cost came from retraining weights, reprocessing embeddings, and rewriting prompts, none of which was about the new model's quality.
Multiple choice
2. Why can't embeddings simply be copied over to a new vendor's model?
A. Embeddings are always deleted automatically after 90 days.
B. An embedding only means something relative to the specific model that generated it, so a different model produces incompatible numbers for the same text.
C. Embeddings require a special license to move between companies.
D. Vendors legally own any embeddings generated on their platform.
Show hint
Look at the knowledge spark on embeddings.
Show answer
B. Embeddings are model-specific representations, not portable data, so switching models usually means regenerating them from the original source text.
Fill in the blank
3. Fill in the blank: MemoDraft's projected migration cost, with a portable data and eval layer, is about $___, compared to PulseWatch's real $210,000.
Show hint
Look at the cumulative cost line chart.
Show answer
$25,000. Reached by week 3 and flat afterward, since only the model tier itself needs to be swapped.
Short answer, where it wouldn't matter
4. Name a tool at Halyard where this vendor lock-in concern would NOT apply, and say why not.
Show hint
Look at "what I would leave alone."
Show answer
Model answer: The plain rules-based fraud-flagging checklist, with no model involved. There's nothing to fine-tune, embed, or export, so lock-in doesn't apply.
Short answer, apply it yourself
5. Think of an AI tool you've used or heard about that was built entirely on one vendor's platform. Which of the four locks, weights, embeddings, prompts, or raw data, would be hardest for that team to escape?
Show hint
Think about whether the tool involved fine-tuning, a search feature, or just prompting a general model.
Show answer
Model answer: A company that fine-tuned a support chatbot entirely inside one vendor's console would likely find the fine-tuned weights hardest to escape, since retraining elsewhere means starting that specific process over.
Short answer, work the number
6. If Halyard had at least kept a local copy of the raw training data, but not the embeddings or eval set, how much of PulseWatch's $210,000 migration cost would that have saved?
Show hint
Look at which cost components in the stacked bar chart relate to raw data versus embeddings and evals.
Show answer
Model answer: Not much of the largest cost, the $140,000 weights retraining and $40,000 embeddings reprocessing would still be needed. Keeping raw data alone mainly helps rebuild the eval set faster, a smaller slice of the total.
Before you close the answer
Why this works
Tests whether you can name the specific mechanisms of AI-stack lock-in instead of a vague warning, and whether you'd design a low-cost anchor against it before a vendor problem forces the question.
Follow-up traps
"Isn't keeping your own copy of everything just extra storage cost forever?" Response: storing extracted structured data and an eval set is cheap and small compared to reprocessing 600,000 documents' worth of embeddings after the fact, which is the real cost being avoided.
"Doesn't this just mean never fine-tuning at all?" Response: no, fine-tuning is still worth doing when it earns its place, the point is containing lock-in to the model layer alone, not avoiding every vendor dependency entirely.
If pressed
MemoDraft's eval set is versioned with a timestamp and a model-tier tag on every entry, so a future migration can immediately tell which memos were evaluated against which model, instead of needing to re-run the entire eval set from scratch to trust a comparison.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
