The direct answer
Before any account's default model flips, run its saved prompts through an automated check that compares the new model's SQL structure against what the old one produced. Any account whose saved prompts fall below a 90 percent structural match goes into a short dual-run window, both models answer and the customer sees the difference, instead of switching on the same date as every account with no saved prompts at all.
Do this, in order
Run an automated compatibility check on every account's saved prompts before the cutover, not after complaints start.Why: this is the decision the whole migration plan turns on.
Route any account that fails the check into a short dual-run window instead of switching it on the general schedule.Why: a tuned prompt library took real hours to build, and it's the one thing a blanket cutover destroys without telling anyone.
Switch every account with no saved prompts to the new model right away, no gating.Why: the new model is already a plain improvement for them, and holding them back protects a risk they don't have.
Show the customer a structural diff the moment their saved prompt's output changes shape.Why: nothing else in the product tells a power user this was a model swap and not their own mistake.
Set the pass bar as a threshold on each account's own saved query set, not a promise of identical SQL forever.Why: model output is probabilistic, and a deterministic promise would be a bar nobody can actually clear.
Watch the override rate inside each dual-run window, not the average accuracy number across all accounts.Why: a climbing override rate on one account's queries shows the fix isn't working weeks before a wrong number reaches someone's report.
How to answer this, stage by stage
Seven moves. Splitting accounts by whether they have tuned prompts, not by how big the customer is, is where the real answer lives.
1
Ground it in one real product before naming a framework
Say it like this
"Say Fenlow is a data-analytics platform. Somewhere inside it there's a box where an analyst types a plain English question, like 'show freight cost by region last quarter,' and gets working SQL back. It's rolling a model swap out to fourteen hundred enterprise accounts, and about sixty of them spent real weeks building their own saved examples to get the old model writing SQL in exactly the shape their dashboards expect."
Why this works
Grounds a broad migration question in one specific product before naming a method, so the answer can't drift into a generic release-notes checklist.
2
State your structure in one line
Say it like this
"I'd use GUARD here, because 'how do you handle customers who tuned prompts to the old model' is really a risk question dressed up as a support question. Who's actually affected and how differently, where the cost of a bad migration lands hardest, who can't tell a broken prompt from their own mistake, the actual migration rule I'd write, and how I'd find the affected accounts before they open a ticket."
Why this works
Two seconds that show a plan before any specifics, and it signals this isn't a generic version-bump announcement wearing an AI company's name tag.
3
Name both groups, not just "our customers"
Say it like this
"There are two groups on Fenlow. Casual accounts, who type a question and take whatever SQL comes back, no saved examples, no tuning. And power accounts, about sixty of them, who spent real hours teaching the old model their exact table aliases, their date logic, their column order, so the output could feed straight into a saved dashboard."
Why this works
This is GUARD's G step. It's not "our users," it's two groups whose relationship to the product is fundamentally different, and a migration that treats them the same is where the trouble starts.
4
Show where the cost lands hardest
Say it like this
"For casual accounts, the new model is just better, first-try accuracy went from 84 to 91 out of 100. For the sixty power accounts, it's the opposite. Their saved prompts, built against the old model's exact habits, dropped from a 97 percent structural match on their own saved queries to 61. The new model isn't wrong, it's SQL that's correct in a shape their prompt library never taught it to produce, and weeks of somebody's work stopped working on the same morning."
Why this works
This is U. It names the exact group the harm concentrates in, and why: they're the ones who invested effort the swap quietly erases.
5
Name who can't tell what happened
Say it like this
"Zsofia Nagy runs analytics for Quillan Freight, one of those sixty accounts. Two days after her account switched, her freight-cost report showed a 40 percent jump that made no sense. Nothing on her screen said 'the model changed.' It looked exactly like a mistake she'd made, or a real cost spike, and she only caught it because she happened to check it against last month's number an hour before it went to her CFO."
Why this works
This is A, GUARD's hardest step. The gap isn't that Zsofia wasn't careful. It's that the product gave her nothing to be careful about until she went looking on her own.
6
Give the actual migration rule, not a general "announce it"
Say it like this
"Here's the rule I'd write in before the cutover, not after: run every account's saved prompts against the new model first, quietly, and check the structure of what comes back against what the old model used to produce. Anything under a 90 percent match goes into a two-week dual-run window, both models answer, the customer sees the difference side by side. Everyone else, the accounts with nothing tuned, switches over immediately, because they've got nothing to lose and no reason to wait."
Why this works
This is R. A named, surgical migration path for the group that actually carries the risk, not a blanket cutover or a blanket delay applied to all fourteen hundred accounts alike.
7
Say how you'd detect it early, then close
Say it like this
"And I'd watch two things before anyone complains. How many accounts fail the compatibility check before cutover even starts, so I know the real size of the sixty instead of guessing. And the override rate inside each dual-run window, because an account still rejecting the new model's output in week two needs more time, not a deadline. So: check compatibility first, dual-run whoever fails it, and switch everyone else right away."
Why this works
Closes on the direct answer in one breath, and shows the migration rule gets checked against real behavior, not just written down and shipped.
Let's learn
Fenlow is a data-analytics platform. Type a question in plain English and it writes the SQL for you.
Before the swap, on the old model, an analyst who just typed a question and took whatever came back got the right query on the first try about 84 times out of 100. About sixty of Fenlow's fourteen hundred enterprise accounts didn't stop there. Over weeks, they built a saved set of examples and instructions, teaching the old model their exact table names, their date logic, their column order, until their first-try match on their own saved queries climbed to 97 out of 100.
Knowledge spark: what's a few-shot example?
A sample question paired with the exact SQL you want back, tucked into the prompt so the model copies the pattern. Build fifteen of them and you've basically taught the model your team's house style.
Fenlow swapped in a new model. It's better on paper, and it's cheaper to run. For the casual accounts, it really is better: first-try accuracy climbed to 91 out of 100. But the sixty accounts with a saved prompt library dropped to 61 out of 100, because the new model writes SQL that's correct, just shaped differently, and their prompts never taught it the new shape.
First-try accuracy, casual accounts vs. accounts with tuned prompts
Same swap, same day. One group's number went up. The other's fell off a ledge.
Casual accounts, old model
84%
Casual accounts, new model
91%
Tuned-prompt accounts, old model
97%
Tuned-prompt accounts, new model
61%
One average accuracy number, watched across all 1,400 accounts, would have shown a straight win. It never split by which sixty accounts had something to lose.
Here's the part that's easy to miss. Those extra wrong queries aren't the real problem, a wrong query just gets deleted and retried. The real problem is that nothing on the screen tells the sixty tuned accounts why it's suddenly happening. It looks exactly like their own mistake.
We didn't just make more queries wrong. We took away the one thing power accounts had stopped checking: that their saved prompts still meant what they'd taught them to mean.
At its worst, this reached Zsofia Nagy, a senior analyst at Quillan Freight. Her freight-cost report jumped 40 percent two days after the switch, from a join the new model built differently. It almost went to her CFO before she caught it.
The choice I would take back
We never kept a way to compare a saved prompt's output across model versions. So when the output changed shape, there was nothing to check it against, no record, no diff, nothing. That was fine when there was only ever one model running. It stopped being fine the day there were two.
What I would leave alone. The casual accounts don't need any of this. No saved prompts, no dual-run, no compatibility check, just switch them over, because the new model is already a plain upgrade for them, and gating it would only slow down a win nobody's waiting to be protected from.
The lesson. A model swap isn't one event. It's two different products landing on two different kinds of customer at the same time, and we built a migration plan for the customer who had nothing to lose.
Now here is the same thing as a story
Read the short version above if you're pressed for time. Read this one when you want to feel why the two groups needed two different plans, not just know that they did.
Rustam Aliyev has run Fenlow's query-generation team for four years. He can look at a support ticket about bad SQL and tell you inside a minute whether it's a real model problem or a customer who typed a vague question.
When Fenlow trained and shipped its new model, Rustam's rollout plan was simple, and for most of the fourteen hundred accounts, it was also right. Accounts got switched over in batches through March. Support tickets about wrong SQL actually went down. By April the new model was answering nine out of ten first-try questions correctly, and nobody on the team was watching that number worry anymore.
So the team stopped checking it account by account. The dashboard showed one number, average first-try accuracy, climbing every week. That number was true for thirteen hundred and forty accounts.
Zsofia Nagy runs analytics for Quillan Freight, a Fenlow customer since year one. She'd spent the better part of two quarters building a prompt library: fifteen saved examples that taught the old model to always alias the freight table as f, always bucket dates by fiscal quarter, always exclude cancelled shipments unless she said otherwise. It worked so well she stopped checking the SQL Fenlow handed back. She just ran it.
One of them has a hand on the switch. The other only has a stack of paper that used to work.
Her account switched over on a Tuesday, in the same batch as three hundred other mid-size accounts. Nothing announced it. No email, no banner, nothing inside the product.
Two days later, she pulled her regular freight-cost report for the exec review. The number was up 40 percent from last month. Freight cost doesn't move 40 percent in thirty days at Quillan. So she went looking, and found the new model had joined two tables in a way that looked completely reasonable and quietly double-counted a shared lane. Same prompt. Same question she'd asked a hundred times before. Different SQL, because the new model builds joins its own way, and her prompt library had never had to say a word about it, because it never needed to.
She caught it an hour before the number went to her CFO. If she hadn't reconciled against last month out of habit, it wouldn't have been caught at all. It would have looked like a real number.
We didn't cost Zsofia one wrong join. We cost her the warning that would have told her to go looking for it.
I want to say the problem is that the new model got the join wrong. It didn't, not exactly. The join was a perfectly reasonable way to answer her question, if you'd never seen how she wanted it answered. The real problem is that nothing on Zsofia's screen could tell her a model swap had happened at all. From where she sat, this looked identical to a mistake she might have made herself.
The team had considered something simpler: leave the old model running behind an opt-out toggle, for anyone who noticed something wrong and asked for it back. It's cheaper to build than a compatibility check. We turned it down for the reason it was tempting: it depends entirely on the customer noticing the change and knowing there's a toggle to reach for, which is exactly the gap that almost put a wrong number in front of Zsofia's CFO. A toggle protects the person who already knows to look. It does nothing for the person who doesn't know there's anything to look for.
Months earlier, when the team scoped the new model's rollout, the question in the room was whether the new model was better on average. It was. Nobody in that meeting asked a second question: for the accounts with a saved prompt library, does "better on average" mean anything at all.
I would put that second question back in the room. Before any account with saved prompts gets switched, its prompts run against the new model first, quietly, and get checked against what the old model used to produce. Anything under a 90 percent structural match goes into a two-week window where both models answer and Zsofia can see, side by side, exactly what changed and why.
Same Tuesday, same switch. This time the compatibility check catches Quillan Freight's account before cutover, flags it at 61 percent match, and routes it into the dual-run window automatically. Zsofia gets an email two weeks before her account would have switched, not a surprise two days after: "Your saved queries produce different SQL under the new model. Compare them here before we move you over." She finds the join difference on a Thursday afternoon, with a full week to check it, not an hour before a board number goes out.
What I'd tell the Rustam who signed off on that first rollout plan: "the model's average is better" and "every customer's workflow is still safe" sound like the same sentence in a launch review. They aren't. One is a number. The other was true for thirteen hundred and forty accounts and false for sixty, and we never built a way to tell which sixty until one of them almost found out the hard way.
GUARD, for a migration that isn't the same event twice
This is a risk question, so the framework is GUARD. "How do you handle customers who tuned prompts to the old model" sounds like it wants a support playbook, which is exactly why it's easy to answer with a generic announcement plan instead of naming the one thing that's actually different here: the customers who invested the most effort are the ones who lose the most when nothing tells them what changed.
G, groups. Casual accounts, who type a question and take whatever SQL comes back. And power accounts, about sixty of Fenlow's fourteen hundred, who spent real hours building a saved prompt library against the old model's exact habits.
U, unequal. The new model is a straight improvement for casual accounts. The cost lands hardest on power accounts, whose tuned prompts suddenly produce SQL in a shape they never taught the model, undoing weeks of real invested work in a single cutover.
The step that should sit here, and doesn't.
A, ability to contest. Zsofia sees the gap first, an hour before her CFO does, by accident. Nothing in the product told her a model swap had happened at all, because the class of accounts that needed a signal, the tuned ones, had never been split from the ones that didn't.
R, reduce. Run every account's saved prompts through an automated compatibility check before its cutover date, comparing the new model's SQL structure against the old one. Route anything under a 90 percent match into a two-week dual-run window instead of a blanket switch. This isn't free: the new model runs about 30 percent cheaper and faster per query, which is exactly why Fenlow wants everyone on it, and the dual-run window spends roughly double the inference cost on those sixty accounts, on purpose, to protect reliability they'd already earned rather than pretend quality, speed, and price can all be free at once.
D, detect. Count how many accounts fail the compatibility check before a single one gets switched, so the size of the affected group is measured, not guessed. And watch the override rate inside each dual-run window, since an account still rejecting the new model's output in week two is telling you something the average accuracy number never will.
Knowledge spark: why 90 percent, not 100?
Model output is probabilistic. Two SQL queries can differ in small, harmless ways and still return the same result. A pass bar of "identical every time" is a promise nobody can keep. A calibrated threshold on a real eval set, checked before cutover, is a bar you can actually clear.
Where this answer would fail
If the fix is "send an email about the upgrade" or "watch support tickets more closely," none of it counts. This is a case of silent output drift: the model changes, nothing in the product errors or warns, and a prompt that used to be reliable degrades without a single exception thrown. The guardrail has to be the compatibility check itself, run in shadow mode against every saved prompt before a single account gets switched, so drift shows up as a number before it shows up as a wrong invoice.
And if you want to be sure it really works, try it somewhere else
Saltmere Veterinary Partners runs the same gap through an AI tool that drafts structured visit summaries from a vet's notes, nothing to do with SQL or freight.
G, groups. General-practice clinics, who type or dictate notes and take whatever structured summary comes back. And Saltmere's oncology clinic, which spent months building custom terminology templates so the old model's summaries matched the exact wording their insurers required for claims.
U, unequal. General clinics barely notice the swap. The oncology clinic's claims start bouncing, because the new model still writes a clinically correct summary, just with different terminology than the templates were built to enforce.
A, ability to contest. Yewande Ogunleye, Saltmere's operations lead, has no way to see a bad claim coming. The tool logs every summary as "complete," identical whether the wording matches an insurer's required terms or not.
R, reduce. We had defaulted every clinic to the same cutover date, right for a clinic with no custom templates and wrong for the one that had spent months building its own. The fix: check a clinic's custom templates against the new model's output before its cutover date, and dual-run any clinic that fails, instead of migrating everyone on the calendar date that was convenient for engineering.
D, detect. Track each clinic's claim rejection rate weekly against its own baseline, not the network average, since one clinic's rate tripling is invisible inside a number blended across forty clinics.
Oncology clinic's claim rejection rate, the six weeks after its cutover
Same clinic, same paperwork. The new segment is entirely terminology mismatches the old templates used to prevent.
Total claims rejected
22%
3 percent was the clinic's normal rejection rate, in green, the dashed line marks it. 19 points on top, in red, is new terminology mismatches, invisible in the network-wide average because 39 other clinics didn't move at all.
Swap the trigger and it still runs
- Speed: if Fenlow rolled the swap out faster, the sixty tuned accounts would hit the mismatch sooner and in a bigger batch, which only shrinks the window a compatibility check would have to run in. It doesn't remove the need for one.
- Cost: a cheaper model tempts more teams to skip the compatibility check and lean on "the average accuracy is higher," which means more tuned accounts switched on day one, not fewer.
- The model gets better: a new model that's right 99 percent of the time instead of 91 makes skipping the check look even safer, and that's exactly backwards. The rarer the structural mismatch, the longer it runs before anyone notices the pattern.
Where people run it wrong
- Treating one blended accuracy number as proof that every customer's workflow is equally safe to migrate.
- Writing a compatibility check, but only running it after a customer complains, instead of before any account in that class gets switched.
- Gating every account, including the casual ones with nothing tuned, which just delays a real improvement for people who never needed protecting from it.
How to use it live
Ask one question before answering: "which of these customers actually built something against the old model's specific behavior, and does the rollout plan check that before it switches them?" That's usually the exact question the interviewer wanted asked, and it buys a few seconds to think.
Flashcards (click a card to flip it)
1 · THE FRAMEWORK
Which framework fits a question about handling customers who tuned their prompts to the old model, and why?
Tap to flip
ANSWER
GUARD, for risk. The real question isn't whether the new model is better on average, it's which customers pay for the swap, who can't tell it happened, the actual migration rule, and how to find them before they complain.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Rustam Aliyev, a senior PM on Fenlow's query-generation team. He's run rollouts for the AI SQL assistant for four years.
3 · THE HABIT
What did Rustam's team never build, even as the new model's average accuracy kept climbing?
Tap to flip
ANSWER
A way to check a saved prompt's output against the old model before switching an account over. They watched one average number for all 1,400 accounts and never split it by whether an account had tuned prompts.
4 · THE SWITCH
What's the two-setting switch this answer turns on?
Tap to flip
ANSWER
Either an account's saved prompts get checked against the new model before cutover, or they don't. There's no middle setting where a high average accuracy quietly covers for the sixty accounts a prompt library was built for.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Fenlow kept no record comparing a saved prompt's output across model versions. So when the output changed shape after the swap, there was nothing to check it against, no diff, no history, nothing.
6 · THE NUMBER
Fill in the blank: tuned-prompt accounts' structural match rate dropped from ______ to ______, while casual accounts' accuracy rose from ______ to ______.
Tap to flip
ANSWER
97 to 61 for tuned accounts. 84 to 91 for casual accounts. Same swap, opposite direction, because nobody split the rollout by which group they were switching.
7 · THE REPLAY
Same swap, the compatibility check gates it this time. What changes for Zsofia?
Tap to flip
ANSWER
Her account gets flagged at 61 percent match before cutover and drops into a two-week dual-run window. She gets an email showing the SQL difference and catches it on a Thursday afternoon, with a week to spare, not an hour before a board number goes out.
8 · TRANSFER
Section four runs GUARD again on a different product. Which one, and what does the reduce step become?
Tap to flip
ANSWER
Saltmere Veterinary Partners' AI visit-summary tool. Reduce: check a clinic's custom terminology templates against the new model before its own cutover date, and dual-run any clinic that fails, instead of migrating every clinic on the same calendar date.
Check yourself Score: 0 / 0
Multiple choice
1. Which two groups does the G step name in Rustam's story, and what separates them?
- A. Casual accounts with no saved prompts, and power accounts with a hand-tuned prompt library built against the old model.
- B. Rustam's team and Fenlow's finance team, who disagree about the cost of running two models.
- C. Zsofia and her CFO, who disagree about the freight-cost number.
- D. Fenlow and Quillan Freight, who disagree over who owns the mistake.
Show hint
Look for who built something against the old model's specific behavior, and who never touched a setting at all.
Show answer
A. B, C, and D name real people or organizations in the story, but not the split the G step names: accounts with nothing tuned, and accounts that invested real effort against the old model's exact habits.
Fill in the blank
2. Zsofia's freight-cost report jumped ______ percent, about ______ days after Quillan Freight's account switched over.
Show hint
Both numbers are in the story, right where Zsofia checks the report before the exec review.
Show answer
40 percent, and 2 days. A jump that size doesn't happen at Quillan in a normal month, which is exactly what made her stop and check it before it went out.
True or false
3. True or false: the mistake happened because the new model's SQL was factually wrong.
Show hint
Check what the story says about the join itself, not just the number it produced.
Show answer
False. The join was a reasonable way to answer the question, just structured differently than Zsofia's prompt library expected. Nothing about the model's logic was broken, it just wasn't the shape her saved prompts had taught it.
Short answer, apply it yourself
4. Pick an AI product you use yourself that lets you customize its behavior, a saved prompt, a template, a set of rules. If the underlying model changed tomorrow, what's one thing you've tuned that might quietly stop working, and how would you want to find out?
Show hint
Think about something you built through trial and error, not something you set once from a menu.
Show answer
Model answer: "I use an email tool with a custom auto-reply template I rewrote three times to get the tone right. If the model changed, that tone could drift without any error message, it would just start sounding a little off. I'd want the tool to run my template against the new model before switching me over, and tell me plainly if the output shifted, instead of me noticing weeks later that people started replying differently."
Multiple choice
5. Why can't Fenlow just tell power accounts to "keep an eye on their queries" instead of building a compatibility check?
- A. Because "keep an eye on it" isn't a rule anyone can point to, and it still puts the entire job of noticing a model swap on the customer least likely to know one happened.
- B. Because Quillan Freight's analysts don't have the training to read SQL.
- C. Because the new model would need to reach 100 percent accuracy first.
- D. Because watching queries closely would slow down the migration too much.
Show hint
Ask what "keep an eye on it" actually changes about the moment the report goes to the CFO.
Show answer
A. B, C, and D aren't the real issue. Watching harder doesn't put a signal in front of the customer before the output changes shape. The fix has to change what the product tells the customer, not how carefully the customer checks afterward.
Short answer
6. If the structural match rate had only dropped from 97 to 90 instead of 97 to 61, would the same fix, a compatibility check plus a dual-run window, still be the right one? Why or why not?
Show hint
Check where 90 sits against the pass bar named in the GUARD recap.
Show answer
Model answer: "The compatibility check would still be the right thing to build, but a drop to 90 clears the pass bar, so that account wouldn't trigger the dual-run window at all, it would just switch over. The fix isn't a fixed reaction to any drop, it's a threshold. A small drop that still clears the bar and a big drop that doesn't are different calls, and the whole point of a calibrated threshold instead of a blanket rule is that it tells the two apart."
If they push back
Why this works
Tests whether you'll treat "the model is better on average" as the whole answer, or go looking for the customers that average is hiding. Most candidates stop at the accuracy dashboard.
Follow-up traps
"It's only sixty accounts out of fourteen hundred, is a compatibility check really worth building for four percent of customers?"
Response: those sixty are disproportionately the enterprise accounts paying for consistency, and losing one to a broken dashboard costs more than the check does to build.
"Isn't a dual-run window just delaying the inevitable, they still have to migrate eventually?"
Response: it's not delay, it's sequencing. The window exists to catch the structural mismatch before it ships a wrong number, not to avoid the swap.
If pressed
The compatibility check doesn't diff two SQL strings character by character. It parses each query's structure, table aliases, join order, column list, and compares that, because two queries can be byte-different and still return the identical result set, and a string diff would flag hundreds of harmless changes a structural diff would ignore.
