CaseAdvancedAI Opportunity & Model Strategy / Model selection from a PM lens / #13
Your chosen model is deprecated in 90 days. Walk through your response plan.
ORDERninety days felt like one deadline, it was five, in a forced order
What actually happens in the ninety days after a vendor tells you a model is going away, on a laptop with a cracked hinge, at a cubicle inside Halloway Department of Labor's digital intake team.
The direct answer
Don't treat ninety days as one deadline. It's five tasks with a forced order: decouple the model call behind one config point, build a golden set from claims already decided, canary the new model against real inflow, cut over, then decommission the old contract. The first task isn't optional and it isn't fast, it's the one that unblocks every other one, so it starts on day one even though it feels like the least urgent thing to a caseworker watching the calendar.
Do this, in order
Decouple the model call behind one adapter, starting day one.Why: nothing else in the plan can run in parallel until one place, not forty, controls which model answers.
Build a golden set from claims already decided by a human in the last 90 days.Why: it's evidence you already own, costs nothing to collect, and tells you where the new model disagrees with real outcomes.
Canary the new model against a slice of live inflow before trusting it fully.Why: a wrongful denial that reaches a claimant is far harder to undo than a slower migration.
Cut over only after the canary clears, not on a fixed calendar date.Why: the 90-day clock is the vendor's deadline, not a certificate that the new model is ready.
Decommission the old contract last, and only after a rollback window closes.Why: cancelling early removes your only fallback if the new model needs a second look.
How to answer this, stage by stage
The interviewer already knows the calendar math. What they're testing is whether you can find the one task that has to happen first, and defend why.
Stage 1
Scope it to one real system
Say it like this
"Let's ground this in one system. EligiScan reads six data sources per unemployment claim and drafts an eligibility recommendation a caseworker signs off on. That's the system I'm migrating in ninety days."
Why this works
Stops the answer from becoming an abstract "migration checklist" with nobody's claim actually riding on it.
Stage 2
Say the structure out loud
Say it like this
"I'll run this as ORDER. Outcome, what all of this is protecting. Reversibility, which piece is hardest to undo. Dependency, what has to happen before what. Evidence, what I can learn cheaply first. Rank, the actual sequence."
Why this works
Tells the interviewer you have a repeatable way to sequence a forced deadline, not just a to-do list.
Stage 3
Reframe: it isn't one deadline, it's a dependency chain
Say it like this
"Ninety days sounds like one countdown. It's really five tasks, and most of them can't start until the first one finishes, because right now the model's name is hardcoded in forty different places, not one."
Why this works
This is where a strong answer separates from "we'll just start testing the new model right away."
Stage 4
Give the one decision: what starts first
Say it like this
"Day one, I'm decoupling the model call behind a single adapter. It feels like the least urgent task, since it doesn't touch the new model at all. It's actually the most urgent, because every other task in the plan is blocked until one place, not forty, decides which model answers."
Why this works
This is the direct answer, and it names the dependency, not just a preference for caution.
Stage 5
Prove it with the compressed failure
Say it like this
"Two years ago, a different agency's rushed model swap went out to every claim at once on the deadline date. Our best estimate, from their own after-action report, is 14 wrongful denials in the first week, because nobody had checked the new model against real decided claims first."
Why this works
Compresses the whole argument into the one failure a fixed-deadline, no-sequencing plan actually produces.
Stage 6
Name what you'd leave alone
Say it like this
"I wouldn't touch the six data-source lookups or the case-note screen the caseworker sees. Neither one cares which model produced the recommendation, they just read whatever the adapter hands them."
Why this works
Shows judgment about what's actually in scope, not a blanket "rebuild everything" instinct under deadline pressure.
Stage 7
Name the AI-specific reasoning and the trade-off
Say it like this
"The honest reason this isn't a generic software cutover is that a new model can pass every unit test and still disagree with the old one on real eligibility judgment calls, in ways you can't see until you check it against claims a human already decided. That check costs time the calendar doesn't want to give up. I'd give up the full ninety-day buffer before I'd give up the check."
Why this works
Names the load-bearing AI-specific judgment and states plainly what's traded for it.
Stage 8
Close on the one line
Say it like this
"So: decouple, build the golden set, canary, cut over, decommission last. In that order, because the order is the plan."
Why this works
Restates the direct answer in one breath, which is what a live interview actually rewards.
Let's learn
EligiScan reads a claimant's wage records, employer separation reason, prior claims, and ID verification, then drafts an eligibility recommendation for a caseworker to sign off on.
Before EligiScan, Kerem Aydin re-derived every number himself from six systems, about 35 minutes a claim.
With EligiScan, Kerem reviews the cited numbers instead of re-deriving them, about 6 minutes a claim, for roughly 900 claims a week across the intake team.
Here's the turn: the deprecation notice isn't really a technical problem. It's a sequencing problem wearing a technical costume. The team's instinct was to treat "90 days" as a single deadline for "have the new model live." The actual work only has one true starting point, and it's the least glamorous task on the list.
The first fork in the plan isn't about the new model at all. It's about whether you can even point at it from one place.
At its worst, a rushed cutover with no sequencing produces wrongful denials that reach real claimants before anyone notices the new model disagreed with the old one.
The choice I would take back
When EligiScan first launched, the team hardcoded the vendor's exact model version directly into more than 40 prompt templates and case-note formatters across the intake pipeline, instead of behind one adapter. That made sense when nobody expected the model to ever change. It stopped making sense the moment a real deadline arrived and "point everything at the new model" turned into 40 separate changes instead of one.
What I would leave alone: the six data-source lookups and the case-note screen caseworkers use don't need to change at all. Neither one cares which model produced the recommendation behind the adapter.
The lesson: a deadline doesn't tell you what order to do things in. Only the dependencies do, and the first dependency is almost never the one that feels urgent.
Laid end to end, the ninety days are almost entirely preparation. The actual cutover is the shortest phase.
Rollout of the new model across live inflow, day by day
Nothing touches a live claim until day 36, and even then only one in twenty do, for 24 days, before the cutover finishes.
Now here is the same thing as a story
The short version above is what you'd say in a planning meeting. Read this one for what the ninety days actually felt like from Kerem's cubicle.
Every Tuesday, Kerem Aydin pulled his queue, glanced at the wage figures EligiScan had already cited, and signed off on the claims that checked out, usually 140 of them before lunch.
The notice arrived on a Friday afternoon: the vendor's model would stop serving requests in 90 days, full stop, no extensions.
One door swings back open any time. The other one doesn't, once a claimant has already changed their plans around a wrong answer.
The first team meeting spent forty minutes debating which vendor to pick next, an important question and, it turned out, not the one blocking anything yet.
Knowledge spark: why can't you just point the same prompts at a new model?
A model swap isn't like swapping a database driver. The same instructions can produce a different eligibility judgment from a different model, even when both technically "work." You have to actually check the judgment, not just the plumbing.
It was a junior engineer, three days in, who found the real blocker: the model's exact version string was hardcoded inside 43 separate prompt templates and formatter functions, built over two years by different people who never expected to touch it again.
Ninety days felt like one deadline. It was five tasks, and forty-three hardcoded lines were standing in front of all of them.
The most urgent tasks are also the easiest to take back if you're wrong. That's not a coincidence, it's the plan.
Once the adapter shipped, on day 9, the team pulled 90 days of already-decided claims and ran the new model against every one, quietly, with no live claimant touched at all. It disagreed with the human-confirmed outcome on 61 of about 3,000 claims, mostly a specific separation-reason category the old model had been tuned on for months.
When the original plan was first sketched, someone said, "we've got ninety days, let's just aim for a clean cutover on day 89 and leave a day of buffer." It sounded reasonable. Nobody had yet found the 43 hardcoded lines, or the 61 disagreements a real check would surface.
Wrongful-denial incidents, two response plans
The rejected plan's estimate comes from a comparable agency's own after-action report on a rushed swap two years earlier.
What I'd tell myself, hearing the junior engineer explain those 43 lines: the deadline was never the hard part. Finding out what was actually blocking everything else was.
ORDER, when the clock is what forces the sequenceNot a countdown. A dependency chain with one true starting point.
Every step after the first is waiting on it, whether anyone planned for that or not.
O
Outcome. What is all of this protecting?
Accurate, on-time eligibility decisions for claimants, without a gap in service when the old model shuts off.
Without a named outcome, ranking the five tasks is just opinion.
R
Reversibility. Which decision is hardest to undo?
A wrongful denial a claimant has already acted on, changed a budget, missed a payment, is far harder to undo than a slower migration. So speed loses to a canary check, every time.
This is the hardest step: naming which mistake you truly cannot take back.
D
Dependency. What unblocks what?
The golden set, canary, cutover, and decommission all wait on one thing: the model call living behind a single adapter instead of 43 hardcoded lines.
Reality forces this order, no judgment call required once you've found it.
E
Evidence. What can you learn cheaply first?
Running the new model against 90 days of already-decided claims costs nothing but compute, and it surfaced 61 real disagreements before a single claimant was touched.
Cheap evidence beats an expensive live test every time it's available.
R
Rank. State the order, defend the top pick.
Decouple, golden set, canary, cutover, decommission. Decouple wins the top spot because nothing else can even start without it, not because it feels urgent.
The order is the whole answer. Everything else is detail.
The recap, one line per letter: outcome is uninterrupted, accurate eligibility decisions, reversibility is that a wrongful denial outranks a slower rollout, dependency is that the adapter unblocks every later task, evidence is testing against 90 days of already-decided claims for free, rank is decouple then golden set then canary then cutover then decommission, in that order.
And if you want to be sure it really works, try it somewhere elseSame five letters, a farm cooperative's crop-advisory tool instead of a benefits office. A different reversal this time: the co-op never wrote down which decisions the model's advice was actually changing.
Ingrid Solheim runs agronomy for Bramwell Grain Cooperative, whose members get planting and pest-treatment advice from FieldAdvise. When their model vendor announced a 60-day deprecation ahead of planting season, Ingrid's team found their real blocker wasn't hardcoded calls, it was that nobody had ever logged which specific advisory decisions members actually acted on versus ignored. Mapped onto ORDER: outcome is members getting advice they can trust through a live planting window; reversibility is that a bad pest-treatment call, once sprayed, can't be undone for the season, ranking above a faster switch; dependency is that you can't build a golden set of "decisions that mattered" without first logging which advisories members actually followed; evidence is pulling three seasons of field-outcome data the co-op already had sitting in old spreadsheets; rank is log real decisions first, then golden set, then canary against a handful of test plots, then full switch before planting season closes.
The same four parts protect a benefits office and a grain cooperative equally well, because neither plan is really about the vendor.
Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "decouple first, because everything else is blocked on it, then golden set, canary, cutover, decommission," and stop.
Cost: there's no budget for a full canary period before the deadline. Say so honestly, and shrink the canary to the highest-stakes claim category only, rather than skipping it.
The model got better, for real: if the new model genuinely tests better across the board, the golden set and canary still earn their cost, because they tell you exactly which claim types you can now trust faster, not just whether the swap is safe at all.
Where people run it wrong.
They treat the vendor's deadline as the date the new model goes live everywhere, instead of the date the old model must be gone.
They start picking a replacement model before finding out what's actually blocking the switch.
They decommission the old contract the moment the new model ships, leaving no way back if week two surfaces a problem.
How to use it live. When an interviewer hands you a deprecation deadline, ask yourself: what's the one task that, if skipped, blocks everything after it? Name that task first, and the rest of the order follows on its own.
Flashcards (tap any card to flip it)
1 · THE FRAMEWORK
What framework fits a forced-deadline response plan?
Tap to flip
ANSWER
ORDER: outcome, reversibility, dependency, evidence, rank. It ranks tasks by what's hardest to undo and what blocks what, not by a fixed calendar.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Kerem Aydin, a caseworker at Halloway Department of Labor, who reviews EligiScan's cited numbers on about 900 claims a week.
3 · THE HABIT
What did the team assume, early on, that turned out to be wrong?
Tap to flip
ANSWER
That picking a replacement vendor was the urgent first question, when the real blocker was 43 hardcoded references to the old model scattered across the pipeline.
4 · THE DEPENDENCY
What's the one task that blocks all the others?
Tap to flip
ANSWER
Decoupling the model call behind a single adapter. The golden set, canary, cutover, and decommission all wait on it.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Hardcoding the vendor's exact model version into 43 separate prompt templates and formatters instead of one adapter, back when nobody expected to ever change it.
6 · THE NUMBER
Fill in the blank: the new model disagreed with the human-confirmed outcome on ___ of about 3,000 already-decided claims.
Tap to flip
ANSWER
61 claims, mostly clustered in one separation-reason category the old model had been tuned on for months.
7 · THE REPLAY
Same 90-day notice, ordered plan in place instead of a day-89 big-bang cutover. What changes?
Tap to flip
ANSWER
An estimated 14 wrongful denials in week one drops to 0 reaching a claimant, because the 1 real disagreement gets caught safely inside the canary slice first.
8 · CROSS PRODUCT TRANSFER
Section 4 runs this again for a different product. Which one, and what's the new missing piece?
Tap to flip
ANSWER
Bramwell Grain Cooperative's FieldAdvise. The missing piece there wasn't hardcoded code, it was that nobody had logged which advisories members actually acted on.
Check yourself Score: 0 / 0
True or false
1. True or false: the response plan treats day 89 as the target date to switch every claim over to the new model at once.
True
False
Show hint
Look at the rollout line chart and the "rank" step.
Show answer
False. The plan canaries at 5 percent starting day 36 and only reaches 100 percent by day 85, after the canary clears, not on a fixed calendar date.
Multiple choice
2. Why does decoupling the model call behind one adapter come before everything else?
A. It's the fastest task, so it's a quick early win.
B. Every later task is blocked until one place, not 43, controls which model answers.
C. The vendor requires it contractually.
D. It's the task caseworkers asked for first.
Show hint
Look at the "dependency" step and the flow diagram.
Show answer
B. The golden set, canary, cutover, and decommission all depend on the model call living behind a single adapter first.
Fill in the blank
3. Fill in the blank: a comparable agency's rushed, un-sequenced cutover is estimated to have caused ___ wrongful denials in its first week.
Show hint
Look at the wrongful-denial bar chart.
Show answer
14 wrongful denials. Against 0 reaching a claimant under the ordered plan, where the one real disagreement was caught safely in canary.
Short answer, where it wouldn't matter
4. Name a part of EligiScan that genuinely doesn't need to change during this migration.
Show hint
Look at "what I would leave alone."
Show answer
Model answer: The six data-source lookups and the case-note screen. Neither cares which model produced the recommendation behind the adapter.
Short answer, apply it yourself
5. Think of a tool at work or school with a hard deadline attached to a change. What's the one task hiding underneath it that everything else actually depends on?
Show hint
Ask what nobody can start on until one specific thing is done first.
Show answer
Model answer: Switching a school's grading platform before a semester deadline: nothing else can start until every teacher's old gradebook export is mapped to the new system's fields.
Short answer, work the number
6. If the canary window had to shrink from 24 days to 10 because of the deadline, would the plan's order still hold?
Show hint
Think about whether the sequence changes, or just the time spent on one step.
Show answer
Model answer: Yes, the order stays the same, decouple, golden set, canary, cutover, decommission. Shrinking canary time raises risk, but skipping a step in the sequence raises it far more.
Before you close the answer
Why this works
Tests whether you treat a deadline as a single date, or find the real dependency chain hiding underneath it. Most candidates jump straight to "pick a replacement model."
Follow-up traps
"What if legal insists everything must be off the old model by day 90, no exceptions?" Response: the decommission date doesn't move, but the cutover date inside it does, shrink the canary window instead of skipping it, since the highest-stakes claim types still get checked first.
"Isn't building a golden set from old decisions just more work you're inventing?" Response: it's the fastest, cheapest evidence available, since it costs compute time against data you already own, not a new live risk.
If pressed
The golden set held one category back on purpose: claims tied to a rare, recently-changed separation-reason rule, since that's exactly where a model tuned on older data is most likely to disagree with a human reviewer, and it did, on 61 of about 3,000 claims.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
