InterviewAdvancedModel Fluency & the AI PM Role / Working with ML engineers and researchers / #21
Tell me about a time you changed a technical decision by asking the right question rather than having the answer.
SPARK a cold start line buried in a Driftpath architecture review, six weeks of pilot logs nobody had pulled yet, at Marrowlight Media
Marrowlight Media runs a streaming platform with about 14,000 titles in its catalog. Silvano Farrowmere leads the team rebuilding the home screen's recommendation model, a new sequence-based ranker called Driftpath. Ludmilla Havercroft is the AI PM in the room the day his team asks for sign-off to cut the ranking fully over to it.
The direct answer
When you do not have the authority to say what is wrong, ask one narrow question that only real evidence can answer, aimed at the exact spot your doubt lives, not a vague concern and not an opinion wearing a question mark. Build it from what would actually settle the doubt: a real account, a real log, a real stretch of time, so the room cannot do anything with it except go check.
Do this, in order
Ask one narrow, checkable question instead of asserting an opinion you are not positioned to have.Why: a question built from real evidence cannot be waved off the way a feeling can.
Build the question backward from the exact evidence that would settle your doubt.Why: naming a real account, a real log, and a real time window is what turns a hunch into something the room has to go check.
Point at production reality, not the model's design in the abstract.Why: an offline test can look perfect and still say nothing about what happens over six real weeks.
Keep your own conclusion out of it, and stay genuinely curious.Why: the moment a question carries a hidden verdict, it reads as concern dressed up as curiosity, and the room defends instead of checks.
Let the team own the analysis. Do not pre-run it yourself and present it as a fact.Why: it keeps this a PM question, not a technical opinion you were never positioned to have.
Once the gap is real, help build the guardrail instead of declaring victory.Why: catching a gap and fixing a gap are two different jobs, and the second one is the one that actually ships.
How to answer this, stage by stage
Nobody is grading whether Ludmilla was right about Driftpath. They are grading whether she could turn a doubt she could not yet defend into one sentence the room could not shrug off.
1
Scope the story to one real decision, one real room
Say it like this
"This happened at a media company I was at, Marrowlight, we run a streaming platform. We were in an architecture review where the ML team was about to fully switch the home screen's ranking over to a new model. I was the AI PM in the room. I didn't own the modeling decision. I could ask questions. I couldn't make the call."
Why this works
Grounds the story in one concrete decision immediately, and states her actual authority level honestly, which is the whole tension the rest of the answer turns on.
2
Name what was actually nagging you, precisely
Say it like this
"About a day before that meeting, I'd read the deck twice. There was one line near the bottom of a slide: cold start, fallback to popularity. One line. Nobody else seemed to be looking at it. But I kept thinking about what a brand new account's screen actually looks like on day one, when there's no watch history at all for the model to work from."
Why this works
Shows the doubt was specific and grounded from the start, not a vague unease, which is what makes the eventual question sharp instead of generic.
3
Say out loud why you did not just assert the fix
Say it like this
"I'm not an ML engineer. If I'd stood up and said, I think your fallback is going to create a popularity bias loop, I'd have been guessing at a mechanism I hadn't verified, in front of the people who actually built it. And if I got the mechanism wrong, the real gap, whatever it turned out to be, would've gotten lost in me being wrong about the details."
Why this works
This is the keep out move, said plainly, and it is the part interviewers are actually listening for: judgment about your own position, not technical bravado.
4
Build the question backward from the evidence that would settle it
Say it like this
"So I asked myself: what's the one thing I could ask that only real data could answer, that isn't me guessing? Not is there a bias problem. Just, literally, has anyone looked. If they had, fine, I'd learn something. If they hadn't, the gap would show itself, without me having to name it."
Why this works
This is the actual method, stated as a method, the part a listener can lift straight into their own answer.
5
Ask it, in the room, in one breath
Say it like this
"Before we cut the fallback over, has anyone pulled the actual ranking logs for accounts that signed up in the last 48 hours, and looked at what they're being shown, out to about six weeks in?"
Why this works
One real cohort, one real artifact, one real time window. Nothing in it asserts an answer, so there's nothing in it to argue with, only something to go check.
6
Read the room, and let the silence do its work
Say it like this
"Silvano kind of stopped. Someone else said, we tested the fallback logic, it defaults to popularity, that's standard. And I said, right, I've seen that slide, I mean the real logs, has anyone actually watched what a new account's screen looks like a few weeks in. There were maybe five seconds where nobody said anything. Then Silvano said, I don't think we have, and wrote it on the whiteboard."
Why this works
Names the exact moment the room's certainty cracked, which is what makes it a story instead of a claim about herself.
7
Say what decision actually changed
Say it like this
"They didn't cut the full switch over that week. They pulled the logs from a pilot that had already been running for six weeks on a small opt-in group. The top 50 titles' share of what new accounts were shown had climbed from 39 percent in week one to 64 percent by week six, and only 310 titles out of about 14,000 had ever reached a new account at all. So instead of shipping Driftpath alone, they shipped it with a cold start floor, a guaranteed slice of less popular and newer titles in a new account's first sessions, plus a coverage number that had to clear a bar before any future ranking change could go out."
Why this works
The outcome is countable, and it shows the room did the technical work. She didn't do it for them, which keeps the story honest about who actually had the authority.
8
Close on the one line
Say it like this
"The habit that mattered wasn't catching a bug. It was building the question backward from the evidence, instead of forward from an opinion I wasn't positioned to have."
Why this works
Restates the whole answer's payoff in one breath, which is exactly what an interviewer remembers after you stop talking.
Let's learn
Marrowlight Media runs a streaming platform, and its home screen depends on a recommendation model to decide what a viewer sees first. For years the ranking ran on an older hybrid system, part collaborative filtering, part content tags. Nobody loved it, but it was steady, and steady is worth something.
Five people in a room, and only one of them is reading the small line near the bottom of the deck.
Under that old system, a new account's recommendations, tracked across their first month, drew from about 2,200 distinct titles out of the roughly 14,000 in the catalog. Nobody had engineered that spread on purpose. It was just what a system built on tags and rough similarity happened to produce.
Driftpath was meant to replace it with something sharper: a sequence model trained on what people actually watched in order, not just what they clicked. In an early pilot, opted in by about 900 new sign-ups, it ran for six weeks before the review meeting where Silvano asked for sign-off to switch the whole home screen over. Nobody had pulled those six weeks of logs yet. The offline test only checked whether the fallback code ran without failing for an account with no watch history. It did. That was the whole check.
Top 50 titles' share of new account impressions, pilot weeks 1 to 9
Drift, before the reviewRecovery, after the floor
Nobody watched this line move for six weeks, because nobody had asked to look at it. Ludmilla's question was the first time anyone did.
Same account, same six weeks. The only thing that changed was whether anyone had checked.
Knowledge spark: what's a feedback loop in a recommender?
The model's own past picks become tomorrow's training data. Recommend a few titles heavily, they get more clicks, the model reads those clicks as proof people want more of the same, and recommends them even more. Nothing has to be broken for this to run away. It just has to go unwatched.
The gap wasn't the twenty five points on that chart. It was what the model would learn from them, every week, without anyone watching.
Distinct catalog titles that reached a new account's rail
Old systemPilot, before the reviewPilot, after the floor
The catalog holds about 14,000 titles. For six unwatched weeks, a new viewer's entire world was 310 of them.
What it costs at its worst: if Driftpath had shipped whole, the concentration keeps climbing on its own, because each week's narrower slate becomes next month's training signal. By the time it shows up anywhere a dashboard would flag, months of biased engagement data are already baked in, and undoing it means retraining against a distorted record, not flipping one setting back.
The choice I would take back
The team's own definition of cold start handled was that the fallback code ran without erroring, checked once, offline, in a single session. That standard made sense while the fallback was simple and the untested territory was small. It stopped making sense the moment that same fallback decided what an entire cohort of new viewers saw, for weeks, unwatched.
What I would leave alone: the ranking for viewers with a real watch history. That path had months of real engagement behind it and had already been tested hard. There was no equivalent gap to go looking for there, and the cold start floor never touched it.
There was a real cost to fixing this, not just a bug found and closed. The diversity slots the floor injects get clicked less than the popular ones do: about 4.1 percent on the explore slots against about 11.3 percent on the popularity slots. That's a cheap, visible cost, absorbed on purpose, against a hidden and expensive one: a catalog that quietly narrows for new viewers until discovery itself stops working.
The lesson: an eval that proves a fallback runs is not the same as an eval that proves it's safe over time. The fix isn't distrusting the team's engineering. It's making sure someone always asks what a decision looks like in its sixth week, not just its first request, before it ships to everyone at once.
Now here is the same thing as a story
The short version above is what you'd actually say in the room. Read this one for the two minutes that decided whether Driftpath shipped whole, or shipped with a floor under it.
Marrowlight's fourth floor conference room runs architecture reviews every other Thursday, 10am, the one meeting where a model change actually turns into a release. Ludmilla Havercroft has owned the recommendation surface for two years. She reads every architecture deck twice before the meeting it belongs to, out of habit, not because anyone asked her to.
The night before this one, at her kitchen table, she read the Driftpath deck the first time straight through, nodding along. It was good work. The sequence model outperformed the old hybrid ranker on every offline metric the team tracked. Second read, slower, she stopped on a line near the bottom of slide nine: cold start: fallback to popularity. One line, no chart under it, no test result next to it. She almost let it go. Somebody more technical would flag it if it mattered, she told herself, and closed the laptop.
One night of reading, one meeting, one moment where she decided not to let it go after all.
Thursday, 10am. Silvano Farrowmere stood at the whiteboard, sketching the same architecture from the deck, boxes and arrows, clean and confident. Four teammates around the table, coffee, easy nodding, one thumbs up when he got to the throughput numbers. He was closing toward the ask: full cutover, next release, retire the old fallback path entirely.
Ludmilla still had that one line in her head. Day one, a brand new account, no watch history at all. What does the model actually do with an empty sequence? The deck said popularity. She thought about what popularity feeding on itself looks like after six weeks, not after one request. She didn't have a mechanism to name. She had a question she could build.
Three real things, none of them an opinion: an account, a log, a stretch of time.
As Silvano reached for a show of hands, she raised hers instead. "Before we cut the fallback over, has anyone pulled the actual ranking logs for accounts that signed up in the last 48 hours, and looked at what they're being shown, out to about six weeks in?"
Silvano stopped mid sentence. A mid-level engineer answered first, a little too fast: "We tested cold start offline, it defaults to popularity ranking, that's standard practice." Ludmilla nodded. "Right, I've seen that slide. I mean the real logs. Has anyone actually watched what a new account's screen looks like a few weeks in, not just whether the fallback code runs." Nobody answered for almost five seconds. Then Silvano, quieter: "I don't think we have." He picked up the marker and wrote it on the whiteboard as an open item, under the architecture he'd been so sure of ninety seconds earlier.
The twenty five points weren't the danger. Feeding them back into training, every week, was.
They didn't cut the release that week. Over the next two weeks, Silvano's team pulled the six weeks already sitting in the pilot's logs. Week one, the top 50 titles held 39 percent of what new accounts were shown. Week six, 64 percent. Only 310 of the catalog's roughly 14,000 titles had ever reached a new account at all, against about 2,200 under the old system. Nobody had built that. It had simply happened, quietly, while everyone watched the offline metrics instead.
Driftpath shipped three weeks later, not alone, but with a cold start floor built in: a guaranteed slice of newer and less popular titles in a new account's first sessions, and a coverage number, the top 50 share of new-account impressions, that had to hold under a set line for three straight pilot weeks before any future ranking change could go to full rollout. By week nine, the share had settled back to 41 percent, and 1,850 titles were reaching new accounts again.
What I'd tell myself, sitting in that review: doubt is not evidence, and you are not owed the confidence to skip the step of finding out. Ask the one question only real data can answer, and let the room supply the rest.
SPARK, off one quiet question in a review room
Not a script for sounding sharp in an interview. SPARK is what actually separates a candidate who can turn a doubt into evidence from one who turns it into an opinion.
SSituation. Who is this person, and how does the room decide without them, today?
Ludmilla Havercroft, AI PM on Marrowlight's recommendation surface, sitting in an architecture review where Silvano's team is about to commit the home screen to Driftpath. She does not own the modeling decision. She owns the outcome if it goes wrong for a whole cohort of viewers.
One person, one room, one real decision already in motion. Never a category of engineer.
PPayoff. What habit does this moment build?
Turning a doubt she cannot yet defend into one narrow question that only real evidence can answer. The payoff isn't that she was right about cold start. It's that the room now has a way to check, rather than a feeling to argue about.
Name the thing the habit produces, not the outcome that happened to follow it this one time.
AAnchor. The one moment everything else hangs on.
The actual question, asked out loud, in the room, before the sign-off vote: "Before we cut the fallback over, has anyone pulled the actual ranking logs for accounts that signed up in the last 48 hours, and looked at what they're being shown, out to about six weeks in?" A real cohort, a real artifact, a real time window. Nothing in it states a conclusion.
Concrete enough to be checked, not just admired. This is the actual answer to the question.
Same doubt, two different sentences. Only one of them survives the room.
RRisk. What breaks the first time it lands wrong?
A vaguer version, "I'm a little worried about bias here," reads as concern-trolling with nothing behind it, easy to wave off with the same "we tested that" line that almost worked. An accusatory version, framed as a disguised verdict, puts Silvano's team on the defensive instead of curious, and the room spends its energy defending the plan instead of checking it. Either way, trust goes down, not up, and the real gap stays hidden.
Not "the question was wrong." What the room does next, in either direction.
Three branches off the same doubt. Only one of them actually gets the logs pulled.
KKeep out. What I deliberately did not do.
She did not assert a technical diagnosis she wasn't positioned to have, "this will cause popularity bias," because she hadn't verified the mechanism and being wrong about it would have buried the real gap. She also did not pull the logs herself beforehand and present her own analysis as settled fact, which would have looked like going around the team and risked being wrong without their tools or their context. Both stayed out on purpose.
Shows judgment instead of overreach. Ties straight back to Risk: the wrong move here costs trust, not just accuracy.
The recap, one line per letter: situation is an AI PM without modeling authority watching a cutover decision form in real time, payoff is a habit of converting doubt into evidence instead of opinion, anchor is the actual question asked, naming a real account cohort and a real time window, risk is either extreme, vague or accusatory, costing trust instead of building it, and keep out draws the line at asserting a technical answer or pre-empting the team's own analysis, every time, on purpose.
One alternative Ludmilla rejected on the way to this: staying quiet and trusting that "someone more technical" would catch it if it mattered. She turned that down because the deck had already made it to a sign-off vote without anyone catching it, which was the actual evidence that nobody else was going to. Silence wasn't neutral. It was one more vote for a plan nobody had actually checked.
And if you want to be sure it really works, try it somewhere else
Same five letters, a public library consortium instead of a streaming platform, and this time the failure mode isn't a popularity loop, it's a language nobody benchmarked.
Brightwater Library Systems builds the catalog search behind a regional library consortium's website. Loredana Bellinghurst is interviewing for an AI PM role there. Orinthia Dashwood, the engineering lead, is presenting a plan to replace the old keyword-and-tag search with an embeddings-based semantic ranker, trained mostly on English-language catalog records and reviews, and to retire the keyword fallback in the same release.
Same shape of moment, a different catalog, a different language gap this time.
Mapped onto SPARK: situation is Loredana in the review where the full cutover gets decided, with no authority to set the model's training data herself. Payoff is the same habit: one checkable question instead of a vague "what about bias" objection. Anchor is her actual question: "Have we pulled real search logs for queries typed in languages the benchmark set doesn't cover, Vietnamese or Yoruba script for example, and checked where the correct catalog item actually lands in the results?" Risk is the same shape as Marrowlight's: a vague fairness objection with nothing behind it gets waved off in one sentence, the specific one does not. Keep out draws the same line: she does not run her own multilingual audit and present it as a verdict, and she does not name the training-data gap as fact before anyone has checked.
Swap the trigger and it still runs.
Speed: the interviewer cuts you off after ninety seconds. Keep only the anchor, the question itself and the one line about what it found. The rest is context you add if asked.
Cost: you never get a follow-up meeting to learn the outcome. The story still holds if it ends at "the room agreed to go check." The habit is the question, not the resolution you happened to get.
The model got better, for real: say Driftpath's offline accuracy had cleared every metric the team was already tracking. The habit doesn't change. A model can score well on everything you're measuring and still have a blind spot in whatever you never thought to measure. Good numbers are a reason to look harder at what wasn't counted, not a reason to stop asking.
Where people run it wrong.
They turn the question into a speech, three sentences of throat-clearing before the actual ask, and the room tunes out before the specific part lands.
They ask a question they already know the answer to, fishing for someone to admit a mistake in front of the room, which reads as a trap the moment anyone notices.
They soften it into something so vague, "are we sure this is solid?", that anyone can say yes and move on, and nothing actually gets checked.
How to use it live. Before you ask anything in a room where you are not the technical authority, silently finish this sentence: "the only way to know is to look at ___." If you cannot name a specific, real thing, a real log, a real account, a real stretch of time, you are not ready to ask yet. You are about to share an opinion instead.
Flashcards (tap any card to flip it)
1 · THE METHOD
What framework fits telling a real story about changing a decision by asking one good question?
Tap to flip
ANSWER
SPARK: situation, payoff, anchor, risk, keep out. Used here in its most literal form: a real behavioral story about one specific question, asked out loud, in one specific meeting.
2 · THE PEOPLE
Who is this answer about?
Tap to flip
ANSWER
Ludmilla Havercroft, AI PM at Marrowlight Media, a streaming platform. Silvano Farrowmere leads the team building Driftpath, the new recommendation model, and is the one who says "I don't think we have."
3 · THE PAYOFF
What habit does the anchor question build?
Tap to flip
ANSWER
Turning a doubt you can't yet defend into one narrow question that only real evidence can answer, instead of asserting an opinion you're not positioned to have.
4 · THE ANCHOR
What was the actual question, word for word?
Tap to flip
ANSWER
"Before we cut the fallback over, has anyone pulled the actual ranking logs for accounts that signed up in the last 48 hours, and looked at what they're being shown, out to about six weeks in?"
5 · THE OLD STANDARD
What old decision does this answer take back?
Tap to flip
ANSWER
The team's definition of "cold start handled" was that the fallback code ran without erroring, checked once, offline, in a single session, never checked against real accounts over real weeks in production.
6 · THE NUMBERS
Fill in the blank: the top 50 titles' share of new account impressions climbed from ___ percent in week one to ___ percent by week six.
Tap to flip
ANSWER
39 percent to 64 percent. Only 310 of about 14,000 titles ever reached a new account in that window, against about 2,200 under the old system.
7 · THE RISK, SURVIVED
What could have gone wrong if the question had been asked differently?
Tap to flip
ANSWER
A vague version ("I think there's a bias problem") would have read as second-guessing and been easy to wave off. The specific version survived because it named a real cohort, a real artifact, and a real time window, nothing to argue with, only something to go check.
8 · CROSS-PRODUCT TRANSFER
Section 4 runs SPARK again on a different product. Which one, and what's the question about?
Tap to flip
ANSWER
Brightwater Library Systems' catalog search. Loredana Bellinghurst asks whether anyone pulled real search logs for queries in languages the benchmark set didn't cover, not just the English-only test set.
Check yourself Score: 0 / 0
Multiple choice
1. What made Ludmilla's question land as curiosity instead of second-guessing?
A. She had already pulled the logs herself and presented her own analysis as fact.
B. She named a specific account cohort, a specific artifact, and a specific time window, with no stated conclusion.
C. She phrased it as a firm statement instead of a question, so it sounded confident.
D. She asked it privately to Silvano after the meeting instead of in front of the room.
Show hint
Check the Keep Out step in the SPARK recap, and compare it to what she actually asked.
Show answer
B. Naming a real cohort, a real artifact, and a real time window, with no conclusion attached, is what made it checkable instead of accusatory. Option A is exactly what she deliberately did not do.
True or false
2. True or false: if Ludmilla had simply said, "I'm a little worried this might have a popularity bias problem," it would have produced the same result.
True
False
Show hint
Look at how the room actually responded when the engineer said "we tested that."
Show answer
False. A vague concern names no specific evidence to check, and the story shows the room could and did wave off exactly that kind of statement ("we tested the fallback logic, that's standard"). The specific version, naming real logs and a real cohort, couldn't be waved off the same way.
Fill in the blank
3. Fill in the blank: before the floor was added, only ___ of Marrowlight's roughly 14,000 titles ever reached a new account's rail in six weeks. After the floor, that recovered to about ___.
Show hint
Check the bar-chart numbers in Let's learn, and flashcard 6's neighborhood.
Show answer
310. 1,850. The old hybrid system, before Driftpath existed, had reached about 2,200 titles over a comparable window.
Short answer, name the reversal
4. What old decision would Ludmilla's team take back, and why did it make sense when it was first set?
Show hint
Look at the key point box titled "The choice I would take back," in Let's learn.
Show answer
Model answer: The team's standard for "cold start handled" was that the fallback ran correctly offline, in a single session, with no multi-week production check required. That made sense while the fallback was simple and the untested territory was small. It stopped making sense once that same fallback decided what an entire cohort of new viewers saw, for weeks, unwatched.
Short answer, where it wouldn't matter
5. Name a place in Marrowlight's own recommendation system where this same habit, asking a pointed evidence question before a launch, genuinely would not have been needed.
Show hint
Check "What I would leave alone" in Let's learn.
Show answer
Model answer: The ranking for viewers with a real watch history. That path had months of real engagement data behind it and had already been tested hard, so there was no equivalent unverified gap worth going looking for.
Short answer, apply it yourself
6. Think of a decision at your own work, one you didn't have the authority to simply override, where you had a real doubt. What's the one narrow, checkable question you could have asked instead of stating an opinion?
Show hint
Name a real cohort, a real artifact, and a real time window, and leave your own conclusion out of it.
Show answer
Model answer: A retailer's demand-forecasting team was about to widen a promotional discount rule storewide. Instead of raising an objection: "Has anyone pulled the sell-through numbers for the three stores where we already ran this discount for a month, broken out by whether the customer was new or returning?"
Before you close the answer
Why this works
Tests whether a candidate knows when they don't have standing to assert a technical answer, and whether they have the discipline to convert a doubt into one checkable question instead of an opinion, a hedge, or silence.
Follow-up traps
"What if you'd asked and they said everything's fine? Would you have just dropped it?" Response: yes, because the question was built to be answered by real evidence either way. A flat, healthy spread in the logs would have been a real answer too, and pushing further with no new evidence would have been the exact opinion-in-disguise move she was trying to avoid.
"Isn't this just you doing the engineer's job for them?" Response: no. She never analyzed the logs herself or proposed the fix. She asked whether anyone had looked, and let the team run the analysis and design the guardrail, which is what kept it a PM question instead of a technical opinion.
If pressed
The cold start floor wasn't free. Blending in the exploration candidates raised the home screen's ranking latency from about 54 milliseconds to about 63 milliseconds per request, a cost the team accepted on purpose rather than risk shipping the narrowing loop unwatched again.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
