InterviewIntermediateModel Fluency & the AI PM Role / Working with ML engineers and researchers / #15
Explain how you would onboard yourself to an existing ML codebase as a new PM.
BOUND · pricing a new PM's real ramp-up at Thistledesk, an AI tool that routes customer-support tickets to the right team
Thistledesk reads an incoming customer-support ticket and decides which of fourteen internal teams should handle it. Henrike Ashcombe just joined as its product manager. Hakon Deveraux is the research engineer who owns the routing model. Stonebrook Appliances, a home-appliance company, runs its whole warranty and repair desk through Thistledesk, and Gulnara Petrosyan, Stonebrook's head of support operations, is the one who eventually calls asking why a ticket about a burning smell sat for nineteen hours.
The direct answer
Don't onboard to the product spec. Onboard to the eval set. Spend your first real days inside the eval reports, the known failure modes, and the actual ticket-to-routing data flow, not the training code, before you let yourself have an opinion on the roadmap. Budget two focused days on the eval set and the failure list, one day pairing with an engineer on the real flow, and a wider range if the codebase turns out to be poorly documented.
Do this, in order
Read the eval set and the last few eval reports before you read anything else, including the spec.Why: that's where the model's real, already-known failure modes live. Everything else can wait a day.
Own a real time budget out loud: two days on the eval set and failure list, one day pairing on the actual data flow.Why: three days is a number you can defend. "I'll read the docs" is not a plan, it's a hope.
Check which end of the range you're on inside the first hour, not the third day.Why: ask for a 30-day-old eval report and the last routing override. Fast answers mean three days. Silence means budget for eight.
Read the version history for why past fixes were made, not just what shipped.Why: without it you'll re-propose something the team already tried and rolled back.
Set a hard rule: no roadmap opinion before you've touched a real failure example, no matter how confident the spec sounds.Why: this is the rule Henrike's first week broke, and it's the one thing that would have stopped it.
Sanity-check the budget against what skipping it actually cost.Why: even the worst-case eight days is a rounding error next to the ten weeks a mis-scoped roadmap fix can lose.
How to answer this, stage by stage
Nobody's grading whether you can say "eval set" with a straight face. They're grading whether you know what a new PM actually owes a codebase with a model in it, versus what any new hire owes any team.
01
Scope it to one concrete team and one real product
Say it like this
"Let's make this real. Thistledesk routes support tickets to the right team, and I've just joined as its PM. That's the onboarding I'm going to walk through, not onboarding in general."
Why this works
Pins a broad-sounding question to one product instead of letting it drift into generic new-job advice.
02
Name the method before touching a number
Say it like this
"I'd size this with BOUND. Break down what onboarding actually requires. Own the real numbers. Give a range instead of one lucky guess. Check the range against something real. Then say which single fact swings it most."
Why this works
Two seconds of structure signals a real estimate is coming, not a vibe dressed up as a plan.
03
Break down what onboarding needs, and what it doesn't, the B step
Say it like this
"I don't need to read every line of Hakon's training code. I do need the eval set and the last few eval reports, the known failure modes, the version history, so I know why past fixes were made, and the real path a ticket takes from a customer's inbox to an agent's screen."
Why this works
This is the line that keeps "onboard yourself" from turning into "read the whole codebase," which nobody actually has time for.
04
Own the numbers, the O step
Say it like this
"Two focused days on the eval set, the recent reports, and the open failure list. One more day pairing with Hakon, watching a real ticket move from intent scoring to team assignment. That's three days, minimum, before I'm allowed an opinion in a roadmap review."
Why this works
A number with a source survives a follow-up question. "A week or so" doesn't.
05
Give a range, and a fast way to check which end you're on, the U step
Say it like this
"Three days if the eval reports are current and the failure list is real. Up to eight if none of that's been kept up and it's all living in one engineer's head. I'd know which inside the first hour: ask for an eval report from the last thirty days and the last time someone overrode a routing decision. Fast answers, I'm near three. Digging through Slack, I'm near eight."
Why this works
Turns "it depends" into something you can actually check before lunch on day one.
06
Run the sanity check, the N step
Say it like this
"Worst case, eight days sounds like a lot to ask a new hire to sit on their hands. But the real fix at Stonebrook slipped ten weeks because I skipped this step. Eight days is a rounding error next to ten weeks."
Why this works
Turns a budget of days into a decision you can defend against "isn't that slow," not just a number on a slide.
07
Name the direction, then close on what onboarding actually produces, the D step
Say it like this
"The one thing that swings this most is whether the eval reports and the failure list are already a living document, or only live in Hakon's head. Not how smart the new PM is, and not how many product areas they already know. So here's what I'd actually walk out of day three with: a written list of the model's real known failures, and permission to say 'that's already tracked' the next time I'm tempted to pitch around it."
Why this works
Ends on the actual deliverable of onboarding, not a feeling of being ready.
Let's learn
Four boxes. The second one, scoring intent, is exactly where a ticket with two problems in it loses the second one.
Thistledesk reads a support ticket and decides, in under a second, which of Stonebrook's fourteen teams should get it next.
Before Thistledesk, a Stonebrook triage lead read and routed about 600 tickets a day by hand, roughly 40 seconds each, close to seven hours of a shift just keeping the queues balanced. With Thistledesk running, it routes correctly 94 percent of the time, in under a second, and that same triage lead's day dropped to about 40 minutes of spot-checking the tickets that got flagged as escalations.
Knowledge spark: what's an eval report?
A written check-in on how the model is actually doing right now, not how it did at launch. It lists what's wrong, how often, and on which kind of ticket. A team that keeps these current can answer "what's broken" in a sentence. A team that doesn't has to reconstruct the answer from memory every time someone asks.
Both feel productive by lunch. Only one of them actually teaches you what the model gets wrong.
Here's the turn. That six percent Thistledesk gets wrong was never really the problem. The problem is what a new PM does when she doesn't yet know which slice of that six percent is dangerous, and proposes a fix for a problem that was never the real one.
Thistledesk didn't need a new feature in week one. It needed someone to finish reading the last eval report first.
What it costs at its worst: for six weeks before Henrike joined, Thistledesk's eval reports had flagged the same open issue, safety-adjacent tickets that mention two problems at once get misrouted to the slow general queue instead of the urgent safety team, 31 percent of the time. In week nine, a ticket that read "dryer smells like burning plastic, also can you reschedule my routine filter cleaning" sat 19 hours in that slow queue before anyone caught it. Nothing caught fire. It was the closest call Stonebrook had seen.
The choice I would take back
Thistledesk's standard onboarding for a new PM was a half-day product walkthrough and a slide deck of the roadmap, no time set aside for the eval set at all. I'd take that back. No PM gets a vote in a roadmap review until they've spent real days inside the eval reports and the failure list, not the spec.
What I would leave alone: a PM joining Thistledesk's billing-settings team, which has no model behind it at all, doesn't need any of this. A half-day spec read and a shift shadowing support is genuinely enough there. This treatment is for the parts of the product a model is actually deciding.
The lesson: sounding informed in week one and being informed are not the same thing. Letting a new PM speak confidently before they've seen a real failure example isn't a shortcut, it's a bill that comes due later, and it's always bigger than the days it would have taken to read the report first.
Now here is the same thing as a story
The short version above is what you'd actually say out loud in an interview. Read this one for the Monday the pitch missed.
Henrike Ashcombe had spent six years as a PM on tools with no model in them, dashboards mostly, and she was good at the part that mattered most there: reading a spec fast, understanding what shipped and why, and having a sharp opinion by the end of the week. Thistledesk was her first product with a model actually deciding something. Nobody told her that changed the job.
Her first two days went the way onboarding always had. A product walkthrough. A demo of the routing dashboard. A slide deck of what was on the roadmap for the quarter. By Wednesday morning she felt ready, the way she always had by Wednesday morning.
Monday's roadmap review was actually the following Monday, but in her head she'd already been building toward it since Wednesday. She'd noticed something in the demo: agents seemed to trust the routing model unevenly, more on some tickets than others. Her idea was a small badge on each ticket showing its emotional tone, calm, frustrated, urgent, so agents could triage faster within their own queue. It was a good idea for a lot of products. She pitched it with real confidence.
Three panels, one Monday. The middle one is the moment the room found out what onboarding had actually skipped.
Hakon Deveraux waited for her to finish, then said it as gently as he could manage. "The ticket that almost became a real problem last month wasn't an angry one. The customer wrote pretty calmly: 'dryer smells like burning plastic, also can you reschedule my routine filter cleaning.' A tone badge would have shown that ticket as calm. It still would have gone to the wrong queue."
The room went quiet in the specific way a room goes quiet when everyone else already knew something the person talking didn't. Hakon's team had flagged the real issue in three straight eval reports over six weeks: tickets that mention two problems at once get routed to whichever team matches the first-mentioned issue, and when the second, buried issue is a safety one, it lands in the slow general queue instead of the two-hour safety desk about 31 percent of the time. A fix was already scoped. It just hadn't shipped, because it kept losing the argument for priority to whatever felt newest.
Knowledge spark: what's a dual-intent ticket?
One message that actually contains two separate problems, like a routine request and a safety concern in the same paragraph. A model built to pick one team has to pick which intent wins, and it often weighs whichever one takes up more of the message, not whichever one is more urgent.
Nobody blamed Henrike out loud. But her pitch did something worse than get rejected: it sounded finished enough that two people in the room walked out thinking product had a plan for the trust problem. Hakon's multi-intent fix, already scoped, got pushed from "next sprint" to "later this quarter." It slipped about ten weeks.
Over those ten weeks, roughly a dozen safety-adjacent, two-issue tickets a week kept landing in the wrong queue. Most sat an extra several hours past the two-hour bar. In week nine, one of them was the burning-plastic dryer ticket. It sat 19 hours before a general-queue agent, reading it cold, realized what it actually said and escalated it by hand.
We didn't lose ten weeks to a bad idea. We lost them to an idea that sounded finished enough to make a real one look optional.
Gulnara Petrosyan called that week, not furious, just done being patient. Henrike didn't have a good answer for her, and she knew exactly why: she'd never actually read the document that would have told her the real problem before she stood up and pitched a different one.
What she did next: she asked Hakon for an afternoon, not to pitch anything, just to read. The three eval reports. The open failure list. The two lines in the version history explaining why a keyword-weighting attempt at this exact problem had been tried and rolled back eight months earlier, because it over-triaged routine tickets as urgent and buried agents in false alarms. None of that took more than a day and a half. By the second day, she understood the actual shape of the bug well enough to ask Hakon a sharper question than anyone else had that quarter: was the fix scoped to catch two-issue tickets specifically, or any ticket the model was simply unsure about?
Run it forward with the classifier that finally shipped in week eleven, once it had priority again: the misroute rate on safety-adjacent dual-intent tickets dropped from 31 percent to 6 percent within two weeks of going live. The queue delay that had climbed to 14 hours at its worst came back down under 3.
What Henrike would tell her first-week self: a demo makes you feel like you understand a product. Only a failure example actually teaches you what it doesn't do yet, and those two feelings are not the same thing, no matter how alike they feel from the front of a roadmap review.
BOUND, for pricing the ramp-up nobody puts on a calendar
Not a way to sound careful about a new job. BOUND turns "get up to speed" into a number Henrike, and whoever manages her, could actually plan a quarter around, instead of a vague sense that reading the spec was probably enough.
BBreak it down. What does this actually require, and what doesn't it?
A new PM doesn't need every line of Hakon's training code, and doesn't need to be able to retrain the routing model herself. She does need four things: the eval set and recent reports, the known and tracked failure modes, the version history and why past changes were made, and the real path a ticket takes from a customer's inbox to a routed queue.
Skip this split and "onboard yourself" either turns into reading a whole repository nobody has time for, or shrinks to reading a spec that teaches nothing about what the model actually gets wrong.
Three real must-haves, two things a new PM can safely skip in week one.
OOwn the numbers. Where does each one actually come from?
Two days on the eval set, the recent reports, and the failure list. One more day pairing with the model's owner on the real ticket-to-team flow. Three days, not a guess, it's roughly what it took Henrike to actually catch up once she started doing it the right way round, and it matches how long the team itself spends reviewing misroutes each sprint.
A number only counts as owned if you can say where it came from when someone pushes on it. "About a week, probably" is a guess wearing a number's clothes.
Where three days of real onboarding actually goes, best case and worst case
Eval set, reports, failure modesPairing on the real data flow
The eval-set piece is what stretches, not the pairing day. A team with no living failure list turns a 3-day ramp into an 8-day one.
UUse a range, not one number.
Three days when the eval reports are current and the failure list is real. Up to eight when none of it's been kept up and it's living only in one engineer's head. Check which end you're on inside the first hour: ask for a report from the last 30 days and the last routing decision someone overrode. A fast, specific answer means three. Hunting through old Slack threads means eight.
A single number here repeats the exact mistake Henrike made: sounding certain about something nobody could actually be certain about yet.
The gap between the two ends isn't laziness on either side, it's whether the team ever wrote its own failures down.
NNail the sanity check. Does the number survive contact with something real?
Worst case, eight days feels like a lot to ask a new hire to sit on. But the actual cost of skipping it, in this exact story, was a ten-week slip on a fix that mattered, and 19 hours a ticket sat wrong the week it mattered most. Eight days is under a sixth of ten weeks.
This is the step a rushed onboarding plan skips. It's what turns "give the new PM time" from a nice-to-have into a defensible budget line.
Safety-adjacent ticket queue delay, by week, before and after the real fix shipped
Average queue delay, safety-adjacent ticketsSafety desk target, 2 hours
The line was already climbing in week 3, the same week the room decided the trust problem had a plan. Nobody looked at this chart again until week 9.
DDirection. Which assumption moves this most, and what does onboarding actually produce?
Not the new PM's experience level, and not how many products they've shipped before. The single biggest swing factor is whether the eval reports and the failure list are already a living, current document, or whether that knowledge only exists in one engineer's head. So onboarding doesn't end with a feeling of being ready. It ends with a written list of the model's real known failures, and the standing rule that no roadmap opinion gets voiced before that list has been read.
Naming the fact that actually swings the estimate, instead of the biggest number in it, is what separates a real estimator from a confident guesser.
Same three days either way. Only one of the two paths spends them on the thing that actually teaches you something.
One alternative the team considered and rejected: a formal, week-long onboarding curriculum with a written quiz on the whole codebase, modeled on how the engineering org onboards new hires. It lost, because most of what actually mattered fit inside three real days, and a manufactured curriculum would have added ceremony without adding a single real failure example. A second alternative, closer to what Henrike's old habit actually was: have every new PM shadow live support calls for a week before touching the product at all. That one teaches real empathy for the agents. It teaches nothing about what the model itself gets wrong, which is exactly the generic version of onboarding that would work fine for any new hire and doesn't answer this question at all. The AI-specific failure sitting under all of this is a silent misroute: a wrongly routed ticket looks exactly as confident as a correctly routed one, with nothing in the interface signaling doubt to the agent who picks it up. The guardrail is a per-ticket confidence score, and any ticket the model scores as both dual-intent and low-confidence gets a mandatory human glance before it's filed quietly into a queue. And the trade-off is real and accepted on purpose: three to eight days of a new PM not shipping anything, in exchange for a roadmap that isn't built on a guess about what's actually broken.
And if you want to be sure it really works, try it somewhere else
Same five letters, a radiology triage tool instead of a support desk, and this time the number that swings the estimate isn't how well-kept the eval set is, it's whether anyone questioned what the eval set's own right answers are worth.
Pleuraview, built by Corvane Health, reads a chest X-ray and flags the ones likely to show a collapsed lung for urgent review before a radiologist opens the file. Jareth Rourke just joined as its product manager. Lucrezia Silvergate is the informatics engineer who maintains its eval set and its labeling pipeline.
Run BOUND on it. Break it down: Jareth doesn't need to understand every layer of Pleuraview's model architecture, but he does need to know how the eval set's own labels were made, since in radiology the "right answer" isn't a fixed fact the way a routing decision eventually is. Own the numbers: Pleuraview's eval set is built from three radiologists voting on each scan, and on about 9 percent of scans, they don't agree. Use a range: two to five days depending on whether Corvane Health's labeling disagreements are already logged per scan or have to be reconstructed by re-reading each contested case.
Different tool, same shape of question. Here the eval set itself needed a second look, not just a first one.
Where Pleuraview's answer genuinely differs
Henrike's gap was that she never opened the eval set at all. Jareth's team already knew to open it on day one, radiology PMs learn that fast. Their blind spot was different: nobody had ever asked how confident the eval set's own ground truth was. A scan Pleuraview gets "wrong" on a contested 9 percent might really be the model disagreeing with one out of three humans on a genuinely hard call, not a mistake at all. Treating every eval miss as equally real, on that slice, would have sent Jareth chasing failures that were never failures.
Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: separate what a new PM needs from what they can skip, own a real number for the knowable part, and turn the unknowable part into a checked range, never a guessed one.
Cost: no budget to hold a new PM off the roadmap for more than two days. Shrink the range honestly instead of hiding it, and say plainly what "good enough to have an opinion" has to mean by day two if that's all the calendar allows.
The model got better, for real: say the routing model's overall accuracy jumps to 98 percent right as the new PM starts. The discipline barely changes, because a higher headline number can hide a worse slice just as easily as a lower one can, maybe more easily, since nobody goes looking underneath good news.
Where people run it wrong.
They let "get up to speed" stand as the whole plan, instead of naming what it actually has to include.
They read the eval set once and call it done, instead of checking whether it's still being kept current.
They treat a confident-sounding idea in week one as a sign of readiness, instead of a sign nobody's checked it against a real failure yet.
How to use it live. Before answering, ask yourself one plain question out loud: "what would I need to see before I'd trust my own opinion on this." Whatever document that is, that's where onboarding actually starts.
Flashcards (tap any card to flip it)
1 · THE FRAMEWORK
What framework fits sizing a new PM's real onboarding to an ML codebase?
Tap to flip
ANSWER
BOUND: break it down, own the numbers, use a range, nail the sanity check, name the direction. Built for splitting a genuinely knowable estimate from one people usually guess at instead.
2 · THE CAST
Who is this answer about?
Tap to flip
ANSWER
Henrike Ashcombe, new PM at Thistledesk. Hakon Deveraux, the research engineer who owns the routing model. Gulnara Petrosyan, head of support operations at Stonebrook Appliances, the client running its warranty desk through Thistledesk.
3 · THE SPLIT
What does a new PM actually need to learn, and what can they skip?
Tap to flip
ANSWER
Needs: the eval set and recent reports, the known failure modes, the version history, and the real data flow. Can skip: reading every line of the training code.
4 · THE OWNED NUMBER
Fill in the blank: ___ days on the eval set and failure modes, ___ day pairing on the real data flow.
Tap to flip
ANSWER
Two days on the eval set, recent reports, and failure modes. One day pairing on the real ticket-to-team flow. Three days minimum, in the well-documented case.
5 · THE RANGE
What's the real range, and how do you check which end you're on fast?
Tap to flip
ANSWER
Three to eight days. Ask, in the first hour, for a recent eval report and the last time a routing decision got overridden. Fast answers mean three. Hunting for them means eight.
6 · THE SANITY CHECK
Does the worst-case 8-day budget survive a smell test against something real?
Tap to flip
ANSWER
Yes. Eight days is under a sixth of the ten-week slip Stonebrook's real fix took once Henrike's rushed pitch sent it down the priority list.
7 · THE DIRECTION
Which single fact swings this estimate the most?
Tap to flip
ANSWER
Whether the eval reports and failure list are already a living, maintained document, or exist only in one engineer's head. That, not the PM's experience level, is what moves the estimate from three days to eight.
8 · CROSS-PRODUCT TRANSFER
Section 4 runs BOUND again on a different product. Which one, and what's genuinely different about its answer?
Tap to flip
ANSWER
Pleuraview, Corvane Health's chest X-ray triage tool. There, the twist isn't a skipped eval set, it's that the eval set's own ground truth is a 3-radiologist vote that disagrees 9 percent of the time, so not every "miss" is a real miss.
Check yourself Score: 0 / 0
True or false
1. True or false: the right way to onboard to an ML codebase as a new PM is to start with the training code, since that's where the model's actual behavior comes from.
True
False
Show hint
Check the B step and the decision-tree diagram, "Where the real shortcut is."
Show answer
False. Start with the eval set and real failure examples. The training code is explicitly one of the things a new PM can skip in week one, it's where the leverage isn't.
Multiple choice
2. Why couldn't Henrike's tone-badge pitch have caught the burning-plastic dryer ticket?
A. Thistledesk doesn't support tone detection on any ticket type.
B. The ticket's tone was calm, so a badge sorting by tone wouldn't have flagged it, and the real problem was a second, buried intent getting dropped, not the ticket's emotional register.
C. Stonebrook's agents don't read tone badges.
D. The ticket never reached the general support queue at all.
Show hint
Check Hakon's correction in the story, and the knowledge spark on dual-intent tickets.
Show answer
B. The customer wrote calmly. A tone badge sorts by emotion, not by whether a second, more urgent issue is buried in the message.
Fill in the blank
3. The real onboarding budget is ___ days at the well-documented end, and ___ days at the poorly-documented end.
Show hint
Check the U step and the stacked bar chart's two totals.
Show answer
3 days, and 8 days. The eval-set-and-failure-modes piece is what stretches between the two, from 2 days to 6, not the pairing day.
Short answer, name the reversal
4. What old decision would Henrike, or Thistledesk, take back, and why did it make sense before?
Show hint
Look at the key point box titled "The choice I would take back," in Let's learn.
Show answer
Model answer: The standard onboarding for every new PM was a half-day spec walkthrough and a roadmap deck, with no time set aside for the eval set. It made sense for a normal software team, where the spec really does describe most of what you need to know. It broke down the moment a model with its own failure modes entered the picture.
Short answer, apply it yourself
5. Think of a system you had to learn fast at a job or in school, one with a part you couldn't fully inspect yourself. What would the "eval set first" version of onboarding to it have looked like?
Show hint
Look for whatever document already lists what that system gets wrong, before reading anything that describes what it's supposed to do.
Show answer
Model answer: Joining a call center as a new supervisor: instead of reading the training manual first, the "eval set" would be the last month of escalated calls and complaint tickets, the real record of where the process actually breaks, read before sitting in on a single normal shift.
Short answer, work the number
6. If Henrike had spent her real three days before the Monday roadmap review instead of after the near miss, what would she most likely have pitched instead of the tone badge?
Show hint
Check what the eval reports had already flagged for six weeks, in Let's learn.
Show answer
Model answer: She'd likely have pushed to reprioritize Hakon's already-scoped multi-intent classifier fix, the one the eval reports had flagged as the real open issue for six weeks, instead of proposing something new that competed with it for the same sprint.
Before you close the answer
Why this works
Tests whether a candidate treats onboarding to an ML-backed product the same as onboarding to any feature team, or knows the specific artifacts, the eval set, the failure list, the version history, that only matter once a model is actually deciding something. Most candidates can say "get up to speed fast." Fewer can say exactly what that has to include.
Follow-up traps
"What if the eval set itself turns out to be wrong or stale?" Response: that's exactly why the version history matters too, so a new PM can see when it was last checked and trusted, and it's why the Pleuraview example in Section 4 pushes the question one layer deeper: sometimes it's not the eval set that's wrong, it's the confidence you're allowed to have in its own ground truth.
"Isn't three to eight days a long time to hold a new PM off the roadmap?" Response: it's a sixth of the ten weeks a mis-scoped roadmap decision cost once, in this exact story, and it's shorter than most teams' onboarding for a role with far less at stake.
If pressed
The version history matters for a reason that never came up directly: eight months before Henrike joined, someone had already tried to fix this exact dual-intent problem by weighting safety-related keywords higher in the routing score. It got rolled back within two weeks, because it over-triaged routine tickets that merely mentioned a part like "circuit" or "spark" as urgent safety issues, burying agents in false alarms. A new PM who never reads that history is one who might confidently re-propose the same fix, and get the same rollback, a second time.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
