InterviewAdvancedEval-Driven Specification / Writing a PRD for an AI feature / #26
Which section of an AI PRD is most often missing, and what damage does that cause?
The direct answer
The eval section is the one most often missing: a fixed, segmented test set with a pass or fail bar that every new model version has to clear before it ships, not just a promise to watch a number afterward. Its absence is the hardest gap to undo, because the first sign of trouble is usually a real person, not a dashboard. Write that section before anything else, or the rest of the document is resting on nothing.
The ranking, by what breaks first if skipped
Write the eval section first: a segmented test set with a pass or fail bar, gated before any model version ships.Why: it's the section most often missing, and its absence is what turns a model regression into someone's performance review instead of a same-day fix.
Name the outcome every other section is protecting, in one line, before ranking anything.Why: without it, ranking sections is opinion wearing a method's clothes.
Rank candidate gaps by what's hardest to undo, not by how painful they'd be to write.Why: a missing logging section is annoying. A missing eval section lets real damage land on a real person before anyone can catch it.
Check what the other sections quietly assume the eval section already provides, before trusting them.Why: a rollback plan that looks complete on paper is often resting on a baseline that was never actually built.
Run the cheap check: pull the last five PRDs your team shipped and search for a named held-out test set.Why: it turns "I think this is the common gap" into a number anyone can go verify.
Leave low-stakes sections, like UI copy, loose and fix them after launch.Why: rigor spent there doesn't prevent any lasting damage, so it's the cheapest place to move fast.
How to answer this, stage by stage
Seven moves, in the order I'd actually say them. The trap in this question is naming a section you personally find tedious to write. An interviewer isn't asking which section is boring. They're asking which gap you can't walk back once code ships.
1
Anchor the question to one real system
Say it like this
"Let me make this concrete. Say it's Cutwater Fulfillment, a pick-path app called Waylight that tells warehouse pickers which aisle and bin to go to next on their scanner. I've sat through three PRDs on that product, and I'm going to talk about the one section that's missing from basically all of them."
Why this works
A generic PRD question invites a generic list of headings. Naming a real system forces a real, defensible answer.
2
Reframe what's actually being asked
Say it like this
"This isn't 'name the six sections of an AI PRD.' Every PM already knows those. The real question is which one section, when it's the one that's missing, does damage you can't walk back. That's harder, and it's the one worth answering."
Why this works
Splits a strong answer from a recited table of contents before you've said a single section name.
3
State the outcome every section is competing to protect
Say it like this
"Every section of this document exists to protect one thing: that we can tell, before a picker ever walks it, whether a new version of the route is actually better or worse, for every kind of aisle it touches, not just on average."
Why this works
Without naming the outcome first, ranking sections is just a gut feeling with a label on it.
4
Rank the candidate gaps by what's hardest to undo
Say it like this
"A missing logging section, I can add next sprint and start collecting data from that day forward, nothing's lost. A missing rollback plan, I can still hotfix once I notice something's wrong. A missing eval section is different: nobody notices anything's wrong until a person does, and by then it's already happened to them."
Why this works
Naming candidates and ranking them shows the method instead of asserting a favorite section.
5
Show what the missing section quietly breaks downstream
Say it like this
"The rollback section looks fine on paper: 'roll back if picks-per-hour drops.' But that number needs a baseline sensitive enough to catch trouble. Without a segmented eval section, the only baseline you've got is a warehouse-wide average, and a 15 percent slice going bad can't move that far enough to trip it."
Why this works
This is where the interviewer sees you understand a document's sections depend on each other, not just sit side by side.
6
Give the cheap check that proves it's the common gap
Say it like this
"Here's how I'd check this isn't just a hunch: pull the last five AI PRDs your team shipped and search for 'held-out' or 'pass bar.' At Cutwater, zero out of five had either. All five said some version of 'we'll monitor it in production.'"
Why this works
A number you can go verify in ten minutes beats a strong opinion every time.
7
Name the section and close on one line
Say it like this
"So: the eval section, a fixed test set with a pass bar per segment, gated before ship. That's the one most often missing, and it's the one where the damage lands on a real person before it ever lands on a dashboard."
Why this works
Ends on the literal answer to the question, in a sentence an interviewer could write down.
Let's learn
An AI PRD is the document a team writes before an AI feature gets built, spelling out what it does, how the team will know it's working, and what happens the moment it's wrong.
Say Cutwater Fulfillment runs three warehouse floors for regional grocery and general-merchandise orders. Waylight is the app on every picker's handheld scanner. Instead of walking a fixed route down every aisle, or working off a paper list sorted by bin number, a picker's scanner tells them, order by order, the shortest walk between only the bins that order actually needs.
Knowledge spark: what a held-out test set is
A batch of real, tricky examples the model never trains on, kept aside just to test new versions against, with a clear pass or fail line. Not a dashboard you watch after launch. A test you run before anything reaches a real person.
Before Waylight, an average order took a picker about 14 minutes, most of it walking, and one person cleared roughly 4 orders an hour. After it shipped on the main dry-goods floor, that dropped to about 6 minutes an order, closer to 9 orders an hour.
Eighteen months later, Cutwater's routing team shipped a new version of the model, built to cut down on pickers crossing paths on the main floor. It worked there. Nobody had walked it through cold storage first, where the aisles run one way only, because you can't back a loaded cart past someone in a two-degree freezer aisle.
Here is the part that matters. The extra distance was never the real damage. What mattered was what nobody could see happening. Cold storage handles about 15 percent of Cutwater's order volume, 40 pickers out of 260. Over six weeks, their average pick time crept from 9 minutes to 13, as the new model's shorter-looking route sometimes sent a picker the wrong way down a one-way aisle, and the only fix was a full loop back around. Warehouse-wide, the number the team actually watched, barely moved: 6.3 minutes to 6.6. A 15 percent slice can't drag an 85 percent average very far.
Average pick time, cold storage vs. warehouse-wide, six weeks after the new routing version shipped
The amber line is cold storage, 15 percent of order volume. The navy line is the warehouse-wide average, the only number anyone was actually watching. It never left "normal week-to-week noise" the entire six weeks.
We didn't send pickers seven minutes out of their way. We put a good picker's name on a write-up.
Three of Cutwater's most senior cold-storage pickers got pulled into performance conversations that quarter for missing quota. It wasn't their fault. Even after the routing bug got fixed, the cold-storage crew kept ignoring Waylight's suggested route for another two months and went back to working from memory, because trust doesn't reset just because the code did. For those two months, the app built to make them faster was slower than the old paper list had ever been.
The eval section is what the other three are quietly waiting on
The choice I would take back
Eighteen months earlier, when the first Waylight PRD got written, Cutwater ran one warehouse floor, no cold storage, no one-way aisles. The success-metrics section said "watch picks-per-hour in production." That was a reasonable call for one floor where every aisle looked like every other aisle. Nobody rewrote it once cold storage joined the network. I'd take that back: a fixed, segmented test set, one per aisle type, with a pass bar every new model version has to clear before a picker ever sees it.
What I would leave alone. The exact wording on the scanner screen, "Go to Aisle 12, Bin B4" versus "Next: 12-B4," never needed this kind of rigor. Cutwater rewrote that copy three times after launch based on picker feedback, each time for free, because getting it wrong for a week costs nothing lasting.
The lesson. A missing section in a PRD doesn't look missing on the day you skip it. The code still ships. The review still happens. The dashboard still moves, just not by much. The gap only shows up weeks later, in a slice of the floor nobody was watching, in a performance conversation about a picker who didn't do anything wrong. The eval section isn't paperwork. It's the only section built to catch that before a person does.
Now here is the same thing as a story
You don't need this to answer the question. Read it slower, when you want to feel why an eval section earns the top spot, not just recite that it does.
Farhan Iqbal has run product for Cutwater's pick-path system for two years, across all three warehouse floors, and he's the only PM who touches it. Hand him a floor plan and a heat map and he'll tell you inside a minute which aisle is about to back up.
Waylight launched on the main dry-goods floor first, and for the better part of a year it was the best thing Farhan had shipped. Picks-per-hour climbed every quarter. Pickers who used to dread the Tuesday-morning rush started clearing it before lunch.
When Cutwater's routing team started pushing new model versions, roughly one every few weeks as the warehouse network grew, Farhan used to sit with an engineer and watch four or five route replays side by side, old version against new, floor by floor. Then it was two replays, and only on the main floor, because the main floor carried 85 percent of the volume. Then, most weeks, he just opened the picks-per-hour dashboard, saw the line trending up or flat, and moved on to the next fire.
One of these gets fixed next sprint. The other doesn't get a second try.
There wasn't a day it changed. Nobody filed a ticket. Nobody's dashboard turned red. Six weeks after a routing update meant to smooth out main-floor congestion, a cold-storage shift lead caught Farhan near the mezzanine stairs before the morning cutoff and asked why three of her best pickers had gone soft. Not annoyed, more confused. "They're my fastest people. Now they're the ones missing quota."
Farhan pulled the scan timestamps for cold storage specifically, something he hadn't looked at on its own in months, and found it inside an hour: the new routing version, tuned and tested against the main floor's two-way aisles, had no idea cold storage only ran one way. When its shorter-looking path crossed a one-way aisle backward, a picker's only option was a full loop, sometimes four extra minutes, on top of a job that was already the slowest, coldest one in the building.
The four extra minutes were never the real cost. Three senior pickers, people who'd trained half the newer hires on that floor, had already sat through a conversation about missing their numbers, for a problem the app had caused.
Nobody on the cold-storage crew had a chart in their head. They had a feeling about whether Waylight was worth listening to, and it only had two settings: trust it, or work around it. A month of missing quota, blamed on them, flipped it. It didn't flip back just because engineering fixed the model two weeks later.
The bug lasted two weeks. Not trusting Waylight again lasted two months longer.
Eighteen months before any of this, in the room where the first Waylight PRD got signed off, someone had asked whether they needed a real test set before shipping model updates, or whether watching picks-per-hour in production was enough. Cutwater ran one floor back then. Every aisle looked like every other aisle. "We'll just watch the number" was the right call, and it cost nothing, because there was nothing yet to hide behind an average.
I would take that decision back, and here's the difference it makes. With a segmented eval section in place, before that congestion-fix version ever reaches a picker, it gets replayed against a held-out set of cold-storage routes, about ten minutes of automated testing, and it fails the pass bar the same afternoon it's built. It never ships. No shift lead pulls Farhan aside six weeks later. Nobody sits through a conversation about a bug in an app they trusted.
The old design asked one question: did the average get better. The new one asks a harder one: did every section stay at least as good as before, one aisle type at a time.
The thing I'd tell myself, back in that first PRD review: skipping the eval section felt like skipping paperwork. It was actually the only section built to tell me something had broken before a person did.
ORDER, run against every section that could have been the answer
TRACE would look for when a metric already dropped, but nothing had dropped yet when this question gets asked, it's about ranking a gap before it bites, not diagnosing one that already has. GUARD would look for who can't push back, and that's not the shape of this question either. What's actually being ranked is which candidate gap does the earliest, hardest-to-undo damage, so the framework is ORDER.
O, outcome. Every section of this PRD is competing to protect one thing: that the team can tell, before a picker walks it, whether a new route is genuinely better across every kind of aisle it touches, not just on average.
R, reversibility. A missing logging section is retrofittable next sprint, nothing about the past is lost. A missing rollback plan still lets you hotfix once you notice trouble. A missing eval section is different: nobody notices anything's wrong until a real person does, and that can't be undone once it's happened to them.
D, dependency. The rollback section's trigger, "roll back if picks-per-hour drops," quietly assumes a baseline sensitive enough to catch a small slice going wrong. Without a segmented eval section, the only baseline that exists is a warehouse-wide average that a 15 percent slice can't move far enough to trip.
E, evidence. Cheap to check: pull the last five AI PRDs your org has shipped and search for "held-out" or "pass bar." At Cutwater, zero of five had either. All five said some version of "we'll monitor it in production."
R, rank. The eval section first: a segmented test set with a pass bar, gated before ship. Everything else, success metrics, the rollback plan, failure modes, the problem statement, can be written or rewritten after the fact without a real person paying for the gap. This one can't.
The check that keeps ORDER honest
Swap the outcome and the order should move. If a wrong route only ever cost a picker a shrug and thirty extra seconds, the eval section could safely sit lower on the list. It isn't the six weeks alone that earns it the top spot. It's that three people had already sat through a conversation about numbers this document should have caught first.
Run it where a mistake can't be looked up again
A municipal library system runs an AI tool that flags likely duplicate catalog records, two entries for what's really the same book, and merges them automatically overnight to keep the catalog clean for search.
O. Every section of this PRD protects one thing: a patron searching the catalog finds a real, correct record for the book they want, including its right edition and its hold queue.
R. A wrongly flagged duplicate caught inside the library's 30-day retention window is easy: a cataloguer un-merges it, the history is still there. A duplicate caught after that window is not. The book is still on the shelf. Its record, its checkout history, and its hold queue are already purged for good.
D. The nightly auto-purge policy, which frees cataloguing staff from manual cleanup, quietly assumes every merge that reached it was already verified correct. Without an eval section testing the merge model against known-hard pairs, translated editions, multi-volume sets, that assumption was never checked.
E. A cataloguer pulled last quarter's merge log against a list of known-tricky pairs. None of them had ever been run through the merge model before it shipped, only watched afterward as a "merges per night" count.
R. Same answer. The eval section, tested specifically against translated editions and multi-volume sets, comes first. The retention policy, the rollback plan, and everything else assume it already exists.
One quarter's wrongly merged records: caught in time vs. purged for good
Those 7 records, and the hold queues and item history attached to them, are not coming back. No cataloguer's mistake, a merge model that was never tested against translated editions and multi-volume sets before it started running unattended.
Swap the trigger and it still runs
The library switches the merge job from nightly to hourly, to keep the catalog fresher. The order doesn't move. The eval section still gets checked more often, not skipped.
A cheaper record-matching vendor cuts licensing cost 40 percent but reads foreign-language titles worse. Same order. The eval section matters more with a noisier model, not less, since it's the only thing that would actually catch the noise.
A new merge model version scores meaningfully higher on the vendor's own benchmark. Doesn't reorder anything. Better on someone else's benchmark is a claim until it clears your own segmented test set.
Where people run it wrong
Treating a rising "merges per night" count as proof the tool is working, when it only measures how often it acted, never whether it acted correctly.
Writing the retention or rollback policy before the eval section exists, so a policy that looks efficient on paper quietly assumes a check that was never built.
Spending time-boxed hours on the parts that are satisfying to build, like the merge review screen, instead of the parts that are hardest to notice going wrong.
If you're asked this cold
Say the outcome out loud before naming a single section. "Every part of this document is trying to protect one thing: the record a patron finds is the right one." Then rank from there. Naming the outcome first is what turns "here's a section I happen to like" into an actual argument.
Flashcards (click a card to flip it)
1 · THE FRAMEWORK
Which framework fits "which section of an AI PRD is most often missing, and what damage does it cause," and why not TRACE or GUARD?
Tap to flip
ANSWER
ORDER, for ranking candidate missing sections by how hard the damage is to undo. TRACE diagnoses a metric that already dropped, and nothing has dropped yet here. GUARD looks for an unequal power imbalance, and this is a document gap, not a policy question.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Farhan Iqbal, the only product manager covering the pick-path system across Cutwater Fulfillment's three warehouse floors, two years into the job.
3 · THE HABIT
What habit let a bad routing model reach every cold-storage picker's scanner unnoticed?
Tap to flip
ANSWER
Farhan used to watch route replays segment by segment before every new model version shipped. Over months of frequent releases, that thinned to just checking whether the warehouse-wide average moved the right way.
4 · THE BLIND SPOT
What's the gap this whole answer turns on?
Tap to flip
ANSWER
A PRD that said "we'll monitor picks-per-hour in production" instead of naming a fixed, segmented test set with a pass bar per aisle type, so a regression in a 15 percent slice of the floor had nowhere to get caught before it reached a picker.
5 · THE OLD DECISION
What decision would you take back, and why did it make sense at the time?
Tap to flip
ANSWER
Writing "monitor picks-per-hour in production" instead of a segmented eval section, 18 months earlier, when Cutwater ran one warehouse floor with no cold storage and no one-way aisles. There was nothing to segment yet.
6 · THE NUMBER
Zero of the last ___ AI PRDs Cutwater shipped named a held-out test set with a pass bar; all ___ said some version of "we'll monitor it in production."
Tap to flip
ANSWER
5 and 5. That's the cheap check that shows this gap wasn't a one-off, it was the pattern.
7 · THE REPLAY
Same six weeks, PRD with a segmented eval section this time: what changes?
Tap to flip
ANSWER
The new routing version gets replayed against the cold-storage held-out set before it ships, about ten minutes of automated testing. It fails the pass bar the same afternoon it's built. It never reaches a picker's scanner. Zero write-ups.
8 · THE TRANSFER
Section 4 runs ORDER again on a different product, in a different industry. Which one, and what plays the role of the eval section there?
Tap to flip
ANSWER
A municipal library's AI duplicate-record merger. The equivalent is a held-out set of known-tricky pairs, like translated editions and multi-volume sets, with a pass bar the merge model has to clear before it's allowed to run unattended.
Check yourself Score: 0 / 0
True or false
1. True or false: the exact wording on Waylight's "next bin" instruction screen needed the same eval rigor as the routing model itself. Say why.
True
False
Show hint
Ask which mistake reaches a real person before anyone can catch it, and which one is cheap to fix after launch.
Show answer
False. The screen copy is cheap to test after launch and fix the same week. The routing model's mistakes reach a picker's shift before anyone can catch them, so only that one needs a pre-ship gate.
Multiple choice
2. Which PRD section does this answer say is most often missing, and causes the earliest, hardest-to-undo damage?
A. The rollout and rollback plan
B. The problem statement
C. The eval section: a segmented test set with a pass bar
D. The success-metrics section, defining the north star
Show hint
Ask which section decides whether a mistake is caught before it ships, versus watched for after.
Show answer
C. Success metrics describes what to watch after launch. The eval section is the pre-ship gate, a fixed test set a version has to pass before it reaches anyone, and that's what was missing.
Fill in the blank
3. Cold storage's average pick time drifted from 9 minutes to ______ minutes over six weeks, while the warehouse-wide average moved only from 6.3 to ______ minutes.
Show hint
It's the number that never left "normal week-to-week noise" on the dashboard Farhan actually watched.
Show answer
13 and 6.6. A 15 percent slice going from 9 to 13 minutes barely dents an 85 percent average that stayed near 6, which is exactly why nobody caught it from the dashboard.
Multiple choice
4. What does the dependency step (D) in this answer's ORDER argue?
A. The rollback plan's trigger assumes a baseline the eval section is supposed to provide, and can't without it
B. The problem statement must be written before the scope section
C. The success-metrics section and the eval section are the same thing
D. Cold storage should be left out of the PRD entirely
Show hint
Ask what "roll back if picks-per-hour drops" is quietly assuming already exists.
Show answer
A. A rollback trigger needs a baseline sensitive enough to catch trouble. Without a segmented eval section, the only baseline is a warehouse-wide average that a small slice can't move.
Short answer, apply it yourself
5. Pick an AI feature you use or are building. Which section of its PRD, if missing, would cause damage nobody could undo once a real person had already lived through it?
Show hint
Look for the section that decides whether a mistake is caught in testing or lived through by a real person first.
Show answer
Model answer: "A meal-delivery app's AI substitution picker, the feature that swaps an out-of-stock item for a similar one. The section that's hardest to go back and fix is the eval set for allergens: once it silently swaps peanut butter cookies for regular ones and someone's already eaten, no rollback plan helps."
Short answer, the number question
6. If cold storage handled 40 percent of order volume instead of 15 percent, would the warehouse-wide dashboard still have hidden the regression for six weeks? Say what changes and what doesn't.
Show hint
Reversibility is about what happens once a mistake is caught, not about how big the slice is.
Show answer
Model answer: "No, not for six weeks. At 40 percent of volume, a jump from 9 to 13 minutes drags the warehouse-wide average enough to move outside normal noise much sooner, maybe within a week or two. What doesn't change: only a segmented eval section, not a bigger dashboard number, would have caught it before a single picker got written up. A worse dashboard complains sooner. It still doesn't say why or where."
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
