InterviewAdvancedEval-Driven Specification / Writing a PRD for an AI feature / #8
Should the PRD name a specific model? Argue both sides.
The direct answer
No, don't name a specific model in the PRD's requirements. Name the eval bar instead, the exact behavior the feature has to hit, and let engineering pick whichever model clears it, today and after the vendor changes something underneath it. Put today's model choice in a separate implementation note that's allowed to go stale, never inside the requirement itself.
What to actually do, in order
Don't put the model's name in the requirement. Put the eval bar there instead.Why: this is the one call the rest of the argument hangs on. Get it backward and every other bullet is just decorating a PRD that's already wrong.
Write the bar as a number checked on a schedule, not a one-time pass at launch.Why: the failure here was a 5 percent bar that existed on paper and nobody was actually running against it, week to week.
Put today's model name in a separate implementation note, and expect it to be wrong by next quarter.Why: a note that's allowed to go stale can't mislead anyone. A requirement that's allowed to go stale, can.
Name who eats each kind of mistake before you argue either side.Why: that's what turns "the model matters" into something you can check, instead of a feeling about caution.
Write down the real cases where you'd name the model on purpose.Why: a compliance rule, a spike with no eval suite yet, a capability only one vendor has. Without these, the pick sounds like a rule instead of a judgment call.
Pick the number that tells you a swap already happened, not the day someone approved it.Why: a model's name doesn't drift. Its behavior does, quietly, and that's the thing actually worth watching.
How to make this argument out loud
Eight moves. This is a commit question with a real exception built into it, so say the position first, then earn the exception. Don't bury it in the middle.
1
Scope it to one concrete product
Say it like this
"Let's say this is one product: a bot that reads a pull request and posts comments on it, like a fast, literal code reviewer. I'll answer for that, not for 'AI features' as a category."
Why this works
Grounds an abstract question in something specific before either side of the argument starts.
2
State the position before any reasoning
Say it like this
"My answer is no. Don't write the model's name into the requirements. Write the behavior it has to hit, and let whoever's building it pick the model that clears that bar."
Why this works
PICK rewards commitment. A hedge here reads as "it depends," which fails the question before the reasoning even starts.
3
Reframe what the question is really asking
Say it like this
"The real question isn't 'can the PRD mention a model.' It's 'what does this document have to still be true about, six months from now, after the vendor changes something you never asked them to change.'"
Why this works
Moves the argument from a naming preference to what a requirement is actually supposed to do.
4
Name who feels each kind of mistake
Say it like this
"If engineering picks a weaker model to save cost, the eval run fails fast, and it's their Tuesday that gets a little annoying. If the PRD just names a model and the vendor quietly updates it behind that same name, nobody's job changes, until three months later the whole engineering org's pull request comments are worse and nobody agreed to that."
Why this works
This is the impact step. It separates visible cost from invisible cost instead of talking about "quality" as one blur.
5
Put a number on the asymmetry
Say it like this
"At the company I'm thinking of, an eval run caught a bad model swap in about a week. A PRD that just named the model didn't catch a real drift for eight weeks, and by then two-thirds of the bot's comments were getting dismissed without anyone reading them."
Why this works
This is C, the heart of the pick, said as something you could check, not a feeling about which side sounds more careful.
6
Give the cases where you'd name the model on purpose
Say it like this
"I'd name it anyway in three cases. One, a compliance rule makes you log exactly which model touched an output. Two, it's a two-week spike with no eval suite built yet, so the model's name is the only anchor you have. Three, the feature genuinely needs something only one vendor's model can do. That's naming a capability, not decorating a doc."
Why this works
This is K. It shows the position has real edges instead of being a rule memorized from a blog post.
7
Say what stays true either way
Say it like this
"None of this means the model doesn't matter. Someone still has to pick one, test it, and write down which one shipped. It just doesn't belong in the sentence that says what the feature has to do."
Why this works
Stops the answer from sounding like "never mention the model anywhere," which isn't actually the position.
8
Close on the line
Say it like this
"So: no, don't name it in the requirement. Name the bar. Name the model in a note you expect to be wrong by next quarter, and build the thing that checks whether it still is."
Why this works
Restates the position in one breath, the way an interviewer wants an answer to end, not trail off.
One more thing before the walkthrough moves on: the interviewer is listening for whether you can hold a position and still show judgment about when it stops applying. A flat "never name the model, ever" answers the letter of the question and fails the spirit of it.
Let's learn
Corvid Software builds tools developers use every day, and one of them is a bot that reads a pull request the moment someone opens it and posts comments right there in the code, the way a fast, literal reviewer would. Missing error handling. A null check that isn't there. A function that got copied and never renamed.
One diff in, a handful of comments out, before a person opens the file
Before the bot, a pull request waited for a teammate to have a free twenty minutes. A full review, reading the diff, checking the tests, leaving comments, waiting on the fixes, checking again, averaged 40 minutes. With the bot catching the routine stuff first, that dropped to 24 minutes, and by the end of the first quarter, 89 percent of what it flagged got fixed before the pull request merged.
Here's the turn. When the PRD got written, the team did the thing that felt like the responsible choice. It named the model: "Use Halyard 3.1 to generate PR comments," sitting in the requirements section like naming a model was the same thing as specifying a behavior. Four months in, the vendor pushed a small update behind that same name and the same address the code called. Nothing in the PRD changed, on paper. So nobody went and checked whether the bot still behaved the way it did on launch day.
The model didn't quietly get worse. The PRD quietly lost the ability to tell anyone when it did.
What each PRD approach cost in time
Eval-bar PRD (my pick)
Model-named PRD (what actually happened)
Detect the drift
Eval-bar PRD
1 week
Model-named PRD
8 weeks
Earn back the team's trust
Eval-bar PRD
2 weeks
Model-named PRD
9 weeks
Both bars measure the same two things in the same unit: how long before someone notices, and how long before developers trust the bot's comments again. Naming the model in the requirement didn't cause the drift. It just removed the thing that would have caught it in week one instead of week eight.
At its worst, the false positive rate, the bot flagging code that had nothing wrong with it, climbed from 4 percent to 13 percent over eight weeks, and by then 61 percent of its comments were getting dismissed without anyone reading them. A new hire, three days into the job, got four comments on a clean pull request and asked in the team channel if the bot was always this trigger happy. Nobody had a good answer, because nobody had been watching it since launch.
The choice I would take back. Writing "use Halyard 3.1" straight into the requirements section, as if naming a model were the same thing as specifying what the feature had to do.
Knowledge spark: what an eval bar actually is
A fixed test, run the same way every time, on a set of examples nobody's allowed to quietly change. Corvid's was 400 real pull requests with a known right answer for each one. The bar is a number the bot has to hit against that set, checked on a schedule, not just once at launch and never again.
What I would leave alone
The two-week spike a team runs to see if a model can even do PR commenting at all, before anyone's committed to building it for real. There's no eval suite yet, nothing for the doc to be held to, and the PRD isn't a promise, it's a bet. Naming the model there is fine. Leave that one alone.
The lesson. A requirement's job is to still be true after the world underneath it changes without asking your permission. If one quiet update from a vendor can make your own PRD wrong, you didn't write a requirement. You wrote a snapshot, and filed it under the wrong heading.
The week nobody could explain what the bot was doing
Use this one when you've got room to actually feel the eight weeks. The short version above has the same shape, just none of the Tuesday.
Tomasz Wieczorek is the only product manager on developer tooling at Corvid Software. He'd had the job two years when the review bot's PRD went out for sign off, in a spec review where six people read the same shared doc on the same shared screen.
The bot launched in March, and for three months it was exactly what everyone hoped. It posted its first comment nine minutes after launch, and it was right. By the end of week one, engineers were merging pull requests without waiting for a human to catch the boring stuff first. Tomasz watched the weekly numbers every Monday morning with his coffee, mostly because he liked watching something he'd shipped actually work.
He checked every Monday for the first month. Then every other Monday, once the numbers stopped moving. By month three he only opened the dashboard when someone in standup asked how the bot was doing, and mostly, the answer was fine.
In week seven of a drift nobody yet knew was happening, a new engineer, three days into the job, opened a pull request that had nothing wrong with it and got four comments back. One told her to add a null check on a variable that couldn't be null. She didn't know the bot well enough to just ignore it, so she asked in the team channel: is this thing always this trigger happy, or did I just get unlucky.
Same bot, two very different ways of being wrong
Nobody could answer her. Not because it was a secret, but because nobody had looked. Tomasz pulled the numbers that afternoon and found the bot's false positive rate had climbed from 4 percent at launch to 13 percent, and it hadn't happened overnight, it had been climbing for eight straight weeks. It crossed 5 percent, the number written into the company's own quality standard, back in week two. Nobody was running that check against the bot, because the thing the PRD actually held anyone to was a model's name, not a number.
By the time Tomasz found it, 61 percent of the bot's comments were getting dismissed without anyone reading them first, which is another way of saying the bot had quietly stopped doing its job while still technically running.
Tomasz remembered the meeting where they wrote the PRD. Someone had asked what model the bot should use, and "Halyard 3.1" felt like the concrete, engineering-friendly answer, more concrete sounding than "a model good enough to hit our comment quality bar," which sounded vague even though it was the thing that actually mattered. So the model's name went into the requirements section, and the bar it was supposed to hit went into a slide that never got attached to the doc.
Here's the replay. Same vendor, same quiet swap behind the same model name. But this time the requirement says the bot has to clear a 5 percent false positive rate against a fixed set of 400 pull requests, checked every Monday. The first Monday after the vendor's update, the automated run fails. Tomasz gets a message, not a complaint from a new hire seven weeks later. By Wednesday, engineering has rolled back to the last version that passed. Total damage: three days, not eight weeks.
What I'd tell myself, if I could: naming the model felt like being specific. It wasn't. It was being specific about the wrong noun.
PICK, spelled out for a PRD
This is a tradeoff wearing a documentation question's clothes, so PICK is the tool here, not BOUND, even though there's real arithmetic in it.
P, position. Don't name the model in the requirement. Name the eval bar. Keep the model's name in a separate note everyone expects to be wrong by next quarter.
I, impact. Engineering feels the cheap mistake: a weaker model chosen to save cost gets caught by the weekly run and costs a Tuesday to fix. Every developer at Corvid feels the expensive one: a silent drift nobody's watching, for as long as it takes someone to notice on their own.
C, cost asymmetry. The cheap side is loud. An eval run fails, someone gets pinged, it's fixed by Wednesday. The expensive side is silent. The model's still called the same name, the PRD's still technically true, and the only thing that changes is how often developers actually read what it posts.
K, kill criteria. Name the model anyway if a compliance rule requires logging exactly which model touched an output, if it's a two-week spike with no eval suite built yet, or if the feature genuinely needs a capability only one vendor's model has, a capability, not a preference.
Knowledge spark: what makes something a kill criterion
A fact you could actually go check, not a feeling. "The vendor might raise their prices" isn't one. "The state board requires logging the exact model version" is, because you can point to the rule and the paragraph it's written in.
The false positive rate, week by week, against the line that mattered
Comments flagging code that had nothing wrong with it
5% bar, written into the PRD but never scheduled as a check
The line crosses the PRD's own 5 percent bar in week 2. Nobody was running that check against it, because the requirement on paper was a model's name, not a number. It only got investigated in week 8, when a new hire couldn't get a straight answer about why the bot kept flagging clean code.
Try the same pick at a pharmacy counter
A small independent pharmacy chain uses an AI tool that checks a new prescription against everything a patient is already taking, and flags a possible interaction before the pharmacist signs off. Same question, a different desk, a different kind of paper trail.
Here the position flips. Name the model, in the pharmacy's own dispensing procedure. The state pharmacy board's audit rule requires every flagged or cleared interaction to record exactly which model and version made the call, so an inspector can trace one specific dispensing back to one specific tool.
P. Name the model, in the procedure that governs dispensing, because a regulator requires it. Keep the eval bar as the separate, ongoing check that the named model still deserves to be named. I. Skip the logging and an error that reaches a patient leaves the pharmacy unable to show which version of the tool cleared the prescription, so the pharmacist carries the liability alone for a decision software helped make. Log it, and an audit costs the owner about ten minutes of paperwork a quarter. C. Not logging is cheap and invisible right up until a bad interaction reaches a patient, then it's the difference between "our tool made a documented, reviewable call" and "we don't actually know what checked this." Logging it is a small, constant cost paid by everyone, every time, on purpose. K. This flips back the day the state board moves to a behavior-based standard instead, asking for the pass rate on a fixed test set rather than a model's name. That's not the rule today, so today, name it.
What stays the same at the pharmacy counter
The routine allergy flag a pharmacist sees on almost every fill. A wrong flag there gets caught in seconds, because the pharmacist is already reading the patient's own chart at that exact moment. Logging the model's name for that flag adds paperwork with nothing behind it to protect.
Swap the trigger and it still runs
Speed: the tool clears a check in half a second instead of two. Doesn't move the pick, in dev tools or at the pharmacy counter. Speed doesn't answer who checks that the model's still doing its job.
Cost: the vendor triples its price. Also doesn't move it. Price was never the risk. The drift was.
The model gets better: the miss rate drops toward zero. Moves the bar, doesn't erase the reason to check it. A rare miss in either domain is still not a zero.
Where people run it wrong
Treating "our policy says use model X" as the same thing as "we're checking that model X still behaves the way it did on day one."
Assuming naming the model is either always required or never allowed, instead of asking what the document actually has to survive.
Writing "keep the model name in the PRD for reference" without saying who re-checks it, and how often.
If you're asked this cold
Say the reframe out loud before you pick a side. "Before I answer, I want to know what this document has to still be true about, six months from now." That's true, it buys you a few seconds, and it's already stage one of the real answer.
Flashcards (click a card to flip it)
1 · THE FRAMEWORK
Which framework fits this question, and what's its hardest step?
Tap to flip
ANSWER
PICK, for a tradeoff. The hardest step is C, the cost asymmetry: naming which mistake is cheap and visible (a bad model swap caught by a weekly check) and which is hidden and expensive (a silent vendor update nobody's watching for), then building the pick around the hidden one.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Tomasz Wieczorek, the only product manager on developer tooling at Corvid Software, two years into the job when the review bot's PRD named a specific model.
3 · THE HABIT
What did Tomasz stop doing because the bot worked?
Tap to flip
ANSWER
Checking the weekly comment numbers. Every Monday for the first month, then every other Monday, then only when someone in standup asked.
4 · THE ASYMMETRY
What's the cost asymmetry in this story?
Tap to flip
ANSWER
A bad model swap gets caught by a weekly eval run in about a week. A vendor's silent update, hidden behind a model name the PRD already approved, went unnoticed for eight weeks.
5 · THE POSITION
State the pick in one sentence, the way you'd say it out loud.
Tap to flip
ANSWER
Don't put the model's name in the requirement. Put the eval bar there instead, and let the model's name live in a note everyone expects to go stale.
6 · THE NUMBER
Fill in the blank: the false positive rate climbed from 4 percent to ______ percent over eight weeks before anyone investigated.
Tap to flip
ANSWER
13. It crossed the PRD's own 5 percent bar by week two, six weeks before anyone was actually checking for it.
7 · THE KILL CRITERIA
Name the cases where you'd name the model in the PRD on purpose.
Tap to flip
ANSWER
A compliance rule that requires logging exactly which model touched an output. A two-week spike with no eval suite built yet. A feature that genuinely needs a capability only one vendor's model has.
8 · THE TRANSFER
Section 4 runs PICK again on a different product. Which one, and does the pick flip?
Tap to flip
ANSWER
A pharmacy's AI drug-interaction checker. There the pick flips: a state board audit rule requires logging the exact model version, so naming it is the right call, one of this answer's own kill criteria, actually happening.
Check yourself Score: 0 / 0
True or false
1. True or false: this answer says a PRD should never mention the name of a specific model. Why or why not?
True
False
Show hint
Look at the three kill criteria in the K step.
Show answer
False. The position isn't a blanket ban. It's that the model's name doesn't belong in the requirement itself. The K step names real cases, a compliance rule, a spike with no eval suite yet, a capability only one vendor has, where naming the model on purpose is the right call.
Multiple choice
2. Which of these is the cost asymmetry (the C step) in this answer?
A. A weaker model costs more to run than a stronger one.
B. A bad model swap gets caught by the weekly eval run in about a week. A silent vendor update behind a named model went unnoticed for eight weeks.
C. Developers prefer bots that leave fewer comments.
D. The PRD took longer to write than the bot itself took to build.
Show hint
One of these is something you could check on a calendar. The others are opinions or facts the answer never states.
Show answer
B. A and C might be true but aren't the asymmetry the pick turns on. D is never stated anywhere in the answer. B is the one with a real number on each side, one caught fast and cheap, one caught late and expensive.
Fill in the blank
3. Fill in the blank: by week ______ of the silent drift, the rate of the bot's comments getting dismissed without anyone reading them had reached 61 percent.
Show hint
Same week a new hire asked the question nobody could answer.
Show answer
8. Eight weeks is also how long the false positive rate climbed before anyone investigated it, from 4 percent to 13 percent, well past the PRD's own 5 percent bar.
Multiple choice
4. Why didn't the PRD's own 5 percent bar catch the drift the week it was actually crossed?
A. The bar was set too low to ever be useful.
B. Nobody was running the check on a schedule, because the requirement on paper was a model's name, not a number.
C. The vendor refused to let Corvid test the new version before release.
D. The bot stopped posting comments entirely once the model changed.
Show hint
Ask what the requirement actually said, on paper, versus what it should have said.
Show answer
B. The 5 percent bar existed somewhere as a company standard, but nothing in the PRD tied it to a recurring check. A number nobody is checking on a schedule is a fact, not a requirement.
Short answer, apply it yourself
5. Pick a product you use yourself that's built on an AI model. Name one place a silent vendor update could change its behavior without you noticing for weeks. What would you check, on a schedule, to catch it sooner?
Show hint
Look for the output you trust without re-checking, the one you stopped watching once it started working.
Show answer
Model answer: "My email client auto-drafts replies. I stopped reading them closely after the first month because they were always close enough to fix fast. If the vendor swapped the model underneath, I might not notice for a long time, I'd just be editing a little more without asking why. Checking: once a week, read one auto-draft all the way through before touching it, and time how much I actually have to change." Any answer works if it names something you check less than you used to, and a real way to check it again.
Short answer
6. If the false positive rate had jumped straight to 13 percent in week 2 instead of climbing slowly over eight weeks, would naming the model in the PRD still have caused the same long delay? Walk through it.
Show hint
Ask what actually triggers someone to look, in a PRD that only names a model.
Show answer
Probably a shorter delay, but still a real one. A sudden jump is more likely to get noticed by a developer complaining loudly, the way one bad review would. But without a scheduled check written into the requirement, catching it still depends on someone happening to notice and happening to say something, instead of a Monday run failing on its own. A slow climb is the worse case because it never looks sudden enough to report. A fast jump is luckier, not safer.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
