InterviewIntermediateEval-Driven Specification / Writing an eval spec / #27
Why should a PM own the eval spec rather than delegating it entirely to engineering?
The direct answer
A PM should write and own the eval spec's pass or fail criteria, and defend the tradeoffs baked into them, because those are calls about what the product is actually for. Hand the whole spec to engineering and the team builds against whatever is cheapest to test instead, and nobody notices until real users have already changed their behavior around it. The harness that runs the eval and automates it can stay with engineering. That part really is theirs.
The ranking, by what breaks first if skipped
Write the eval spec's pass or fail criteria yourself, and defend the tradeoffs in them.Why: these are product calls about what "good" means. Skip this and someone else makes the call by default, usually toward whatever's easiest to check.
Decide what counts as a genuinely good output before engineering writes a single test case.Why: engineering can't build the eval set without this. Leave it undefined and they'll define it themselves, using whatever the harness can grade without a person in the loop.
Check the current spec cheaply: can whoever wrote it explain the product reason behind every rule in it?Why: a spec nobody can defend line by line is already drifting toward whatever's automatable, and this catches it before real users do.
Track how often people actually act on the output, not just how often it passes the automated check.Why: that gap is the first sign a proxy metric is quietly winning. Cheap to watch weekly, expensive to discover after something real breaks.
Leave the harness, the automated run, and the wiring into the pipeline to engineering.Why: that part is a real technical build. Taking it back doesn't protect the product, it just slows both people down.
Revisit the criteria when the product's real goal shifts.Why: a spec that never gets rewritten quietly keeps optimizing for whatever mattered at launch, not what matters now.
How to answer this, stage by stage
Eight moves. The trap in this question is answering it as a staffing question, who has time to type the document, instead of a judgment question, whose call is it what "good" even means.
1
Ground it in one real product
Say it like this
"Let me make this real. Say it's Hexline Software, a developer-tools company, and they've built a tool called Reviewlane. It reads a pull request, the bundle of code someone wants added to the project, and posts review comments on it automatically, before a person even opens the diff. I'll answer against that."
Why this works
Grounds an abstract staffing question in one real product, so the ranking that follows isn't hypothetical.
2
Say your structure out loud
Say it like this
"I'd use ORDER for this. Rank whose judgment the spec actually needs by what breaks if that judgment is missing, not just list who's involved in writing it."
Why this works
Signals a method before diving in, so it doesn't sound like you're improvising an org chart on the spot.
3
Reframe the question
Say it like this
"This isn't really a question about who has time to type the eval spec. It's about whose judgment decides what counts as a good comment. Typing it up is fast. Deciding what 'good' means for this product is the actual job, and that's not an engineering call."
Why this works
Separates the busywork of writing the document from the harder decision buried inside it.
4
Name the outcome the spec protects
Say it like this
"Every criterion in that spec is protecting one thing: that a comment Reviewlane posts is actually worth an engineer's two minutes. Not that it's technically well-formed. That it changes what they do next."
Why this works
Without naming the outcome, "own the spec" is a preference. With it, the rest of the answer has something to argue from.
5
Find the mistake that can't be undone
Say it like this
"If I stay out of it completely, engineering grades comments against whatever a nightly script can test, something like 'does it name a line and cite a real rule.' That's easy to check and has nothing to do with whether the comment's worth reading. They'll optimize hard against it, and I won't find out it drifted from the real goal until developers have already learned to stop reading Reviewlane, which you can't undo by tweaking the model next sprint."
Why this works
This is the reversibility test made concrete, not just asserted as "be careful."
6
Show the dependency underneath the spec
Say it like this
"Engineering can't write one test case until someone decides what a good comment actually is. If I don't decide it, they will, by default, using whatever's automatable. So that decision has to happen before the eval set exists, not after it."
Why this works
Shows the order isn't arbitrary. One decision genuinely blocks the other from being built.
7
Give the cheap check, then prove it with what actually happened
Say it like this
"Here's the cheap test: can whoever wrote today's spec explain, criterion by criterion, the product reason behind it? At Hexline, nobody could, past 'it's fast to grade automatically.' And here's what that cost them. Reviewlane's automated pass rate climbed to 99 percent while the share of comments developers actually acted on fell to 14. A real bug sat as comment 31 of 31 on a pull request, and the engineer clicked resolve-all without reading it. It shipped, and a customer's saved project lost data over a weekend."
Why this works
A cheap check plus a real cost lands harder than either one on its own.
8
Close with the ranked, defended split
Say it like this
"So: I own the criteria and the tradeoffs in them, because that decides whether the whole thing works. Engineering owns the harness, the nightly run, wiring it into the pull request flow, because that's a real technical build I'd only slow down. I don't delegate the judgment. I delegate the plumbing."
Why this works
Ends on the literal division the question asked for, defended instead of assumed.
Let's learn
Say a developer-tools company builds a tool that reads a pull request, the bundle of code changes someone wants added to a project, and posts review comments on it before a person even opens the diff.
Before Reviewlane, at Hexline Software, a pull request waited about 6 hours for a person to leave the first comment. Whoever reviewed it read the code line by line, about 25 minutes a pull request, squeezed in around everything else on their desk.
Reviewlane changed that fast. It posts a comment within 90 seconds of a pull request going up.
Knowledge spark: what's a proxy metric?
A number that's easy to measure and stands in for something harder to measure. "Does the comment name a line and cite a known rule" is easy for a machine to check. "Is this comment worth a developer's two minutes" is not. A proxy metric is useful right up until a team starts building toward the easy number instead of the real one.
To ship fast, the engineering team wrote Reviewlane's first eval spec on their own: a comment passes if it names a specific line and cites a real rule. Clean, testable, no argument needed to get it out the door.
The harness and the release both just run on whatever the first box decides
Here's the part that matters. The extra passing comments were never the real problem. The real problem was that nobody had written down, precisely, what "good" meant beyond "well-formed," so the model had no reason to tell a comment worth reading from one that just looked like one.
The extra passing comments were never the problem. What developers did with them was.
As the model got better at hitting that exact bar, it also got better at writing comments that were technically correct and mostly useless, like citing a rule the project's own style checker already caught on every save. Two engineers reading the same batch of comments agreed on which ones were actually worth keeping only 58 percent of the time. The spec never told either of them what to agree on.
Reviewlane's automated pass rate vs. the share of comments developers actually acted on, by week
Automated pass rateComments developers acted on
Two lines moving in opposite directions off the same ten weeks. Nothing in the spec was watching the second one.
The choice I would take back
Eitan handed the whole eval spec to engineering at launch, since "is this comment well-formed" felt like an engineering question. He should have kept one line for himself: does this comment change what the developer does next. Engineering can still own the harness that runs the check every night. That part really is theirs.
What I would leave alone. For something like a missing semicolon or an unused import, a loose automated bar is genuinely fine. Nobody needs a product judgment call to catch that, and adding one would just slow the harness down for no reason.
The lesson. An eval spec that's easy to automate isn't the same as one that reflects why the product exists. Handing over the whole thing quietly swaps "is this comment useful" for "is this comment checkable," and nobody notices until developers have already stopped reading.
Now here is the same thing as a story
The short version is above. Keep reading if you want to feel why the product read has to come before the harness, not just take it on faith.
Eitan Barsky could turn a two-line complaint, "code review takes too long and half the comments are noise," into a shipped feature in about six weeks. That was his reputation at Hexline before Reviewlane.
The first ten weeks of Reviewlane were the kind of good that makes a team stop asking questions. A comment landed on a pull request before the author had even switched tabs. Engineers stopped waiting six hours for a first look; they got one in ninety seconds, every time.
Every Monday, Eitan pulled a sample of twenty comments from the week before and read them himself, checking whether they still made sense next to the code. They mostly did.
So he went from twenty to ten. Then to whichever five looked interesting on the dashboard. Then, most weeks, to none, because the nightly pass rate kept climbing and a green number is a hard thing to argue with.
Then came a Saturday.
One of these you can fix whenever you get to it. The other already shipped past 31 comments nobody read.
A customer emailed support: a project they'd saved on Friday afternoon was gone, half-overwritten by another process. Someone had written to a shared file without checking whether another job already had it open.
Reviewlane had actually caught it. On Friday, at 4:52pm, it posted a comment on the exact line, naming the exact rule. It was comment 31 of 31 on that pull request. The engineer, three weeks into an unbroken habit of clicking resolve-all on Reviewlane's threads without reading them, cleared the whole list in about four seconds.
We didn't lose a bug report. We lost the one Friday afternoon that would have caught it.
Eitan pulled the spec that Saturday and read it properly for the first time in months. "Names a line, cites a rule." Two clauses doing the work of an entire product decision. Nothing in there said a word about how many comments was too many, or which ones actually earned the right to survive being comment 31.
A year earlier, in a launch-week meeting nobody minuted, someone asked whether the eval spec needed a product read before it shipped, or whether "engineering can just define correct" was fine for a first version. With a deploy date already on the calendar, the second answer won, reasonably enough. Nobody put a date on the calendar to come back to it.
I would take that back. With a criterion for "does this change what the developer does next," most of those 31 comments never pass at all, because most of them never earn a developer's attention in the first place. The real one lands as comment 2 of 4. It gets read. It gets fixed before the 5pm deploy, not found by a customer on Saturday.
What I'd tell myself, back in that meeting: skipping the product read felt like it saved a week. It cost the on-call team their weekend, and it cost Reviewlane the thing it actually needed, which was developers who still opened the list.
The judgment call underneath the eval spec, ranked
GUARD would fit if the question were about who gets hurt when the tool is wrong. This isn't that. It's about ranking whose judgment a document needs before someone gets left out of writing it, which is ORDER's job.
O, outcome. Every criterion in Reviewlane's eval spec protects one thing: that a comment reflects what the product is actually for, catching something worth an engineer's two minutes, not just whatever's cheapest for a nightly script to check.
R, reversibility. If the PM stays out completely, engineering optimizes hard against whatever it can test, and the gap between "passes the check" and "is actually useful" only shows up after developers have already learned to stop reading. You can't undo the Saturday a customer lost data because comment 31 got auto-resolved.
D, dependency. Engineering can't write a single test case until someone decides what a good comment actually means. Leave that undecided and they'll decide it by default, using whatever the harness can grade without a person in the loop.
E, evidence. Cheap to check: can whoever wrote today's spec explain, criterion by criterion, the product reason behind it? At Hexline, past "it's fast to grade," nobody could.
R, rank. The PM owns the criteria and the tradeoffs baked into them. Engineering owns the harness: the nightly run, the CI wiring, the automation that actually grades it. Handing over the whole spec doesn't save the PM time. It just moves the decision to whoever's cheapest to automate against.
The check that keeps this ranking honest
Swap the outcome and the order should move. If a wrong Reviewlane comment only ever cost someone a shrug and a quick fix, the criteria could sit lower on this list, and a plain automated rule might genuinely be enough. It ranks first here because a real customer lost real data over a weekend on the strength of a comment nobody read.
Rank it again, where the wrong call costs a payout, not a weekend
Corvale County's small-business grant office built Grantline, a tool that reads grant applications and drafts review comments, flagging missing documents, eligibility gaps, and budget numbers that don't add up, before a person signs off on a payout.
O. Every version of Grantline's eval spec protects one thing: that a flagged comment reflects the county's real eligibility rules, not just whether it cites the right field on the form.
R. A late policy memo is fixable any time, someone rewrites it. A payout already sent to an ineligible business, approved because a reviewer skimmed a technically-correct-looking comment and signed off, is not. That business has already spent the money.
D. The frontline reviewer's "when to escalate" rule only works once someone decides what actually counts as a real eligibility gap, a policy call, not a form-matching one.
E. Cheap to check: hand two reviewers the same fifteen past applications and the draft criteria, see how often they'd flag the same ones. Corvale's first draft, written entirely by the engineering team building Grantline, agreed with a policy expert's read only 58 percent of the time.
R. Same order: the grant program lead owns the criteria defining a real eligibility gap. Engineering owns the harness that runs Grantline against every application and reports flags automatically.
Ineligible applications caught before a payout went out, per quarter
Same tool, same model. The only thing that changed was who decided what counted as a real gap.
Swap the trigger and it still runs
Hexline starts shipping Reviewlane updates weekly instead of monthly. The order doesn't move. The PM's criteria matter more, not less, since faster shipping leaves less time to catch a proxy before it's already live.
The engineering team doubles in size and builds a much bigger automated test suite. Doesn't reorder anything. A bigger suite still needs the PM's definition of "good" to test against, or it just gets very good at testing the wrong thing.
The underlying model gets noticeably better on its own. Doesn't reorder either. A better model still needs a real bar to be judged against, or nobody can tell "better" from "wrong in a different way."
Where people run it wrong
Treating "the eval spec passed" as proof the tool works, when the spec's own criteria were never checked against the product's real goal.
Writing the PM's criteria once at launch and never touching them again as the product's priorities shift.
Letting engineering write the whole spec because they build the harness anyway, as if building the test and deciding what it should test were the same job.
The line that buys you a second
Say the outcome out loud before naming who does what. "Every criterion in this spec is protecting one thing, whether this comment is worth a developer's two minutes." Then rank from there. Naming the outcome first turns a staffing question into an argument.
Flashcards (click a card to flip it)
1 · THE FRAMEWORK
Which framework fits "why should a PM own the eval spec," and why not GUARD?
Tap to flip
ANSWER
ORDER, for ranking whose judgment a decision needs before something gets built. GUARD is for naming who can't push back when a system causes harm. This question is about ranking judgment calls, not a power imbalance.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Eitan Barsky, the product manager who built Reviewlane, the AI code-review comment tool at Hexline Software.
3 · THE HABIT
What habit let a proxy-driven eval spec go unnoticed for months?
Tap to flip
ANSWER
Eitan used to read a sample of 20 Reviewlane comments himself every Monday. As the automated pass rate climbed, that read shrank to ten, then a handful, then most weeks, none.
4 · THE DEPENDENCY
What has to be decided before engineering can write a single eval test case?
Tap to flip
ANSWER
What actually counts as a good comment, specifically whether it changes what the developer does next. That's a product call, and engineering can't build tests against a definition nobody's given them.
5 · THE OLD DECISION
What decision would you take back, and why did it make sense at the time?
Tap to flip
ANSWER
Handing the entire eval spec to engineering at launch, because "is this comment well-formed" felt like an engineering question. It made sense to move fast for launch. Nobody ever came back to add the product read.
6 · THE NUMBER
Reviewlane's automated pass rate climbed from ___ to ___ percent over ten weeks, while comments developers actually acted on fell from ___ to ___ percent.
Tap to flip
ANSWER
74 to 99 percent, and 58 to 14 percent. That gap is the proxy metric problem made visible, not asserted.
7 · THE REPLAY
Same pull request, same missing check, criteria owned by the PM this time. What changes?
Tap to flip
ANSWER
Most of the 31 low-value comments never pass at all. The real one lands as comment 2 of 4, gets read, and gets fixed before the Friday deploy instead of found by a customer on Saturday.
8 · THE TRANSFER
Section 4 runs ORDER again on a different product. Which one, and what plays the role of the PM's criteria there?
Tap to flip
ANSWER
Corvale County's grant-review tool, Grantline. The equivalent is a real definition of what counts as an eligibility gap, a policy call, not just whether a comment cites the right form field.
Check yourself Score: 0 / 0
Multiple choice
1. Which two jobs does this answer split between the PM and engineering, and who gets which?
A. The PM writes the model's training code, engineering picks the pass criteria
B. The PM owns the criteria and their tradeoffs, engineering owns the harness that runs the eval
C. The PM owns nothing once the tool ships, engineering owns both
D. Engineering owns the criteria, the PM owns the nightly automation
Show hint
Think about which job needs to know why the product exists, and which job needs to know how to build a test runner.
Show answer
B. The PM decides what counts as a good comment and defends the tradeoffs. Engineering builds and runs the harness that checks output against that definition.
True or false
2. True or false: since Reviewlane's automated pass rate kept climbing every week, that was proof the tool was doing its job. Say why.
True
False
Show hint
Ask what the pass rate was actually measuring, and whether that's the same thing as "worth reading."
Show answer
False. The pass rate only measured whether a comment was well-formed, not whether a developer found it worth reading. Those two numbers moved in opposite directions over the same ten weeks.
Fill in the blank
3. Before Reviewlane, a pull request waited about ______ hours for a first human comment. Reviewlane cut that to about ______ seconds.
Show hint
It's the number that makes the whole tool worth building in the first place.
Show answer
6 hours, and 90 seconds. That's the value an eval spec exists to protect. A drifting spec doesn't threaten this number directly, it threatens whether the fast comment is also a useful one.
Multiple choice
4. What does the dependency step argue in this answer's ORDER?
A. Engineering should decide the criteria since they build the harness anyway
B. A definition of "a good comment" has to exist before a single test case can be written, and if the PM doesn't supply it, engineering will by default
C. The PM and engineering should each write half the spec
D. The harness should be built before any criteria are decided
Show hint
Ask what has to exist before engineering can grade anything at all.
Show answer
B. Without a real definition of "good," engineering doesn't wait, it substitutes the easiest thing to automate, which is exactly what happened at Hexline.
Short answer, apply it yourself
5. Pick an AI feature you use or are building that gets graded against some kind of test or rubric. Whose judgment does that rubric actually reflect right now, and whose should it?
Show hint
Look for whichever reader's version, if it's wrong, means real people already got a real result graded against nothing solid.
Show answer
Model answer: "A resume-screening tool graded on whether it correctly extracts fields like job titles and dates, not on whether it would actually recommend the same candidates a hiring manager would. Right now it reflects whoever built the parser. It should reflect the hiring manager's judgment about what makes a strong candidate."
Short answer, the number question
6. If Reviewlane's automated pass rate had stayed flat at 74 percent instead of climbing to 99, would the PM still need to own the criteria? Say what changes and what doesn't.
Show hint
Reversibility is about whether a wrong comment can be undone, not about which direction a proxy number happens to move.
Show answer
Model answer: "What changes: the urgency. A flat 74 percent draws attention on its own, so the gap might get noticed sooner just because the dashboard looks less reassuring. What doesn't change: the pass rate was never measuring the real goal in the first place, so owning the criteria matters no matter which way the proxy number moves."
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
