Describe how the PRD should handle the eval spec: inline, linked or separate?

The four questions that settle where the eval spec lives

This is a tradeoff between two real documents, not a style question, so PICK does the work: commit to one, then show why the other one costs more than it looks like it does.

One more thing before this moves to the long version: most candidates hear "inline, linked, or separate" and answer it as a documentation-tidiness question, whichever keeps the PRD shortest. Say who actually gets hurt by each choice, in real hours or real dollars, and you've shown the judgment the question is actually testing.

Let's learn

The feature is a strip of small markers under Reeltide's video player, one dot per scene change, with a short label at each, put there by nobody the viewer will ever meet.

Before Chapter Marks, a viewer hunting for one moment in a 40-minute upload dragged the scrub bar back and forth, guessing, for about 100 seconds on average. A creator writing chapter titles by hand for that same video spent close to 18 minutes doing it after the upload finished.

Now Chapter Marks writes the whole chapter list during the upload itself. A viewer taps the list and lands on the right second in about 4 seconds. That's the win, and it's real.

Here is the turn. Every now and then a marker lands a few seconds off the real scene cut, and that alone is not the real problem, most viewers never notice five seconds on a 40-minute video. The real problem is that two different documents disagreed about how many seconds counted as "off," and nobody in the room who signed off on Chapter Marks knew that.

At its worst, that looks like this. The PRD for Chapter Marks promised markers within 3 seconds of the real scene cut, the number the marketing page turned into "precise, second-accurate chapters." The actual eval criteria lived in a shared doc used by three other features too. Someone else's PM, working on a different feature entirely, loosened that shared tolerance to 8 seconds and never touched the Chapter Marks PRD, because it wasn't their document to touch. Engineering built exactly to what the shared doc said. It just no longer said what the PRD promised.

What I would leave alone. A two-person internal prototype that never ships past its own team. Nobody outside that pair will ever open the PRD, so there's no second author who could quietly edit a linked number even if one existed.

The lesson. We built the shared doc to keep every PRD short and consistent. That was the right call when two features existed. Nobody ever revisited it once there were four, and "consistent" quietly turned into "nobody's specifically watching this."

Now here is the same thing as a story

The short version is above. Read this one when you want to feel why it matters, not just know the rule.

Ingmar can smell a PRD that's about to grow legs before anyone else in the room. Four years on Reeltide's platform team, and he's the one people hand a draft to before it goes to review, just to see what he cuts.

When the quality pod formed, eighteen months back, someone suggested pulling every feature's eval criteria into one shared doc, so PRDs would stay short and every team would grade things the same way. It was a good idea. At the time, two features existed, and Ingmar wrote both of them himself.

For months, that felt like a win. PRDs stayed five, six pages. Reviewers read them in ten minutes flat and said so, out loud, in standups. Ingmar linked to the shared eval doc, glanced at the summary line at the top to confirm nothing had obviously changed, and moved on.

Then he stopped glancing at the summary line. He'd click the link, see the doc load, and close the tab before it finished scrolling into view. Then he stopped clicking at all. The tag in the PRD said "current," and four months running, it had been.

Then, in a Tuesday sync, Reeltide's support lead mentioned it the way you'd mention traffic. A creator's ticket had come in: "Chapters are landing in the wrong scene entirely on my top video." Not a complaint about a whole feature failing. Just one line, read off a list of eleven other tickets.

Same PRD decision, two very different costs

Ingmar opened the shared eval doc for the first time in months. The chapter-timing tolerance no longer read 3 seconds. It read 8. Someone else's PM had widened it four months earlier, for a completely different feature, one that genuinely needed more slack. The edit was reasonable on its own. It had just landed, silently, on three other features that never asked for it.

It was never really about the five seconds. Chapter Marks had never had a number in the room. It had a document, and the document only had two states: linked and current, or something to worry about later.

Auditing the four features that shared that one doc took ten engineering days. Every one of them had shipped, every one had "passed" whatever the shared doc said that week, and every one of them was quietly grading against a promise the PRD itself no longer matched.

Eighteen months before any of this, in the room where the quality pod first proposed the shared doc, someone did ask whether each PRD should just keep its own copy of its own numbers. It felt like duplicate work for two features that would obviously stay in sync, since the same three people wrote both. They built the shared doc, linked both PRDs to it, and moved on to shipping the next thing.

Ingmar moved Chapter Marks' eval spec inline the month after the audit. Three seconds, in the PRD itself, owned by him, dated. Run the same drift again, and the other PM's edit never touches this document at all, because it isn't their document to edit.

One design lets a stranger's edit quietly rewrite your promise. The other makes that literally impossible, because the words live in the file with your name on it.

And the thing I'd tell myself, back when we built that shared doc: I wanted every PRD I touched to stay five pages. I never asked what happens the day someone I've never met edits the one page that decides whether "accurate" still means what it meant when I wrote it.

PICK, the four letters mapped onto Chapter Marks

This is a tradeoff about where one number lives, not a rule for organizing documentation in general, so PICK is the tool.

Try it on a claims photo, not a video at all

Ambercross Insurance runs an AI feature that reads photos of car damage and estimates repair cost, so an adjuster can approve small claims same day. Its eval spec has to say how close that estimate must land to what a licensed in-person assessor would say, and the same question comes up: does that number live in the PRD, or somewhere it points to.

P. Here the position flips. Keep the tolerance, within $150 of a licensed assessor's number, in one owned, versioned, linked "AI Assessment Standards" doc, not duplicated inline across every claims PRD.
I. An engineer on any one of Ambercross's six claims products feels a duplicated inline copy the moment it silently diverges from the other five, since nothing forces six separate PRDs to agree. Compliance feels a bare, unowned link the day a regulator asks which number is the real one and nobody can say for certain.
C. Here, six duplicated inline copies are the quiet, expensive risk, each one can drift on its own with nobody checking all six against each other. One owned, versioned, linked doc, audited on a fixed schedule, is the loud, cheap option: a missed audit shows up immediately, because it's the one thing compliance already checks every quarter.
K. Fold it back into each PRD inline the day this tolerance governs only one or two products, or the day compliance drops the quarterly cross-check, the same evidence that would have kept Chapter Marks' number in a shared doc too, if Reeltide had ever built the equivalent check.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0