What does the success criteria section look like when the metric is subjective quality?

How to answer this, stage by stage

Six moves. Name the two numbers before the story, or the answer sounds like a definition instead of something you'd actually put in a PRD.

Let's learn

What happens when "good enough" depends on which person happens to be looking at it? Say we build Backlot, a tool that takes a seller's plain product photo, a lamp on a kitchen counter, and drops it into a staged scene: a matching backdrop, a couple of props, warm light that actually falls the right way.

Before Backlot, a small seller paid a freelance photographer about 75 dollars a shot and waited a week for a batch back, or shot it themselves on a kitchen counter and hoped it looked fine. Backlot turns the same photo into a staged version in under thirty seconds, included in the plan they already pay for.

Here's the turn. A script can check that the product itself rendered cleanly, no missing handle, no cropped edge. Whether the whole staged scene actually looks professional enough to publish is a judgment call. Only a person can make that call. So the team wrote one line into the rubric, looks professional and realistic, and one number into the success criteria: ship when 90 percent of a weekly sample of 50 images score 4 or 5 out of 5. Nobody wrote down whether two people looking at the very same photo would land on the same score.

At its worst, the pass rate held right around 90 percent for eight straight weeks, so the weekly review looked healthy every single time. Underneath it, once two reviewers finally scored the same 50 images instead of splitting the pile between them, they agreed on the verdict barely 56 percent of the time. The number that was supposed to prove the bar was solid had never actually tested whether it was one bar.

Passed on paper. Redone by hand a week later.

What I'd leave alone. A render with a missing product, a floating shadow, or a cropped edge doesn't need two people's agreement. Everyone calls that a fail on sight. Keep that one as a single reviewer and a checklist. Agreement only earns its keep on the calls two reasonable people could actually split on.

The lesson. A pass rate only tells you the number moved where you wanted. It never tells you whether the two people deciding it were pointing at the same bar. For a subjective metric, the real success criteria isn't the score. It's whether two people scoring blind would hand you the same one.

Now here is the same thing as a story

Use this version when you've got the time to sit with it. The short version is above. This is for when you want to feel why it mattered.

Yusuke Handa has run product for Backlot for two years, since before the staging model could reliably handle anything shinier than a ceramic mug.

For the first several months, the rubric earned its trust honestly. Every Friday, two stylists scored a sample of the week's staged photos, and on the rare image where their scores landed on opposite sides of the line, whoever was in the room argued it out loud for two minutes and moved on.

Then the checking faded, in three small steps, none of them looking wrong at the time. First, to save review hours as the weekly sample grew, the team started splitting the 50 images between two reviewers instead of doubling up on each one, so a single image only ever got one opinion. Second, the rubric still said one line, looks professional and realistic, because writing anything longer felt like overbuilding a document for something a trained eye would just know. Third, nobody was saving which reviewer scored which image, only the week's overall pass rate.

Ottilie Baranski had reviewed staged photos since the rubric's first draft, and she was good at it, fast, and fair by her own lights. She passed images that had the right mood even when a shadow fell a little wrong. Nobody told her that was generous. Nobody was checking.

Then Dashiell Okonkwo joined the review rotation in week eight, and Yusuke gave him an onboarding exercise: blind re-score last week's already-passed batch before joining the live rotation.

Dashiell's scores came back, and 22 of the 50 images he flagged as an obvious fail, mismatched reflections, shadows falling the wrong way for the added light, had already passed the week before. Yusuke pulled the actual overlap number. Fifty-six percent. Barely better than a coin flip.

It would be easy to say Ottilie was too soft. She wasn't. She was fast, she was consistent with herself, and every score she gave was a real judgment, carefully made. Nobody decided the bar should be whatever Ottilie's eye called good enough. It just quietly became that, one Friday at a time, because nothing was ever checked against a second opinion.

So here's the decision Yusuke would take back.

Two years earlier, in the meeting where the rubric first got written down, someone asked whether "looks professional and realistic" was specific enough. It felt like overbuilding a document for something a trained eye would just know. Yusuke remembers agreeing they'd tighten it if it ever became a real problem.

I'd write the agreement check into the success criteria from day one instead. Same rubric, same weekly sample, but every image gets scored twice, blind, and the agreement rate rides right next to the pass rate on the same weekly report.

Here's the replay. Same eight weeks, agreement tracked from week one. It crosses the 65 percent floor by week five instead of surfacing by accident in week eight. That crossing is the trigger, not Dashiell's onboarding exercise. Yusuke pulls that week's disagreements and adds three checkable lines to the rubric: lighting direction has to match the backdrop, reflections have to match what's actually in frame, texture has to read as real up close. Agreement climbs to 81 percent by week eight. Backlot still ships on schedule. In the first month live, sellers ask to redo a staged photo by hand about 24 times instead of the 187 it actually took before anyone rewrote the rubric.

One success criteria section watches whether this week's sample passed. The other watches whether passing still means the same thing it did in week one.

And the thing Yusuke would tell himself, back in that first meeting: the pass rate was never the risk. Writing a rubric loose enough that nobody would ever be caught disagreeing with it was.

LEAD, applied to a bar two people have to agree on

This reads like a formatting question, what goes in one section of a document. Underneath it, it's still asking whether "good enough" means the same thing to everyone allowed to say yes. That's LEAD, run on a rubric instead of a single score.

And if you want to be sure it really works, try it somewhere else

An AI tool called Aftercare drafts the discharge note a vet hands a pet owner at checkout, in plain language, so they know what to watch for at home. A vet at Denhollow Animal Hospitals reads each draft before it goes out and scores it on one line: clear enough for a worried owner to follow. Same shape of problem. A pass rate that holds steady can hide the same thing it hid at Backlot, one vet's idea of clear quietly becoming the whole bar, because nobody's checking whether a second vet would call the same note clear.

Same note. Two calls.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0