What is the difference between a model metric and a product metric? Give an example of each.

How to answer this, stage by stage

This question sounds like a vocabulary check. It isn't. The interviewer wants to know whether you'd have caught the six weeks where the score went up and the product got worse. Seven moves get you there.

Let's learn

Here's what happens when a metric goes up for six weeks straight and the product quietly gets worse the entire time.

Docwright is a tool from Hallcross Systems that reads a code change and writes the documentation for it: the comment block above a function explaining what it does, what each parameter means, and what it returns. Before a tool like this, an engineer opening an unfamiliar file either read the whole function line by line or pinged a teammate and asked. At Vantable, a fintech company running Docwright on their payments monorepo, that used to mean about 85 messages a week in the team's Slack channel asking some version of "what does this do."

With Docwright running, that number dropped to about 30 a week within the first month. Engineers stopped asking each other and started reading the generated docstring instead.

Docwright's own internal number, its claim check score, measures something narrower: for every factual claim a generated docstring makes (this parameter validates the input, this function returns null on failure), does that claim actually match the code, checked against the golden set. That score sat at 94 percent when Docwright launched at Vantable, and it climbed to 96 percent over the following six weeks, the same six weeks the Slack questions stayed low. Everyone watching that one number would have called the rollout a clean win.

Here is the turn. The claim check score climbing was not the lie. Every claim it checked really was more accurate against the golden set. The lie was in what the golden set was made of: 500 clean, well typed, well tested functions. Vantable's actual payments code is eight years old, half undocumented, full of quiet assumptions no golden set example ever had. Docwright kept getting better at a test that stopped resembling the job.

At its worst, this costs more than the 55 Slack messages a week Docwright was supposed to remove. Once one engineer, Farro Sable, caught Docwright confidently claiming a validation check existed on a function that had none, he stopped reading any of its docstrings closely. He started deleting them on sight and writing his own, the exact behavior from before Docwright existed, except now his team also had to notice the confident wrong ones first.

The gauge only ever looks at the golden set. The person is the one deciding, every single day, whether to trust what Docwright wrote.

The choice I would take back is not the golden set itself, it's that the golden set was never updated with real edits from production. Nobody sampled Vantable's actual rewrites back into the eval, so the model kept scoring itself against a version of reality that had already moved on without it.

What I would leave alone: Docwright's docstrings for pure utility functions, date formatting, currency rounding, string helpers, never needed a Vantable specific exception. Those functions look the same in the golden set and in Vantable's repo, so the claim check score and the edit rate agree there and always will.

The lesson: a model metric answers "did the model do the thing right." A product metric answers "did the thing right matter to the person holding it." Those are two different questions, and only one of them notices when a person quietly stops trusting you.

Now here is the same thing as a story

The short version is above. Read on for how ordinary week four looked from Isolde's side of the dashboard.

Picture a Slack channel that used to fill up every morning with the same four words, "what does this do," going quiet for weeks, then, without anyone shipping anything new, starting to fill back up again for no reason anyone can point to.

Isolde Renard had run product for Docwright for two years by the time Vantable came on as a customer. She was good at the part of the job most PMs get wrong: trusting the number on the dashboard exactly as much as it had earned, and no more. For the first month of the Vantable rollout, the claim check score climbed every single week, and she checked it every Monday like a ritual. 94, 94.5, 95, 95.6. She had every reason to feel good about it.

Farro Sable, a staff engineer on Vantable's payments team, felt good about it too, at first. He'd been the loudest skeptic in the pilot kickoff, and by week three he was the one telling other teams to turn Docwright on.

Then, on a Tuesday in week four, Farro found a docstring claiming a function validated that a transaction amount was positive before processing it. It didn't. The function trusted its caller completely, and the validation Docwright described lived somewhere else entirely, three files away. Nothing broke that day. Farro caught it in a routine review, the same way he caught everything. But he stopped skimming Docwright's docstrings after that. He started reading every one closely enough to rewrite it, which took about as long as writing it from scratch.

By week six, three more engineers on his team had hit their own version of the same thing, always a small, confident, wrong claim, never the same one twice. None of them filed anything either. Each one just started rewriting on sight and mentioned it to the others over lunch. The Slack question count didn't move yet. It wouldn't, not for weeks. What moved first, if anyone had been watching it, was the edit-or-delete rate: docstrings getting rewritten or deleted within 48 hours of appearing in a pull request. It went from 9 percent in week one to 19 percent by week five, quietly, underneath a claim check score that kept climbing the entire time.

The paper on the left never met Farro. The person on the right is the only one who actually decides whether Docwright's work survives the week.

Isolde had built the claim check score herself, two years earlier, when Docwright's first customers were smaller teams with cleaner, newer codebases. Back then, a fixed golden set of 500 well typed functions was a reasonable proxy for almost any customer's code, because almost any customer's code looked roughly like the golden set. She'd never gone back and asked whether that was still true once an eight-year-old fintech monorepo showed up.

What actually broke the pattern: a new hire on Isolde's own team asked, in a totally ordinary standup, "wait, do we know if people are keeping what it writes, or just that it's technically right?" Isolde didn't have an answer. She pulled the git history for every Vantable repo running Docwright and built the edit-or-delete number that afternoon. It had been climbing for five weeks by the time she looked.

The replay: Isolde shipped an automated claim checker that cross-references every docstring's factual claims against the function's actual signature and type hints before a human ever sees it, flagging anything it can't verify instead of stating it plainly. She also started sampling real edits from Vantable's repo back into the golden set every two weeks, instead of leaving it frozen at launch. Within three weeks, the edit-or-delete rate came back down to 12 percent. The claim check score, briefly, dropped to 91 percent, because the golden set now contained the messy code that had been exposing the model the whole time. That drop was the number finally telling the truth.

What I would tell myself, watching that Monday ritual of checking one climbing number: a score that only ever compares itself to its own test can climb forever without ever once checking in with the person it's supposed to be helping.

LEAD, walked through once more

This is a question about which number to trust and when, not a story about a habit with two settings, so LEAD fits, not a recovery framework built for diagnosing a sudden drop.

Two things worth saying out loud, since this is exactly where an AI PM answer earns its name instead of borrowing a generic metrics framework. First, the alternative most people reach for is an explicit thumbs up or down button on every docstring. It got looked at and rejected, because fewer than 3 percent of engineers at Vantable ever clicked it, so it was sparse and it skewed toward whoever bothered, while the edit rate reads everyone's real behavior automatically out of version control. Second, the real fix wasn't "make the model always right." It was a calibrated bar: the claim checker flags anything it can't verify against the function's actual signature, and the whole system ships when it clears a target verified-claim rate on a refreshed eval set, most of the time, not every time. There's a real cost trade-off sitting underneath this too. Feeding the model the full call graph, every caller and callee, catches more hallucinated claims but roughly triples the token cost and adds real seconds of latency per function. Fine for the handful of functions in one pull request. Expensive enough that Vantable's one-time backfill of 40,000 existing functions used the cheaper, shallower version instead, and those landed in a review queue rather than auto-merging, accepting a higher hallucination rate there in exchange for not paying for deep context on code nobody was actively changing.

And if you want to be sure it really works, try it somewhere else

Same four letters, a completely different industry, so the method proves itself instead of repeating a story that happened to be prepared in advance.

Draycott Archives sells Shelfmark, an AI tool that reads a scanned manuscript or ledger and drafts the catalog description archivists use to help researchers find it later.

L, link. Not Shelfmark's own description-quality score. Whether researchers can actually find the item they're looking for, measured by how often the archive's reference desk still gets asked to hunt something down by hand.
E, early signal. Denny Emberly, who runs product for Shelfmark, watches the correction rate: how often an archivist edits or flags a generated description within a week of it going live. It started climbing on 19th century handwritten ledgers weeks before reference desk requests for those same items went back up.
A, abuse. Shelfmark's own similarity score, how closely a description matched professionally catalogued reference items, kept improving, because the reference set was mostly clean, typed 20th century documents. Handwritten ledgers with faded ink and archaic phrasing were never well represented in it, so the score climbed while the ledger descriptions quietly got worse.
D, decision. Under 10 percent correction rate for a document type, ship as-is. 10 to 20, route that document type to a human-drafted-first workflow instead of AI-first. Above 20, pull Shelfmark off that document type until the reference set includes real examples of it.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to naming both metrics and which one you'd actually watch, don't build up to it.
Cost: refreshing the golden set every two weeks turns out to need a part-time reviewer's time every sprint. Don't skip it to save the hours. A frozen eval set is exactly how a model quietly drifts out of sync with its own product for months.
The model got better, for real: say Docwright's underlying model gets a genuine capability jump and the claim check score jumps ten points overnight. Still check the edit rate before believing anything changed for Vantable specifically, a better model can still be handed a golden set that doesn't represent the job.

Where people run it wrong.
They treat the model's own eval score as the finish line, because it's the number the model team already has, instead of asking who has to act on the model's output.
They build an explicit feedback button, assume silence means approval, and never check how many people actually clicked it.
They let the golden set sit frozen at launch, so a model can keep "improving" against a version of the job that stopped existing months ago.

How to use it live. Say the two definitions before naming any product: "A model metric grades the model against a fixed test. A product metric grades whether it actually changed what a real person does." That buys you the room to pick the right example for whatever product the interviewer throws at you next, instead of reciting Docwright's numbers on reflex.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0