Describe how you would measure whether onboarding taught the right mental model.

Farrowgate is a marketplace for buying and selling used goods between neighbors. ValueMark suggests a listing price from photos and a condition description. Bettina Voss sells off her late grandmother's household items on Farrowgate most weekends.

How to answer this, stage by stage

Nobody's grading whether you can name a metric. They're grading whether the metric you pick would have caught the problem before the damage was done.

Let's learn

Here is a question worth sitting with: what would it look like, weeks before anyone complains, if onboarding had quietly taught someone the wrong idea?

Farrowgate is a marketplace for secondhand household goods. ValueMark suggests a listing price for anything a seller photographs and describes.

Before onboarding taught anything about ValueMark, sellers guessed prices from nothing but instinct, often wildly off in both directions. After onboarding, most sellers accept ValueMark's suggestion outright, about 74% of the time, unedited.

At its worst: a seller who never formed a real mental model finally hits a rare, oddly-shaped item, a hand-carved chest with sentimental but no obvious resale value, accepts a low suggested price without question, and only feels the loss weeks later scrolling similar items that sold for triple.

What I would leave alone: a seller whose prediction accuracy is already high by listing three doesn't need the check repeated on every future listing. Once the mental model is there, more checking is just friction with nothing left to measure.

The lesson: the metric that would have caught this isn't a bigger number further downstream. It's a smaller, earlier one that asks the seller to commit to a guess before they see the answer.

Now here is the same thing as a story

The short version above is what you'd say defending this metric in a product review. Read this one for how the gap actually showed up for Bettina.

Bettina lists most of what's left of her late grandmother's house on Sunday afternoons, one box at a time, coffee going cold on the counter.

The whole check fits in ten seconds. It asks for a guess, not a click, and that's the entire difference.

In her first week, onboarding walked her through exactly what ValueMark is: a suggestion built from photos and condition, not an appraisal. She nodded along, the way anyone does through an onboarding screen, and moved on to her first listing.

Bettina's line on the chart above is this flow, in numbers. She got it wrong first, and that's exactly what made the later guesses meaningful.

By her fifth listing, Bettina's prediction accuracy had climbed to 68%. She'd learned that ValueMark tends to price sentimental-but-common items lower than instinct suggests, and rare, well-preserved items higher. That's the mental model onboarding was actually trying to build.

Both sellers click accept about as often. Only one of them is actually thinking about the number before they do.

Another seller in the same cohort, someone who never engaged with the guess-first check at all, kept an identical 74% acceptance rate the entire time. On paper, that looked just as healthy as Bettina's.

Bettina and the other seller start in nearly the same spot. Only the prediction check shows which direction each of them is actually moving.

The gap between the first mark and the last one is four weeks of warning a dispute-rate metric alone would never have given anyone.

Six weeks later, that other seller listed a rare item and accepted a suggested price well under what a specialist later told them it was worth. The complaint that reached Farrowgate's support team blamed the tool. The real story was a mental model that had never actually formed, sitting quietly behind a completely normal-looking acceptance number the whole time.

A seller who always guesses "same as my number" can pass the check without ever really predicting anything, which is exactly why the check items have to include cases where the model actually surprises people.

The old measurement asked only whether sellers used ValueMark. The new one asks whether they understand it, weeks before an unusual item ever puts that understanding to the test.

We built the acceptance-rate dashboard first because it was the easiest number to pull and it always looked good. It took one specialist's appraisal, arriving after the fact, to see that a healthy-looking acceptance rate and a real mental model were never actually the same measurement.

LEAD, in one screenNot the dispute rate three months later. The guess a seller makes before they ever see the answer.

The recap, one line per letter: link is fewer disputes and less seller regret, early signal is the prediction-accuracy trend across the first ten listings, abuse is a seller gaming the check by always guessing their own number, and decision is a targeted re-explainer only for sellers still stuck at chance.

And if you want to be sure it really works, try it somewhere elseSame four letters, a code-review tool instead of a marketplace. Different domain, and the mental model being checked is about risk, not price.

Mergewise is a code-review tool that scores whether a pull request looks safe to auto-merge. Foster Amankwah is a backend engineer whose team turned on the auto-merge suggestion last quarter, and the real risk is a developer treating a high safety score as a certification instead of a probabilistic estimate.

Mapped onto LEAD: link is fewer bad auto-merges reaching production, the real cost the whole feature exists to avoid. Early signal is asking developers, right after onboarding, to predict whether Mergewise will flag a sample pull request as safe or risky before revealing the score, tracked across their first ten reviews. Abuse is a developer who always guesses "safe" since most pull requests genuinely are, looking accurate without ever engaging with the cases that actually diverge from that default. Decision is showing a real near-miss example to anyone still guessing "safe" every time by their tenth review, since that's the pattern most likely to miss a genuine risk later.

Two of these four branches exist specifically to catch the developer whose guess never actually varies.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "measure whether they can predict it, not whether they click it," and stop.
Cost: there's no engineering time this quarter to build a live prediction-check screen. Say so, and start with a single post-onboarding quiz instead of a running check across ten listings.
The model gets better, for real: even as ValueMark's accuracy improves, sellers still need a mental model calibrated to when it's wrong, since a rarer miss on a better model is still a miss someone has to catch.

Where people run it wrong.
They measure acceptance or click-through rate and call it comprehension, when the two can look identical and mean completely different things.
They check comprehension once, right after onboarding, and never again, missing that the real test is whether it holds up over real use.
They build the check entirely from average-case examples, missing the edge cases the mental model actually needs to cover.

How to use it live. When someone asks how to measure a mental model, ask yourself first: what would this person have to predict, correctly, to prove they actually understand it? Measure that guess, not the click that follows it.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0