What research would you run to test whether your uncertainty display works?

Trellis Home matches homeowners with contractors for repair jobs and shows a Match Confidence tag, High, Medium, or Low, on every recommendation. Naomi Castellano is the UX researcher who joined the team to find out whether that tag actually changes what people do, not just whether they notice it.

How to answer this, stage by stage

The interviewer isn't checking whether you can name a research method. They're checking whether you'd trust a survey that says "yes, I saw it," over behavior that says otherwise.

Let's learn

Say a marketplace builds a feature that tells homeowners how much to trust its top contractor pick. How would you actually know it works?

Trellis Home's Match Confidence tag appears on every recommended contractor: High, Medium, or Low, based on how much history the algorithm has on that contractor and how well they match the specific job. Before the tag existed, homeowners booked the top-ranked contractor directly about 71% of the time, with no signal at all about how sure the match really was.

The middle box is the one most teams skip, watching what people do, not just asking them about it afterward.

After the tag shipped, the team's only proof it worked was a satisfaction survey: 78% of respondents said they'd noticed the tag, and Trellis Home called that a launch success.

Here's the turn: noticing a tag and acting on it are not the same fact, and nobody had ever checked whether the second one was actually happening. The real risk wasn't that the tag failed. It's that nobody could tell the difference between it working and it just sitting there.

At its worst, an untested confidence display can quietly do nothing while everyone believes it's doing its job, right up until leadership asks for proof and the team realizes they never built a way to check.

What I would leave alone: the underlying match algorithm itself doesn't need this scrutiny, its accuracy was already tracked separately. This research is about whether the display of that accuracy changes behavior, a different question entirely.

The lesson: a feature that shows people something is only as good as your ability to prove it changed what they did next. Without that, "it works" is a guess wearing a launch announcement.

Now here is the same thing as a story

The short version above is what you'd say out loud in an interview. Read this one for how the real gap actually surfaced.

The research lab at Trellis Home empties out by 6, except on Tuesdays, when Naomi Castellano runs live sessions with homeowners testing new features. Six weeks into her first quarter there, in a routine planning meeting, she asked a simple question: how do we know the Match Confidence tag is actually changing anyone's behavior, versus just sitting there?

Nobody on the existing team had a real answer. They had impression counts, showing the tag rendered correctly on-screen. They had the satisfaction survey, showing 78% of respondents recalled seeing it. Neither one said anything about what a homeowner actually did next.

Naomi's team recruited 40 real homeowners with an active repair need and watched them use a live prototype in moderated sessions, rather than asking afterward whether they'd "noticed" anything. Watching mattered: several participants said, unprompted, that they'd seen the tag, then booked the top match anyway without opening its reasoning, exactly the gap a survey alone would have missed entirely.

The survey and the actual booking told two different stories. Only one of them was true.

Alongside the moderated sessions, the team logged real behavior across the full platform, split by confidence tier instead of averaged together. In week 1 of the study, homeowners viewed extra contractor profiles at nearly the same rate regardless of tag, around 30% for both Low and High confidence matches. That flat line was the actual finding: the tag wasn't changing anything yet.

The leading signal would have rung in week 3. The dispute-rate data wouldn't have caught up until week 9.

The team ran an A/B test on the tag's wording, comparing the original generic label against a version naming the specific reason, "Only 2 similar jobs completed nearby" instead of just "Low confidence." Measured against real behavior, not preference, the specific-reason version produced a real, growing gap between tiers within about 8 weeks.

Three possible findings, three different actions. Nobody had written this down before the study started.

LEAD, mapped onto this one research planNot a lecture on research methods in general. LEAD is what tells you which number would have caught the gap first.

Four parts, and none of them is a satisfaction survey question on its own.

The recap, one line per letter: link is genuinely calibrated booking behavior, not tag recall, early signal is the browse-more rate split by tier, abuse is mistaking curiosity clicks for real behavior change, and decision is a pre-set action for each of the three possible findings.

And if you want to be sure it really works, try it somewhere elseSame four letters, a food-delivery app's ETA confidence display instead of a contractor marketplace. A different lagging outcome proves the same early signal works.

Corvid Eats shows a delivery-time estimate with a small "less certain, traffic is heavy" note attached when its ETA model has lower confidence, and product researcher Bertrand Achebe wants to know whether that note actually changes anything, or just makes an occasionally-late delivery feel more forgivable after the fact.

Mapped onto LEAD: link is whether customers' actual patience and reorder behavior improve when expectations are honestly set, not just whether complaint tickets read as less angry. Early signal is the rate of customers who check the live map tracker again within 5 minutes of an uncertain-ETA order, versus a confident one, since checking more suggests the note registered as real information, not decoration. Abuse is a naive version of this research counting total app opens as "engagement," when a customer anxiously refreshing the map is actually a sign of unresolved uncertainty, the opposite of the note working well. Decision: if checking behavior on uncertain orders drops toward the confident-order baseline over a few weeks, the note is genuinely reassuring people appropriately; if it stays elevated indefinitely, the wording needs a concrete next step, like "we'll text you if this changes," not just an acknowledgment of doubt.

All four of these were true of Trellis Home's original launch survey. None of them were true of the redesigned study.

The old decision at Corvid Eats wasn't quite Trellis Home's absent-logging gap, it was closer to a packaging choice: the ETA note was written once at launch and never revisited, treated as a finished, static piece of copy rather than something whose real effect needed checking against actual customer behavior over time, the same way Trellis Home once assumed its own tag was working simply because it had shipped.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "log real behavior by confidence tier, not just impressions, and set an action for each possible finding before you start," and stop.
Cost: there's no budget for a full moderated study right now. Say so, and start with the free part: split existing behavioral logs by confidence tier retroactively; that alone often reveals whether a gap even exists before spending on new research.
The model gets better, for real: if the underlying match algorithm improves so much that Low-confidence tags become rare, the same behavioral-logging discipline still matters, it just has less to show on a shrinking slice of traffic.

Where people run it wrong.
They treat a satisfaction survey's "yes, I noticed it" as proof the feature works.
They count engagement, like clicks or opens, without checking whether it changed a real decision.
They launch research with no pre-agreed action, so every finding becomes a debate instead of a decision.

How to use it live. When someone asks how you'd test an uncertainty display, ask back: what would someone do differently, in their own hands, if this were actually working? Design the study around answering that, not around whether people remember seeing a label.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0