Design a survey instrument to measure perceived reliability.

A survey instrument built around a single specific request the person can actually remember, checked against what the system logs say really happened.

How to answer this, live

Nobody is grading whether you can name "ask about a recent interaction" as a phrase. They're grading whether you can show why a vague reliability question is actually broken, out loud, in front of them. Seven moves get you there.

The plain version

What happens when a survey says your product is fine, and the product is not fine?

Briarwood Home makes a voice hub. It sits on a shelf or a counter. You tell it to turn on a lamp, warm up the living room, or run a whole scene, like "movie night," which is really six small commands stacked into one word.

For three years, Briarwood asked its customers one question every quarter, by email: "Overall, how reliable is your Briarwood Hub, from one to five?" About 9,000 people answered each time, out of roughly 180,000 households. The score sat near 4.2. Steady. Nobody worried about it.

At its worst, a survey like this can sit calm for months while something real breaks underneath it, because a general "how reliable is it" question has no fixed thing anyone is picturing when they answer. Someone who had a great morning and someone whose living room lamp just turned on by itself can both circle a 4.

What I would leave alone: a general "how do you feel about the Hub" question is genuinely fine for things like price or the way the device looks on a shelf. Those really are about overall feeling, not one fact you could check. The mistake was using that same soft question for something as checkable as "did it do what I asked."

The lesson: a survey question is only as good as the memory it asks for. Ask about a feeling in general and you get today's mood. Ask about one real thing that happened this week and you get a fact you can go check.

The same thing, as a story

The short version sits above. Read on for the quarterly review where a support director asked Roksana a question the survey couldn't answer.

Roksana Anholt built Briarwood's quarterly reliability survey herself, back when the company had one product line and about 40,000 households. For most of two years it did exactly what it was supposed to do. It sat near 4.2, and when something genuinely broke company-wide, it dipped, just a little, just enough to matter.

Then Briarwood shipped a firmware update that changed how the Hub resolved a request when a scene touched more than one device with a similar name, like two lamps in the same room. The update was meant to make scenes faster to build. It also made the Hub guess wrong more often about which lamp a scene meant, and when it guessed wrong, it still said "Done," because from the Hub's own point of view, it had completed a scene. It just wasn't the scene the person meant.

Nobody noticed for a while, because nothing about the number Roksana watched gave her a reason to look. Support tickets mentioning a scene "doing the wrong thing" climbed from about 40 a week to around 130 a week over ten weeks. The quarterly survey, over that same stretch, moved from 4.3 to 4.2 to 4.2 to 4.1. Inside the range it always sat in.

The trigger wasn't a crisis. It was one sentence in a quarterly review. The VP of Support looked at the reliability slide, then at her own inbox, and said, "If this has been flat at 4.2 for a year, why is my scene-complaint queue three times bigger than it was in the spring?"

Roksana didn't have an answer. She pulled the raw text responses from the last three surveys to look for a pattern by hand. A good chunk of respondents had written some version of "I don't really use scenes, not sure what you mean." The number wasn't hiding a scene problem. It had never been asking about scenes in the first place.

Worse, she couldn't check a single complaint-shaped comment against what actually happened, because the survey had been built two years earlier with no way to connect one person's answer to their own activity log. That decision made sense at launch, when the team was four people and there was no golden-set pipeline to feed anyway. Nobody had come back to change it once the company had both.

What the old instrument actually was: one soft question, asked the same way every quarter, with no way to tie any single answer back to a real request.

So Roksana rebuilt the survey around one decision. Instead of asking how reliable the Hub is overall, the new version asks: "Think about the last time, in the past three days, you asked your Hub to do something. What did you ask for?" Then, about that one specific thing: "Did it do that exact thing?" And, with a clear opt-in, "Can we check this against your Hub's activity log for just this one request?"

The whole design hangs on this one card. Every other question in the survey is optional. This one is not.

The next time a similar regression shipped, seven months later, Roksana reran the comparison on purpose, as a test. Under the old survey, the flat quarterly score gave no warning until support tickets forced the issue, around week ten. Under the anchored, log-matched survey, the self-reported failure rate among people whose anchored request was a scene command crossed 15 percent by week three, on a sample of about 60 responses, well past the bar she'd set for a real signal. Of those, log matching confirmed roughly 90 percent were genuine mismatches, not people misremembering. Seven weeks of exposure became three.

What the anchor design deliberately does not try to catch: whether people quietly gave up on saying "movie night" out loud and started tapping the scene button in the app instead. That's a real trust signal, and it matters, but it lives in Briarwood's own usage logs, not in anything a person would think to volunteer on a survey about one recent request.

SPARK, in one screen

Not a diagnosis of one bad quarter. SPARK run on the actual design of the instrument, using a real regression as the way to check whether the design would have worked.

Same regression, run through two instruments. One instrument stays quiet the whole time. The other one rings early enough to matter.

Three things worth saying plainly, since this is where the real judgment sits. The alternative rejected was the industry-standard single reliability scale, the exact question Briarwood had used for three years: it lost because it cannot be traced to a command category, it stays flat regardless of what's actually breaking underneath it, and it is genuinely vulnerable to the respondent's mood that day rather than any fact about the product. The AI-specific failure named is silent wrong execution, a form of the Hub confidently doing the wrong thing and reporting it as done, which no ordinary satisfaction survey design would ever think to check for, because the guardrail against it (log matching) only makes sense once you already know your own model can be confidently wrong. The trade-off accepted on purpose: an anchored, three-day recall window and a short, tightly scoped survey trades some of the depth a longer, more reflective quarterly survey could offer, for a much higher chance that what people report actually happened the way they say it did. And the threshold that turns a number into a real signal, not morning noise: a command category only counts as a live problem once its anchored, log-matched self-reported failure rate crosses roughly 15 percent within a rolling two-week window, on at least 40 anchored responses for that category, not on one bad batch of five.

Run it again, somewhere with no smart speaker in sight

Same five letters, a city's 311 phone line instead of a smart hub, and the anchor idea holds up with a completely different kind of AI system underneath it.

The City of Aldercreek runs an AI phone line for 311. Residents call to report a pothole or schedule a bulk trash pickup, and the assistant handles most calls without a human ever picking up. Colette Deschamps analyzes call data for the city's public works department.

S, situation. Today, without a good instrument, Aldercreek relies on the automated post-call rating: press a number, one to five, right after you hang up. It has averaged 4.4 for over a year. Colette has no way to tell whether that number reflects the assistant actually doing the right thing, or just sounding polite and answering fast.
P, payoff. Stop citing "4.4 out of 5, satisfied" in a public works meeting. Start citing whether the specific thing a caller asked for actually got logged the way they asked for it.
A, anchor. A short follow-up text two days after the call: "What did you call about?" then "Did what happened match what you were told would happen?" matched, with consent, against the actual work order created from that call.
R, risk. The old post-call rating mostly measures how pleasant the voice sounded and how short the hold was, not whether the request landed correctly. The AI-specific failure here: the assistant mishears a cross-street name and logs a bulk pickup request under the wrong address, then confirms the pickup out loud, so the caller hangs up thinking it worked. Guardrail: log-matching catches the address mismatch even when the caller never noticed anything wrong.
K, keep out. How many days it actually takes public works to clear a logged pothole is a real operational number, already tracked in the work-order system. It doesn't belong in a phone survey about whether the call itself went the way the caller expected.

Swap the trigger and it still runs.
Speed: an interviewer gives you ninety seconds. Skip straight to the anchor: "ask about one real request, in the last few days, and check it against the log." Everything else is why that one line is correct.
Cost: the log-matching pipeline is eight weeks out. Don't drop the anchor while you wait, keep asking the anchored question and read the free text by hand for outright contradictions until the pipeline ships.
The model got better, for real: say the intent-recognition model's own offline accuracy genuinely improved that quarter. That still isn't proof every category is fine. A model that is more accurate on average can still be badly wrong on one category it wasn't tuned for, and neither the old scale nor an improved accuracy score would show you that, only the anchored, log-matched category breakdown would.

Where people get this wrong.
They add the anchored question, then still lead every review with the old blended score, because that's the one number leadership already knows how to read.
They treat a stable anchored score as proof of nothing wrong, instead of checking whether the category it's measuring is even getting enough real traffic to say anything yet.
They respond to one bad category by adding a fifth or sixth survey question instead of asking whether the real fix is a guardrail in the product, not another question on the form.

How to use it live. Open with the anchor, not the framework name: "I'd ask about one real recent request, not a general feeling, and check it against the log." That gives you a concrete decision on the table before the interviewer can steer you toward a longer, softer list of survey questions.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0