CaseAdvancedDesigning for Uncertainty & Trust / Onboarding users to probabilistic products / #16
Design the moment where you ask a user for the data the feature needs.
SPARK the product is Stackwell, an AI incident-triage assistant built into an engineering team's on-call tools
Stackwell watches a team's service metrics, flags anomalies, and drafts a first-pass incident summary before a human ever opens the page. Emerson Vail is the sole SRE at a 40-person startup, and inherited the on-call rotation three months ago from a teammate who left.
The direct answer
Don't open onboarding with a blank settings form asking the user to describe their systems from memory. Show them ten real, specific moments pulled from their own history, and ask a one-tap question about each: was this a real problem, or noise? That's the whole data-ask. It uses a judgment the person already has, instead of asking them to invent one on the spot.
Do this, in order
Ask for a judgment on real past events, not a description written from memory.Why: a person can tag ten spikes they remember. Almost nobody can accurately describe "normal" for their own system in the abstract.
Make each ask a single tap, not a field to fill in.Why: a ten-tap review gets finished. A ten-field form gets abandoned halfway, and half a calibration is worse than none.
Ask the one person who will actually judge the output, not a persona or a team-wide survey.Why: the calibration has to match the standard whoever is on call will hold it to at 2am.
Give the model a usable floor from ten tags, and keep improving it from real pages afterward.Why: waiting for a perfect calibration before launch means launching never; ten tags is enough to be better than a generic default.
Leave the deep historical log ingestion for later, once one person's setup proves the idea works.Why: pulling weeks of every team's logs on day one is exactly the slow, heavy ask this design is trying to avoid.
How to answer this, stage by stage
The interviewer isn't grading whether you drew a pretty screen. They're grading whether the moment you designed actually gets answered honestly, by a real person, in under a minute.
Stage 1
Scope it to one person, one product
Say it like this
"I'll answer this for Stackwell, an AI incident-triage tool, and for Emerson, the one SRE who'd actually set it up."
Why this works
Keeps the answer from drifting into a generic "collect user data" essay.
Stage 2
Say your structure out loud
Say it like this
"I'll use SPARK. Situation, what he does today. Payoff, the habit I want to build. Anchor, the actual design decision. Risk, what breaks the first time I'm wrong. Keep out, what I won't build yet."
Why this works
Tells the interviewer you have a method, not just an opinion.
Stage 3
Ground it in what happens today, no product
Say it like this
"Right now, when the pager goes off, Emerson opens four dashboards and decides real-or-noise using thresholds he's memorized. There's no screen for that. It's in his head."
Why this works
Shows the interviewer you know what "normal" even means before you design around it.
Give the one decision
Say it like this
"The data-ask moment is a ten-tap review screen. It shows Emerson ten real spikes from his own metric history, and asks 'incident, or noise,' one tap each. No blank fields, nothing written from memory."
Why this works
This is the answer, stated concretely enough to argue with.
Stage 5
Prove it survives being wrong
Say it like this
"Say the very first live page after launch is a false alarm. Because the threshold came from Emerson's own ten tags, the screen can say 'you called something like this noise before,' and he downgrades it in one tap instead of losing trust in the whole tool."
Why this works
Answers SPARK's hardest step: does the anchor actually survive its own risk?
Stage 6
Name what you'd deliberately skip
Say it like this
"Day one, I would not ask for a full historical log import, and I would not ask every teammate to separately define normal. Just the primary on-call owner, to start."
Why this works
Shows judgment about scope, not just enthusiasm for building everything at once.
Stage 7
Name the rejected alternative
Say it like this
"I considered a free-text setup form asking people to describe their infrastructure and thresholds. I rejected it, because it asks for a judgment nobody can actually produce accurately from memory."
Why this works
Proves this was a real choice between real options, not the only idea that occurred to you.
Stage 8
Close on the one line
Say it like this
"Ask for a judgment the person already has, on data they already trust, in one tap. That's the whole design."
Why this works
Restates the decision once more, so it's the last thing the interviewer hears.
Let's learn
What happens the first time a tool asks a stranger a question before it has earned the right to be trusted with an answer?
Stackwell is an AI tool that watches an engineering team's metrics, flags anything unusual, and drafts a short summary of what might be going on, before a human even opens the incident channel.
Knowledge spark: what is cold start?
A new AI feature has no memory of what "normal" looks like for a brand-new team. Its first guesses are generic, built from other teams' patterns, not this one's. That gap between generic and calibrated is the cold start problem.
Before Stackwell existed, Emerson decided real-or-noise using thresholds nobody had ever written down. He just knew them. A tool that wants to do his job for him has to somehow get that same knowledge, without making him write an essay about it.
Setup completion rate, by ask design
Almost three times as many people finish the ask when it's ten taps on their own data instead of a form asking them to describe their systems cold.
Skipping setup and using a blank text field look different, but they land in nearly the same useless corner: low value, either way.
At its worst: a team launches with a generic default threshold because nobody finished the setup form, and Stackwell pages someone for a routine deploy blip on its very first night live.
The decision I would take back
We built one long setup form covering every configuration option at once, because it let us ship a single onboarding screen instead of designing several smaller moments. That made sense when the team was small and the form took two minutes. It stopped making sense once the form grew past twenty fields and most people quit halfway through.
What I would leave alone: asking for the team's Slack channel and paging schedule can stay a normal short-text form. That data doesn't require judgment, just typing, so a plain field costs nothing.
The setup screen was never really asking Emerson for data. It was asking him to trust a stranger's guess about his own job, before it had done anything to earn that trust.
The lesson: the moment you ask someone for data isn't a technical step before the real product starts. It's the first real interaction the product has with a person, and it teaches them, correctly or not, whether this thing is worth their patience.
Now here is the same thing as a story
The short version above is what you'd say defending this design in a review. Read this one for how the screen actually got built.
The laptop lives propped open on Emerson's kitchen counter, hinge cracked from a year of being carried between the couch and the desk. Most on-call nights, it's the only light on in the apartment by 11pm.
Before Stackwell, this whole loop lived in Emerson's head. No screen asked him anything, because no screen needed to.
Engineering leadership rolled out Stackwell after a bad outage, and Emerson was the one who had to set it up. The first version of the setup screen was a form: twenty-two fields, asking him to describe normal traffic ranges, typical deploy cadence, and expected error rates, all from memory, all before he'd seen the tool do anything.
He got through six fields and closed the tab. Not out of protest. He just didn't know the numbers off the top of his head, and guessing felt worse than skipping it.
The redesigned ask has no blank field anywhere on it. Every question points at something Emerson already has an opinion about.
The rebuilt version pulled ten real anomalies straight out of the team's own metric history, from the eight months before Stackwell existed, and asked one question about each: incident, or noise? Emerson tapped through all ten in under ninety seconds, because he already knew the answer to every single one.
None of these four rules mention the model. All four are about what's fair to ask a tired person at 11pm.
Two weeks later, Stackwell paged him for the first time, a genuine spike in checkout latency. Because the threshold had come from his own ten tags, the page felt earned rather than random.
The ten tags weren't the finished calibration. They were the starting line the model kept building on with every real page after.
Same 2am page, two different setup screens behind it. Only one of them earns a quiet night.
The old form asked Emerson to invent a description of his own systems out of thin air. The new screen asked him to judge ten things he'd already lived through. Same goal, calibrating the model, completely different ask.
I built the long form because a single screen was simpler to design and simpler to ship than several small moments spread across the first week. It took watching six fields get filled in and sixteen get abandoned to see that "simpler to build" and "simpler to answer" were never the same thing.
SPARK, the moment itselfNot a form. A judgment call, asked in a shape the person can actually answer.
S
Situation. What he does today.
Emerson decides real-or-noise by memory, checking four dashboards against thresholds he's never written down.
Grounds the design in a real workflow that already exists, no screen involved.
P
Payoff. The habit we want.
Stop checking all four dashboards on a fixed rotation. Only look when something crosses a threshold shaped like his own judgment.
Names what the person stops doing, which is the actual product being shipped.
A
Anchor. The data-ask screen itself.
Ten real past spikes from his own history, one tap each: incident, or noise. No blank fields.
The hardest step, and the one concrete decision everything else hangs on.
R
Risk. What breaks the first time it's wrong.
A false alarm at 2am. Survives because the threshold traces back to Emerson's own tags, so he can correct it in one tap.
Tests whether the anchor actually holds up under its own failure, not just on a good day.
K
Keep out. What waits.
No full historical log import, no team-wide survey, day one. Just the primary on-call owner's ten tags.
Shows restraint, which reads as judgment rather than a wish list.
False-alarm pages per week, first month live
The generic default stays noisy every week. The calibrated version starts quieter and keeps getting quieter, because every correction feeds back into the threshold.
The recap, one line per letter: situation is Emerson's memory-based routine, payoff is trading dashboard-checking for threshold-trusting, anchor is the ten-tap history review, risk is the first false alarm surviving because of that same history, and keep out is holding off on full log ingestion until one person's setup proves the idea.
And if you want to be sure it really works, try it somewhere elseSame five letters, a tool-rental marketplace instead of an engineering team. A completely different domain, and the anchor moves to a different point in time entirely.
Barrowlend is a peer-to-peer marketplace where neighbors rent out tools they own. An AI feature drafts a condition report from a lister's photos, but it needs real examples of "worn but fine" versus "actually broken" to calibrate against, and nobody has that data before their first listing goes live.
Mapped onto SPARK: situation is a lister today, snapping a few photos and writing "good condition" from habit, with no real standard behind the phrase. Payoff is getting them to stop writing vague condition claims and start trusting a specific, checkable description instead. Anchor is moving the data-ask to after the first completed rental, not at listing creation: the renter who just used the tool is asked one question, "worn as described, or worse than described," which produces real calibration data from someone with nothing to gain by lying. Risk is the first miscalibrated report reaching a renter; it survives because early reports are shown with a plain "not yet confirmed" mark until enough renter tags exist. Keep out is skipping any attempt to ask the lister for a condition description at all, since self-reported condition is exactly the biased data the feature exists to replace.
Asking at listing creation felt like the obvious moment. It was actually the one branch guaranteed to produce a biased answer.
Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "ask for a judgment they already have, on data they already trust, in one tap," and stop there.
Cost: there's no engineering time this quarter to build a custom history-review screen. Say so, and start with a five-question phone call instead, the same judgment, a slower channel.
The model gets better, for real: if Stackwell's base accuracy improves, the data-ask moment still matters, a better average model calibrated to nobody's actual team is still a worse fit than a mediocre model calibrated to this one.
Where people run it wrong.
They ask for everything at once, in one long form, because it's simpler to build than several small moments.
They ask a persona instead of the specific person who will actually judge the output later.
They treat the setup screen as a technical step instead of the first real conversation the product has with a person.
How to use it live. When someone asks you to design a data-ask moment, ask yourself first: could the person answer this from memory, in one tap, without guessing? If not, you're asking for data. You're asking for an essay.
Flashcards (tap any card to flip it)
1 · THE FRAMEWORK
What framework fits "design the moment where you ask a user for the data a feature needs"?
Tap to flip
ANSWER
SPARK: situation, payoff, anchor, risk, keep out. The anchor step is the actual answer to a design question like this one.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Emerson Vail, sole SRE at a 40-person startup, who inherited the on-call rotation and used to decide real-or-noise from memory alone.
3 · THE HABIT
What habit does this design want to build?
Tap to flip
ANSWER
Stop checking four dashboards on a fixed rotation. Only look when a threshold shaped like his own judgment actually crosses.
4 · THE ANCHOR
What's the actual design decision, the data-ask moment itself?
Tap to flip
ANSWER
A ten-tap review of real past spikes from the team's own history. One tap per spike: incident, or noise. No blank fields.
5 · THE REJECTED OPTION
What alternative was considered and rejected?
Tap to flip
ANSWER
A twenty-two field free-text setup form. Rejected because it asked Emerson to describe his own systems from memory, which most people simply can't do accurately.
6 · THE NUMBER
Fill in the blank: the free-text form finished at a ___ percent completion rate, versus 89 percent for the ten-tap review.
Tap to flip
ANSWER
31 percent. Almost three times as many people finish the ten-tap version.
7 · THE RISK TEST
What happens the first time Stackwell pages someone for a false alarm?
Tap to flip
ANSWER
The screen can say "you called something like this noise before," since the threshold traces back to his own ten tags, and he downgrades it in one tap instead of losing trust in the tool.
8 · CROSS PRODUCT TRANSFER
Section 4 answers this again for a different product. Which product, and when does its data-ask moment happen?
Tap to flip
ANSWER
Barrowlend, a tool-rental marketplace. Its data-ask happens after a rental ends, asking the renter, not the lister, since the renter has no reason to sugarcoat the tool's condition.
Check yourself Score: 0 / 0
Multiple choice
1. Why does the ten-tap history review work better than the twenty-two field form?
A. It asks fewer total questions overall.
B. It asks for a judgment Emerson already has, instead of a description he'd have to invent from memory.
C. It uses a nicer-looking screen design.
D. It skips calibration entirely and uses a generic default.
Show hint
Look at what each of the ten taps is actually asking for.
Show answer
B. Ten real past spikes are things Emerson already has an opinion about. Describing "normal traffic ranges" from memory is not.
True or false
2. True or false: this answer recommends removing all setup fields and asking nothing on day one.
True
False
Show hint
Look at "what I would leave alone."
Show answer
False. Simple fields with no judgment required, like a paging channel, can stay plain text fields. Only the judgment-heavy ask gets redesigned.
Fill in the blank
3. Fill in the blank: the redesigned screen shows ___ real past spikes, one tap each.
Show hint
Look at the labeled parts diagram of the data-ask screen.
Show answer
Ten. Enough to bootstrap a usable threshold without asking for a huge time investment on day one.
Short answer, name the reversal
4. What old decision does this answer take back, and why did it make sense when it was made?
Show hint
Look at "the decision I would take back."
Show answer
Model answer: Building one long setup form covering every option at once. It made sense while the team was small and the form only took two minutes to complete.
Short answer, where it wouldn't matter
5. Name a piece of setup data where a plain text field is still fine, even with this redesign in place.
Show hint
Look at "what I would leave alone."
Show answer
Model answer: The team's paging channel or Slack destination. It doesn't require judgment, just typing, so a plain field costs nothing.
Short answer, apply it yourself
6. Pick an app you've set up yourself. What's one thing it asked you to fill in that you had to guess at, and how could it have asked for a judgment you already had instead?
Show hint
Think about a setup screen that asked you to describe your habits or preferences before you'd used the product at all.
Show answer
Model answer: Many fitness apps ask new users to type a target pace or weight goal cold. A stronger design would show a few real past workouts and ask which pace felt sustainable, a judgment the person can actually make.
Before you close the answer
Why this works
Tests whether you understand that an onboarding data-ask is a UX decision with a real cost, not a form-design afterthought, and whether you can design a moment that survives the model being wrong on day one.
Follow-up traps
"What if the on-call owner leaves and someone new inherits the setup?" Response: the ten tags stay attached to the team, not the person, and the new owner gets a lighter, five-tap re-confirmation instead of starting over.
"Isn't ten historical spikes too small a sample to calibrate anything real?" Response: it's a floor, not a finished calibration; every real page after launch adds another labeled example, so the threshold keeps improving instead of staying frozen at day one.
If pressed
Stackwell's real backtest pulls the ten spikes from a stratified sample across time of day and metric type, not just the ten largest, so the calibration covers quiet-hour noise as well as peak-hour noise.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
