CaseAdvancedDesigning for Uncertainty & Trust / Onboarding users to probabilistic products / #13
What happens when a user's first interaction produces a bad output?
FLIPS the product is StageAI, a virtual staging tool that generates furnished photos of empty listing rooms
Bellcourt Homes gave StageAI to its listing agents this spring. Marisol Draven has staged homes by hand for five years and agreed to trial it on one listing, an awkward bay-window room she wasn't sure any tool could handle.
The direct answer
A bad first output doesn't cost you one wrong photo. It costs you the whole relationship, because a new user has no track record to weigh it against. Design the first output to flag its own uncertainty and never auto-replace anything without a visible side-by-side, so a bad guess reads as "it flagged a hard room" instead of "this thing doesn't work."
Do this, in order
Flag low-confidence layouts before showing the generated image, not after.Why: this is the one design choice that turns a bad first guess into a caught mistake instead of a lost user.
Never auto-apply the first output. Keep the empty room visible next to it.Why: a side-by-side lets her judge the miss in ten seconds instead of discovering it later, embarrassed, in front of a buyer.
Give her a second, easier room to try in the same session.Why: one bad room shouldn't be the only data point she ever gets before deciding the tool is broken.
Track first-session abandonment separately from later usage drop-off.Why: someone who never comes back after one bad room looks identical to churn on a normal dashboard, and it isn't the same problem.
Leave ordinary rectangular rooms exactly as fast as they are.Why: those are the rooms the model actually handles well, and slowing them down to double-check would cost more than it protects.
How to answer this, stage by stageSix moves, in the order you'd actually say them.
Stage 1
Scope it to one real room, one real agent
Say it like this
"I'll answer this for StageAI, a virtual staging tool, and one listing agent trying it for the very first time on a tricky bay-window room."
Why this works
Turns a hypothetical about "bad first outputs" into something you can trace start to finish.
Stage 2
Say your structure out loud
Say it like this
"I'll use FLIPS. Find the person. Locate the habit. Identify the flip. Pinpoint the old decision. Show the replay."
Why this works
Tells the interviewer this is a real behavior change being traced, not a list of nice features.
Stage 3
Reframe the question
Say it like this
"A bad first output isn't really a quality problem. It's a trust problem with zero cushion, because a brand-new user has no good experiences yet to weigh the bad one against."
Why this works
Moves the answer past "make the model more accurate" into the actual design question.
Stage 4
Name the flip
Say it like this
"She goes from willing to trial a new shortcut, to closing the app and telling her whole team it doesn't work. No middle ground. That's the flip, and it happens inside one session."
Why this works
Identifies the exact verb that snaps, which is the hardest and most important part of FLIPS.
Stage 5
Give the one decision
Say it like this
"Flag the room as low-confidence before she ever sees the staged photo, and keep the empty room right beside it so she can judge the miss herself instead of getting blindsided."
Why this works
This is the concrete fix, defensible on its own, not just a wish for "better onboarding."
Stage 6
Close on the one line
Say it like this
"So the goal isn't a perfect first output. It's a first output that fails in a way she can catch, so one bad room becomes a caught mistake instead of a closed tab."
Why this works
Restates the decision and the reason in one breath.
Let's learn
StageAI reads a photo of an empty room and generates a virtually furnished version in about ten seconds, meant to replace the cost of physical staging or a manual photo edit.
Before it, Marisol either staged a room in person, several hours and a rental truck, or hired an editor to manually furnish a photo, usually a full day's turnaround.
Knowledge spark: what's "depth estimation"?
How a model guesses the actual shape and depth of a room from a flat photo: where the walls bend, how far back a corner sits. An unusual room shape, like a bay window alcove, can fool that guess in ways a normal rectangular room never does.
Marisol's very first room was the bay-window alcove she'd picked specifically because it was hard. StageAI generated a confident-looking staged photo, a sofa placed squarely across what turned out to be the alcove's recessed window seat.
Five steps, and the third one, finding the exact flip, is the one this whole answer turns on.
And here's the turn: the mistake itself wasn't the real cost. A bad photo is a bad photo, five minutes to notice and discard. The real cost is what a brand-new user does with zero other data points to weigh it against.
Most trust breaks slowly over months. A first interaction breaks it in one sitting, because there's no history yet to soften the fall.
At its worst: Marisol closes the app, tells three colleagues at Bellcourt's Monday meeting that "the AI staging thing doesn't work," and none of them ever open it either, based entirely on one room StageAI was never going to handle well without a warning.
The decision I would take back
We shipped the very first version with no confidence flag on unusual room shapes, because in our internal testing, the depth-estimation problem never actually came up. That made sense when our test set was mostly standard rectangular rooms. It stopped making sense the first time a real user's room had a bay-window alcove our test set had never included.
What I would leave alone: ordinary rectangular rooms, the vast majority of listings, generate cleanly and fast. Slowing those down with the same flag-and-check ceremony would cost more trust than it protects.
We didn't lose one bad photo. We lost the only shot a brand-new user gives you before deciding for good.
The lesson: a first interaction has no cushion. Whatever a probabilistic product shows on day one has to survive being wrong, because there's no track record yet to catch it.
Now here is the same thing as a storyThe short version is above. Read this for how the actual fix got found.
Marisol can walk into an empty room and see, almost instantly, exactly which corner needs a reading chair and which wall needs to stay bare. Five years of staging homes by hand taught her that.
She picked the bay-window room on purpose, half as a real test and half, she'd admit later, hoping to catch the new tool out.
We built for a dial, trust rising and falling gradually. What actually happened on day one was a switch.
StageAI's answer came back in ten seconds: a clean, confident-looking photo, sofa and coffee table placed squarely across the alcove's recessed window seat, as if the seat weren't there at all.
Marisol didn't check a settings menu or read a help article. She closed the laptop, texted her manager "this thing put a couch through a window," and moved the listing back to her usual in-person process for good.
With the flag in place, the whole session runs to a different ending in the same ten minutes.
With the fix in place, that same ten-second generation now comes back with a small flag: "Unusual layout, low confidence in furniture placement," and the empty original photo stays right beside the staged one instead of quietly replacing it.
None of these four pieces fix the model's guess. They just make the guess safe to be wrong.
Marisol sees the flag, laughs at the couch-through-a-window attempt instead of feeling burned by it, and tries a second, ordinary bedroom in the same listing. That one comes back clean, no flag, and she uses it in the final listing photos.
By the end of that quarter, instead of zero, she'd used StageAI on eleven more listings, always skipping straight past any room the flag caught and trusting the ones it didn't.
I would take back shipping with no confidence flag at all. It made sense when our own test rooms never surfaced the problem. It took one real bay-window alcove, and one lost user who almost never came back, to see that a model's blind spot and a user's first impression are the same event.
FLIPS, the flip that happens in one sittingFive letters. The third one is the whole method.
F
Find the person.
Marisol Draven, a five-year listing agent at Bellcourt Homes who stages by hand and picked the hardest room in the house to test StageAI first.
A named person with real skill, not a generic "new user."
L
Locate the habit.
Before StageAI, she trusted her own eye completely and had no shortcut habit yet to protect. This was her first ever attempt to hand a judgment call to a tool.
The habit under threat isn't a product habit yet. It's her willingness to try a shortcut at all.
I
Identify the flip.
Willing to trial a shortcut, to closing the tab and telling her team it's broken. No middle setting, and it happened inside one ten-minute session.
The hardest step. A first-session flip has none of the slow decay a habit-based flip usually shows.
P
Pinpoint the old decision.
Shipping with no confidence flag on unusual layouts, since our internal test rooms never surfaced the depth-estimation problem a real bay-window room did.
A specific, reasonable-at-the-time call, not a vague "we should have tested more."
S
Show the replay.
Same bay-window room, same wrong guess, but flagged and shown beside the original. She catches it in ten seconds, tries a second room, and uses StageAI on eleven more listings that quarter.
Ends in something countable: eleven listings, not zero.
Return rate after a bad first output, by design
Same bad first room, same model. One design choice about how it fails changed whether a user ever came back.
Error rate on unusual layouts, over the model's first two quarters
The model kept improving on unusual rooms all along. The flag is what carried users through the months before it fully caught up.
The recap, one line per letter: find the person is Marisol picking the hardest room on purpose, locate the habit is her total trust in her own eye with no shortcut habit yet, identify the flip is willing-to-trial snapping to closed-for-good in one session, pinpoint the old decision is shipping with no confidence flag, and show the replay is eleven listings instead of zero.
And if you want to be sure it really works, try it somewhere elseSame five letters, a veterinary clinic instead of a listing photo.
Thistlemoor Veterinary Clinic uses VetPulse, an AI tool that reads a vet tech's intake notes and drafts a triage summary before a vet sees the animal. Anneke Brandt is a vet tech there and tried VetPulse for the first time on an unusual case, a dog with symptoms that didn't match any single common condition cleanly.
Mapped onto FLIPS: find the person is Anneke, three years into vet tech work and confident reading symptoms herself. Locate the habit is her full trust in her own triage judgment, with no AI shortcut habit yet formed. Identify the flip is willing-to-try snapping to refusing to open VetPulse again, inside one shift, after its first summary confidently pointed toward the wrong condition. Pinpoint the old decision is shipping the triage summary with no flag for cases that didn't cleanly match its training patterns. Show the replay: with a flag reading "atypical symptom combination, treat as a starting point only," Anneke catches the mismatch immediately and still uses VetPulse on her next four typical cases that same week.
Different animal, same design principle: the flag has to arrive before trust gets tested, not after.
Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "flag the low-confidence case before she sees the output, and never auto-apply it," and stop.
Cost: no engineering time to build a real confidence model this quarter. Say so honestly, and start with a simple rule-based flag, anything the model's training data barely covered, since a rough flag still beats none.
The model gets better, for real: if StageAI's overall accuracy improves, unusual rooms are still where it's weakest, and a rarer miss on exactly those rooms is more dangerous, since users start expecting it to just work everywhere.
Where people run it wrong.
They treat a first-session dropout the same as ordinary week-two churn, when it's a completely different, much less forgiving problem.
They try to fix this by making the model more accurate everywhere, instead of designing for the specific moment it's still going to be wrong.
They auto-apply the first output to look impressive, removing the one pause where a new user could catch a miss themselves.
How to use it live. When someone asks what happens after a bad first output, ask yourself one question first: does this person have any other good experience yet to weigh it against. If the answer is no, design for that moment specifically, not for the average case.
Flashcards (tap any card to flip it)
1 · THE FLIP FAMILY
What flip family is this?
Tap to flip
ANSWER
Abandonment flip: uses it once, quietly never opens it again, compressed into a single first session instead of unfolding over months.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Marisol Draven, a five-year listing agent at Bellcourt Homes who stages homes by hand and picked the hardest room to trial StageAI first.
3 · THE HABIT
What did she trust completely before ever opening StageAI?
Tap to flip
ANSWER
Her own eye for staging a room, built over five years, with no AI shortcut habit yet in place to protect or threaten.
4 · THE FLIP, IN THIS STORY
What's the two-setting switch here?
Tap to flip
ANSWER
Willing to trial the tool, versus closing it and declaring it broken to her whole team. No middle setting, and it happened in one session.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Shipping with no confidence flag on unusual room layouts, since the internal test set never included a room like the bay-window alcove.
6 · THE NUMBER
Fill in the blank: with no confidence flag, only ___ percent of users who saw a bad first output ever came back.
Tap to flip
ANSWER
12 percent. With the flag in place, that number rose to 68 percent.
7 · THE REPLAY
Same bay-window room, redesigned first output. What changes?
Tap to flip
ANSWER
A flag reads "unusual layout, low confidence." She catches the couch-through-window miss in ten seconds, tries a second room, and uses StageAI on eleven more listings that quarter.
8 · CROSS PRODUCT TRANSFER
Section 4 answers this same question again for a different product. Which product, and which flip family?
Tap to flip
ANSWER
VetPulse, a veterinary triage-summary tool. Same abandonment flip family, this time triggered by an atypical symptom case instead of an odd room shape.
Check yourself Score: 0 / 0
Short answer, recall the flip
1. What was the flip in Marisol's story, and what were its two settings?
Show hint
Look at the "I" step.
Show answer
Model answer: Willing to trial the tool, versus closing it and declaring it broken to her team. No middle ground, and it happened within one session.
Multiple choice
2. Why couldn't Marisol have just "given it a bit more of a chance" instead of flipping right away?
A. Because she was too busy to try again.
B. Because a brand-new user has no other good experience yet to weigh a bad one against, so one bad room is the whole track record.
C. Because StageAI required a paid upgrade to try a second room.
D. Because her manager told her to stop using it.
Show hint
Look at the "small move, big snap" diagram.
Show answer
B. With no track record to soften it, the first bad output is the entire relationship, not one data point in a longer one.
True or false
3. True or false: this answer recommends adding the same confidence-flag check to every room StageAI generates, including ordinary rectangular ones.
True
False
Show hint
Look at "what I would leave alone."
Show answer
False. Ordinary rooms, the vast majority of listings, are left exactly as fast as they already are. The flag targets unusual layouts specifically.
Fill in the blank
4. Fill in the blank: error rate on unusual layouts started at 34 percent in month 1 and dropped to ___ percent by month 8, as more edge cases got added to training.
Show hint
Look at the line chart.
Show answer
9 percent. The flag is what carried real users through the months before the model fully caught up.
Short answer, name the reversal
5. What old decision does this answer take back, and why did it make sense when it was made?
Show hint
Look at "the decision I would take back."
Show answer
Model answer: Shipping with no confidence flag on unusual layouts. It made sense because internal testing never surfaced the problem, until a real user's room did.
Short answer, apply it yourself
6. Pick a product you tried once and never opened again. What would have changed your mind if its first bad moment had been designed differently?
Show hint
Think of an app or tool that got something wrong on your very first try, and whether it explained itself at all.
Show answer
Model answer: Many people describe abandoning a translation or voice tool after one confidently wrong answer, and say a simple "not sure about this one" note would have kept them trying it.
Before you close the answer
Why this works
Tests whether you'll treat a first bad output as a design problem with zero trust cushion, rather than assuming better average accuracy will eventually fix it on its own.
Follow-up traps
"Isn't flagging uncertainty just going to make the tool look unreliable from day one?" Response: the opposite happened in practice. Return rate rose from 12 percent to 68 percent, because a flagged miss reads as honesty, and a silent miss reads as broken.
"What if you just improve the model so this never happens?" Response: worth doing in parallel, but it took eight months to bring the error rate from 34 percent to 9 percent, and a new user's first session can't wait that long for a fix.
If pressed
The real confidence flag used a simple proxy at launch, whether the room's detected wall angles fell outside a normal range, rather than a fully separate uncertainty model, since a rough signal shipped in week one beat a precise one that would have taken a quarter to build.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
