The direct answer
Build the synthetic test set from a real distribution shape, not out of your own head: get anonymized case-mix data from a clinical partner and write test messages that match it, then run the prototype through a stress test of real, documented atypical cases before you trust its accuracy number. The risk was never that made-up data is a little off. It's that data invented from imagination reproduces exactly what you already expected to see, and the patient who doesn't talk like a textbook is the one it misses.
Do this, in order
Build synthetic cases from a real, anonymized case-mix distribution, not from imagination.Why: this is the one practice the whole answer turns on.
Run the prototype against a held-out set of real, documented atypical cases before trusting its accuracy number.Why: it catches exactly the blind spot a hand-written test set can't see.
Treat an offline accuracy score as unproven until it clears that stress test.Why: a number from an invented set measures how well you copied your own assumptions, not how the tool handles a real patient.
After launch, check real traffic against what the synthetic set predicted, broken out by symptom pattern.Why: this is how you catch the miss nobody thought to test for in the first place.
Leave the low-stakes, textbook-shaped complaints alone.Why: forcing a real-distribution rebuild on every category slows down the ones invented data was never actually wrong about.
When a launch date is already set on an offline number, say plainly that the number hasn't been checked against real patterns yet.Why: the honest conversation costs a delay now; the alternative costs a missed patient later.
How to answer this, stage by stage
Eight moves. Naming both groups, showing the exact pattern the invented data missed, and writing the real practice are where the real answer lives.
1
Ground it in one real product before naming a framework
Say it like this
"Say a telehealth startup builds a chat tool. A patient types what's going on, and it tells them: handle it at home, book a doctor this week, or go get help right now. No real patient message has touched it yet, because the sign-off to use real data on this feature hasn't come through."
Why this works
Grounds a risk question in one real product before naming a method, so the answer can't stay abstract.
2
State your structure in one line
Say it like this
"I'd use GUARD here, because 'how do you prototype without real customer data' is really asking who gets protected by leaving real data out, and who ends up carrying the gap that decision creates. Who's affected, where the blind spot lands hardest, who can't tell the test was ever unrealistic, the actual practice you'd add, and how you'd catch what it missed."
Why this works
Two seconds that show a plan exists before you say a single specific thing.
3
Name both groups, not just "the patient"
Say it like this
"There's the patient, whose real messages never get touched and stay private the whole time this thing gets built. And there's the team, who has to build and test the entire tool on cases nobody real ever actually said, because that's the only data they're allowed near."
Why this works
This is GUARD's G step. The same privacy decision gives the two groups two completely different positions.
4
Show where the blind spot lands hardest
Say it like this
"The team wrote three hundred test messages themselves, off a public symptom checklist. Chest pain, shortness of breath, the textbook heart attack. It scores ninety-six percent against that. But a lot of real patients, especially women having a heart attack, describe nausea, an ache in the jaw, feeling tired, no chest pain at all. That pattern was never on the checklist, so it was never in the test set either."
Why this works
This is U. It names the exact real-world pattern the synthetic data never included, not "the model might be biased" in the abstract.
5
Name who can't tell the test was unrealistic
Say it like this
"Marites, the clinical reviewer who signs off on launch, sees ninety-six percent and approves it. She has no way to know that number came from cases the product manager invented herself, not from anything shaped like a real patient population. She's trusting a number she can't actually check."
Why this works
This is A, GUARD's hardest step, and the one a synthetic-data question usually skips.
6
Write the actual practice, not a values line
Say it like this
"Here's what I'd add. Get anonymized case-mix numbers from a partner clinic, just the shape, how often each symptom cluster shows up, the age spread, how patients tend to phrase it, no real messages. Write the synthetic set to match that shape. Then build one more set: real, published, atypical case write-ups. The prototype has to pass those before anyone trusts the ninety-six percent."
Why this works
This is R, and it's the actual answer. A specific practice with a name, not "be more careful with data."
7
Say how you'd detect it after launch
Say it like this
"Once it's live, I'd run every incoming message through the model in shadow mode first, without acting on it, and compare its call to what the nurse line actually did, broken out by symptom pattern. If the atypical cluster starts disagreeing with the nurses more than the textbook cluster does, that's the flag, before a patient ever gets sent home who shouldn't have been."
Why this works
This is D. It names exactly what you'd watch, not "monitor the model."
Say it like this
"So: build the synthetic set from a real shape, stress-test it against real atypical cases before trusting the number, and watch for the gap by symptom pattern after launch. That's the whole answer."
Why this works
Closes on the direct answer in one breath, and shows the fix isn't blanket caution about synthetic data.
Let's learn
The tool is a chat box a person opens on their phone when something feels wrong, before they've decided whether it's worth calling anyone.
This is Amberlace Health's newest feature. A patient types what's going on, and it tells them one of three things: handle it at home, book a doctor this week, or go get help right now.
Before this feature existed, every one of those same questions went to a phone line. Two nurses covered most nights, working down a paper checklist, and between them they triaged about ninety calls, roughly nine minutes each. Real messages, real people, real numbers.
Knowledge spark: what's synthetic data?
Fake messages, written to stand in for real ones, so nobody's actual information has to be used. Nothing wrong with a person's actual information staying out of it. The catch is that a fake message is only as good as whatever it was based on.
The new chat tool answers in under two seconds. In its first round of testing, it got ninety-six percent of its test cases right.
Here's the catch. None of those test cases came from a real patient. The team isn't allowed to touch a single real message yet, because the sign-off for using real patient data on this feature hasn't come through. So the product manager, Bridget Nolan, wrote all three hundred test cases herself, off a public ER symptom checklist. Chest pain. Shortness of breath. The textbook version of a heart attack.
When the team later ran the same tool against forty real, anonymized transcripts a partner clinic donated, just to see, its accuracy on the ordinary, textbook-shaped cases held at ninety-six percent. On the cases where a patient described a heart attack the way a lot of real patients actually do, nausea, an ache in the jaw, feeling unusually tired, no chest pain anywhere, its accuracy dropped to sixty-one percent.
Triage accuracy, textbook-shaped cases vs real atypical cases
Same tool, same test day. The only thing that changed was whether the case sounded like something a real patient would actually say.
Textbook-shaped cases, written from the checklist
96%
Real atypical cases, donated by a partner clinic
61%
The tool never got worse. The test just never asked it about the patient the checklist forgot to describe.
The real gap wasn't four points of accuracy. It was a patient nobody had ever written a test case for.
At its worst, this costs Amberlace Health a patient who has a real heart attack and gets told to rest and drink fluids, from a tool that looked, on paper, like it was ready to launch.
The choice I would take back
Bridget wrote every test case herself, entirely from imagination, because that felt like the safe, privacy-first way to test before real data was allowed near the product. That made sense when nobody had thought to ask a clinical partner for the shape of its real cases, not the cases themselves. It stopped making sense the day that ninety-six percent became the number leadership used to plan a launch.
What I would leave alone. For a stuffy nose or a mild headache, a made-up test message and a real one read almost the same. Rebuilding that whole slice of the test set from a real distribution wouldn't move a single score. Save the real-shape effort for the categories where a real patient's own words are doing the work.
The lesson. Not touching real patient data protects the patient. It does not automatically protect the test. Those are two different jobs, and a startup can do the first one right and still build a test that quietly agrees with everything it already assumed.
Now here is the same thing as a story
Read the short version above if you're pressed for time. Read this one when you want to feel why the shape of the data matters, not just know that it does.
Bridget Nolan can turn a vague clinical requirement into a working prototype faster than almost anyone on her team. She's been the product manager for Amberlace Health's intake group for two years, and she's the one people call when a feature needs to move fast without cutting a privacy corner.
When legal flagged that the new triage chatbot couldn't touch a single real patient message until a new consent flow shipped, months out, Bridget didn't wait around. She sat down with a public ER triage checklist and wrote three hundred believable patient messages herself: fevers, cuts, breathing trouble, chest pain, the whole list.
For three weeks, this went great. Every time she added a new symptom category, the tool's score held around ninety-five, ninety-six percent. Her clinical advisor called the test set thorough in a review meeting. Leadership started sketching where a triage bot might launch first.
At first, she checked every new category against a medical reference before writing the message. Then she started reusing the same handful of sentence patterns, since they scored just as well. Then, three weeks in, she stopped adding new categories at all, and just reran the same three hundred cases build after build, watching the number hold.
Then came a slow Tuesday evening.
Constance Herrick, a nurse helping pilot-test the internal build, typed in her own symptoms as a joke. Then, almost as an afterthought, she pasted in a real transcript a friend had shared for a training deck, already anonymized: a patient describing nausea, a dull ache along her jaw, and feeling unusually tired. No chest pain anywhere in it. The tool's answer came back: rest and drink fluids, check in tomorrow if it's not better.
Constance knew enough to know that wasn't right. She flagged it to Bridget the next morning, not as an incident, just a "hey, this one looked off."
Same message. Two very different starting points.
Bridget didn't wave it off. She spent the better part of two days chasing down real, anonymized transcripts anyone could legally share, forty of them from a partner clinic, and ran the tool against them properly. Ordinary cases held at ninety-six percent. The atypical ones, the jaw ache, the tiredness, no chest pain, came back at sixty-one.
We didn't lose four points on the real cases. We lost the four weeks it would have taken to find out, if Constance hadn't happened to paste in the wrong transcript on a slow Tuesday.
I want to say the problem is that Bridget cut a corner. She didn't, not really. She never had a number in her head either, just a feeling: the test set is working, or it isn't. Three straight weeks of ninety-six percent had switched her feeling to "it's working," and there was no in-between setting where she kept wondering what the checklist had left out.
Back when the eval plan got signed off, in a meeting with legal and her clinical advisor, the question on the table was simple: how do you test this without touching real patient data. The answer everyone landed on, sensibly, was that Bridget would write the cases herself, entirely invented, so there was zero chance of real information leaking in before consent existed. Nobody in that room thought to ask whether a partner clinic could share just the shape of its real cases, the counts, the age spread, the way symptoms tend to cluster, without sharing a single real message.
I would go back and put that question in the room. Bridget builds her three hundred cases against a real case-mix distribution from the start, and adds the atypical stress test before the first review meeting, not four weeks after it. The sixty-one percent number shows up in week one instead of week four, before three weeks of phrasing work go into a set built the wrong way, and before Constance ever has to paste in that transcript at all.
One design tells you the number is good. The other tells you what the number is actually good at.
What I'd tell the Bridget who sat down to write those three hundred cases: "not touching real data" and "building a realistic test" were never the same promise, and I only ever kept the first one.
The five checks, run against a test set built entirely from imagination
This is a risk question, so the framework is GUARD. "How do you prototype without real customer data" sounds like a privacy question, which is exactly why nobody thinks to ask whether the workaround is realistic, only whether it's safe.
G, groups. Bridget's team, who get to build and test fast because no real patient data is anywhere near the product yet. And the patient, whose privacy is protected the whole time, but who has no way of knowing whether the tool that's about to triage her was ever tested against anyone who talks like her.
U, unequal. A patient with a textbook-shaped complaint, a stuffy nose, a sprained ankle, gets a test set built the safe way and a tool that reads her fine. A patient whose real heart attack looks like nausea and a tired jaw, not chest pain, gets a tool tested entirely on a checklist that never once described her.
The check that should sit here, and doesn't.
A, ability to contest. Marites, the clinical reviewer who signs off on launch, sees a ninety-six percent number and approves it. Nobody showed her how the test set was built, only what it scored, so she has no way to check whether that number came from anything shaped like a real patient population.
R, reduce. Get anonymized case-mix numbers, just the shape, from a partner clinic: how often each symptom cluster shows up, the age spread, common ways patients phrase it. Build the synthetic set to match that shape instead of a checklist. Then run one more set, real, published, atypical case write-ups, and require the tool to pass it before the ninety-six percent counts for anything.
D, detect. Once it's live, route real incoming messages through the model in shadow mode first, without acting on its call, and compare its read to what the nurse line actually did, broken out by symptom pattern. A gap that shows up specifically on the atypical cluster, and nowhere else, is the flag.
Where this answer would fail
If the fix is "collect more synthetic examples" or "add a disclaimer," none of it counts. A real case-mix shape from a named partner, and a stress test built from real write-ups, are things you can put on a launch checklist and check whether they happened.
And if you want to be sure it really works, try it somewhere else
A port authority's cargo risk-screening tool runs into the same gap, in a room with nothing to do with medicine.
G, groups. Rashida Kwarteng, a risk analyst who builds a solo tool flagging containers worth a manual inspection, using manifest data she wrote herself because the real shipping records are commercially sensitive and off-limits before a data-sharing agreement is signed. And the port's inspection unit, who inherit whatever the tool flags once leadership sets a target date for cutting the inspection backlog.
U, unequal. A container carrying an ordinary, common commodity gets screened by a tool trained on invented manifests that look a lot like it, and reads fine. A container using a real, documented smuggling pattern, one that shows up in actual seizure records but never made it into Rashida's imagined examples, sails through the same screen.
A, ability to contest. The inspection unit hears about the rollout date in a planning memo, with no chance to ask whether the tool's catch rate was ever checked against a real shipment mix, just the invented one Rashida wrote.
R, reduce. The same practice: pull anonymized shipment-volume and commodity-mix numbers, just the shape, from the port authority's own statistics office, build the synthetic manifests to match it, and require the tool to pass a stress test built from real, historical seizure write-ups before its catch rate counts for anything.
D, detect. Run the tool in shadow mode against real incoming manifests for a full quarter without acting on its flags, and compare its picks to what human inspectors actually chose, broken out by cargo category.
Swap the trigger and it still runs
- Speed: a faster model just means the atypical blind spot gets found two weeks sooner instead of two months, but the fix stays the same, a real-shape set and a real stress test before the number gets trusted.
- Cost: a cheaper model means more teams prototype solo on invented data more often, which means more blind spots getting built in before anyone thinks to ask about the shape.
- The model gets better: a sharper model makes the invented test set score even higher, which makes it easier to trust, not harder, because a better-looking number is exactly what gets a launch date approved.
Where people run it wrong
- Treating "we didn't use real data" as proof the product is safe, when it's actually proof the test was never checked against anything real.
- Waiting until a tool is live to find its blind spots, instead of stress-testing against real-shaped data before launch.
- Turning the fix into "collect more synthetic examples," which just makes the same invented pattern bigger, not truer.
How to use it live
Ask one question before you answer: "who wrote the test data, and did they write it from something real?" That buys you a second to think, and it's usually the exact question the interviewer wanted you to ask.
Flashcards (click a card to flip it)
1 · THE FRAMEWORK
Which framework fits a question about prototyping without real customer data, and why?
Tap to flip
ANSWER
GUARD, for risk. The real question isn't whether synthetic data is acceptable, it's who's protected by leaving real data out, where the blind spot in invented data lands hardest, who can't tell the test was ever unrealistic, the actual practice you'd add, and how you'd catch what it missed.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Bridget Nolan, a product manager at Amberlace Health, a telehealth startup. She can turn a vague clinical requirement into a working prototype fast, and built the triage chatbot's three-hundred-case test set entirely by hand.
3 · THE HABIT
What did Bridget stop checking as the test set kept scoring well?
Tap to flip
ANSWER
Whether each new test category matched anything a real patient might actually say. She went from checking a medical reference for every category, to reusing the same phrasing, to rerunning the same three hundred cases and watching the number hold.
4 · THE SWITCH
What's the two-setting switch in this story?
Tap to flip
ANSWER
Either the synthetic test set is shaped by real case-mix data and stress-tested against real atypical cases, or a launch decision gets made on a number that never measured that risk. There's no partial version of "we checked."
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
The eval plan had Bridget write every test case herself, entirely from imagination, so no real patient data went near the product before consent existed. That made sense when nobody had thought to ask a partner clinic for the shape of its real cases. It stopped making sense the day that test set's score became the number leadership planned a launch around.
6 · THE NUMBER
Fill in the blank: accuracy on textbook-shaped cases was ______ percent. Accuracy on real atypical cases was ______ percent.
Tap to flip
ANSWER
96 and 61. Same tool, same launch date on the table. What changed was only whether the test case sounded like something a real patient would actually say.
7 · THE REPLAY
Same eval plan, real-shape data built in from the start. What changes?
Tap to flip
ANSWER
The sixty-one percent number shows up in week one instead of week four, before three weeks go into phrasing the wrong kind of test case, and before a nurse has to stumble onto the gap by accident.
8 · TRANSFER
Section four runs GUARD again on a different product. Which one, and what does the reduce step become?
Tap to flip
ANSWER
A port authority's cargo risk-screening tool, built solo by risk analyst Rashida Kwarteng on invented shipping manifests. Reduce: the same real-shape practice, shipment-volume and commodity-mix data from the port authority's statistics office, plus a stress test built from real seizure records.
Check yourself Score: 0 / 0
Multiple choice
1. Which two groups does the G step name in Bridget's story, and what gives them different starting points?
- A. Bridget's team, kept away from real patient data, and the patient, who has no way to know if the tool was ever tested against anyone like her.
- B. The clinical reviewer and leadership, who disagree about the launch date.
- C. Bridget and Constance, who disagree about whether the near miss mattered.
- D. The model and the checklist it was built from.
Show hint
Look for who built the tool and who's on the receiving end of what it decides.
Show answer
A. B, C, and D name real people or things in the story, but not the two groups the G step is built to separate: the one the privacy decision protects, and the one who inherits whatever gap that decision leaves behind.
Fill in the blank
2. The tool scored ______ percent on the textbook-shaped test cases Bridget wrote herself, and ______ percent on the forty real atypical cases the team tested afterward.
Show hint
Both numbers are in the chart, "Triage accuracy, textbook-shaped cases vs real atypical cases."
Show answer
96 and 61. Same tool. What changed was only whether the test case sounded like something a real patient would actually say.
True or false
3. True or false: the risk in this story was that the model's ninety-six percent accuracy score was mathematically wrong.
Show hint
Ask what the ninety-six percent was actually measured against.
Show answer
False. The score was correct for the set it was measured against. The risk was that the set never included a real pattern, an atypical heart attack presentation, so ninety-six percent measured how well the tool matched Bridget's assumptions, not how it would handle a real patient.
Short answer
4. What old decision does this answer take back, and why did it make sense when it was made?
Show hint
Look for the decision made before the chatbot existed, not the fix that came after the near miss.
Show answer
Model answer: "The eval plan had Bridget write every test case herself, entirely invented, so no real data went near the product before consent existed. That was the right call when nobody had thought to ask a partner clinic for just the shape of its real cases. It stopped being right the day that invented test set's score became the number a launch date got planned around."
Short answer, apply it yourself
5. Pick an AI tool you've seen tested only on made-up or example data. What's one real-world pattern that data probably never included?
Show hint
Think about who the maker imagined when they wrote the examples, and who might not match that picture.
Show answer
Model answer: "A customer-support chatbot tested only on neatly written example tickets. It probably never saw a message typed in a hurry, half in one language and half in another, from someone who's annoyed and skipping punctuation, exactly the kind of message that shows up most on a bad day."
Multiple choice
6. In this story, where would rebuilding the test set from a real-shape distribution matter least?
- A. A stuffy nose or a mild headache, where a made-up message and a real one read almost the same.
- B. A patient describing an atypical heart attack, without chest pain.
- C. Any case the clinical reviewer would use to approve launch.
- D. Any case a shadow-mode comparison would flag after launch.
Show hint
Ask which category the checklist and a real patient would describe almost identically anyway.
Show answer
A. B, C, and D are exactly where the real-shape data and the stress test earn their place. A is what this answer would leave alone: ordinary, textbook-shaped complaints where invented and real phrasing land in the same place.
