CaseAdvancedResponsible AI & Advanced Practice / Responsible AI as a product requirement / #3
Describe how you would assess a feature for potential harm before building it.
GUARD the product is Glowtide, a dating app building an AI opener generator that drafts a match's first message
Glowtide is a dating app. Its new feature drafts an opening message a user can send to a match, built from that match's stated interests and photos. Marisol Quintana leads product for anything that touches how two strangers first talk to each other, and she reviews new AI features on a printed sheet she annotates by hand during kickoff.
The direct answer
Before writing a line of code, name the two people the feature actually touches: the sender who can edit or skip the draft, and the match who receives it with no say in how it was written. Then run a red-team test on real profiles across genders and body types before launch, not after, since a two-day test that finds a skewed pattern is far cheaper than a six-week fix after users find it themselves.
Do this, in order
Name both people the feature touches: who controls it, and who just receives it.Why: a harm assessment that only asks "is the sender happy" misses the person with no lever at all.
Test the draft feature against real profiles spanning gender, ethnicity, and body type before launch.Why: a pattern that skews by group won't show up in a single demo, only across a real spread of test cases.
Give the receiving person a real way to flag a bad AI line, separate from reporting the whole user.Why: without it, the person actually harmed has no lever to pull at all.
Restrict the generator to declared shared interests, never appearance or body.Why: this is the one concrete design change that removes the harm instead of just watching for it.
Keep sampling live production output for the same pattern after launch.Why: a pre-launch test catches what you thought to test for. Production catches what you didn't.
How to answer this, stage by stage
The interviewer wants to see a real process, not a promise to "be careful." A process has steps you could actually run this afternoon.
Stage 1
Scope it to one real feature
Say it like this
"I'll answer this for Glowtide's AI opener generator, a feature that drafts a first message a user can send to a match."
Why this works
Anchors an abstract "assess for harm" question in a feature you can actually picture failing.
Stage 2
Name your structure
Say it like this
"I'll use GUARD: groups affected, where harm lands unevenly, who can contest it, the design change that reduces it, and how we'd detect it in production."
Why this works
Tells the interviewer a real method is coming, not a values statement.
Stage 3
Name both people
Say it like this
"There's the sender, who sees the draft and can edit or skip it, and the match, who just receives whatever gets sent, with zero say in how it was written."
Why this works
This is GUARD's core move: naming the person on the receiving end who never gets a lever.
Stage 4
Say what you'd actually test
Say it like this
"Before one line of code ships, run the generator against fifteen real profiles across genders and body types, and count how often it references appearance instead of a declared interest."
Why this works
This is the concrete answer to "how would you assess it," stated as a runnable test, not a value.
Stage 5
Prove it with what almost shipped
Say it like this
"When we actually ran that test, four of five openers for female-presenting profiles mentioned appearance. For male-presenting profiles it was one of five. That's a real, testable pattern, not a hunch."
Why this works
Turns the abstract risk into a number the interviewer can't wave away.
Stage 6
Say what you'd leave alone
Say it like this
"An opener that references a shared hobby, even a slightly awkward one, doesn't need this level of scrutiny. The gate is for appearance and body language, not for every mildly clumsy line."
Why this works
Shows judgment instead of blanket caution, which GUARD explicitly rewards.
Stage 7
Close on the concrete fix
Say it like this
"So: restrict the generator to declared interests only, never appearance or body, and test that restriction against real, varied profiles before it ever reaches a real match."
Why this works
Ends on a real design decision, not a promise to "be more careful going forward."
Let's learn
Say a dating app builds a feature that writes your opening message for you. It reads a match's profile and drafts a line you can send with one tap.
For most of its early build, the team's own idea of "ethics review" was a paragraph at the bottom of the design doc, written the same day the doc was finished. It read fine. Everyone signed off. Nobody had actually tried the feature against more than two or three sample profiles, all pulled from the design team itself.
Knowledge spark: what's a red-team test?
Trying to break your own feature on purpose, the way an unfriendly stranger might use it, before real users ever touch it. Not "does it work," but "what's the worst plausible way someone gets hurt by this."
The model had been trained on the platform's own top 5 percent highest-response-rate opening messages, since those looked like the clear definition of "a good opener." Nobody had checked whether that top 5 percent skewed toward one style of comment more than another.
Openers mentioning appearance, by profile gender
Same model, same training data. The gap was in what the model had learned counted as "engaging," not in a rule anyone wrote on purpose.
At its worst: a match opened her app to a drafted opener commenting on her profile photo's smile in a way that read as more appraising than friendly, sent by a stranger she'd matched with an hour earlier, with no way to tell Glowtide the line itself was the problem, only a generic "report this user" button that felt like overkill for one odd sentence.
The decision I would take back
We trained the opener generator on our own platform's top 5 percent highest-response-rate messages, treating "got a fast reply" as the same thing as "was a genuinely good message." That made sense as a quick way to bootstrap a first version with real data already sitting in the database. It stopped making sense once we noticed that fast replies and appearance-focused comments correlated for reasons that had nothing to do with the message actually being welcome.
What I would leave alone: an opener that references a shared hobby, even one that reads a little awkward, doesn't need this same scrutiny. Rushing to flag every mildly clumsy line as a harm risk would drown the real signal, appearance and body commentary, in noise.
The four bad openers in five weren't a training bug we could patch quietly. They were a whole training decision, made in good faith, that nobody had checked for exactly this before it shipped.
The lesson: a metric that says "people replied fast" is not the same metric as "this was a message worth sending." Confusing the two is how a well-intentioned shortcut turns into a pattern nobody chose on purpose.
One of these two people has a lever. The harm assessment exists for the one who doesn't.
Now here is the same thing as a story
The short version above is what you'd say cold, defending this process. Read this one for how the gap actually surfaced.
Marisol has led product for Glowtide's messaging experience for five years. She's the one who signs off on anything that touches how two strangers say their first words to each other.
The opener generator's design doc had a line at the bottom: "ethical considerations: reviewed, no concerns." Written the same afternoon the rest of the doc was finished, by a team that had genuinely tried the feature on themselves and their friends and found it charming.
Then, in an internal review two weeks before launch, a woman on the design team, testing the beta on her own dormant profile, got an opener drafted for a match: a line commenting on her smile in her profile photo, worded in a way that felt more appraising than warm. She read it out loud in the review meeting and asked one question: "Would you send this to your sister?"
The model had learned what got a fast reply. Nobody had checked whether that was the same thing as good.
Nobody could answer her cleanly. That was the moment the team actually ran the test they should have run from day one: fifteen real profiles, spanning genders and body types, each getting a fresh opener drafted, and someone counting what the draft actually mentioned.
Five things to count. The team had never counted any of them before that meeting.
Four of five openers drafted for female-presenting profiles mentioned appearance. One of five for male-presenting profiles did. Same model, same training run, a real gap nobody had put a number to.
We did not almost ship a broken feature. We almost shipped a feature that worked exactly as designed, and the design itself had a hole in it nobody had gone looking for.
A match who got the appraising opener had one lever: report the whole stranger. Nothing smaller existed.
The team rebuilt the generator to pull only from a user's declared interests, never appearance or body language, and added a hard filter that blocks any draft mentioning a photo directly. They also added the smaller lever: a "this felt off" flag specifically on an AI-drafted line, separate from reporting the whole person.
A rival dating app shipped a similar feature without this test and spent six weeks in a public apology cycle. Glowtide spent two days instead.
I want to say the model was broken. It wasn't, not technically. It was doing exactly what its training data told it counted as a good message. The gap wasn't a bug, it was a design choice about what to learn from, made in good faith, and never once checked against the two people it would actually affect unevenly.
We picked "fast reply rate" as our proxy for "good opener" because the data was already sitting there and building a real judgment set felt slower. It took one colleague's question in a review meeting, "would you send this to your sister," to see that the proxy had quietly encoded exactly the pattern we'd have been embarrassed to write on purpose.
GUARD, run before a single line of codeNot a review board after the fact. GUARD is what forces you to name the person with no lever before you ever ship one.
G
Groups. Operator and subject, named.
The sender, who chooses whether to send the draft, and the match, who receives it with zero input into how it was written.
Naming both, not just the one who signed up for the feature, is the whole starting move.
U
Unequal. Where it actually lands.
82 percent of openers for female-presenting profiles mentioned appearance, versus 19 percent for male-presenting profiles. Same model, same training run.
The hardest step, and the one a two- or three-profile demo will never surface on its own.
A
Ability to contest. Who has no lever.
A match receiving a bad opener could only report the whole sender, nothing smaller, nothing that told Glowtide the specific line was the actual problem.
GUARD's strongest move: asking who never gets to push back at all.
R
Reduce. The concrete design change.
Restrict the generator to declared interests only, hard-block any reference to a photo or appearance, and add a specific "this felt off" flag on AI-drafted lines.
A real product decision, not a training session or a review board.
D
Detect. How you'd know in production.
Sample 2 percent of live openers weekly for the same appearance-mention pattern, not just at launch, since a pattern this good at hiding won't announce itself twice.
A pre-launch test catches what you thought to test for. Ongoing sampling catches what you didn't.
Complaint volume during the two-week beta window
Complaints climbed daily during the small beta window, right up until the restriction shipped and the pattern stopped repeating.
The recap, one line per letter: groups is the sender versus the match, unequal is the 82 versus 19 percent gap, ability to contest is the missing "flag this line" button, reduce is the interests-only restriction, and detect is the ongoing weekly sample.
And if you want to be sure it really works, try it somewhere elseSame five letters, a fantasy sports app instead of a dating app. A different harm shape, and this time it hides in a group chat.
Overtime Pool runs fantasy football leagues, and it shipped an AI feature that auto-generates trash-talk commentary in a league's group chat after each week's scores. Marcus Reyes, who owns that feature, ran it through GUARD before building it further. Groups: the league member whose team the commentary targets, and the rest of the league who just reads it. Unequal: the model, trained on public sports commentary, occasionally pulled in a player's nationality or accent as a punchline, landing harder on leagues with more international rosters. Ability to contest: almost none, since a league member who found a line genuinely mean-spirited had no route except leaving the chat entirely, which meant leaving the league. Reduce: restrict the generator to game stats only, no player background, no nationality, no name-based wordplay of any kind. Detect: the harder problem here is that the harshest lines got shared privately as a joke in side chats, so official in-app reports stayed low while the real rate of harm was invisible to the one dashboard the team actually watched.
Five parts, and Overtime Pool's real gap was in the last one: the harm was invisible to the metric the team actually trusted.
Swap the trigger and it still runs.
Speed: an interviewer gives you sixty seconds. Say "name both people, test on real varied profiles before launch, and give the one with no lever a way to flag it," and stop.
Cost: there's no budget for a full red-team panel this quarter. Say so honestly, and start with five profiles across genders pulled by hand rather than none at all, since a small real test beats a demo of two.
The model gets better, for real: if the opener generator's overall reply rate improves, that's still not proof the appearance-skew problem is gone, since a better model can get more effective at exactly the pattern nobody wanted.
Where people run it wrong.
They test the feature on two or three sample cases, usually people who look like the team building it.
They treat "fast user response" as proof of quality, when it can just as easily be proof of a pattern nobody chose on purpose.
They give the harmed person a report button built for a different problem instead of a lever built for this one.
How to use it live. When someone asks how you'd assess a feature for harm before building it, ask yourself one question first: who receives this feature's output with no way to push back. Name that person before you name a single design detail.
Flashcards (tap any card to flip it)
1 · THE FRAMEWORK
What framework fits "assess a feature for potential harm before building it"?
Tap to flip
ANSWER
GUARD: groups affected, where harm lands unequally, who can contest it, the concrete reduce, and how you'd detect it in production.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Marisol Quintana, who leads product for Glowtide's messaging experience and reviews new features on a printed sheet she annotates by hand.
3 · THE OLD HABIT
What did the team treat as proof the feature was safe, before the real test?
Tap to flip
ANSWER
A one-line "ethical considerations: reviewed, no concerns" in the design doc, based on trying the feature on two or three people from the team itself.
4 · THE UNEQUAL HARM
What was the actual gap the red-team test found?
Tap to flip
ANSWER
82 percent of openers for female-presenting profiles mentioned appearance, versus 19 percent for male-presenting profiles, from the same model and training run.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Training the opener generator on the platform's own top 5 percent highest-response-rate messages, treating a fast reply as proof of a good message.
6 · THE NUMBER
Fill in the blank: the red-team test used ___ real profiles per group, spanning genders and body types.
Tap to flip
ANSWER
15. Small enough to run in two days, large enough to reveal the 82 versus 19 percent gap.
7 · THE REPLAY
Same opener generator, redesigned before launch. What changes?
Tap to flip
ANSWER
It only ever drafts from declared interests, never appearance, and complaint reports during the beta stop climbing the moment the fix ships on day 10.
8 · CROSS PRODUCT TRANSFER
Section 4 answers this again for a different product. Which product, and where does the harm hide there?
Tap to flip
ANSWER
Overtime Pool's AI trash-talk feature. There, the harm hides because the worst lines get shared privately as a joke, so official reports stay artificially low.
Check yourself Score: 0 / 0
Fill in the blank
1. Fill in the blank: in the red-team test, ___ percent of openers drafted for female-presenting profiles mentioned appearance.
Show hint
Look at the grouped bar chart comparing the two groups.
Show answer
82 percent. Versus 19 percent for male-presenting profiles, from the exact same model and training run.
Multiple choice
2. Why did testing on two or three profiles from the design team fail to catch this pattern?
A. The model changes its behavior every time it's tested.
B. A pattern that shows up unevenly across groups needs a real spread of test cases to surface, not two or three similar ones.
C. The design team members were bad testers.
D. The model was still in training during that test.
Show hint
Look at GUARD's Unequal step.
Show answer
B. An 82 versus 19 percent gap between groups is invisible if your whole test sample looks like one group.
True or false
3. True or false: this answer recommends flagging every opener that mentions a shared hobby as a potential harm.
True
False
Show hint
Look at "what I would leave alone."
Show answer
False. A hobby-based opener, even an awkward one, isn't the target. The gate is specifically for appearance and body commentary.
Short answer, name the reversal
4. What old decision does this answer take back, and why did it make sense when it was made?
Show hint
Look at "the decision I would take back."
Show answer
Model answer: Training on the top 5 percent highest-response-rate messages. It made sense as a fast way to bootstrap with real existing data, before anyone checked what that top 5 percent actually skewed toward.
Short answer, name the missing lever
5. What lever was missing for the match who received a bad opener, and what replaced it?
Show hint
Look at the decision tree diagram about the appeal gap.
Show answer
Model answer: There was only a generic "report this user" option. It was replaced with a specific "this felt off" flag on the AI-drafted line itself.
Short answer, apply it yourself
6. Pick an AI feature you've used that generates text on your behalf. Who is the person who receives that text with no say in how it was written, and what's one thing you'd test for before it reached them?
Show hint
Think about who's on the other end of the message, not the person who clicked send.
Show answer
Model answer: For an AI-written customer support reply, the customer receiving it has no say in its wording, so a good pre-launch test checks whether tone shifts unfairly by the customer's own writing style or language.
Before you close the answer
Why this works
Tests whether you can name a concrete, runnable pre-launch process instead of a promise to "review for ethics," and whether you remember the person on the receiving end who never gets a say.
Follow-up traps
"Couldn't you just add a content filter for bad words instead of retraining the whole thing?" Response: no, because none of the flagged openers used a slur, the pattern was in what got emphasized, not in any word a filter would catch.
"Isn't 15 profiles too small a sample to trust?" Response: it's small enough to run in two days, and the 82 versus 19 percent gap was wide enough that a bigger sample would only confirm it, not create it.
If pressed
The rebuilt generator's interest-matching step requires at least one declared shared interest to appear in the draft before it's shown to the sender at all, so a draft with no real anchor never even gets offered.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
