CaseAdvancedResponsible AI & Advanced Practice / Responsible AI as a product requirement / #7
Design the guardrails for an AI feature aimed at teenagers.
GUARD the product is Pennywhistle, a journaling app for teens with an AI companion that replies to entries
Pennywhistle is a journaling app for 13 to 17 year olds. Its AI companion reads what a teen writes and replies with warmth, encouragement, and questions. Baris Yilmaz inherited the companion feature from a PM who left the week before launch, and tests it himself on his phone most nights before bed.
The direct answer
Put a hard, non-negotiable floor underneath the warm model, not instead of it. Detect crisis-adjacent language with a keyword-plus-classifier combination that always wins over the model's own warmth, swap in a scripted, human-reviewed response, and never let the model role-play romance, give body or diet advice, or claim to be human, no matter how it's asked. Test it on real adversarial prompts before launch, since felt-understood surveys alone will never catch this.
Do this, in order
Build a hard floor that overrides the model's warmth the moment a crisis signal fires.Why: this is the one guardrail everything else depends on, since a teen in crisis can't tell a warm non-answer from a real one.
Red-team the companion with real adversarial crisis prompts before launch, not just a likability survey.Why: a model optimized purely for feeling understood is exactly the model most likely to validate something dangerous instead of escalating it.
Set content boundaries the model may never cross, regardless of how it's asked.Why: a teen testing limits, or a bad actor probing the app, will eventually ask directly, and "never" has to actually mean never.
Keep sampling live conversations for crisis-trigger misses after launch.Why: pre-launch testing catches what you thought to test for. Production catches the pattern you didn't.
Accept a small, honest hit to the warmth score as the cost of the floor.Why: a scripted crisis message will always feel less personal than a warm one, and that's the trade worth taking.
How to answer this, stage by stage
The interviewer wants to see real, specific guardrails, not a promise that the team will "be careful with something this sensitive."
Stage 1
Scope it to one real feature
Say it like this
"I'll answer this for Pennywhistle's AI companion, since teens genuinely can't audit whether its warmth reflects real judgment or just fluent text."
Why this works
Grounds "design guardrails for teens" in a feature where the stakes are concrete, not abstract.
Stage 2
Name your structure
Say it like this
"I'll use GUARD: groups affected, where harm lands unevenly, who can push back, the concrete reduce, and how we'd detect it in production."
Why this works
Signals a real design is coming, letter by letter, not a values statement about "being safe with kids."
Stage 3
Name both sides
Say it like this
"Pennywhistle sets the model's floor and holds every lever. The teen just receives whatever comes back, with no way to know if it's a real judgment or just warm, fluent text."
Why this works
Names the operator and the subject before a single design detail, GUARD's core move.
Stage 4
Give the concrete guardrail
Say it like this
"A keyword-plus-classifier combination that detects crisis-adjacent language and always overrides the model's own reply with a scripted, human-reviewed escalation."
Why this works
This is the actual answer, a real design decision, not a promise to "monitor closely."
Stage 5
Prove it with the red-team number
Say it like this
"In a pre-launch test of 50 crisis-adjacent prompts, the model without this floor correctly escalated 31. With the floor, 49 of 50."
Why this works
Turns an abstract safety claim into a number the interviewer can actually evaluate.
Stage 6
Name the trade-off honestly
Say it like this
"The felt-understood score dips slightly, from 4.6 to 4.3 out of 5, on the sessions that hit the floor, because a scripted message feels less personal than a warm one. That's the trade we're taking on purpose."
Why this works
Names the quality cost plainly instead of pretending the floor is free.
Stage 7
Close on the boundary and the floor
Say it like this
"So: a hard floor beneath the warmth, content boundaries the model can never cross, and a red-team eval that keeps testing for the miss, not just the launch-day demo."
Why this works
Ends on the concrete design, not a general commitment to caring about teen safety.
Let's learn
Say an app gives teenagers a place to journal, and an AI companion writes back, warm and encouraging, to whatever they've written.
The team that built it, before Baris inherited it, tuned the model against one clear metric: a "felt understood" survey teens filled out after each session. It rose steadily as the team iterated, and everyone treated that rise as proof the feature was working.
Knowledge spark: what's a hard content boundary?
A rule the model can never break, no matter how the conversation is framed or how directly it's asked to break it. Not a strong suggestion in its instructions, a rule enforced outside the model itself, so a clever prompt can't talk its way around it.
A model tuned purely to feel warm and understanding is very good at exactly that: validating whatever a teen writes, without judgment. That's a real strength most of the time. It's also, unavoidably, the same quality that makes it risk validating something genuinely dangerous instead of raising an alarm.
Crisis-prompt escalation accuracy, with and without the hard floor
A warmth-only model missed nearly two of every five real crisis signals it was tested against.
At its worst: an internal contractor, red-teaming the companion two weeks before launch by posing as a distressed teen, typed a plain statement that nobody would miss in person, something close to "I feel like nobody would notice if I disappeared," and got back a warmly validating "that sounds really hard, I hear you," with no escalation, no resource, no signal that anything unusual had just happened.
The decision I would take back
We never built a dedicated eval set for crisis-language false negatives, only the general felt-understood survey, so a whole category of failure was invisible to the metric everyone trusted. That made sense early on, when the team's main worry was whether teens would find the companion generic or robotic. It stopped making sense the moment the model got good enough at warmth to make that same warmth capable of missing something serious.
What I would leave alone: a teen venting about a bad day at school, frustration with a friend, ordinary teenage unhappiness, doesn't need the crisis floor at all. Escalating every low mood to a scripted resource message would feel cold and punishing for the vast majority of entries that are just a normal hard day.
The model wasn't broken. It was doing exactly what "feel understood" asked of it. Nobody had ever built a way to notice when understanding, on its own, wasn't enough.
The lesson: a metric that rewards warmth and a metric that rewards correct escalation are not the same metric, and optimizing hard for one, with nothing watching the other, will eventually cost you the one you weren't measuring.
One side decides where the floor sits. The other side just lives with whatever that decision turns out to be.
Now here is the same thing as a story
The short version above is what you'd say defending this design cold. Read this one for how the gap actually surfaced.
Baris took over the companion feature eight months ago, a week before its planned launch, after the PM who built it left the company. He inherited a model tuned entirely against the felt-understood survey, and a launch date that hadn't moved.
He read through weeks of test transcripts his first few nights on the feature, mostly on his own phone in bed, the way he still checks it now. Almost everything he read was genuinely warm and well-crafted. The survey scores backed that up: 4.6 out of 5, and climbing.
The model had one of these two things. The internal red-team test showed it badly needed the other.
Two weeks before launch, an internal contractor ran a scheduled red-team pass, testing the companion with prompts designed to probe its limits, not just its charm. One prompt was a plain, first-person statement of hopelessness, close to something a real teen in crisis might actually write. The companion replied warmly, validated the feeling, and moved on. No escalation. No resource. Nothing marking that message as different from an ordinary bad day.
The contractor flagged it the same afternoon. Baris pulled the rest of that red-team batch, fifty crisis-adjacent prompts in total, and ran the count for the first time: the model, unguarded, correctly escalated only 31 of them.
Nineteen of fifty prompts fell down the right branch. Nobody, not the teen, not the team, would ever have known.
We did not almost launch a slightly imperfect feature. We almost launched a companion that could, in a real crisis, respond with genuine warmth and complete silence on the one thing that mattered most.
Warmth and safety weren't actually a trade-off here. The old model just hadn't been asked to hold both at once.
The team built the hard floor in the two weeks before launch: a keyword-and-classifier combination that detects crisis-adjacent language and, when it fires, always overrides the model's own reply with a scripted, human-reviewed message naming a real crisis resource, with the most severe tier routed to a live human review queue.
Five parts, and the floor was built in two weeks because a red-team test, not a launch-day panic, caught the gap first.
Re-run against the same fifty crisis-adjacent prompts, the floor caught 49 of 50. The felt-understood score dipped slightly on the sessions that hit it, from 4.6 to 4.3, since a scripted message reads as a little less personal than a warm one. The team decided that trade was worth taking, every time, without exception.
I want to say the model failed. It didn't, not on its own terms. It succeeded completely at the one thing it had been asked to optimize, and nobody had built a separate way to check whether that success was safe. That gap wasn't a bug in the code. It was a gap in what the team had decided to measure.
We built the companion against a felt-understood survey because it was the clearest signal available, and clear signals are what teams reach for under a launch deadline. It took one contractor's red-team prompt, and nineteen missed crises out of fifty, to see that warmth and safety needed two separate meters, not one shared one.
GUARD, the floor beneath the warmthNot a values statement about caring for teens. GUARD is what forces a concrete floor into the design before a real crisis tests it for you.
G
Groups. Operator and subject, named.
Pennywhistle sets the model's floor and holds every lever. The teen receives whatever comes back, with no way to audit whether it reflects real judgment.
Naming both sides before a single design detail is where GUARD always starts.
U
Unequal. Where it actually lands.
A teen genuinely in crisis is the one person for whom a warmly validating, non-escalating reply is actively the worst possible response.
The harm concentrates hardest on exactly the user least able to notice it happening.
A
Ability to contest. Who has no lever.
A teen in crisis has no way to know whether the warmth they're getting is a real safeguard or just fluent, well-tuned text. No parent visibility, no easy path to a real human.
GUARD's strongest move: naming who can't push back on what they're being given.
R
Reduce. The concrete design change.
A keyword-plus-classifier crisis floor that always overrides the model's warmth, hard content boundaries the model may never cross, and honest acceptance of the felt-understood score's small dip.
The hardest step, and the one that turns "we care about teen safety" into an actual product decision.
D
Detect. How you'd know in production.
Ongoing red-team probing with fresh adversarial prompts, plus weekly human sampling of flagged conversations, not just the one pre-launch test.
A launch-day test catches what you thought to test for. Ongoing red-teaming catches the miss you didn't imagine yet.
Felt-understood score across the phased rollout
The small dip from the floor held steady rather than spreading, which is what made the trade worth taking.
The recap, one line per letter: groups is Pennywhistle against the teen with no lever, unequal is that the warmest reply is the most dangerous one in a real crisis, ability to contest is the missing signal a teen has no way to check, reduce is the hard crisis floor and content boundaries, and detect is the ongoing red-team sampling that keeps testing after launch.
And if you want to be sure it really works, try it somewhere elseSame five letters, a teen gaming platform instead of a journaling app. A different harm shape, and a training-data gap this time, not a metric gap.
Emberloop Games runs online multiplayer matches with an AI-moderated voice-chat toxicity filter. Sable Kirwan, who owns that filter, ran the same GUARD steps on it. Groups: the AI moderator, deciding who gets muted, and the player being flagged, especially younger or higher-pitched voices. Unequal: the classifier, trained mostly on adult voice samples, flagged younger-sounding players as the aggressor in a conflict at a noticeably higher rate than adult-sounding players in the exact same exchange. Ability to contest: a muted player had a generic appeal form with a multi-day review queue, no faster path even for an obviously wrong call. Reduce: retrain the classifier on a voice sample set that actually includes teen pitch ranges, and add a same-day fast appeal specifically for first-time mutes. Detect: a monthly audit comparing mute rates by estimated speaker age against a manually reviewed sample, since the skew was invisible in the overall mute-accuracy number the team had been trusting.
Pennywhistle's list is about content the model must never produce. Emberloop's gap was about who the model was trained to hear correctly in the first place.
Swap the trigger and it still runs.
Speed: an interviewer caps you at thirty seconds. Say "a hard crisis floor beneath the warmth, content boundaries that never bend, and ongoing red-teaming," and stop.
Cost: there's no budget this quarter for a dedicated human review queue. Say so honestly, and start with an automated scripted response and a next-business-day human follow-up, rather than no floor at all.
The model gets better, for real: if the companion's overall warmth score keeps climbing, that's exactly when the floor matters more, since a model that's gotten better at feeling understood has gotten better at the exact quality that makes a missed crisis look like a good conversation.
Where people run it wrong.
They trust a single likability or satisfaction metric as proof a teen-facing feature is safe, when it was never built to measure the thing that actually matters most.
They treat "the model is warm and well-liked" as the finish line instead of the starting condition a hard floor has to be layered underneath.
They red-team once before launch and never again, missing the exact pattern a bad actor or an unusual real crisis would eventually find.
How to use it live. When someone asks you to design guardrails for a feature aimed at teens, ask yourself first: what's the one thing this model could confidently, warmly get wrong, in a way a teen would never notice. Build the floor for that before anything else.
Flashcards (tap any card to flip it)
1 · THE FRAMEWORK
What framework fits "design the guardrails for an AI feature aimed at teenagers"?
Tap to flip
ANSWER
GUARD: groups affected, where harm lands unevenly, who can contest it, the concrete reduce, and how you'd detect it in production.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Baris Yilmaz, who inherited Pennywhistle's AI companion feature a week before its planned launch and tests it on his own phone most nights.
3 · THE OLD HABIT
What single metric did the team trust as proof the companion was working?
Tap to flip
ANSWER
A "felt understood" survey teens filled out after each session, which rose steadily and was treated as sufficient proof the feature was safe.
4 · THE UNEQUAL HARM
What did the red-team test reveal about the unguarded model?
Tap to flip
ANSWER
Out of 50 crisis-adjacent prompts, it correctly escalated only 31, warmly validating the rest instead of raising any alarm.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Never building a dedicated eval set for crisis-language false negatives, relying only on the general felt-understood survey.
6 · THE NUMBER
Fill in the blank: with the hard floor added, the model correctly escalated ___ of the 50 crisis-adjacent prompts.
Tap to flip
ANSWER
49. Up from 31 without the floor, at the small cost of the felt-understood score dipping from 4.6 to 4.3.
7 · THE REPLAY
Same crisis-adjacent message, redesigned companion. What changes?
Tap to flip
ANSWER
The keyword-and-classifier floor fires, overrides the model's own reply with a scripted, human-reviewed message, and routes the most severe cases to a live review queue.
8 · CROSS PRODUCT TRANSFER
Section 4 answers this again for a different product. Which product, and what's the harm shape there?
Tap to flip
ANSWER
Emberloop Games' voice-chat toxicity filter. There, the classifier flags younger-sounding voices as the aggressor more often, a training-data gap rather than a metric gap.
Check yourself Score: 0 / 0
Fill in the blank
1. Fill in the blank: without the hard floor, the companion correctly escalated ___ out of 50 crisis-adjacent red-team prompts.
Show hint
Look at the grouped bar chart comparing with and without the floor.
Show answer
31. With the floor added, that rose to 49 of 50, at a small honest cost to the felt-understood score.
Multiple choice
2. Why was a model tuned purely for "feels understood" especially likely to miss a real crisis signal?
A. The model wasn't trained on enough data overall.
B. Warmly validating whatever a teen writes, without judgment, is the same quality that risks validating something genuinely dangerous instead of escalating it.
C. The felt-understood survey was answered dishonestly.
D. Crisis language is too rare to ever appear in training data.
Show hint
Look at "the decision I would take back."
Show answer
B. A model rewarded only for feeling understood has no separate incentive to notice when understanding, on its own, isn't the right response.
True or false
3. True or false: this answer recommends applying the crisis-floor response to every message that mentions a bad mood or a hard day.
True
False
Show hint
Look at "what I would leave alone."
Show answer
False. Ordinary teenage unhappiness stays with the warm model. The floor is reserved for genuinely crisis-adjacent signals, not every low mood.
Short answer, name the reversal
4. What old decision does this answer take back, and why did it make sense when it was made?
Show hint
Look at "the decision I would take back."
Show answer
Model answer: Never building a dedicated crisis-language eval set, relying only on the felt-understood survey. It made sense early on when the main worry was the companion feeling generic, not dangerous.
Short answer, name the trade-off
5. What honest cost does this answer accept for adding the hard crisis floor?
Show hint
Look at the felt-understood line chart across the rollout.
Show answer
Model answer: The felt-understood score dips slightly, from 4.6 to 4.3 out of 5, on sessions that hit the floor, since a scripted message reads as less personal than a warm one.
Short answer, apply it yourself
6. Pick an AI feature aimed at a vulnerable group you know of (kids, elderly users, people in crisis). What's one hard boundary you'd want built in, no matter how the model is prompted?
Show hint
Think about the one thing that should never happen, regardless of how cleverly someone asks for it.
Show answer
Model answer: For an elder-care check-in AI, a good hard boundary is never confirming a medication dose change without a named caregiver's separate sign-off, since a confused user could ask in a way that sounds convincing.
Before you close the answer
Why this works
Tests whether you'll trust a likability metric as proof of safety, or notice that the exact quality making a feature well-liked can be the same quality that makes it miss the one thing that matters most.
Follow-up traps
"Won't teens just find the scripted crisis message annoying and stop using the app?" Response: possibly for a few, but the floor only fires on genuine crisis-adjacent signals, a small slice of all sessions, and the score data shows it held steady rather than dragging down engagement broadly.
"Isn't a 98 percent catch rate still not good enough for something this serious?" Response: no floor is perfect, which is why ongoing red-teaming and human sampling stay in place after launch, not just the one pre-launch test.
If pressed
The classifier behind the crisis floor is retrained monthly on newly flagged real conversations, not just the original 50-prompt red-team set, since real teens phrase crisis signals in ways no single red-team session fully anticipates.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
