ConceptAdvancedResponsible AI & Advanced Practice / Responsible AI as a product requirement / #14
How do you decide which harms are in scope for your product to prevent?
ORDER the product is Larkspur Coach, Larkspur Financial's budgeting and spending-advice app
Larkspur Financial makes Larkspur Coach, an app that reviews a user's spending and suggests where to cut back. Adaeze Nnamdi is the product manager who decides which risks the team actually builds against, out of every risk anyone has ever raised.
The direct answer
Rank candidate harms by how hard they are to undo, not by how often they come up in a meeting or how recently someone mentioned them. A harm that's rare but compounds and can't be reversed, like a controlling partner using shared account visibility to monitor someone's spending, outranks a harm that's common but recoverable, like an occasionally clumsy savings tip. Scope follows that order, not the loudest voice in the room.
Do this, in order
Rank every candidate harm by how hard it is to undo, before anything else.Why: reversibility, not frequency, is what decides how much damage a harm actually does over time.
Name the one outcome every candidate harm is actually competing to protect.Why: without a shared outcome, ranking harms becomes a debate about whose example feels scariest.
Map what has to exist before you can even detect a harm, like defining account visibility as a trackable setting.Why: you can't rank or fix a harm your product doesn't have a concept for yet.
Pull cheap evidence, like existing support tickets, before committing full engineering scope.Why: a pattern already sitting in your own data is far cheaper to find than one you have to go looking for blind.
Ship the hardest-to-undo fix first, even if it protects the fewest people.Why: you can improve a common, recoverable harm later. You can't take back the harm nobody could reverse.
Keep low-severity, easily fixed harms on the list, just ranked honestly last.Why: they still matter. They just don't get to skip the line because someone raised them most recently.
How to answer this, stage by stage
Seven moves. This one leans on naming reversibility early, since that's the whole argument.
Stage 1
Ground it in one real product
Say it like this
"I'll answer this for a budgeting app that gives spending advice, since 'which harms are in scope' means something concrete there: an open list of things that could go wrong, and a limited roadmap to actually address them."
Why this works
Keeps the answer from becoming an abstract policy discussion.
Stage 2
Say your structure out loud
Say it like this
"I'll use ORDER. The outcome everything competes to protect, reversibility, what depends on what, what evidence I can get cheaply, then the actual rank."
Why this works
Shows a repeatable method, not a gut call dressed up as a framework.
Stage 3
Name the outcome
Say it like this
"Every candidate harm is competing to protect the same thing: whether a user's own financial decisions stay their own, and aren't quietly turned into a tool against them."
Why this works
Gives the ranking a shared target, so it isn't just a list of scary anecdotes.
Stage 4
Rank by reversibility, out loud
Say it like this
"A controlling partner using shared account visibility to monitor someone's spending ranks first, even though it's rare, because it compounds and the person harmed often can't undo it. A clumsy savings tip ranks near the bottom, because it costs a moment of annoyance and nothing more."
Why this works
This is the actual answer to the question, said as a real ranking with real examples.
Stage 5
Name what has to exist first
Say it like this
"Before we can detect the abuse case at all, account visibility has to become something our product actually tracks and can gate. Right now it isn't even a setting."
Why this works
Shows the ranking isn't just a wish list, it accounts for what actually has to be built to act on it.
Stage 6
Say what cheap evidence you'd pull
Say it like this
"I'd pull our own support tickets for phrases like 'my partner sees this' before building anything, since that pattern may already be sitting in data we have."
Why this works
Demonstrates you'd validate the ranking cheaply before committing a roadmap to it.
Stage 7
Close on the one line
Say it like this
"So: rank by how hard a harm is to undo, not how often it comes up. The rare, irreversible one ships first."
Why this works
Leaves the interviewer with the decision rule itself, not a list of examples.
Let's learn
Say a budgeting app reviews a user's spending and suggests specific places to cut back, right down to naming a subscription or a category by name.
Before this team had any real process, "which harms matter" got decided the way most things get decided under deadline pressure: whichever concern someone raised most recently, or most loudly, in a planning meeting became the one the team addressed that quarter.
Knowledge spark: what does "in scope" actually mean here?
It means a harm the team has committed to detect and reduce on purpose, with an owner and a fix, rather than something that might get addressed if anyone happens to remember it later.
For a while, this seemed to work fine. The team fixed a tone problem where the coach's copy read as judgmental. They fixed a bug where a subscription got miscategorized. Nothing felt urgent enough to argue about.
Here is the turn. The volume of complaints was never a good measure of the actual damage being done. A harm that happens rarely but compounds, and that the person harmed may not be able to escape or undo, doesn't generate ten loud complaints in a planning meeting. It generates silence, right up until it doesn't.
Estimated users affected per year, by harm category
By raw frequency, the abuse case ranks dead last. It ships first anyway, because frequency was never the number that mattered.
At its worst: a controlling partner uses the shared account view Larkspur built for convenience to track exactly where their partner's money goes, tightening control over time, and the person being watched has no idea the app is the reason their spending is suddenly so visible to someone else.
The decision I would take back
We decided, early on, that any linked account holder could see full transaction detail by default, since that seemed like the obviously helpful choice for couples budgeting together. That made sense when we imagined every linked account as two people cooperating. It stopped making sense the moment we had to imagine a linked account as two people who are not cooperating at all.
What I would leave alone: the occasional clumsy or slightly judgmental savings tip doesn't need the same scrutiny. It's real, it's worth fixing eventually through better copy, but it costs a moment of annoyance, not a compounding, hard-to-escape harm.
The harm that never showed up loudly in a meeting was the one that mattered most. Silence isn't the same thing as safety.
The lesson: deciding what's in scope by who spoke up most recently quietly ranks harms by how comfortable people are complaining about them, not by how much damage they actually do.
Now here is the same thing as a story
The short version above is the ranking rule. Read this one for how Adaeze actually built it.
Adaeze Nnamdi has managed Larkspur Coach for two years. Every planning cycle, she kept a running list of "things someone flagged," and every cycle, the list got triaged the same informal way: whatever felt most pressing that week.
The second milestone is the one that changed how every harm afterward got ranked.
Then a support rep flagged something odd: a user had messaged asking, almost in passing, whether their partner could "turn off" seeing their transactions. The rep almost closed the ticket as a simple settings question. Something about the phrasing made her escalate it instead.
It turned out to be exactly what it sounded like. A user's partner had been using the account's shared visibility feature, built for couples budgeting together, to monitor and question every purchase. Nothing had gone catastrophically wrong yet. But the pattern was there, quietly, and nobody had ever asked whether "shared visibility" needed a second use case in mind.
Both of these were technically on the same brainstorm list. Only one of them belonged there with equal weight.
Adaeze pulled together every harm anyone had ever raised, six months of scattered Slack messages and meeting notes, and mapped them for the first time on the two axes that actually mattered.
The harm in the top left corner was the one the loudest-voice method would have ranked last, every single time.
We were not choosing to ignore the abuse case on purpose. We were ranking by which harm got mentioned most, and mentioned most just isn't the same thing as matters most.
She built a simple triage rule going forward: any new candidate harm gets sorted by how hard it would be to undo, not by how it arrived or how many people brought it up.
The abuse case lands in the leftmost branch every time it's re-evaluated, regardless of how it's reported.
Four pieces. The old process only ever had a rough version of the first one, and even that was never written down.
The visibility fix shipped first: transaction-level detail is now hidden from a secondary account holder by default, with the primary holder choosing explicitly what to share. The essential-expense protection shipped second: rent, medication, and childcare categories now require extra confirmation before the coach suggests cutting them. The tone problem, real but far more recoverable, stayed on the list, just honestly ranked behind both.
What I would tell myself, a year earlier: the loudest harm in the room was never a measure of the worst one. It was just a measure of who felt safest speaking up about it.
ORDER, the rank Adaeze actually builtNot "which harms feel scariest." ORDER forces you to rank by what can't be undone, not by what gets mentioned most.
O
Outcome. What every candidate is competing to protect.
Whether a user's own financial decisions stay their own, and never get quietly turned into a tool against them.
Gives the whole ranking a shared target instead of a debate over anecdotes.
R
Reversibility. The whole argument.
Abuse via shared account visibility ranks first, since it compounds and the person harmed often can't undo it, even though it affects the fewest people by raw count.
Flips don't flip back; this is why reversibility beats frequency as the ranking rule.
D
Dependency. What unblocks what.
Account visibility had to become a real, trackable product setting before the team could detect or gate misuse of it at all.
Some ranking order is forced by what has to exist first, not just by judgment.
E
Evidence. What you could learn cheaply.
A search of existing support tickets for phrases like "my partner sees" surfaced the pattern before any new research had to be commissioned.
Cheap evidence you already have beats an expensive study you'd have to go build.
R
Rank. State the order, defend the top pick.
Visibility abuse first, essential-expense cuts second, translation quality third, tone and copy last, all four kept in scope, none dropped.
A ranking with a defensible top pick is a decision. A list with no order is just a list.
Financial-harm-flagged support tickets per month, as fixes ship in ranked order
Each ranked fix, shipped in reversibility order, cut into the total a little more than the last. Nothing was shipped in raw frequency order.
The recap, one line per letter: outcome is keeping financial decisions genuinely a user's own, reversibility is ranking the rare abuse case above the common tone issue, dependency is building account visibility as a real setting first, evidence is mining existing tickets before new research, and rank is a defended order with all four harms still in scope.
And if you want to be sure it really works, try it somewhere elseSame five letters, a home security camera app instead of a budgeting app. A completely different field.
Ferrowatch Home makes SentryView, a feature that flags "suspicious activity" from a household's security cameras and alerts whoever has app access. Conrad Aabel owns the decision about which harms SentryView is actually built to prevent.
Mapped onto ORDER: outcome is that a household's own sense of safety and privacy stays intact, for everyone the cameras can see, not just whoever holds the app. Reversibility ranks a harm where a landlord or ex-partner retains hidden camera access after they should have lost it far above a harm where the AI mislabels a delivery driver as "suspicious," since the first can enable ongoing surveillance nobody consents to, and the second is corrected the next time someone glances at the same clip. Dependency is that access needs an expiration and a visible access log before revocation can even be checked. Evidence is pulling existing support tickets for "my ex still gets alerts" before building a bigger access-review feature. Rank places access-revocation enforcement first, false "suspicious activity" tagging second, cosmetic UI complaints last.
Same triage logic. A camera feature needs it for exactly the same reason a budgeting app does.
Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "rank by how hard a harm is to undo, not how often it's raised, and the rare irreversible one ships first," and stop.
Cost: there's no engineering time this quarter for a full access-control rebuild. Say so honestly, and start with a manual audit flagging any account access older than six months for a required re-confirmation.
The model gets better, for real: if the "suspicious activity" model's accuracy improves overall, that's still not a reason to deprioritize the access-revocation problem, since a better model doesn't touch who gets to see the footage in the first place.
Where people run it wrong.
They rank harms by how many times something gets mentioned in meetings, which mostly measures who feels safe complaining.
They treat every harm as equally reversible, missing that some damage compounds while other damage resets on its own.
They wait for a full incident before ranking anything, instead of ranking off a near miss while it's still cheap to fix.
How to use it live. If an interviewer asks you to just list the harms instead of ranking them, resist it. A list with no order is a brainstorm. The actual skill being tested is the ranking rule itself.
Flashcards (tap any card to flip it)
1 · THE FRAMEWORK
What framework fits "how do you decide which harms are in scope for your product to prevent"?
Tap to flip
ANSWER
ORDER: outcome, reversibility, dependency, evidence, rank. Reversibility is the step that carries the whole argument.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Adaeze Nnamdi, the product manager who decides which harms Larkspur Coach is actually scoped to prevent.
3 · THE HABIT
What did the team stop doing once the real ranking rule existed?
Tap to flip
ANSWER
They stopped triaging harms by whoever raised them most recently or most loudly in a planning meeting.
4 · THE RANKING RULE
What's the actual rule this answer uses to rank harms?
Tap to flip
ANSWER
Rank by how hard the harm is to undo, not by how often it happens. A rare, compounding harm outranks a common, recoverable one.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Making full transaction detail visible to any linked account holder by default, since it made sense only when imagining every linked account as two people cooperating.
6 · THE NUMBER
Fill in the blank: by raw frequency, the abuse-via-visibility harm affects about ___ users a year, the fewest of all four categories measured.
Tap to flip
ANSWER
40. It still ranks first, since frequency was never the number this ranking is built on.
7 · THE REPLAY
Same near miss, ranked scope already in place. What changes?
Tap to flip
ANSWER
Shared visibility already defaults to hiding transaction detail, so the pattern the support rep caught never has a chance to develop in the first place.
8 · CROSS PRODUCT TRANSFER
Section 4 answers this again for a different product. Which product, and what ranks first there?
Tap to flip
ANSWER
Ferrowatch Home's SentryView camera app. There, enforcing access revocation, so an ex-partner or former landlord can't retain hidden camera access, ranks first.
Check yourself Score: 0 / 0
Fill in the blank
1. Fill in the blank: the abuse-via-visibility harm affects roughly ___ users a year, the rarest of the four harms measured, but ranks first in the actual scope.
Show hint
Look at the bar chart of estimated users affected per year.
Show answer
About 40. It still ranks first, since reversibility, not raw frequency, is the ranking rule.
Multiple choice
2. Why does this answer rank the rare abuse case above the far more common tone-and-copy issue?
A. Tone issues are cheaper to fix, so they don't count as real harms.
B. The abuse case compounds and is hard for the person harmed to undo, while the tone issue is recoverable in a moment.
C. Support tickets about tone are automatically deprioritized by policy.
D. The abuse case affects more total users over a customer's lifetime.
Show hint
Look at the Reversibility step and the quadrant diagram.
Show answer
B. Reversibility, not frequency or lifetime volume, is the ranking rule ORDER is built around here.
True or false
3. True or false: this answer recommends dropping the tone and shaming-copy issue from scope entirely, since it ranks last.
True
False
Show hint
Look at "what I would leave alone" and the priority list's last bullet.
Show answer
False. It stays in scope, just honestly ranked last, since it's real but far more recoverable than the other three harms.
Short answer, name the reversal
4. What old decision does this answer take back, and why did it make sense when it was made?
Show hint
Look at "the decision I would take back."
Show answer
Model answer: Making full transaction detail visible to any linked account holder by default. It made sense while every linked account was imagined as two people cooperating on a shared budget.
Short answer, where it wouldn't matter
5. Name a place in this same product where a rare, low-volume harm genuinely would not need to jump the queue this way.
Show hint
Think about a rare harm that's still easy to undo.
Show answer
Model answer: A rare glitch that miscategorizes a single transaction. It's uncommon and mildly annoying, but the user can just fix the label themselves in a few seconds.
Short answer, apply it yourself
6. Pick a product you use yourself. Name one rare, hard-to-undo harm it could cause that probably never comes up in a typical feedback meeting.
Show hint
Think about a harm that's quiet, not loud, and difficult to reverse once it happens.
Show answer
Model answer: A common one: a shared photo-storage app where a former partner retains access to a household's photo library long after a breakup, quietly, with no prompt ever asking either person to review who still has access.
Before you close the answer
Why this works
Tests whether you can build a real ranking rule for an open-ended list of harms, instead of just naming harms, and whether you'll protect against a rare, severe one even when it never shows up loudly in a meeting.
Follow-up traps
"Isn't it wasteful to spend engineering time on a harm affecting only 40 people a year?" Response: reversibility, not headcount, is the ranking rule; a harm that compounds and traps someone is worth fixing even at low volume, since the alternative is knowingly leaving an unrecoverable harm unaddressed.
"How do you know you've found every candidate harm worth ranking?" Response: you don't, up front, which is exactly why the triage rule has to apply to every new harm as it's reported, not just the ones known at launch.
If pressed
Larkspur's real fix also logs every change to account-visibility settings with a timestamp, so if a dispute ever arises about who changed what, there's a real record instead of two conflicting accounts.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
