CaseAdvancedResponsible AI & Advanced Practice / Responsible AI as a product requirement / #11
Explain how you would test for bias in a ranking or recommendation feature.
GUARD the product is SmartMatch, Delwick Mobile's plan and device recommendation ranker
Delwick Mobile is a phone carrier. SmartMatch ranks which plan and device bundle to show each customer first, in the app and in store. Denzel Osei is the product manager who owns the ranking model and, since a question from a new teammate, its bias testing.
The direct answer
Don't test the ranker by asking whether it treats groups "the same" on average. Test it by measuring, per group, how often the actual lowest-cost plan that fits a customer's usage gets shown at all, since that's the number the ranker can quietly starve for one group while looking fine overall. Then give every customer a visible second option, the cheapest fit, next to the top pick, so nobody's stuck evaluating a deal against a menu they can't see.
Do this, in order
Measure how often the actual best-value plan gets shown at all, broken out by group.Why: a ranker can look fair on "average predicted savings" while quietly hiding the best deal from one group.
Always show the lowest-cost fitting option next to the top recommendation.Why: a customer can't push back on a deal they never knew was possible.
Check which features the ranker leans on, and flag proxies like zip code directly.Why: a model doesn't need a protected field to reproduce its pattern; a proxy does the job just as well.
Route "this feels too expensive" complaints to a fairness review, not just a pricing queue.Why: the only complaint path in place was routing every relevant signal straight past the team that needed to see it.
Run the group-level check monthly, automatically, not only when someone happens to ask.Why: the gap in this story sat unnoticed for months because nobody had a standing reason to look.
Keep optimizing for predicted conversion where the stakes are genuinely low.Why: not every ranked list needs this scrutiny; the ones that decide what someone pays every month do.
How to answer this, stage by stage
Seven moves. This one leans hard on naming who can't push back, since that's the actual test GUARD is built around.
Stage 1
Name one real ranking feature
Say it like this
"I'll answer this for a plan and device ranker at a phone carrier, since 'bias in a ranking feature' means something specific there: whether some customers get shown worse deals than others, systematically."
Why this works
Grounds the answer in a real, checkable outcome instead of a generic fairness lecture.
Stage 2
Say your structure out loud
Say it like this
"I'll use GUARD. Groups affected, where it lands unequally, who can't contest it, the design change that reduces it, and how I'd detect it in production."
Why this works
Shows a repeatable method, not a one-off gut check.
Stage 3
Name both groups
Say it like this
"There's the team that set the ranking objective, optimizing for predicted conversion. And there's the customer, who only ever sees one offer, with no way to know a cheaper fit existed."
Why this works
This is GUARD's strongest move: naming the operator and the subject, not just "users" in the abstract.
Stage 4
Name where it lands unevenly
Say it like this
"If zip code or a language proxy correlates with lower rates of seeing the actual cheapest fitting plan, that group is paying a quiet tax nobody decided on purpose."
Why this works
Turns "test for bias" into one specific, measurable claim instead of a vague worry.
Stage 5
Name who can't push back
Say it like this
"A customer can complain that a plan is expensive. They can't complain that they were never shown the cheaper one, because they don't know it exists."
Why this works
The single strongest line in a GUARD answer: naming the exact thing the subject can't see to object to.
Stage 6
Give the design change and the detection plan
Say it like this
"Show the cheapest fitting option next to the top pick, always. And run a monthly check on the rate each group actually sees that best-value plan, not just the average predicted savings."
Why this works
Gives a concrete product change, not a policy document or a training session.
Stage 7
Close on the one line
Say it like this
"So: test the rate the best deal actually gets shown, by group, not just the average outcome. And give every customer a visible cheaper option to compare against."
Why this works
Leaves the interviewer with the concrete test, said once more, cleanly.
Let's learn
Say a phone carrier builds a ranker that decides which plan and device bundle to show a customer first, in the app or in a store.
Before SmartMatch existed, a sales rep picked the plan by hand, usually the one closest to what the customer already had. It worked, but slowly, and it depended on which rep you happened to get.
SmartMatch replaced that with a model trained to predict which offer a customer was most likely to accept. Conversion rates went up. Average handling time went down. On every metric the team was watching, it looked like a clean win.
Knowledge spark: what's a proxy variable?
A feature that isn't a protected category itself, like income or race, but correlates with one closely enough to reproduce its pattern anyway. Zip code is a classic proxy: it's just a location, but it can track income and other things a model was never supposed to use directly.
Here is the turn. "Most likely to accept" and "actually the best deal for this person" are not the same target, and nobody had ever checked how far apart they'd drifted for different groups of customers. The model wasn't doing anything as crude as looking up a customer's income. It didn't need to. Zip code alone carried enough of that signal to reproduce the same pattern.
Rate customers were shown their actual best-value plan, by income tercile
Low-income customers were shown their actual cheapest fitting plan barely more than four times in ten, while high-income customers saw it nearly three times in four.
At its worst: two customers with the same usage pattern get shown different top offers, one closer to the cheapest fit, one further from it, and neither of them can see the other's screen to know anything was uneven.
The decision I would take back
We collapsed "find the best fit for this customer" and "predict what they're likely to accept" into a single ranking objective, because it was simpler to build and to explain to leadership as one clean metric. That made sense while the two targets mostly pointed the same direction. It stopped making sense the moment they quietly diverged for one group of customers and nobody had kept a way to tell.
What I would leave alone: ranking which accessory or add-on to suggest after a plan is chosen doesn't need this scrutiny. The stakes are a ten-dollar case, not a monthly bill that compounds for years, so a slight ranking skew there costs far less than it would here.
The customer being shown the worse deal has no idea a better one was ever an option. That's not a pricing complaint. That's a complaint nobody had a form for.
The lesson: a ranking feature can hit every metric the team is watching and still be quietly unfair, because the metrics were never built to notice the one thing that mattered: whether the best option was actually visible to everyone.
Now here is the same thing as a story
The short version above is the test plan. Read this one for how Denzel actually found the gap.
Denzel Osei has managed SmartMatch for two years. He knows the model's dashboard by heart: conversion rate, average handling time, customer satisfaction after a purchase. Every one of them had been trending the right direction since launch.
Two people, one decision. Only one of them ever gets to see what else was possible.
Then a new data scientist joined the team. Reviewing the model's feature importance chart in her first week, she asked a question nobody had asked out loud before: why is zip code one of the top three features driving the recommendation?
Nobody had a good answer. It had always been in the model. It had always seemed to help. Nobody had ever separated "helps predict what a customer will accept" from "helps predict what a customer can afford," and those aren't the same question.
Every one of these had been true for months. The list just hadn't existed yet.
Denzel ran the group-level check that had never been run before: not the average predicted savings, which looked fine everywhere, but the actual rate each income tercile was shown its true cheapest fitting plan. Low-income customers saw it 41% of the time. High-income customers saw it 74% of the time. Same usage patterns. Same model. Wildly different visibility into the same menu.
We did not build a model that overcharged anyone directly. We built one that quietly decided who got to see the whole menu and who only got to see one dish.
Every one of these customers who called in had already told the company something was off. The path just never routed it anywhere useful.
Denzel's fix had two parts. First, a design change: show the cheapest fitting plan next to the top recommendation, every time, for every customer, so nobody has to guess whether a better deal existed. Second, a detection change: a monthly automated check on the best-value visibility rate, broken out by income tercile, zip code cluster, and language preference, with any gap over ten points triggering a review before it could sit unnoticed for months again.
Four pieces. The old process only ever had the shape of the first one, and even that wasn't checked by group.
The gate doesn't block every recommendation. It only stops the ones where the gap is real and nobody would otherwise notice.
Within two months of the fix shipping, the visibility gap between income terciles had nearly closed. Nobody's top recommendation got worse. The customers who'd been quietly missing the best fit finally started seeing it.
What I would tell myself, back when SmartMatch first launched: a clean conversion number was never proof the ranking was fair. It was proof the ranking was good at the one thing anyone had thought to measure.
GUARD, the check Denzel actually ranNot "is the model biased." GUARD forces you to name who can't push back, and to fix that specifically.
G
Groups. Operator and subject, named.
The ranking team, who set the objective as predicted conversion. The customer, who only ever sees the one offer the model chose to show.
Names both sides instead of talking about "users" in general.
U
Unequal. Where the harm actually lands.
Low-income customers, tracked through a zip code proxy, saw their true best-value plan 41% of the time versus 74% for high-income customers.
Turns "bias" into one specific, measured gap instead of a general worry.
A
Ability to contest. The strongest move.
A customer can complain a plan feels expensive. They cannot complain about a cheaper plan they never knew existed, and nothing in the support flow ever routed that gap anywhere.
Names the exact thing the subject has no way to push back on.
R
Reduce. The design change.
Always show the cheapest fitting plan next to the top recommendation, for every customer, regardless of group.
A concrete product change, not a policy memo about fairness.
D
Detect. How you'd know without waiting for a complaint.
A monthly automated check on best-value visibility rate, by income tercile, zip code cluster, and language preference, any ten-point gap triggers review.
Catches the next version of this problem before a customer, or a regulator, has to point it out.
Low-income tercile: best-value visibility rate, by month
The rate had been flat and low for four months before anyone asked why. Two months after the fix, it had more than doubled.
The recap, one line per letter: groups is the ranking team against the customer, unequal is a 33-point visibility gap between income terciles, ability to contest is a complaint path with nowhere to route "I was never shown the cheaper plan," reduce is always surfacing that cheaper plan, and detect is a monthly group-level check with a hard trigger.
And if you want to be sure it really works, try it somewhere elseSame five letters, a grocery delivery app instead of a phone carrier. A completely different field.
Pallmoor Grocery Delivery runs QuickPick, a ranker that decides which store and which promotion to show first when a customer opens the app. Yara Petrov owns the ranking model.
Mapped onto GUARD: groups is the merchandising team, who set QuickPick's objective around predicted basket size, against the customer, who only sees the one store surfaced first. Unequal is that customers in delivery zones with fewer nearby stores get shown a narrower set of promotions overall, since the model was trained mostly on dense urban zones where competition kept promotions frequent. Ability to contest is that a customer can complain a delivery fee felt high, but has no way to know a lower-fee store option in their zone was ranked lower and simply never surfaced. Reduce is always showing the lowest total-cost option for a customer's zone as a second card, not just the highest predicted-basket-size pick. Detect is a monthly check on promotion-frequency-shown by delivery zone density, flagging any zone falling far below the median.
Same four pieces. A grocery ranker needs them for exactly the same reason a phone plan ranker does.
Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "measure how often the actual best option gets shown, by group, and always surface a visible cheaper alternative," and stop.
Cost: there's no engineering time this quarter for a full monthly audit pipeline. Say so honestly, and start with a single quarterly manual pull of the visibility rate by group, even a rough one, rather than waiting for the automated version.
The model gets better, for real: if overall conversion improves, that's still not proof the visibility gap closed. A better-converting model can get better specifically by serving the group already seeing the best deal even more of it.
Where people run it wrong.
They test for bias on the model's average output instead of on what each group is actually shown.
They check fairness once at launch and let it go unaudited as the model keeps learning from new data.
They treat "no explicit protected field in the model" as proof the model can't be biased, ignoring the proxies doing the same job.
How to use it live. If an interviewer pushes on "but the model doesn't use race or income directly," don't argue the model's inputs. Ask what output gap you'd actually measure to check, and name it. That's the whole test.
Flashcards (tap any card to flip it)
1 · THE FRAMEWORK
What framework fits "explain how you would test for bias in a ranking or recommendation feature"?
Tap to flip
ANSWER
GUARD: groups, unequal, ability to contest, reduce, detect. Ability to contest is the hardest, strongest step.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Denzel Osei, the product manager who owns SmartMatch, Delwick Mobile's plan and device ranker.
3 · THE HABIT
What did the team never do, for two full years, until a new hire asked?
Tap to flip
ANSWER
Check the model's group-level visibility rate for its own best-value plan. Only the average predicted savings was ever watched.
4 · WHO CAN'T PUSH BACK
What can a customer contest, and what can't they?
Tap to flip
ANSWER
They can complain a plan feels expensive. They can't complain about a cheaper plan they were never shown, since they have no way to know it existed.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Collapsing "best fit for the customer" and "predicted to accept" into one ranking objective, since it made sense while both targets mostly pointed the same way.
6 · THE NUMBER
Fill in the blank: low-income customers were shown their true best-value plan only ___% of the time, versus 74% for high-income customers.
Tap to flip
ANSWER
41%. That 33-point gap is the number the old dashboard never measured.
7 · THE REPLAY
Same low-income customer, redesigned ranker. What changes?
Tap to flip
ANSWER
They see the cheapest fitting plan right next to the top pick, every time. Visibility for that group rises from 41% to 81% within two months.
8 · CROSS PRODUCT TRANSFER
Section 4 answers this again for a different product. Which product, and what's unequal there?
Tap to flip
ANSWER
Pallmoor's QuickPick grocery ranker. There, customers in lower-density delivery zones see a narrower set of promotions, since the model trained mostly on dense urban zones.
Check yourself Score: 0 / 0
Short answer, name the flip
1. Why did zip code alone let the model reproduce an income-linked pattern, even with no income field in the training data?
Show hint
Look at the knowledge spark about proxy variables.
Show answer
Model answer: Zip code correlates closely enough with income that it carries much of the same signal, so a model doesn't need the protected field itself to reproduce its pattern.
Multiple choice
2. Why does this answer measure "rate the best-value plan is shown," instead of "average predicted savings across all customers"?
A. Average predicted savings is illegal to measure by group.
B. The average looked fine while a 33-point visibility gap sat hidden inside it, by group.
C. Predicted savings only applies to device purchases, not plans.
D. Visibility rate is easier to compute than savings.
Show hint
Look at the bar chart comparing income terciles.
Show answer
B. The whole point of the GUARD check is that a fine-looking average can hide a real, unequal gap underneath it.
True or false
3. True or false: this answer recommends applying the same level of bias-testing scrutiny to every ranked feature in the app, including low-stakes ones like accessory suggestions.
True
False
Show hint
Look at "what I would leave alone."
Show answer
False. Low-stakes rankings, like a ten-dollar accessory suggestion, don't carry the same recurring financial harm as a monthly plan choice.
Short answer, where it wouldn't matter
4. Name a ranked feature in this same app where this level of bias testing genuinely would not matter as much.
Show hint
Think about what costs little and doesn't repeat monthly.
Show answer
Model answer: Which phone case or accessory to suggest after checkout. The financial stakes are small and one-time, unlike a plan choice that compounds every month.
Short answer, apply it yourself
5. Pick a ranked list or recommendation you see regularly. What group might be seeing a systematically worse version of it, and how would you check?
Show hint
Think about who the ranking was mostly trained on versus who else uses it.
Show answer
Model answer: A common one: a job-search app's ranked listings might surface fewer remote roles to users outside major cities, since the ranking may have been trained mostly on urban search behavior.
Short answer, the number
6. If the low-income visibility rate had started at 60% instead of 39%, would the same fix, always showing the cheapest fitting plan, still matter? Why or why not?
Show hint
Think about whether the fix depends on the size of the starting gap.
Show answer
Model answer: Yes, though with less urgency. Any real, sustained gap by group still means some customers systematically can't see their best option, which the fix addresses regardless of how large the gap starts out.
Before you close the answer
Why this works
Tests whether you can find a fairness problem that hides inside a metric everyone is already watching, and whether you'll name the specific group that has no way to contest what they can't see.
Follow-up traps
"Couldn't you just remove zip code from the model entirely?" Response: that alone doesn't fix it, since other features can carry the same correlated signal; the actual fix is measuring the output gap directly, not just scrubbing one input.
"Isn't showing a second, cheaper option just going to hurt conversion?" Response: possibly a little, but the visibility check exists specifically because conversion was already the thing masking this gap; a small conversion cost is the trade being accepted here on purpose.
If pressed
Kestrel's real fix also logs which second option was shown and whether the customer picked it, so the team can tell over time whether the cheaper option is a genuine choice people take or just a box nobody clicks.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
