ConceptAdvancedDesigning for Uncertainty & Trust / Trust, transparency and explainability in UX / #17
What is the relationship between explainability and regulatory exposure?
GUARD the product is RollCheck, an AI tool that flags likely duplicate voter registrations for county election offices
Picture a list of forty thousand names, and no way for a single one of them to ask why they're on it. Civic Roll Project builds RollCheck for county election offices. Naledi Khumalo leads the data team, and pulls the fresh voter file the day it lands, every cycle, before anyone else touches it.
The direct answer
The less a system can explain an adverse decision about one specific person, the more that silence becomes the evidence used against it. Regulators and courts don't need to prove your model is biased to act on it; they only need to show that nobody, including you, could explain why this person was flagged and that person wasn't. Explainability isn't a nice interface touch. It's the thing that stands between an outcome you can defend and one you can't.
Do this, in order
Give every affected person a specific, plain-language reason for an adverse decision, not a generic notice.Why: an unexplained decision is the exact gap a regulator or plaintiff uses to argue the process was unfair.
Pair every reason with a real appeal path and a stated deadline.Why: a reason nobody can act on is barely better than no reason at all.
Track the appeal rate by demographic group, not just the overall flag rate.Why: a near-zero appeal rate across every group usually means nobody understands their own flag, not that every flag is correct.
Match the depth of explanation to how adverse and how hard to undo the decision is.Why: a spam filter and a voter-roll purge don't need the same bar, and treating them the same wastes effort on the low-stakes one.
Keep a record of who reviewed each flag and what fields triggered it.Why: when a regulator eventually asks, "who reviewed it" is usually the first question, and it needs a real answer.
Leave the underlying matching logic alone for the vast majority of clearly non-duplicate records.Why: the exposure lives in the explanation gap, not in the matching itself, which works fine for most of the file.
How to answer this, stage by stage
This is a concept question with no product handed to you. Scope it fast, then spend your time on the actual relationship, not on describing the tool.
Stage 1
Scope it to one system
Say it like this
"I'll answer this with RollCheck, a tool that flags likely duplicate voter registrations for county election offices, since a real example makes the relationship concrete instead of abstract."
Why this works
Grounds a legal-sounding question in one inspectable system instead of a general essay.
Stage 2
Say your structure out loud
Say it like this
"I'll use GUARD. Groups affected, where the harm lands unevenly, who can actually contest it, the design change that reduces it, and how you'd detect it in production."
Why this works
Shows the interviewer you're running a real method, not free-associating about compliance.
Stage 3
Reframe the question
Say it like this
"This isn't 'explainability is a nice-to-have.' It's that an unexplained adverse decision is exactly the evidence a regulator needs to prove the process wasn't fair, whether or not the model was actually wrong."
Why this works
This is the actual relationship the question is testing, not a restatement of it.
Stage 4
Give the one decision
Say it like this
"Send every flagged voter the specific fields that triggered the match, name, date of birth, address overlap, plus a plain appeal path with a real deadline. Not a generic postcard."
Why this works
Matches the direct answer exactly, which is what makes it survive a follow-up question.
Stage 5
Prove it with a failure
Say it like this
"When the postcard gave no real reason, renters got flagged at three times the rate of homeowners, and almost nobody contested it in time, since nobody understood what they were contesting. That gap is what drew the state inquiry, not the flag rate itself."
Why this works
Shows the mechanism, silence becoming evidence, with a number attached instead of asserting it.
Stage 6
Close on one line
Say it like this
"Explainability and regulatory exposure move together. The less you can say about why, the more a regulator gets to assume the worst about why."
Why this works
Restates the direct answer in a sentence short enough to actually remember.
Let's learn
Civic Roll Project builds RollCheck, a tool that reads voter registration files across county lines and flags records that look like the same person registered twice.
Before RollCheck, county clerks cross-checked by hand, catching maybe a few hundred likely duplicates a cycle out of millions of records, far below the real number sitting in the data.
Knowledge spark: what's disparate impact?
A rule that looks the same for everyone on paper but lands harder on one group in practice. Nobody has to intend it for it to count. A regulator only has to show the pattern, then ask you to explain it.
RollCheck flags around 40,000 records a cycle. Every flagged voter gets a form postcard: your registration needs review, respond within 20 days.
Flag rate, renters vs homeowners
Renters move more often than homeowners, which is exactly what trips RollCheck's matching logic. The rate gap is real even though nobody designed it on purpose.
Complaints to the state election board, by month
Complaints climbed for five straight months before the news story, then fell fast once the new postcard gave people something they could actually respond to.
The turn. The extra flags were never really the problem. The problem was that not one of those 40,000 people could see the actual fields that triggered their flag, or what to do about it in a way that felt real instead of bureaucratic.
At its worst: fewer than 2 percent of flagged voters responded to the postcard in time. Most weren't ignoring a fair result. They didn't understand what they were being asked to contest, so they didn't contest anything, and unresponded flags moved toward removal from the rolls.
The gap sits in the third box. Nothing about the matching logic caused it. The missing reason did.
The decision I would take back
We sent a generic form postcard instead of the specific matching fields that triggered each flag, since a plain notice was cheaper to print and easier to get state approval for on a tight timeline. That made sense before anyone looked at who was actually responding to it. It stopped making sense once the response rate stayed under 2 percent for three cycles running.
What I would leave alone: RollCheck's underlying matching logic, which correctly clears the overwhelming majority of records as clearly not duplicates. The exposure never lived in the matching. It lived entirely in what happened after a match got flagged.
Now here is the same thing as a story
The short version above is what you'd say defending RollCheck to a state election board. Read this one for how the postcard actually got rewritten.
Naledi Khumalo has led Civic Roll Project's data team through four election cycles. She built RollCheck's matching model herself, and can usually tell from the flag reason codes alone which cases are real duplicates before a human ever opens the file.
The first two cycles with RollCheck felt like a real win. Clerks who used to spend weeks cross-checking by hand now had a clean list every Monday morning, sorted by confidence.
Nobody noticed the postcard problem happening. It built up over months: the response rate sat under 2 percent from the start, and everyone read that as evidence the flags were simply correct, since so few people pushed back.
Naledi's team held the lever the whole time. The flagged voter never even knew there was one.
A near-zero appeal rate isn't proof that every flag was fair. It's proof that nobody understood what they were being asked to appeal.
Then a local reporter cross-referenced flagged addresses against county rental listings and published the 3-to-1 gap between renters and homeowners. Within a month, the state election board opened an inquiry, not into whether RollCheck's matching was accurate, but into whether flagged voters had been given a fair chance to respond.
RollCheck landed in two of the three branches at once: a legal notice requirement, and a measurable disparate impact.
The team considered pausing RollCheck's flags entirely for a full cycle while they investigated, and rejected it. Suspending duplicate detection would have let real duplicate registrations sit unresolved for another two years, trading one kind of harm for another.
Naledi's team had the first item. They had never written down the other three in a form a voter could read.
The replay: the new postcard names the exact fields, matching name and birth date, an address that overlaps a prior registration, and a 20-day appeal window explained in one paragraph a stranger could follow. The response rate climbed from under 2 percent to 34 percent within the first cycle it ran.
I chose the generic postcard because it was simpler to get approved on a tight rollout schedule, and simple felt safe. It took a reporter finding the pattern in public rental listings, something Civic Roll Project's own team had access to the whole time, to see that the postcard's vagueness was never neutral. It fell hardest on exactly the people least equipped to fight back.
GUARD, the shape of the exposureNot a policy memo. GUARD is what forces you to name who can't push back, before a regulator does it for you.
G
Groups. Who this actually touches.
Naledi's team, who set and can adjust the flag; the flagged voter, who cannot see or challenge the match.
Names both sides of the lever before anything else.
U
Unequal. Where it lands hardest.
Renters flagged at 2.4%, homeowners at 0.8%, roughly three times higher, since renters move more often.
Turns a vague fairness worry into a specific, measurable gap.
A
Ability to contest. The hardest step.
A generic postcard with no matching detail. Under 2% responded, not because they agreed, but because they didn't understand it.
The single design choice that turned an accuracy question into a legal one.
R
Reduce. The design change.
Name the matching fields, state the deadline in plain language, give a real appeal path.
A product decision, not a training deck or a review board.
D
Detect. How you'd know, before the regulator tells you.
Track appeal rate by group. A flat, near-zero rate everywhere is the warning sign, not the all-clear.
Turns the fix into something you monitor, not a one-time patch.
The recap, one line per letter: groups are the clerk and the flagged voter, unequal is the 3-to-1 renter gap, ability to contest is the postcard nobody could act on, reduce is naming the actual matching fields, and detect is watching the appeal rate itself as the real signal.
And if you want to be sure it really works, try it somewhere elseSame five letters, a telecom carrier instead of an election office. A different sector, same silence turning into evidence.
Kestrel Mobile runs RiskGate, a tool that flags accounts for possible fraud and can suspend service pending review. Callum Fenner works trust and safety, reviewing flags RiskGate escalates each morning.
Mapped onto GUARD: groups are Callum's team, who can lift a suspension, and the flagged customer, who often finds out only when their phone stops working. Unequal is that prepaid accounts, used disproportionately by lower-income customers, get flagged at a noticeably higher rate than postpaid accounts, since payment-pattern signals behave differently for prepaid plans. Ability to contest is the core problem: customers get a generic text, "your account requires verification," with no reason and a support line with hour-long wait times. Reduce is a specific message naming the actual trigger, an unusual payment method or a location mismatch, plus a same-day callback option instead of a queue. Detect is tracking how many suspended customers ever reach a human reviewer at all, since a low reach-rate hides the same silence RollCheck had.
Same shape as RollCheck's arc, six months faster, because a suspended phone gets noticed immediately, where a flagged voter record doesn't.
The bottom right corner, hard to undo and unexplained, is where regulatory exposure actually concentrates. RollCheck and RiskGate both started there.
Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "silence becomes the evidence against you," and stop there.
Cost: there's no budget to rebuild the notice system this quarter. Say so honestly, and ship the specific reason first; it costs far less than the appeal-path redesign and closes most of the exposure on its own.
The model gets better, for real: if RollCheck's matching accuracy improves next year, the explanation gap doesn't shrink on its own. A more accurate model that still can't say why is still exposed the same way.
Where people run it wrong.
They treat a low complaint rate as proof the system is fair, when it's often proof nobody understood what happened to them.
They fix the model's accuracy and assume that resolves the legal risk, when the risk usually sits in the explanation, not the score.
They write one generic disclosure for every decision, instead of matching its depth to how adverse and how reversible the decision actually is.
How to use it live. When someone asks about explainability and regulatory exposure, ask yourself one question first: if a regulator subpoenaed this exact decision tomorrow, could you show them why it happened to this person and not another? If not, that gap is the exposure, not a detail beside it.
Flashcards (tap any card to flip it)
1 · THE FRAMEWORK
What framework fits "what's the relationship between explainability and regulatory exposure"?
Tap to flip
ANSWER
GUARD: groups, unequal, ability to contest, reduce, detect. The ability-to-contest step is the answer's hardest and most load-bearing step.
2 · THE PEOPLE
Who is this answer about?
Tap to flip
ANSWER
Naledi Khumalo, who leads the data team at Civic Roll Project and built RollCheck's matching model herself.
3 · THE UNEQUAL HARM
Where does the harm land unevenly in this story?
Tap to flip
ANSWER
Renters get flagged at roughly three times the rate of homeowners, since renters move more often and trip the matching logic more.
4 · ABILITY TO CONTEST
What's the actual gap in this story, the one GUARD is built to find?
Tap to flip
ANSWER
A generic postcard with no specific reason and no accessible appeal path, which most flagged voters could not meaningfully act on.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Sending a generic form postcard instead of the specific matching fields, since it was cheaper and faster to get approved under a tight rollout timeline.
6 · THE NUMBER
Fill in the blank: the response rate to the new, specific postcard climbed to ___ percent, from under 2 percent.
Tap to flip
ANSWER
34 percent, within the very first cycle the new postcard ran.
7 · THE REPLAY
Same 3-to-1 flag gap, redesigned notice. What changes?
Tap to flip
ANSWER
Flagged voters can now see the exact fields that triggered their match and a plain 20-day appeal window, and response rates rise enough to catch real errors before removal.
8 · CROSS PRODUCT TRANSFER
Section 4 answers this again for a different product. Which product, and what's different about the exposure?
Tap to flip
ANSWER
Kestrel Mobile's RiskGate. There, exposure surfaces faster, since a suspended phone gets noticed immediately, unlike a flagged voter record.
Check yourself Score: 0 / 0
True or false
1. True or false: this answer argues that RollCheck's matching model needed to be retrained because it was biased.
True
False
Show hint
Look at "what I would leave alone."
Show answer
False. The matching logic worked fine for the overwhelming majority of records. The exposure lived entirely in the missing explanation and appeal path, not in the model itself.
Multiple choice
2. Why did the state inquiry focus on the appeal process instead of RollCheck's matching accuracy?
A. Because matching accuracy can never be measured.
B. Because a fair-looking accuracy number doesn't prove flagged voters had a real way to contest it.
C. Because election boards aren't allowed to review model accuracy.
D. Because the reporter only had access to appeal records.
Show hint
Look at the decision tree titled "what sets the regulatory exposure."
Show answer
B. An accurate model can still fail the fairness question if nobody affected can see or challenge the decision.
Fill in the blank
3. Fill in the blank: renters were flagged at about ___ percent, compared to 0.8 percent for homeowners.
Show hint
Look at the bar chart comparing flag rates.
Show answer
2.4 percent. Roughly three times the homeowner rate, the gap that drew the reporter's attention in the first place.
Short answer, name the reversal
4. What old decision does this answer take back, and why did it make sense when it was made?
Show hint
Look at "the decision I would take back."
Show answer
Model answer: Sending a generic postcard instead of the specific matching fields. It made sense under a tight rollout timeline, when a simple notice was easier and faster to get state approval for.
Short answer, where it wouldn't matter
5. Name a kind of AI decision where this level of explanation and appeal detail genuinely isn't needed.
Show hint
Look at the quadrant plotting stakes against how clear the reason is.
Show answer
Model answer: A spam filter. It's low stakes and fully reversible, a person can always check their spam folder, so a heavy explanation and appeal process would cost more than the decision is worth.
Short answer, apply it yourself
6. Pick an AI-driven decision you've been on the receiving end of. Could you have gotten a specific reason for it if you'd asked, or only a generic one?
Show hint
Think of a loan or credit decision, an account flag, or a content moderation call.
Show answer
Model answer: Most people can recall a generic denial or flag with no real reason attached. The harder, more useful question is whether they knew there was any way to ask for one.
Before you close the answer
Why this works
Tests whether you treat explainability as a legal and design necessity tied to how adverse a decision is, or as an interface nicety you'd add if there's time left.
Follow-up traps
"Isn't this just a lawsuit-avoidance argument, not a real design principle?" Response: no, because the same fix, a specific reason and a real appeal path, is also what let real errors get caught and corrected, which the lawsuit-avoidance framing alone would miss.
"What if giving a detailed reason helps bad actors game the system?" Response: possible for a few adversarial cases, which is why the reason names the matching fields rather than the exact scoring thresholds, giving enough for a fair appeal without a blueprint for gaming it.
If pressed
Civic Roll Project's real consent decree required the appeal rate itself to be reported to the state board quarterly, by demographic group, turning "ability to contest" from a design choice into an ongoing, audited number.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
