How do you handle a right-to-explanation request for a model-driven decision?

Larkspur Credit matches borrowers to personal loans and uses a model to decide who gets approved and at what rate. Sanaa Farooqi is the product manager for the underwriting model. She keeps a printed stack of that week's declined files on the corner of her desk, the ones her team is required to review by hand.

How to answer this, stage by stage

The short version is above. This is what you'd actually say if an interviewer asked you to walk through it live.

Let's learn

Say a loan marketplace uses a model to decide who gets approved and at what rate, in seconds, for thousands of applicants a day.

For its first year, Larkspur's decline letters said one thing: a three-digit reason code and a customer service number. Reviewers spot-checked a sample of declines each week, saw the codes lined up with real underwriting factors, and moved on. The system looked fine, because on the sample it was fine.

Then appeal volume for one kind of applicant, people with thin or unusual income records, started running higher than everyone else's, while the overturn rate on those same appeals stayed near zero. Not because those people were always rightly declined. Because nobody could tell, from the letter, what to even push back on.

At its worst: a state regulator opens an inquiry after a news story about one declined applicant, asks for the reasoning behind ten similar cases, and Larkspur can only hand over ten copies of the same three-digit code.

What I would leave alone: a clean, traditional-file decline, where the top factors are obvious and stable, doesn't need a person in the loop. The letter there is already specific and correct. Adding review to every single case would slow the whole system down for no real gain.

The lesson: a fallback message that was rare and harmless at launch can quietly become the loudest thing your system says to one entire group of people. Check who it's landing on, not just how often it fires.

Now here is the same thing as a story

Use the short version above to defend the design. Read this one for how Priya actually found the gap.

Sanaa Farooqi has run the underwriting model at Larkspur for two years. She built the reason-code system herself, back when a "declined" letter went to maybe two hundred people a week.

For most of that first year, the system earned its keep. Every Friday she pulled twenty declined files at random, read the reason code next to the actual factors the model had weighed, and checked they lined up. They always did. By month nine she was pulling five files instead of twenty. By month fourteen, she mostly wasn't pulling any.

Two people, one decision. Only one of them ever had a lever in their hand.

Then a local news site ran a piece on a rideshare driver who'd been declined twice, six months apart, with the same three-digit code both times, and no one at the company who could tell her why. It hadn't happened to Priya's own team first. It happened to a peer product at a sister company, and the story named the parent brand.

Five steps, and the fourth one was never really a step. It was a wall with a phone number painted on it.

Priya pulled two years of decline letters that week and, for the first time, sorted them by file type instead of reading them one at a time. Thin-file and gig-income applicants weren't just declined more. They were getting the exact same fallback code, over and over, while traditional-file declines almost always carried a real, specific factor.

The bottom-right corner is where a model made the call and the applicant had nowhere to take it.

She went back to the letter itself and mapped what a real explanation would actually need.

A three-digit code was one of these four parts. It was never going to carry the other three by itself.

With the redesign, every letter now names the two to four real factors the model actually weighed for that case, in plain words, plus one concrete next step. Any file the model can't explain that clearly gets pulled for a person to review before the letter ever mails.

Same decision, same day. The only thing that changed was what the letter was able to say about it.

The old design asked whether the model got the decision right. The new one also asks whether the person on the other end of it has anything they can actually do with the answer.

I built one fallback code because it was fast to ship and it covered a case that barely came up. It took a stranger's name in a news story, someone who was never even Priya's own customer, to see that "rare" and "rare for the applicants who most need an answer" are two very different claims.

GUARD, for a decision the applicant never gets to questionNot a checklist. GUARD is what forces you to name both people in the room, not just the one who built the product.

Four numbers, and the second and third ones are the pair that would have caught this months earlier.

The recap, one line per letter: groups is the reviewer and the applicant, unequal is the fallback code landing hardest on thin-file applicants, ability to contest is the missing appeal path, reduce is real factors plus a next step for every decision, and detect is coverage and overturn rate tracked by file type.

And if you want to be sure it really works, try it somewhere elseSame five letters, a streaming service instead of a lender. This time nobody's credit is at stake, but the same missing lever shows up anyway.

Wavelane Streaming uses a model to flag and suspend accounts it scores as sharing logins outside one household. Declan Voss manages that trust and safety model.

Mapped onto GUARD: groups is the trust and safety analyst, who can review flagged accounts and reverse a suspension, and the subscriber, who gets a locked account and a canned message. Unequal is that shared family accounts on older, cheaper legacy plans get flagged far more than accounts on the newer plan built with multiple profiles in mind. Ability to contest is the real gap: the appeal form asks the subscriber to "describe your household," with no way to see what pattern actually triggered the flag. Reduce is naming the actual signal, unusual device count or a distant new login city, in the message itself, plus a one-tap "this is me, I travel for work" option that doesn't require a written appeal. Detect is watching false-suspension rate by plan type, since a gap hiding inside one plan tier is invisible in the overall average.

A different product, the same missing fourth step: a decision with no real way to push back on it.

Swap the trigger and it still runs.
Speed: an interviewer gives you thirty seconds. Say "build in real, specific reasons from day one, and route anything the model can't explain to a person before it ships," and stop there.
Cost: legal says a full explanation engine is a two-quarter build. Ship the two most common real factors first, by volume, and add the rest as you go, rather than shipping nothing until it's complete.
The model gets better, for real: if the underwriting model's overall accuracy improves, the explanation problem doesn't go away. A more accurate model that still can't say why, in plain words, is just a more confident version of the same gap.

Where people run it wrong.
They write a longer, more official-sounding decline letter, without adding a single real, checkable factor.
They treat every request the same, when what people actually need differs: some need a factor, some need a path to reapply, some need a human being.
They measure how many explanation requests got answered, instead of how many actually got resolved.

How to use it live. When you get this question, ask yourself first who in this story has a lever and who doesn't. Say both people out loud before you say anything about the letter itself.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0