Explain why regulated decision-making often demands explainability that current models cannot provide.

Whitmarsh Mortgage is an online home lender. Trueline is the model that scores every mortgage application and decides, on its own, which files never reach a person. Fritjof Meskell is the product manager who owns Trueline's roadmap. Halfdan Solbakken is the senior underwriter who took a phone call he could not finish. Gwynneth Loures runs a landscape gardening business, six years in, and was buying her first home when Trueline turned her down.

How to answer this, stage by stage

Nobody is grading whether you feel bad for Gwynneth. They're grading whether "explainability current models can't provide" turns into an actual mechanism, with a fix specific enough to defend.

Let's learn

Trueline is the model behind Whitmarsh Mortgage's home loans. Every application gets a score, and the score decides three things: an instant approval, an instant decline, or a hand-off to a person.

Four of Trueline's inputs. Two of them are ordinary. Two are the kind of interaction term that doesn't belong to any single, nameable reason.

Whitmarsh built Trueline fourteen months ago to replace an older scorecard, because the old one kept turning down applicants like Gwynneth for looking, on paper, less steady than they actually were. Trueline reads bank cash flow, gig and seasonal income patterns, and the usual credit file, and it approves real applicants the old model missed. It scores about 3,400 applications a month, on a 300 to 850 scale. Below 615, Trueline declines the file on its own, no underwriter ever opens it. From 615 to 664, an underwriter decides either way. Above 664, it approves on its own.

Here's the turn. Whitmarsh never rebuilt the tool that writes that reason. It still runs off a list of fifteen canned codes built five years ago, for the old scorecard, back when every input mapped cleanly to one code each. Trueline doesn't work that way. It weighs over 300 engineered features, many of them interaction terms, one factor multiplied against another, that were never single, nameable things to begin with. When Trueline's real top driver is one of those, the notice generator can't use it. It falls back to whichever of the fifteen old codes sits numerically closest, and prints that instead.

At its worst, this is a real applicant reading a confident, official-looking sentence, believing it's the true reason, and having no way to check whether it is.

The lesson: a model getting better at predicting risk and a model getting better at explaining itself are two separate improvements, and building one doesn't hand you the other for free. Regulated decisions need the second one specifically, and a bigger, more accurate model can quietly move further away from it while everyone celebrates the accuracy number going up.

Now here is the same thing as a story

Stage five above compresses this into four sentences. Here's the six weeks underneath, the part a stand-up answer skips.

Gwynneth Loures has run her own landscaping business for six years. Spring through early fall, she's booked solid, planting beds, laying stone paths, keeping a dozen properties looking cared for. Winter is quiet, some months almost nothing comes in. She'd built her whole financial life around that rhythm: a cushion set aside every August for the slow months, bills paid on time every single year, credit that any lender would call solid.

The old scorecard and its reason list were built for each other. Trueline inherited the list without anyone asking whether it still fit.

Under the old scorecard, an application like hers was actually a coin flip, the model saw the winter dip and read it as risk, full stop. Trueline was supposed to fix exactly that. It looks at cash flow across the whole year, not just any single month, and it approved plenty of seasonal businesses the old model would have turned away. For over a year, that was the whole story: a better model, more real applicants getting a fair look.

Gwynneth's file scored 604. The cutoff was 615.

Whitmarsh's team decided, months earlier, which score band would ever get a person's eyes on it. Gwynneth found out what that decision meant on the day it applied to her.

The letter that reached her inbox that evening read: "Your application was declined. Reason: insufficient stated income relative to obligations." Eleven words, official, final. Nothing about it looked like a guess.

It was one anyway. What actually drove Trueline's score wasn't her income level at all, income covered her mortgage payment comfortably averaged across the year, the same way it had for four years running. The real top factor was a single interaction term: the swing between her summer deposits and her winter ones, multiplied against a seasonal volatility figure tied to her declared industry code. That term had no matching code in Whitmarsh's list of fifteen. The system reached for the nearest one it had, "insufficient income," and printed it as if it were the answer.

Three days later, Gwynneth called Whitmarsh's appeal line. Halfdan Solbakken, a senior underwriter, took the call. He read her the letter's wording back to her, slower, kinder, but it was still the same eleven words.

Halfdan wasn't hiding anything from her. He genuinely didn't have it. Her file had never crossed a person's desk before that call.

"Which number," she asked him. "If my income is the problem, tell me the number I'd have needed, and I'll show you I clear it." Halfdan pulled up her file while she waited on the line. There was a score. There was a printed reason. There was nothing underneath either of those he could turn into an answer to her actual question. He told her, honestly, that he'd have to look into it and call her back. He hung up knowing he had nothing to call her back with.

Halfdan couldn't let it go. He'd underwritten mortgages for eleven years, and he'd never once told an applicant "I don't know why" and meant it literally. He flagged it to Fritjof Meskell, Trueline's product owner, the next morning: "I don't think we could answer a regulator on this one any better than I answered her."

Fritjof brought it to Constanza Prosek, Whitmarsh's fair-lending analyst. She pulled 200 auto-declined files from the two quarters before Gwynneth's, all scored between 590 and 614, and did by hand what the system was supposed to do automatically: read each file and work out the real top driver herself. She compared her answer against the letter each applicant had actually received.

138 of 200 matched. In the other 62, close to one in three, the generated reason cited something that provably wasn't the model's real top factor, and in most of those, the true driver was an interaction term with no code to its name.

Two separate questions got treated as one. A file can be close to the cutoff and still have a clean reason. Gwynneth's had neither going for it.

I keep coming back to the meeting fourteen months earlier, when Whitmarsh decided how Trueline would launch. Rebuilding the reason-code list to match Trueline's real complexity was on the table. It got shelved, not out of carelessness, the fifteen codes had covered every case the old scorecard ever produced, and nobody expected the new model's inputs to be different in kind, only in accuracy. That was a reasonable read of a model that hadn't shipped yet. It stopped being reasonable the day Trueline's real driver for a real file was something the list had no way to say.

Here's the redo, run properly. Gwynneth's file, same score, same 604. But now anything below 615 goes to an underwriter, the same as the 615 to 664 band already did. Halfdan reads her actual numbers: strong personal credit, steady annual income that covers the payment, and a debt calculation that penalizes her seasonal swing more than her real risk warrants. He writes the true reason himself, ties it to an adjustable line item, and tells her exactly what would move her score over. She gets an honest answer in four business days instead of an instant, hollow one.

One design let a fifteen-item list stand in for the truth, because it had always been close enough before. The other one checked what actually drove the decision, every time it mattered.

What I'd tell myself, in that first meeting: we asked whether Trueline would be more accurate than the old model. We never asked whether its reasons would still be real ones.

GUARD, run against a reason nobody could stand behind

This was never really about whether Trueline's score was wrong. It probably wasn't, by its own numbers. GUARD is for naming who pays when a model's real reason and its printed reason stop being the same thing.

Four of these five steps happened exactly as designed. The step that mattered, a person reading the actual file, was never in the path at all.

The trade-off, said out loud: routing the roughly 470 auto-declined files a month back to an underwriter instead of an instant decision adds a few business days of turnaround and costs real underwriter hours, hours that had been freed up specifically to handle Whitmarsh's growing volume. Whitmarsh took that trade on purpose, for the slice of decisions where an unverifiable reason is a genuine, uncontestable harm, and left the fast, fully automated path in place for the clear majority of files where the reason was never in doubt.

And if you want to be sure it really works, try it somewhere else

Same five letters, a trucking company's hiring model instead of a mortgage, and the same explainability gap shows up under a different regulation.

Same fork Whitmarsh faced, a completely different industry. The line an applicant can't check is still the one that decides.

Vantage Freight, a trucking carrier, uses Roadscore to auto-screen driver applicants against safety and reliability signals before a human recruiter ever sees a file. A federal law called the Fair Credit Reporting Act works a lot like Regulation B here: reject someone based on a background-style score, and they're owed a specific reason, not a vague one.

Same rank, mapped onto Roadscore: size it by what the applicant can never check for themselves. Roadscore auto-rejected candidates whose score sat just under its hiring line, and the rejection notice cited "insufficient recent driving history," a code built for Roadscore's earlier, simpler version. After Vantage retrained the model on richer telematics data, the real top factor for a growing share of near-cutoff applicants became an interaction term, hard-braking frequency weighted against the specific mix of routes they'd driven, something the old code never covered. The fix runs the same shape as Whitmarsh's: near-cutoff rejections, and anything scored soon after a model retrain, get held for a human reviewer instead of an automatic notice, so the reason on the letter is the real one.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: never let a model generate a regulated reason it can't verify against its own decision, full stop.
Cost: no budget this quarter to route every near-cutoff file to a person. Do the cheaper version first: show the applicant the actual computed values behind the top two or three factors, not just a category name, so at least what's printed is checkable even without a human rewrite.
The model got better, for real: Trueline's accuracy on seasonal-income applicants improves next year, catching more real approvals than before. That's a reason to re-test whether the reason-mapping gap has narrowed, with fresh audit data, not a reason to assume a better model quietly fixed its own explainability on the way up.

Where people run it wrong.
They treat a confident, official-sounding sentence as proof it's also the true reason, when sounding certain and being verified are two different things a model can produce separately.
They let overall model accuracy stand in for "explainable," when nobody ever checked whether the stated reason for a specific decision actually matches what drove it.
They wait for a denied applicant's phone call, or a regulator's exam, to reveal the gap, instead of asking before launch whether the team can reconstruct a real reason from the file alone.

How to use it live. Before answering, ask out loud: "if this exact applicant asked for the specific number behind this decision, could anyone here actually give it to them?" If the honest answer is no, that question alone is most of the real answer.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0