Describe the readiness review you would run before promoting a POC to production.

How to answer this, stage by stage

Seven moves. Naming who can't tell the call was made on POC-size evidence, and writing the actual checklist item, are where the real answer lives.

Let's learn

The tool is a check that runs on a mortgage file the moment an underwriter opens it, before they've read a single document themselves.

This is Northbridge Bank's newest underwriting tool. It reads the income documents in a loan file, pay stubs, W-2s, 1099s, bank statements, tax returns, and tells the underwriter whether the numbers on them actually agree with each other.

Before this tool, an underwriter checked those documents by hand. That took about 50 minutes a file, on top of everything else in a loan review, and a loan usually took 9 business days to close.

In its first round of testing, the new tool checked a file in under 6 minutes, and it agreed with what the underwriter actually decided on 98 percent of the 150 files in the test.

Here's the catch. Those 150 files weren't a random slice of Northbridge's loan book. They were last quarter's closed loans, and self-employed and gig-income borrowers barely showed up in them, because those loans usually take longer to close and most hadn't finished yet. When the team later ran the tool against a wider set that actually matched the loan book, on ordinary salaried files it still held at 98 percent. On self-employed and gig-income files, the ones with 1099s and bank statements instead of a clean W-2, its accuracy dropped to 69 percent.

At its worst, this costs Northbridge a self-employed borrower who has a perfectly good loan, and gets wrongly told their documents don't add up, from a tool that looked, on paper, ready to launch.

What I would leave alone. For an ordinary, salaried borrower, a POC file and a production file read almost the same. Rebuilding that whole segment's test set wouldn't move its score at all. Save the segmented effort for the document types that actually vary.

The lesson. One blended number can be exactly right and still hide a group it was never tested on. A readiness review that only checks the average is checking the easiest part of the question.

Now here is the same thing as a story

Read the short version above if you're pressed for time. Read this one when you want to feel why segmenting a number matters, not just know that it does.

Ronke Halton has run product for Northbridge Bank's underwriting team for four years. She's the one people ask to turn a vague compliance requirement into something an underwriter can actually click through in a morning.

When the mortgage-document tool was ready for its first real test, Ronke pulled the 150 most recently closed loan files. It was the fastest data to get to, the pilot review was three weeks out, and every file in that batch had already gone through a full underwriting cycle, so there was something real to grade the tool against.

For three weeks, this went fine. Every batch she reran, the tool agreed with the underwriter's actual decision 97, 98 percent of the time. Her analytics lead called the result strong in a review meeting. The underwriting committee started asking when it could touch a live file.

At first, Ronke checked which document types made up each new batch before she reran the test, a W-2 here, a 1099 there, roughly matching what she remembered from the loan book. Then she stopped checking the mix and just reran the same 150 files, since the number kept holding. Then, in her final readiness memo to the committee, she reported one line: 98 percent agreement with underwriter decisions. No mention of what kind of file that 98 percent came from.

Then came a Tuesday, three months after launch.

The bank's quality team ran its regular quarterly audit, pulling 40 declined loans at random to check the underwriters' work, a routine the bank had done for years, long before the tool existed. Fourteen of the forty were self-employed or gig-income borrowers. Most of them had been flagged "inconsistent" over things a person would have waved through, a 1099 total that didn't quite match a bank deposit because of a processing fee, a bank statement covering a different month than the pay period.

Same loan decision. Two very different starting points.

Ronke didn't wave it off. She pulled three months of override logs and broke them out by document type herself, something the launch checklist had never asked anyone to do. Ordinary salaried files: the tool's calls matched what an underwriter would have done, 98 percent of the time. Self-employed and gig-income files: 69 percent.

I want to say the problem is that Ronke cut a corner. She didn't, not really. She never had a number in her head either, just a feeling, the pilot is working, or it isn't. Three straight weeks of 97, 98 percent had switched her feeling to "it's working," and there was no setting in between where she kept asking who that number actually covered.

Back when the eval plan got signed off, in a meeting with her analytics lead and the launch date already on the calendar, the question on the table was simple: is the tool accurate enough to trust with real files. The answer everyone reached, sensibly, was to test it against whatever closed loans were sitting ready, since that was real underwriting data and nothing needed to be invented. Nobody in that room asked whether "closed last quarter" matched the loan book the tool would run on once it was live.

I would go back and put that question in the room. Ronke tests against a sample that matches the real mix from the start, and the 69 percent shows up in week one of the pilot instead of month three of production, before a single self-employed borrower gets a declined loan they never really understand.

One version of the checklist tells you the number is good. The other tells you what the number is actually good at.

What I'd tell the Ronke who wrote that first eval plan: "fast to pull" and "looks like our loan book" were never the same test, and I only ever built the first one.

GUARD, run against a launch decision built on one blended number

This is a risk question, so the framework is GUARD. "What readiness review would you run" sounds like a project-management checklist question, which is exactly why it's easy to answer with process instead of a real check.

The check that should sit here, and doesn't.

And if you want to be sure it really works, try it somewhere else

A county benefits office's eligibility-screening tool runs into the same gap, in a room with nothing to do with mortgages.

G, groups. Vesna Castro, a program analyst who builds a solo tool to screen benefit applications for Northfield County, using the 200 most recently closed cases because they were the fastest data to pull before her pilot review. And the county's applicants, who inherit whatever the tool decides once the board sets a countywide rollout date.
U, unequal. An applicant who filed everything in English, in person, at the counter, gets screened by a tool tested almost entirely on cases like theirs. An applicant who needed an interpreter or submitted documents in another language, a group that's 22 percent of the county's real caseload but only 4 percent of Vesna's 200 test cases, gets screened by a tool that barely saw a case like theirs.
A, ability to contest. The county board approves the rollout because the pilot scored 95 percent agreement with the caseworkers' own decisions. Nobody on the board asks what share of that pilot came from interpreter-needed cases, so nobody there can tell the number was never really tested on them.
R, reduce. The same practice: build a segmented sample sized to the county's real caseload mix, English-only and interpreter-needed alike, and require both segments to clear the bar on their own before the tool goes live.
D, detect. After launch, track the appeal rate by whether the case needed an interpreter, every month, not just overall. A rising appeal rate in one group while the countywide rate looks steady is the flag.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0