When should a human be required to approve an AI action rather than merely able to?

Brookline Loan Servicing uses PathForward, a tool that reads a homeowner's hardship file and recommends approving a repayment plan or denying it and moving the account to the foreclosure track. Delphine Achebe is a loss-mitigation specialist who has worked hardship files for nine years, and who nearly let one bad denial reach a real family.

How to answer this, stage by stage

Nobody is grading whether you can define "human in the loop." They're grading whether you can name the actual line that decides when a checkbox isn't enough.

Let's learn

What happens the moment a machine's decision can't be undone?

PathForward reads a homeowner's hardship file, income, expenses, the reason for the missed payments, and recommends approving a new repayment plan or denying it and moving the account toward foreclosure. Before it, every one of Brookline's roughly 1,400 monthly hardship files went to a specialist team of nine, each file taking about thirty-five minutes to read start to finish, a six-day average turnaround.

Slow, but every single file passed through a person before anything was mailed.

Now PathForward clears about two-thirds of files on its own in under a minute, and only sends the harder or more uncertain ones to Delphine's team, whose average review time has dropped to nine minutes.

Here's the turn: the occasional wrong call was never really the danger. The danger was a design choice that let a denial and the next collections step fire in the same motion, so by the time anyone might catch the mistake, the account was already moving.

The third box used to be a person. Merging steps quietly removed it.

At its worst, a family gets a foreclosure notice mailed before anyone at Brookline even learns their file was misread, and by the time a specialist catches it on appeal, real damage, credit, legal fees, stress, is already done.

One of these two people can stop the letter. The other only finds out it was sent.

What I would leave alone: auto-approved repayment plans don't need a mandatory gate. A wrongful approval costs Brookline money, not the homeowner, and it's easy to catch and correct later. Not every AI decision needs the same brake pedal.

The lesson: the question was never whether PathForward is usually right. It's whether the person on the receiving end has any real window to catch it when it's wrong.

Now here is the same thing as a story

The short version above is what you'd say defending this policy to Brookline's compliance committee. Read this one for how close the near miss actually came.

The loss-mitigation floor at Brookline gets loud around month-end, when the hardship queue fills up ahead of the next payment cycle. Delphine Achebe has worked that floor for nine years, and she is the one who once caught a self-employment income mismatch that would have denied a nurse's repayment plan over a clerical rounding error nobody else had noticed.

For its first four months, PathForward's volume stayed small enough that Delphine's team saw nearly every flagged file the same day it came in. The system worked exactly as designed: model decides, decision executes, letter mails, all in one motion, because volume was low enough that nothing ever really got missed.

The design didn't change between month one and month five. The volume running through it did.

By month four, hardship volume had nearly tripled after a rate reset hit thousands of adjustable loans at once, and PathForward's denial-and-referral motion kept firing at the same speed as always, just far more often.

The same word, "review," described two very different guarantees, and nobody had picked which one this case needed.

In month five, a new hire on Delphine's team asked her, almost in passing, why a denial letter for a widow's file had already gone out before anyone flagged the income document looked incomplete. Delphine pulled the file. The income document was, in fact, incomplete, missing a pension statement that would have qualified the woman for the plan she'd asked for. The denial, and the foreclosure-track referral behind it, had already mailed two days earlier.

Delphine caught it in time to intercept the referral and reverse the denial before the foreclosure track actually advanced, but only because a new hire happened to ask an odd question on a slow afternoon. An audit of the prior six weeks found four more files where the same gap existed, unnoticed, because nothing had gone wrong loudly enough yet.

A file only needs all four rarely. A denial that also starts foreclosure usually has every one of them.

With the redesigned policy, any denial that also triggers a foreclosure-track referral or a credit bureau report is blocked until a specialist actively signs it, a real click, on a real screen, not a default. Run the same month forward: the widow's incomplete file gets caught at the gate itself, before any letter exists to intercept, and the file is resolved in one day instead of nearly reaching a real foreclosure step two days after the fact.

The old design asked whether a person could look. The new one asks whether a person had to.

I let the denial and the referral merge into one motion because splitting them felt like giving back the speed we'd just built. It took a new hire's offhand question to see that speed was never the thing worth protecting on the file that turns out to be wrong.

GUARD, the letters that decide who gets a leverNot a lecture on fairness in the abstract. GUARD is what tells you exactly when "able" stops being enough.

Two questions, asked every time: can it be undone, and does the person affected have time to ask first.

The recap, one line per letter: groups is the specialist who holds the lever and the homeowner who doesn't, unequal is a wrongful denial's cost landing entirely on the homeowner, ability to contest is the missing pause once decision and action got merged, reduce is splitting them back apart with a real sign-off, and detect is watching how long that sign-off actually takes.

And if you want to be sure it really works, try it somewhere elseSame five letters, a county unemployment office instead of a mortgage servicer. A different harm, the same missing pause.

Marrow County's unemployment office uses a model that flags benefit claims as likely eligible or likely fraudulent based on wage-reporting patterns. Anders Kessler reviews the claims it flags, and until recently, a likely-fraudulent flag froze a claimant's payments the same day, automatically, with a form letter mailed a week later.

Mapped onto GUARD: groups is Anders, who can release a frozen payment any time he wants, against a claimant who has no way to contest a freeze before their rent is due. Unequal is that a false fraud flag can cost someone their apartment before the appeal letter even arrives. Ability to contest is the real gap: the freeze and the mailing both fired the moment the model flagged the claim, with no pause for a person to look first.

Same shape, a different office. Before either system, a person read the file before anything moved.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "require approval exactly where the action is hard to undo and outruns the person it affects, leave the rest merely reviewable," and stop.
Cost: there's no budget to add reviewers to every gate this quarter. Say so honestly, and gate only the subset of denials that also trigger foreclosure or fraud freezes, since that's where irreversible harm actually concentrates.
The model gets better, for real: even a model that's rarely wrong still needs the gate, since the whole point was never accuracy, it was giving the affected person a window that used to not exist at all.

Where people run it wrong.
They make everything "able to review" and call it human-in-the-loop, without asking whether anyone actually has time to use that ability before the harm lands.
They gate everything equally, which burns out the very reviewers meant to catch the cases that matter, on cases that never needed a brake pedal.
They ship the gate and never check whether it's still a real look or a five-second click nobody would notice was automatic.

How to use it live. When someone asks when a human must approve, ask two questions back: can this be undone, and does the person it affects have any real time to object before it moves. If both answers are no, that's your required gate.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0