How do you balance a refusal rate that is too high against one that is too low?

Meridian Bank built the Meridian Assistant to answer account questions inside its banking app: balances, card disputes, password resets, and requests to move money. Farah Al-Rashid is the trust and safety product manager who owns the setting that decides when the assistant answers and when it says it can't help.

How to answer this, stage by stage

Six moves. Say them close to this, in your own words, and you'll sound like someone who has actually shipped a refusal setting before.

Let's learn

Every month, Meridian's branch call center used to field about 40,000 questions that had nothing wrong with them at all: "what's my balance," "can you reset my login," "is this charge mine."

The Meridian Assistant launched to take those off the phone lines. At first it refused about 3 requests in 100, mostly ones it genuinely couldn't parse. Customers who got refused just called the branch, same as always. It cost a few minutes and nobody minded much.

Then, one week, the assistant confirmed a change to a joint account's contact details for someone who wasn't actually on that account. Nobody had asked it to be extra careful about ownership changes specifically. It had just never been tested on one.

Compliance's fix was blunt: tighten the refusal threshold everywhere, for every request type, all at once. The refusal rate jumped from 3% to 22%.

Here is the turn. The extra refusals were not, on their own, the real problem. The real problem is that "everywhere" swept up balance checks and password resets right alongside wire transfers and ownership changes. Customers asking the easiest, cheapest questions started getting bounced to the call center too, and call volume climbed back toward where it had started before the assistant ever launched.

At its worst: the assistant became more expensive to run than not having it at all, while a genuinely risky request could still slip through, because the tightening was blunt instead of targeted.

What I would leave alone: the low-stakes tier doesn't need a strict bar even after the incident. A wrong balance figure or a failed password reset costs a phone call, and tightening that tier only pushes customers back to the call center for no safety gain at all.

The lesson: a refusal rate is not one thing to get right. It's a rate that should look completely different depending on what's actually being asked. Chasing one target number is the mistake, not whichever number you land on.

Now here is the same thing as a story

The short version above is what you'd say out loud in the interview. Read this one for how Farah actually found her way to the fix.

Farah Al-Rashid has run trust and safety for Meridian's digital products for five years. She built her reputation catching the small stuff, the setting nobody else thought to check, before it became a headline.

For the first eight months after launch, the Meridian Assistant was a quiet success. It handled about 78,000 requests a month, refused about 3 in 100, and the call center's easy-question volume dropped by nearly a third. Farah checked the refusal dashboard every Monday, saw the same small number, and moved on to other work.

The middle three milestones are the whole story: one blunt tightening, then an audit that found it hadn't actually solved anything.

Then came the joint-account incident. A caller had asked the assistant to update contact details on an account, and the assistant confirmed the change without checking that the caller was actually listed on that account. Nobody got hurt this time. But it could have gone the other way, and everyone in the room after the incident knew it.

The fix that got approved that same week was the simplest one anyone could agree on fast: raise the refusal threshold, for everything, until the model's confidence had to clear a much higher bar before it answered anything at all. Refusal rate went from 3% to 22% inside a month.

Farah didn't love it, but she signed off. It felt like the safe call.

Two months later, compliance ran its regular quarterly audit, pulling a random sample of refused and answered requests. The sample told two stories at once. Complaint volume about the assistant refusing simple, harmless requests had tripled. And buried in the "answered" pile, one more high-stakes request, a request to move funds between accounts under unusual timing, had gone through without the scrutiny it should have gotten. The blunt tightening had made the easy questions much worse and had not actually closed the gap on the hard ones.

Farah's team had been treating both boxes as the same size. They are nowhere close.

We did not actually make the assistant safer. We made it worse at the easy 97% of its job, while the hard 3% stayed exactly as exposed as before.

Farah went back to the decision that had been made in that first meeting. Nobody had asked which requests actually needed the strict bar. They had asked how to make the number go up, fast, and a single global threshold was the fastest lever available.

Four questions decide the bar for a request. A single number can't hold all four at once.

She rebuilt the setting around those four questions instead of one number. Anything reversible, cheap, and not money-related got the loose bar back, close to the original 3%. Anything that moved money, changed ownership, or touched regulated advice got a bar strict enough that a single ambiguous case routes to a human, every time, no exceptions.

The requests worth guarding closely aren't the common ones. They're the rare ones sitting in the bottom left corner.

The step Farah's team skipped the first time is the second box: actually checking what kind of request this was before deciding how strict to be.

Overall refusal rate settled around 9%, but that number now means something completely different than it did before: almost nobody gets refused a balance check, and almost every wire transfer gets a second look.

What I would tell myself, a year earlier: a single refusal number was never the safety measure. It just felt like one, because it was the only number anyone was watching.

PICK, the bar Farah actually builtNot "how strict should we be." PICK forces you to say which error you're willing to eat, and for whom.

The recap, one line per letter: position is one bar replaced by a bar that reads the stakes, impact is a phone call against a fraud claim, cost asymmetry is visible-and-cheap against hidden-and-expensive, and kill criteria is a single high-stakes miss tightening that tier on the spot.

And if you want to be sure it really works, try it somewhere elseSame four letters, a city transit agency instead of a bank. Different flip family entirely this time.

Grayling Transit runs an AI assistant that answers rider questions about routes, delays, and fare disputes through the transit app. Dara Voss manages the assistant and owns its refusal setting.

Mapped onto PICK: position is that a single refusal bar for "how do I get from A to B" questions and "reverse this fare charge" questions is wrong for the same reason it was wrong at Meridian. Impact: a wrongly refused route question sends a rider to a paper schedule for two minutes. A wrongly approved fare reversal, done automatically without a stated reason, quietly drains real revenue and can't be told apart from a legitimate one until someone reconciles the books weeks later. Cost asymmetry: route refusals show up instantly in app-store reviews, fare reversal abuse shows up only in a monthly finance reconciliation, so the finance team was the last to know something was wrong. Kill criteria: if reconciliation ever finds fare-reversal abuse above a set dollar threshold in a month, every automatic reversal above ten dollars gets a human check, no exceptions, until the pattern is understood.

This time the flip underneath isn't a refusal-rate story at all. It's a substitution flip: once riders learned automatic reversals were easy to get, word spread, and people started requesting reversals for charges that were actually correct, because the easy path was right there and nobody was watching who used it.

The shape repeats even though the flip underneath it doesn't: a blunt, one-size rule breaks under real volume, and only a context-aware rule survives it.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "one bar can't fit every stake, so I'd split it by reversibility and whether money moves," and stop.
Cost: there's no engineering time this quarter for a tiered system. Say so honestly, and start with the smallest possible split: one strict rule for anything that moves money, everything else keeps today's bar.
The model gets better, for real: if accuracy improves across the board, that's still not a reason to merge the tiers back into one bar. A better model still deserves a stricter check on the handful of requests where being wrong is expensive.

Where people run it wrong.
They chase a single target refusal percentage instead of asking what that percentage is actually made of.
They tighten everything after one incident instead of tightening only the slice the incident actually came from.
They watch complaint volume as if it were the whole picture, when the expensive error doesn't generate a complaint at all.

How to use it live. If someone pushes you on "but what refusal rate is correct," don't give them a number. Ask them back which requests they're asking about, since the honest answer changes completely depending on what's actually being asked.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0