How do you decide which cases get routed to a human?

Union Trust Bank runs WatchLine, a tool that scores every transaction for money-laundering risk and builds the day's alert queue. Folasade Adeyemi is a financial crimes investigator who reviews the alerts that make it through, out of roughly 50,000 scored every month.

How to answer this, stage by stage

Nobody is grading whether you can name a scoring model. They're grading whether your ranking survives the one case built specifically to duck it.

Let's learn

Picture fifty thousand transactions scored every month, and room to actually look closely at maybe two thousand of them.

WatchLine reads every wire transfer and large transaction Union Trust processes and scores it for money-laundering risk. Before it, a fixed-rule engine flagged around 8,000 transactions a month against simple triggers, round-dollar amounts, sanctioned-country lists, and investigators cleared 92 percent of those as false positives, exhausted by volume that mostly went nowhere.

Each box had to exist before the next one could. The fix wasn't a switch, it was a sequence.

Now WatchLine scores all 50,000 alerts and routes only the riskiest slice, about 2,000 a month, to a human, a queue Folasade's team can actually work through carefully instead of skimming.

Here's the turn: the false positives that stayed in the queue were never the real danger. The danger was a genuinely new laundering pattern, built specifically to not resemble anything WatchLine had ever flagged, scoring calm and ordinary precisely because nothing in its training data looked like it either.

Score alone can't tell these two apart. Only asking about the action itself can.

At its worst, a genuinely new laundering structure runs for months underneath a queue that only ever asked "does this look like trouble we've seen before," and the funds are gone, overseas, well before any human ever looks at the case.

What I would leave alone: routine, well-understood transaction types, common wire patterns the model has scored thousands of times with a strong track record, don't need this override. The fix targets the case built to look ordinary, not every alert equally.

The lesson: a ranking built entirely on resemblance to the past will always have a blind spot shaped exactly like whatever hasn't happened yet.

Now here is the same thing as a story

The short version above is what you'd say defending this routing change to Union Trust's compliance committee. Read this one for how the gap actually got found.

Folasade Adeyemi has investigated financial crime at Union Trust for seven years, and she is the one who once caught a structuring pattern, several deposits each kept just under a reporting threshold, that a junior analyst had cleared as unrelated coincidence.

Under the old routing rule, WatchLine's top four percent by score reached Folasade's team every month, and for most of a year, that rule caught real cases at a rate the compliance team was proud of. The rule wasn't wrong. It was just answering only one question.

Score alone only answers the last branch on this tree. It was being asked to answer all four.

A new laundering ring structured its transactions around what the model had already learned to consider safe, small, varied amounts, unusual but plausible-sounding business justifications, timing spread out enough to avoid any pattern WatchLine had seen before. Every one of those transactions scored in the bottom half of the queue, alongside thousands of genuinely routine ones.

Nobody caught it during the run itself. It surfaced when a partner bank in another jurisdiction flagged a linked account during its own routine review and reached out to Union Trust to compare notes, revealing months of transactions that had cleared Folasade's queue without ever reaching it.

The fix took three real months to build, in order, because each step genuinely needed the one before it.

With the redesigned routing, an alert now routes to a human if it's flagged as statistically unlike anything in Union Trust's known-pattern library, independent of its score, alongside anything tied to a fast-moving, hard-to-reverse transfer, and anything a fixed legal threshold already requires. Run the same laundering pattern forward: its unfamiliarity, the exact thing that let it score low, is now the reason it routes to a human within days, not the reason it stays hidden for months.

Score is still on the list. It's just fourth, not first.

The old queue asked how alarming something looked. The new one also asks how unfamiliar it is, and stopped assuming those were the same question.

I built the routing rule around score percentile because it backtested beautifully against every case we already knew about. It took a call from another bank's compliance team, not our own audit, to see that "backtests well" and "catches what hasn't happened yet" were never the same guarantee.

ORDER, the four questions that decide the queueNot a scoring debate. ORDER is what tells you score should never go first.

The recap, one line per letter: outcome is catching real laundering without drowning in false positives, reversibility is ranking the wire that can't be recalled above the alert that can wait, dependency is the pattern library that had to exist before novelty detection could, evidence is the cheap retrospective check before the full build, and rank is novelty, reversibility, legal floor, then score.

And if you want to be sure it really works, try it somewhere elseSame five letters, an airline's baggage claims desk instead of a bank's crimes unit. A different kind of irreversible.

Falconbridge Airways scores mishandled-baggage claims for likely fraud or error, deciding which claims auto-pay and which route to Rowan Achterberg, a baggage claims agent, for a manual look before any payout goes out.

Mapped onto ORDER: outcome is paying legitimate claims fast without funding serial fraud. Reversibility is that a payout issued as an instant digital gift card is effectively gone the moment it's redeemed, while a claim still pending review can be adjusted freely. Dependency is that detecting a repeat-claimant pattern needs a claimant history database, which had to exist before that rule could route anything at all. Evidence is a cheap backtest checking how many past confirmed fraud cases a repeat-claimant rule would actually have caught. Rank: repeat-claimant pattern first, irreversible payout method second, legal or contractual claim deadlines third, score fourth.

Swap "laundering pattern" for "repeat claimant," and the same dependency chain holds up in a different building.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "route by novelty and reversibility before score, since score only measures resemblance to what you've already caught," and stop.
Cost: there's no engineering budget to build a novelty scorer this quarter. Say so honestly, and start with a manual flag for any transaction type or claim type nobody has categorized before, a cheap proxy while the real system gets built.
The model gets better, for real: even a much more accurate score doesn't fix this on its own, since accuracy is measured against cases you've already seen, and the blind spot lives specifically in the cases you haven't.

Where people run it wrong.
They treat the model's score as the entire routing decision, instead of one input among several.
They build novelty detection last, as a nice-to-have, when it's actually the piece that catches what everything else is designed to miss.
They wait for an external tip, a partner bank, a regulator, a customer complaint, to reveal the gap, instead of sampling their own auto-cleared cases on a schedule.

How to use it live. When someone asks how to route cases to a human, ask what a case would look like that was built specifically to score low. If your ranking has no answer to that, score is doing too much of the work.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0