How do you handle a model that is best for one customer segment and worst for another?

Auberdine Foodservice Supply delivers food and paper goods to restaurants and cafeterias. StockSense is the model that predicts how much each account will order next week, from a chain of forty locations down to a single independent diner. Marisol Etxeberria is the AI PM who has to decide what to do with a model that is excellent for one kind of account and shaky for another.

How to answer this, stage by stage

Nobody is scoring whether you know that segments exist. They're scoring whether you can name which segment's errors are actually the expensive ones, and commit to a pick before you explain why.

Let's learn

The label printer at Auberdine's Ohio warehouse prints a pick list before the sun's up, one for every account on the truck that day.

Before StockSense, a rep built each account's weekly order forecast by hand, about 25 minutes per account. With StockSense, that same forecast comes back in under a minute for all 1,400 of Auberdine's accounts, chains and independents alike.

The four letters, held up as one page. Cost asymmetry is the one most segment problems skip.

Here's the turn: StockSense scores 94 percent accuracy on large chain accounts and 71 percent on small independent accounts, and averaged across the whole book, that looks like a fine 85 percent. The average hides that one segment's mistakes are a phone call, and the other segment's mistakes are a relationship that ends without a word.

At its worst, an independent restaurant runs out of a key ingredient on a busy Friday because StockSense underforecast a new menu item, and the owner switches to a competitor supplier that same week, with nobody at Auberdine finding out until the account has already gone quiet for a month.

What I would leave alone: I wouldn't add a mandatory human check to mid-size group accounts, since their order history sits between the two extremes and a light spot-check already catches what little goes wrong there.

The lesson: an average accuracy number is a place to hide a segment's real problem, not a place to find one.

Now here is the same thing as a story

The short version above is what you'd say defending a routing decision. Read this one for what it felt like watching two quarters pass with nobody able to name the exact week anything changed.

Her name is Marisol. She has run demand planning at Auberdine for five years, and she can spot a supplier's own excuse before they've finished making it.

When StockSense first launched, reps loved it evenly across every account. Chains got their forecasts in seconds. Independents, which used to take the longest to plan by hand because their orders swing with local events and weather, suddenly took no time at all. For months, everyone reviewed everything lightly, and it held up fine.

Two boxes of deliberately unequal weight. Only one of them ever shows up on a dashboard.

As Auberdine's book grew from 800 accounts to 1,400 over about a year, reps simply had less time. Nobody decided, in any single meeting, to stop double-checking independent forecasts. It just happened, a little at a time: reps kept close tabs on the handful of big chain accounts that made up most of their bonus, since a mistake there was visible and painful to explain. The hundreds of small independent accounts, worth less individually, got whatever attention was left over, which was less and less.

Nobody could point to the week it changed. There was no single bad order, no one complaint that started it. Just, slowly, less scrutiny landing on exactly the accounts that needed it most.

The second step is where a thin order history quietly becomes a bad prediction, for either segment.

The real question was never whether StockSense was good enough overall. It was whether Auberdine's own reps were spending their limited checking time on the segment where a mistake actually costs the business something, instead of the segment where it just feels urgent in the moment.

The segment with the thinnest history is also the one with the weakest forecasts. That's not a coincidence, it's the mechanism.

When the automated rollout was first designed, someone said, "let's just turn it on for everyone, we'll tune it later," and it sounded reasonable, since at 800 mostly-chain accounts, there wasn't much of a gap to tune.

Rerun the same two quarters with segment-based routing in place: chains keep running on StockSense alone, independents get a rep's four-minute check before ordering, and independent churn falls back from 11 percent a quarter to 4, close to where it sat before the book of business ever grew.

Three branches, and the busiest one, independents, is the one that finally gets a human step.

What I'd tell myself, watching that churn number climb: an average accuracy score is not a plan. It's a number that looks calm right up until you ask which segment it's actually calm about.

PICK, the decision that ends the whole "which segment do we trust" argumentNot a script for distrusting the model everywhere. PICK is what tells you exactly where to spend a rep's limited attention.

The recap, one line per letter: position is routing by segment instead of one blanket setting, impact is fifty dollars against fourteen thousand, cost asymmetry is choosing to protect against the quiet expensive error over the loud cheap one, and kill criteria is the ninety percent, twelve month bar that would lift the human check.

And if you want to be sure it really works, try it somewhere elseSame four letters, event ticketing instead of foodservice. The segments swap, the quiet expensive error doesn't.

Tomasz Wybicki runs product at Larkspur Ticketing, where FraudGuard flags likely-fraudulent purchases before a ticket ships. Mapped onto PICK: position is routing by buyer type, FraudGuard runs alone on casual, one-ticket buyers, and a human reviews flags on high-volume resellers, where it's worst. Impact: a false flag on a casual buyer costs an annoyed support ticket. A missed fraud case on a reseller account costs thousands in chargebacks across dozens of tickets at once. Cost asymmetry: the casual buyer's error is loud and cheap, a complaint, resolved fast. The reseller's error is quiet until a chargeback wave lands weeks later. Kill criteria: once a specific reseller account has enough clean transaction history, the human review lifts for that account.

Staff had quietly stopped mentioning when they manually overrode FraudGuard's calls on reseller accounts, since it felt like second-guessing a tool leadership had praised. Nobody was tracking how often that override happened, or why, until a chargeback wave made the pattern impossible to ignore.

The same four checks, whether the product ships food or tickets.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "route by segment, protect the error that's hidden and expensive, not the one that's loud," and stop.
Cost: no budget for a human reviewer on the weaker segment. Say so honestly, and start with the highest-value accounts in that segment first, rather than skipping the review everywhere.
The model got better, for real: if a vendor update genuinely improves accuracy on the weak segment, that's exactly when to re-check the kill criteria, not assume the gap has already closed.

Where people run it wrong.
They watch one blended accuracy number and never ask which segment is dragging it down.
They put review effort where mistakes are loudest, not where they're most expensive.
They never write a kill criteria, so a human check meant to be temporary quietly becomes permanent, or gets dropped too early with no evidence behind it.

How to use it live. The moment an interviewer describes a model that's uneven across customer segments, ask yourself: which segment's mistakes are loud and cheap, and which are quiet and expensive? Commit to protecting the quiet one, and the rest of the routing decision follows on its own.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0