How do you test a replacement model against the behaviour users have come to expect?

How to answer this, stage by stage

Nobody is grading whether you can say "run an A/B test." They are grading whether you can size a real equivalence test that would actually catch a dangerous gap before it reaches a pharmacist's counter. Seven moves get you there.

Let's learn

Say we build a model that checks every prescription against a patient's other medications and flags a likely dangerous interaction before the pharmacist hands it over. Before a tool like this existed, pharmacists cross-checked interactions from memory and a printed reference guide, fast for common combinations, slow and easy to miss for anything unusual.

Then Wayfinder deployed Vigil. Every one of the roughly 48,000 daily prescriptions gets checked automatically, about 3.2 percent trigger some kind of interaction alert. Pharmacists stopped relying purely on memory for common combinations and started trusting the alert to catch what they might not think to check.

Now Wayfinder is migrating from Vigil to Custodian, a newer interaction model that's faster and, on internal testing, generates fewer false alarms overall, alerts pharmacists have started to tune out precisely because there are so many low-stakes ones. Fewer false alarms sounds like an unambiguous win. But "fewer alarms overall" says nothing on its own about whether the alarms it does drop are the safe ones to drop.

Say plainly: a lower false-alarm rate is not the problem. The problem is that nobody can tell, from that headline number alone, whether Custodian achieved it by getting smarter about which alerts are truly low-risk, or by quietly getting a little worse at the rare, dangerous combinations that almost never show up in a spot check.

At its worst, Custodian goes live chain-wide having genuinely improved on the common cases while quietly missing the specific MAOI-decongestant combination it was never really tested against, because that combination occurs so rarely in real dispensing history that no amount of "just watch the live data" would have caught it before a patient was harmed.

The choice I would take back. When Custodian's evaluation was first scoped, the plan was to replay the last month of dispensing history, about 1.4 million prescriptions, through both models and compare overall alert rates. That felt exhaustive. It also meant the rare category, occurring maybe four times a year chain-wide, would show up zero or one times in that entire month, nowhere near enough to say anything meaningful about it either way.

What I would leave alone. Wayfinder also runs a much lower-stakes model that flags when a prescription's dosage looks unusually high for a patient's weight bracket, a useful sanity check but rarely the sole line of defense against a serious error. A pure statistical replay test is genuinely sufficient there; there's no category rare enough to need a synthetic set of its own.

The lesson. A test built from real history will only ever be as good as what real history happens to contain. If the case you most need to test for is too rare to show up naturally, the test has to be built on purpose, or it will quietly never happen at all.

Now here is the same thing as a story

The short version is above. Read on if you want to feel how close a one-month replay came to calling the migration done.

Bhavna Sowa has run Wayfinder's clinical-decision-support roadmap for four years. She was a practicing pharmacist for a decade before moving into product, which is why she's usually the person in the room who asks "but how do we know" one layer past where everyone else stops.

Custodian's migration plan initially called for a one-month replay of historical prescriptions through both models, comparing overall alert rates and calling it equivalence-tested if the numbers landed within two percentage points of each other. On that measure, the plan looked airtight: 1.4 million prescriptions, real data, a full month, an industry-standard-sounding tolerance.

Bhavna asked one question before signing off: how many times did the MAOI-decongestant combination, one of the handful of truly dangerous interactions on Wayfinder's watch list, actually appear in that month of data. The pharmacy informatics team ran the count. Zero. Not one occurrence in 1.4 million prescriptions.

Bhavna paused the sign-off and asked the clinical pharmacology team to build a synthetic test set instead: every known dangerous drug pairing on Wayfinder's formulary, constructed as realistic prescription scenarios, confirmed correct by a licensed pharmacologist rather than pulled from real patient records. It took the team about three weeks to build and clinically validate 200 such scenarios, MAOI-decongestant among them.

Four separate things, not one blended "does it match" question. The rare-category item is the one a pure statistical replay could never have answered.

On that synthetic set, Custodian caught 194 of the 200 scenarios, a 97 percent catch rate. Not perfect, but honestly measured, on purpose, against exactly the cases that mattered most, rather than left as an unknown hiding behind a reassuring one-month replay number.

The six misses, once reviewed, all shared a pattern: prescriptions where the dangerous combination involved a less common brand-name formulation of one of the drugs rather than the generic. Custodian's training data had simply seen fewer examples of that brand name. That's a fixable, specific gap, the kind of finding a real test surfaces and a reassuring aggregate number never would have.

What stayed with Bhavna wasn't the six misses themselves, it was how close the original plan came to calling the migration equivalence-tested without ever actually testing the one category where a miss could genuinely hurt someone. At the original kickoff, someone had proposed building a synthetic rare-case set from the start. It was set aside as extra work for a migration that already had "a full month of real data" to point to.

The redesigned testing plan now runs two tracks from day one of every future migration: a statistical equivalence test on common interactions, sized by a real minimum detectable difference, and a synthetic, pharmacologist-built set for every interaction on the chain's designated high-severity watch list, refreshed whenever a new dangerous pairing gets added to that list. Run the same Custodian migration through that design from the start, and the brand-name gap gets found and fixed three weeks earlier, before cutover instead of during a pre-signoff spot check that almost didn't happen.

The thing Bhavna would tell her past self, back at that kickoff meeting: a month of real data feels thorough right up until you ask how many times the case that matters most actually showed up in it.

BOUND: the arithmetic behind the equivalence test

This is a sample-size and test-design question, how much evidence is actually needed to trust a replacement model, not a person's habit flipping between two settings, so BOUND fits and FLIPS doesn't.

B, break it down. Two separate sample sizes, not one. For a common interaction, standard two-proportion statistical test sizing: driven by the baseline alert rate, the smallest gap worth catching, and the confidence and power required. For a rare, dangerous interaction, natural incidence is too low to sample in any usable window, so the "sample size" becomes a synthetic set built from every known dangerous pairing instead.
O, own the numbers. Warfarin-NSAID occurs in about 1.1 percent of relevant prescriptions, roughly 528 cases a day. Wanting to catch a 3-point difference in alert rate at 95 percent confidence and 80 percent power needs about 1,568 historical cases, about three days of dispensing history. The MAOI-decongestant combination occurs only a handful of times a year chain-wide, confirmed by an actual count: zero occurrences in a full month of 1.4 million prescriptions. The synthetic set instead uses 200 pharmacologist-confirmed scenarios covering every known dangerous pairing on the watch list.
U, use a range. Tightening the common-category minimum detectable difference from 3 points to 1.5 roughly quadruples the sample, to about 6,270 cases, near twelve days of history. Loosening it to 5 points drops the requirement to about 565 cases, under a day. The synthetic set's size is less about statistics and more about coverage: 200 scenarios covers every currently known dangerous pairing on the watch list, and grows only as new pairings get added to it.
N, nail the sanity check. Wayfinder's informatics team already replays about 2,000 historical prescriptions a month as routine practice for clinical rule changes; 1,568 cases in one batch is a single day's computational work. Building and clinically confirming 200 synthetic scenarios took about three weeks of pharmacologist time, the real bottleneck in the whole plan.
D, direction. The minimum detectable difference is the single biggest lever on the statistical piece, roughly quadrupling the sample when tightened by half. But the assumption that swings the actual project calendar most is how many pairings sit on the high-severity watch list, since each one needs its own pharmacologist-confirmed scenario, and that list, not the statistics, is what should get scrutinized hardest before committing to a migration date.

One more thing the arithmetic alone doesn't show: 97 percent on the synthetic set isn't treated as a pass on its own, it's compared against Vigil's own catch rate on the identical synthetic set, which came out to 96 percent. Custodian isn't required to hit some absolute number pulled from nowhere, it's required to match or beat the model it's replacing, on the same test, so the bar moves with reality instead of sitting fixed at a number nobody derived from anything.

And if you want to be sure it really works, try it somewhere else

Selvedge Textile Mills runs an AI tool that inspects camera footage of woven fabric on the line and flags likely defects before the roll ships to a garment maker. The mill is migrating that classifier to a newer version.

B, break it down. Same two-part shape. A common, low-stakes defect, like a minor weave slub, has plenty of real inspection history to run a statistical equivalence test on. A rare, severe defect, a foreign-fiber contamination that can trigger a customer recall, occurs too rarely in real footage to test that way, so it needs a constructed test set instead.
O, own the numbers. Weave slubs show up in about 2.4 percent of inspected fabric, roughly 340 flagged segments a day across the mill's lines. Catching a 3-point alert-rate difference at 95 percent confidence needs about 1,510 historical segments, under a week of footage. Foreign-fiber contamination happens only a few times a year mill-wide; a month of footage review found zero real examples. Ozzy Sorrel, the PM running this migration, had the quality team build 60 synthetic contamination scenarios instead, staged fabric samples with known foreign fibers deliberately introduced, filmed and confirmed by a QA lead.
U, use a range. Tightening the slub detection threshold to a 1.5-point difference roughly quadruples the sample to about 6,040 segments, over three weeks of footage instead of under one. The synthetic contamination set's size is bounded by how many distinct fiber types the mill's insurer requires coverage for, not by statistics.
N, nail the sanity check. The mill's QA team already reviews about 1,200 flagged segments a month as routine practice, so 1,510 in one batch is a manageable single week. Staging and filming 60 synthetic contamination scenarios took about two weeks of QA lead time, the real bottleneck, same shape as Wayfinder's pharmacologist hours.
D, direction. Same lever as the pharmacy case: the detectable-difference threshold swings the statistical sample size most, but the project calendar is actually set by how many rare defect types the synthetic set needs to cover, since each one needs a real, physically staged example, not just a number typed into a spreadsheet.

Swap the trigger and it still runs.
Speed: an interviewer caps the answer at ninety seconds. Skip straight to the split: statistical test for common cases sized by a real detectable-difference threshold, synthetic constructed set for rare dangerous cases since real history won't provide enough.
Cost: the pharmacology team can only spare limited hours this quarter. Don't shrink the synthetic set's coverage to fit the budget, shrink the common-category statistical sample instead by accepting a looser detectable difference; that's a real, statable tradeoff, not a silent one.
The model got better: Custodian turns out to beat Vigil on both tracks, including the rare category. That doesn't make the two-track test unnecessary, it's still the only way anyone would have known that for certain, rather than assumed it from a reassuring aggregate number.

Where people run it wrong.
They size one sample from real history and apply it to every category, when a rare category needs a completely different testing approach, not just a bigger version of the same one.
They treat "a full month of real data" as inherently thorough, without ever checking how many times the case that matters most actually appeared in it.
They set a flat, deterministic "must always catch it" bar for a probabilistic model, instead of a calibrated pass rate measured against the model it's replacing.

How to use it live. Say the split before naming a single number: "matching expectations means two different kinds of proof, a statistical test for what's common enough to have real history, and a constructed test for what's too rare to ever show up in it." That buys the room to ask what's actually on this product's own rare-and-dangerous list, instead of reciting a generic A/B testing checklist.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0