Describe the infrastructure work that is worth doing regardless of which model wins.

Verrow Pharmacy runs a tool that checks drug interactions and drafts plain-language explanations for patients before a prescription gets filled. Corwin Aldrich leads the clinical-systems product team, and the question in front of him is what infrastructure to build now, before anyone knows which underlying model Verrow will be running in two years.

How to answer this, stage by stage

Nobody is scoring whether you can guess which model architecture wins. They're scoring whether you can name what to build before that question even gets answered.

Let's learn

Say we build a tool that checks whether a new prescription will interact badly with what a patient already takes, and writes up a plain-language note explaining any risk.

Before it existed, a pharmacist checked interactions by hand against a printed monograph, a few minutes per prescription, catching most issues but occasionally missing a subtler one buried in a patient's full drug history. With the tool, checks happen in seconds, and pharmacists began trusting its patient-facing explanations enough to hand them over directly, without rewriting them.

None of these three depend on which model Verrow ends up running. All three would have caught the same problem regardless.

Here's the turn: the tool's reported accuracy, ninety-six percent against its own golden set, was never the real number. The golden set was built almost entirely from single-drug queries, because those were easy to generate and easy to label. The real pharmacy queue is full of multi-drug, multi-condition cases, and once you measure accuracy against that real mix instead of the golden set, coverage on the hard cases drops to sixty-one percent.

At its worst, a pharmacy chain can run for two years believing its AI tool is catching ninety-six percent of interactions, while the cases that actually carry the most clinical risk are quietly the ones it was never really tested on.

What I would leave alone: I wouldn't touch the single-drug interaction checks. They were genuinely well-tested from day one, and adding more infrastructure there would just be process for its own sake.

The lesson: infrastructure that would have caught the problem doesn't need to know which model is running. It needs to know what the real input actually looks like.

Now here is the same thing as a story

The short version above is what you'd say defending a $180,000 infra ask to a budget committee. Read this one for how a routine audit uncovered a habit nobody had asked anyone to build.

Odessa Finch has filled prescriptions at a Verrow location for nine years, and she can read a patient's full medication list and flag a risky combination almost on instinct.

When the tool first launched, Odessa typed in a patient's entire medication list exactly as it appeared in the chart, three drugs, two chronic conditions, whatever was actually there. For the first several months, this worked fine on the simple cases and produced vague, unhelpful answers on the complex ones, answers that weren't wrong exactly, just thin, generic, unsatisfying.

Every step Odessa took was sensible on its own. Together, they made the hardest cases invisible to the tool.

So, without anyone training her to do it or asking her to, Odessa learned the tool's shape. When a case felt complex, she started splitting it: one query for drug A against the patient's main condition, a second query for drug B, a third for drug C, each one simple enough that the tool answered cleanly and confidently. The actual three-way interaction between all of them together never got asked about at all, because there was no single query left that contained the whole picture.

The tool was measured against the left panel and used, every day, against the right one.

A routine compliance audit, the kind that happens every eighteen months regardless of whether anything looks wrong, pulled two hundred randomly sampled interaction checks from across Verrow's locations for manual review.

Nothing about any single step looked like a violation. Together, they meant a third of the hardest cases were never really checked.

The audit found that thirty-four percent of multi-drug cases, three or more medications, had been pre-split into separate single-drug queries before ever reaching the model, compared to zero percent of naturally simple cases. Nobody had told pharmacists to do this. They'd each discovered it independently, the same way Odessa had, because the tool quietly failed soft on complexity instead of saying so.

What I'd tell myself, reading that audit finding: the model never had to fail loudly for this to happen. It just had to fail quietly enough that a group of careful, competent people invented their own workaround, one at a time, with nobody ever telling them the tool had a blind spot in the first place.

BOUND, the number that doesn't wait on the model to be chosenNot a checklist of best practices. BOUND is what turns "build good infra" into a defensible number with real assumptions behind it.

The recap, one line per letter: break it down is the three-layer equation, own the numbers is 70k, 60k, and 50k from comparable builds, use a range is $350,000 to $470,000 for a bad migration, nail the sanity check is $180,000 against one bad migration inside a 14-to-16-month swap cycle, and direction is model swap frequency as the assumption that matters most.

And if you want to be sure it really works, try it somewhere elseSame five letters, a small-town newspaper instead of a pharmacy chain. Different flip family entirely, the same silent input change.

The Larchmere Ledger uses an AI tool to draft summaries of public court records and city council minutes for its three-person newsroom. Mapped onto BOUND: break it down is the same three-layer equation, a fact-check eval harness, a model-agnostic citation schema, and an observability log of every draft against its source. Own the numbers would put the build at a much smaller scale for a three-person newsroom, maybe $18,000 total. Use a range accounts for how much of that could be open-source tooling versus custom work. Nail the sanity check compares that cost against the cost of one factual correction and the trust it costs a small paper, which is disproportionately high relative to the newsroom's size. Direction is the same insight: how often the underlying model changes matters more than the build cost itself. The flip here is different: verification, the familiar one, but worth naming because it's the flip a missing eval harness would have caught. Nils Kader, a reporter who'd been spot-checking maybe one in five AI-drafted summaries against the source record, started checking every single one after the paper mistakenly published a minor's name pulled from a court record the model had summarized incorrectly, a mistake a citation-schema layer would have caught automatically before it ever reached print.

The same four ideas that protect a pharmacy's patients protect a small newsroom's readers.

The model is the part that changes. These three layers are the part that doesn't, whether it's a pharmacy or a newsroom.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "build the eval harness, the model-agnostic schema, and the observability layer first, because all three pay off regardless of which model wins," and stop.
Cost: no budget for all three this quarter. Say so honestly, and start with the observability layer alone, since you can't even diagnose a problem like Odessa's without it.
The model gets better, for real: if one model family clearly proves dramatically better and cheaper for years running, the honest move is to lean into deeper integration with it and say so, not keep every layer maximally abstracted out of habit.

Where people run it wrong.
They build a golden set from whatever's easy to label, and never revisit it once real usage looks different from the test set.
They treat "we'll figure out the infra once we know which model wins" as a reasonable sequencing, when the infra is exactly what doesn't depend on that answer.
They measure a tool's success by its own reported accuracy, without ever checking whether the input reaching it still resembles the real job.

How to use it live. The moment someone asks what infra to build regardless of the model, ask yourself: what would let me catch a problem even if I have no idea which model caused it? That's the layer worth building first.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0