How do you factor in that the buy option's capability improves without your effort?

Crestholm Mutual is an insurance company. ClaimLoop routes an incoming claim document to the right adjuster team and flags likely fraud cases. Selin Paskowitz is the AI PM weighing whether to build a custom classifier in house, and Graham Orenstein is the VP of Finance pushing to build, to lock in a four-point accuracy edge over the vendor Crestholm currently pays for.

How to answer this, stage by stage

Nobody is scoring whether you can recite build-vs-buy pros and cons. They're scoring whether you noticed that one side of the comparison is a moving target.

Let's learn

Two years ago, a claim document at Crestholm Mutual got sorted correctly by a vendor model eighty-two times out of a hundred.

ClaimLoop routes an incoming claim document to the right adjuster team and flags likely fraud cases, instead of a mailroom clerk sorting by hand. Before ClaimLoop, a mailroom team sorted claims by reading the first page, about 30 seconds each, at 76 percent correct, since claims paperwork is genuinely ambiguous and some documents belong to two categories at once.

With the vendor version of ClaimLoop, routing hits 88 percent correct, in about a tenth of a second, for the same volume.

Here's the turn: the extra four points of accuracy isn't really the decision anymore, since most of it may get won without Crestholm spending a dollar. The actual decision is whether paying real engineering money to grab those points now, instead of waiting, is worth it.

At its worst, Crestholm spends 610 thousand dollars over three years chasing a four-point lead that quietly disappears anyway, and ends up right back where a 420 thousand dollar subscription would have landed it, just with less money left over.

The five letters, held up as one page. The O step, owning the vendor's own trend as a real number, is the one this whole answer turns on.

What I would leave alone: I wouldn't apply this same reasoning to a capability that's genuinely proprietary, like Crestholm's own historical fraud-pattern data. A vendor's general model never sees that data, so it can't improve on it for free, regardless of any trend line elsewhere.

The lesson: the fair fight isn't build's cost against buy's cost today. It's build's cost against how much of the gap the vendor was already going to close for you, for free, by the time you're done building.

Now here is the same thing as a story

Use the short version above in the room. Read this one for what it felt like the day a two-year-old chart quietly answered a six-month argument.

Selin Paskowitz had run the numbers on ClaimLoop three times before Graham Orenstein's team ever asked for a fourth.

Graham had spent four years pushing Crestholm's finance team to own more of its own technology instead of renting it, and he was good at his job: he could spot a recurring vendor fee compounding into real money faster than almost anyone in the building.

When the 92-percent in-house prototype numbers came back from a two-week internal test, Graham brought them straight to Selin's team with a clean case: four points of accuracy, and Crestholm would own every bit of it forever.

Nobody at Crestholm had ever charted the vendor's accuracy over time. It built up slowly: a small internal note here, a shrug there, a passing comment that "the vendor seems better than it used to be," never a real number, until Selin pulled two years of the same fixed eval set against every vendor release and actually plotted it.

Both options end up in nearly the same place. Only one of them costs 190 thousand dollars more to get there.

Selin never had a fixed rule for when a gap was "worth building for" versus "worth waiting on." It came down to one real question: is the vendor's trend actually closing this gap, or standing still? For claims-routing, the trend was closing it. For Crestholm's own fraud-pattern data, there was no trend at all, because the vendor's model would never see that data no matter how many versions it shipped.

The sprint is only a third of the real cost. The other two thirds is upkeep, just to hold a lead the vendor's own trend was already closing.

Back when Crestholm's procurement policy was first written, comparing vendors once at renewal made sense, since the underlying models barely changed between one renewal and the next. It stopped making sense the moment providers started shipping real accuracy gains every few months on their own.

The base case is measured, not guessed. But the whole decision should still survive the low end of this range, not just the middle.

Rerun the argument with the range included: even at the low end, 1.5 points a year, the vendor still closes most of the gap within about three years, just more slowly. At the high end, it's closed within a year. Either way, building buys time, not a permanent edge, and the sanity check holds: 610 thousand dollars is real money to buy something the vendor's own roadmap was going to deliver regardless.

One version spends 610 thousand dollars and ends up roughly where the vendor lands anyway. The other spends 420 thousand and gets there just as fast, since most of the gap was never going to require Crestholm's money to close.

What I'd tell myself, watching Graham's clean four-point case land on the table: a gap you haven't charted over time isn't a gap yet. It's just a number you haven't asked "compared to what, and by when" about.

BOUND, the five checks this decision actually neededNot a script for always choosing buy. BOUND is what makes sure build only wins when the gap is genuinely durable.

The recap, one line per letter: break down build's real cost against buy's real cost as equations, own the vendor's measured 3-points-a-year trend instead of assuming it, use the 1.5-to-4.5 range instead of one number, nail the sanity check that 610k for a temporary lead against 420k for durable convergence, and name the vendor's own improvement rate as the one assumption that would flip the whole call.

And if you want to be sure it really works, try it somewhere elseSame five letters, a shipping terminal instead of an insurance office. This time the vendor's trend has gone flat, and that changes the answer.

Noor Zubrycki runs container-damage detection at Tallowreach Terminal, deciding whether to keep buying a vendor's computer-vision damage-detection API or build one in house. Mapped onto BOUND: break it down the same way, build's cost against buy's cost over three years. Own numbers: here the vendor's accuracy has moved only about 0.3 points a year over the last 18 months, essentially flat, because container-damage detection is a small market the provider hasn't prioritized, versus an in-house team that could realistically gain 2 points a year with dedicated investment. Use a range: there's a real, if small, chance the provider suddenly reprioritizes logistics and the trend jumps, so the range runs from flat to 1.5 points a year. Nail the sanity check: 180 thousand dollars to close a 6-point gap, plus 60 thousand a year to hold a 2-point pace, totals 360 thousand over three years, against a flat vendor that likely never closes the gap on its own, versus 270 thousand to keep buying and stay 6 points behind indefinitely. Direction: the one assumption that flips this is whether the vendor's parent company reprioritizes logistics, a real but low-probability risk, so the base case favors building, the opposite of Crestholm's call, because the trend line itself points the other way.

Same five checks, but the branch that fires depends entirely on which way the vendor's own trend line points.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "chart the vendor's trend, project it forward, and compare the cost of closing the same gap at every point, not just today," and stop.
Cost: no time to build a real two-year trend before deciding. Say so honestly, and use the widest defensible range for the vendor's improvement rate instead of guessing a single number.
The vendor got better, for real: if the vendor's trend accelerates past your projection, that's not a reason to panic-build, it's exactly the signal the range and the quarterly recheck exist to catch early.

Where people run it wrong.
They compare today's gap and stop, treating the vendor's number as if it were frozen.
They price the engineering sprint but forget the ongoing upkeep needed to keep any lead open.
They treat "the vendor might improve" as a vague risk instead of a real, chartable, sourced trend.

How to use it live. The moment an interviewer asks how you'd factor in a vendor's free improvement, ask yourself: what would this comparison look like if I ran it again in eighteen months? If the honest answer is "roughly the same," that's the whole argument, said out loud, in one breath.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0