How do you evaluate whether proprietary data is actually a moat?

Fenwick Credit underwrites consumer loans using six years of proprietary alternative data on top of standard bureau scores. Priyanka Osei runs the data science team that used to run a quarterly exercise estimating how long a competitor would take to catch up. Then the model's accuracy crossed a number that felt like proof, and the exercise quietly stopped.

How to answer this, stage by stage

Nobody is scoring whether you believe your own data is special. They're scoring whether you can put a number on what it would cost someone else to stop caring that it's yours.

Let's learn

Here is what happens when a genuinely good number gets mistaken for a permanent one.

Fenwick Credit's default-prediction model improved from an AUC of 0.71, using bureau data alone, to 0.86, using six years of proprietary alternative data layered on top. In years one and two, every additional 10,000 labeled outcomes added about 0.9 points of AUC. By years five and six, that same 10,000 outcomes added only about 0.1 points. The gains were real. They were also flattening, quietly, the whole time.

The five letters, held up as one page. Direction is the step this question is really testing.

For the first three years, Priyanka's team ran a quarterly exercise: estimate what it would cost a well-funded rival to replicate Fenwick's edge without the proprietary data, using current data-market prices and typical model-tuning timelines.

Here's the turn: once the model crossed 0.85 AUC in year four, that exercise started to feel unnecessary. The model was clearly excellent. Nobody framed skipping the quarterly estimate as a real decision. It just stopped feeling worth the two days it took each quarter, exactly the two days that would have caught what was actually happening.

At its worst, the board keeps hearing "our data moat compounds every year" in the same words used at the founding pitch, while the actual arithmetic behind that claim quietly stops being checked at all, right as the one number that mattered most, whether accuracy gains were still compounding, started telling a very different story.

What I would leave alone: I wouldn't run the full replication estimate for a minor competitor with no real funding behind them, since the exercise is worth its cost only against a rival who could plausibly spend the money.

The lesson: a moat claim is only as good as the last time someone actually tried to break it on paper. A good model score is evidence the moat existed once. It is not evidence it still does.

Now here is the same thing as a story

The short version above is what you'd say defending the underwriting model's competitive position in a board deck. Read this one for how a genuinely good quarter quietly ended the one check that would have caught the erosion.

For three years, the last Tuesday of every quarter belonged to Priyanka's replication exercise: pull current bureau and alt-data feed prices, estimate a rival's tuning timeline, and present a range to the board. It never took more than two days, and it always came back reassuring.

One number kept getting watched because it was easy to feel good about. The other quietly stopped, because it never had that problem.

In year four, the model's AUC crossed 0.85 for the first time, a genuine milestone the whole team had worked years toward. The board meeting that quarter ran long on celebration. The replication exercise, due that same week, got pushed. Then it got pushed again the next quarter. By year five, nobody had scheduled it at all.

Four parts. Fenwick's team used to price all four every quarter. For eighteen months, nobody priced any of them.

Nobody decided the moat was safe forever. The team was simply busy, the model was genuinely good, and an exercise that had always come back reassuring stopped feeling like it was worth the two days.

Eighteen months after the last replication check, a rival lender launched a competing product. Independent industry benchmarks showed it had closed about 80 percent of the accuracy gap to Fenwick, in 14 months, using a purchased alternative-data feed and a tuning approach that wasn't particularly novel.

The rival landed almost exactly inside the range Fenwick's own team used to estimate, back when anyone was still running the numbers.

When the quarterly check first got pushed, someone said, "let's skip it this one quarter, the board's thrilled with the model and we have a launch to finish," and it sounded completely reasonable, since nothing about that specific week suggested anything had changed.

Fenwick's advantage had quietly moved from the first branch to the last one, and nobody was running the check that would have shown it.

Rerun the same eighteen months with the quarterly replication estimate still running: year five's check would have shown marginal gains flattening to 0.1 points per 10,000 outcomes, a clear signal the "moat" had become a volume lead. That finding alone would have justified investing in a genuinely new proprietary signal, one a rival couldn't simply purchase, well before any competitor's launch, instead of learning about the erosion from an industry benchmark report.

What I'd tell myself, learning about a rival's benchmark result eighteen months after the last time anyone at Fenwick priced the gap: the model's own score was never proof the moat was safe. It was proof the moat had worked, in the past tense, and the past tense is exactly the part that needs re-checking most.

BOUND, the estimate that would have caught this at year fiveNot a pep talk about proprietary data. BOUND is what forces a real number onto a claim that usually just gets asserted.

The recap, one line per letter: break it down is data price times volume plus tuning and compliance, own numbers is sourcing each figure from real market prices, use a range is 9 to 24 months instead of one false-precise figure, nail the sanity check is comparing $26M to Fenwick's own funding round, and direction is that whether gains are still compounding swings the estimate more than everything else combined.

And if you want to be sure it really works, try it somewhere elseSame five letters, a teleradiology startup instead of a lender. Different flip family entirely, the same overconfident number.

Farida Nabulsi leads data at Halcyon Diagnostics, which claims its ten-year, forty-million-scan annotated library is a moat for its diagnostic-assist model. Mapped onto BOUND: break it down is annotation cost per scan times scans needed for parity. Own numbers assumes double-annotation costs about $18 a scan versus $6 for a single read, and a rival needs roughly 2 million well-annotated scans for parity. Use a range gives 12 to 30 months depending on how many radiologists a rival can hire at once. Nail the sanity check compares that timeline against how often hospital data-sharing agreements typically get renegotiated, since regulation could reshuffle access either way. Direction is whether Halcyon's own recent scans are still double-annotated, or have quietly shifted toward single-read only.

The flip here is scope, not over-trust. As new hospital partners joined, scan volume tripled within two years, and Halcyon's radiologists could no longer double-annotate every incoming scan the way they had for the archive's first three years. They shifted, one hospital contract at a time, to single-read annotation for most new volume, keeping double-annotation only for a shrinking, hand-picked sample. The "forty million scans" number kept growing. The share of it that was ever high-quality kept shrinking at the exact same time.

The archive got bigger at exactly the moment its average quality got worse, and the headline number never showed either change.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "price what it would cost a rival to replicate your edge, in months and dollars, and recheck that number on a schedule," and stop.
Cost: no time to build a full replication model before the next board meeting. Say so honestly, and estimate the single biggest lever, whether your gains are still compounding, first.
The data landscape got worse for competitors, for real: if new privacy rules make alt-data harder for anyone to buy, that genuinely strengthens a moat. The direction of a perturbation isn't always bad news; the discipline is checking either way.

Where people run it wrong.
They treat a good accuracy score today as proof the underlying advantage is permanent.
They run a replication or moat estimate once, usually for a pitch deck, and never schedule it again.
They give one confident number instead of a range, which hides exactly how shaky the underlying assumptions are.

How to use it live. The moment an interviewer asks whether your data is a real moat, ask yourself: what would it cost a well-funded rival, in real months and dollars, to not need your data at all? Answer that with a range, name what swings it most, and the rest of the response follows.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0