How do you evaluate a competitor's quality claims without access to their evals?

Picture a press release with a big, confident number in it, and no way for you to check where that number came from. Netreef builds an AI tool that identifies fish species from dock and vessel photos, for the catch reports fleets are legally required to file. Soren Kallio is the marine data analyst who tracks fishing-fleet compliance for Netreef, and this is what happened the week a rival, Tallyfin, published a species-identification accuracy claim nobody outside their own building could verify.

How to answer this, stage by stage

Nobody is scoring whether you believe the competitor's number. They're scoring whether you can build a defensible range around it, out loud, with the arithmetic showing.

Let's learn

Here is what happens when a competitor publishes a quality number with no way for anyone outside their building to check it.

A single accuracy number is four different questions wearing one costume.

Netreef's own species-identification model runs against a golden set of 12,000 labeled catch photos across 40 species, tested blind, on data the model has never seen. Overall, it scores 91.4 percent. Broken out, it scores 97 percent on common, visually distinct species like salmon and cod, and only 68 percent on visually similar juvenile groundfish, a group that makes up just 8 percent of total catch volume but causes most of the actual regulatory violations.

Here's the turn: Tallyfin's marketing page claimed 99.2 percent species-identification accuracy, with no methodology, no third-party validation, and no mention of whether that number was blended or measured per species. The claim itself wasn't necessarily false. But treated at face value, it implied Tallyfin had beaten Netreef's own five-year, 12,000-image result, on a problem where the hardest cases matter most, using nothing anyone outside Tallyfin could check.

What I would leave alone: a small rival with no public accuracy claim at all doesn't need this treatment. There's nothing to discount or sanity-check yet; the honest answer there is simply "no claim exists to evaluate."

The lesson: a number nobody can check isn't evidence. It's a starting point for an estimate, and refusing to build that estimate isn't caution, it's just letting the loudest number in the room win by default.

Now here is the same thing as a story

The short version above is what you'd say defending this estimate to Soren's own director. Read this one for how a simple question from someone new turned into an actual method.

His name is Soren Kallio. He has tracked fishing-fleet compliance data for five years, long enough to know which dockside lighting makes a young hake look like a young cod in a bad photo, and which regional fleets file catch reports late every single quarter.

Nothing dramatic happened here. Someone just asked a question the team had quietly stopped asking itself.

A new analyst joined Netreef's team in her second week and read Tallyfin's marketing page out loud in a meeting: "99.2 percent species-identification accuracy." She turned to Soren and asked, simply, "How do we know that's real?" Soren opened his mouth to answer and realized he didn't actually have one. Nobody did. The number had been sitting in a competitor-tracking slide for two months, unchallenged, mostly because it sounded confident and nobody had built a way to push back on it.

One of these is a number. The other is a number wearing a number's clothes.

Soren spent the afternoon building an actual estimate instead of a shrug. He started with Netreef's own honestly-measured number, 91.4 percent overall, achieved on a golden set built over five years, tested blind. He assumed, based on published industry patterns for self-reported AI accuracy claims with no released methodology, that such numbers typically overstate real, independently-checked performance by 15 to 20 points. Applied to Tallyfin's 99.2 percent, that lands the estimate around 80 to 84 percent, which is suspiciously close to, and possibly below, Netreef's own number.

This is what a real number looks like once you're willing to show your own hard cases too.

Here is the decision Netreef would take back. Telling customers years ago that the system was "lab-grade reliable," with no published breakdown, was fine when nobody else was making claims to compare it to. It stopped being fine the exact week a rival made an equally vague, equally unverifiable claim, because now two unverifiable numbers were sitting side by side, and neither company could point to anything solid.

Soren's team started publishing their own species-level breakdown the same quarter, the 97 and 68 percent figures included, not hidden. Six weeks later, a prospective customer asked both companies for their per-species numbers. Tallyfin didn't answer. Netreef did, in a single afternoon, because the number had already been broken down the way Soren had learned to do it for a rival's claim.

BOUND, in one screenNot a verdict on whether the number is true. BOUND is what turns an unverifiable claim into a defensible range.

The same four branches decide what Soren actually says out loud, once the estimate is built.

The recap, one line per letter: break it down is stating the equation before touching numbers, own numbers is a stated 15-to-20-point self-report discount, use a range is 75 to 90 percent instead of one guessed figure, nail the sanity check is comparing it against Netreef's own harder-won 91.4 percent, and direction is naming that blended-versus-per-species reporting would move the estimate the most.

And if you want to be sure it really works, try it somewhere elseSame five letters, a small-town news service's AI writing assistant instead of a fisheries tool. A different claim, the same method.

Millbrook Wire runs an AI tool that drafts short local-news articles from public records for small papers. A rival service claimed its AI drafts were "98 percent factually accurate" in a funding announcement, with no published fact-checking methodology. Mapped onto BOUND: break it down is separating factual accuracy on simple record-based facts, like a permit number, from accuracy on inferred claims, like a quote's tone. Own numbers assumes self-reported factual-accuracy claims in AI writing tools typically run 10 to 15 points optimistic against a human fact-checker's blind review, based on published newsroom AI pilots. Use a range lands the estimate around 83 to 88 percent. Nail the sanity check compares it against Millbrook Wire's own fact-checked rate of 89 percent on simple record facts alone, before ever touching inferred claims, which the rival's number never separated out. Direction says the single most valuable missing fact is whether the 98 percent excludes inferred claims entirely, since record facts alone are far easier to get right than anything requiring judgment.

Not every claim earns a full BOUND estimate. This is what decides which ones do.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "treat the number as an upper bound, discount it with a stated assumption, and give a range, not a verdict," and stop.
Cost: there's no time to research real self-report optimism rates before the next customer call. Say so honestly, and use a conservative, clearly-labeled placeholder discount until better data exists, rather than skipping the estimate entirely.
The model gets better, for real: if a rival eventually does publish real, third-party-verified methodology behind their number, that's not a threat to this method, it's the evidence test finally landing, and the range collapses into a real number worth trusting.

Where people run it wrong.
They either accept a rival's number at face value or dismiss it outright, instead of building an actual estimate in between.
They give a single guessed number instead of a range, pretending to more precision than the evidence supports.
They never check the claim against their own honestly-measured result, missing the sanity check that would have caught an implausible number.

How to use it live. The moment someone quotes you a rival's unverifiable number, ask yourself: what would I have to assume about how they measured this for that number to be real? Say the assumption out loud, and build the range from there.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0