Describe the minimum eval dataset you need before shipping anything.

Wraithbind Security sells a triage tool that scores incoming security alerts as likely real or likely noise, so a small SOC team can focus on what matters. Tomasz Wiercinski owns the eval process behind every new version, and had to explain, after a near miss, why a model that scored 97 percent on paper had never actually been tested against the company's own past incidents.

How to answer this, stage by stage

Nobody is testing whether you know what an eval set is. They're testing whether you'd have caught a real incident quietly falling out of what "passing" meant.

Let's learn

Every week, Wraithbind's model handled about 40,000 incoming alerts, and its own eval set was rebuilt each release from the most recent 10,000 labeled ones, a fast, clean, convenient slice to work with.

That approach worked fine for the first two releases, when the company's confirmed real incidents were still recent enough to be inside the window. By release six, the two oldest of the company's three ever-confirmed incidents had simply aged out of "the most recent 10,000," quietly, with nobody deciding to remove them on purpose.

The fourth step is the one Wraithbind's convenience sample skipped entirely, without anyone deciding to.

Here's the turn: the model's aggregate accuracy never dipped. It sat at a comfortable 97 percent release after release, because the eval set was dominated by common, easy alert categories that the model handled fine. The number everyone watched told a completely reassuring story, right up until it didn't matter anymore.

At its worst, a model version ships that would have failed on a pattern resembling a past real incident, and nobody finds out until an analyst catches it by instinct rather than by any signal the eval process actually produced.

What I would leave alone: the recent-alert sampling itself isn't wrong for the bulk of the eval set. Most alert categories are common enough that a fresh, rotating sample genuinely reflects current conditions well. The gap was only ever the confirmed incidents, which needed to be permanent, not refreshed.

The lesson: a sampling method that's honest about common cases can be quietly dishonest about rare ones, and rare is exactly where the real cost of a mistake lives.

Now here is the same thing as a story

The short version above is what you'd say defending a new release-gate policy to engineering leadership. Read this one for how a near miss surfaced a gap nobody had gone looking for.

Ines Marchetti had worked the overnight SOC shift at Wraithbind's largest client for two years, long enough to trust her gut about which alerts felt wrong before the tool even finished scoring them.

The four things Wraithbind's convenience sample was quietly missing three of, without anyone deciding to skip them.

The new model version had passed its release eval comfortably, 97.2 percent, a hair above the last one. Nobody flagged anything, because nothing looked different from any other release in the past year.

Two numbers, both real, telling two very different stories about the same release.

Three weeks after the release, an alert came through resembling, in a way Ines couldn't fully articulate at first, a pattern from an incident the company had confirmed eighteen months earlier: unusual authentication timing paired with a specific lateral-movement signature. The model scored it low-priority, likely noise. Ines escalated it anyway, on instinct.

The category that mattered most sat exactly where a convenience sample was least likely to reach.

It was real. A genuine, active intrusion attempt, structurally close enough to the 18-month-old confirmed incident that a model actually tested against that case should have caught it easily. When Tomasz's team checked, the current eval set didn't contain that incident at all. It had aged out two releases earlier.

Two of these four had a real, defined process behind them. The other two were assumed, not built.

Rerun the same release cycle with the confirmed-incident replay set locked in permanently: the eighteen-month-old incident never ages out, the model's recall against it gets checked every single release, and the drop to 50 percent recall is caught in the eval pipeline before ship, not three weeks later by an analyst's gut feeling on a Tuesday night.

Nobody decided, on any single release, to stop testing against the past. The convenience sample just never promised to keep it.

What I'd tell myself, hearing that a real incident had aged out of the very thing meant to prevent it recurring: convenience sampling isn't dishonest by design. It just quietly forgets exactly the cases it was never built to remember on purpose.

LEAD, in one screenNot a case for testing everything exhaustively. LEAD is what tells you exactly which small, deliberate set actually earns its keep.

The recap, one line per letter: link is analysts never missing a breach or drowning in noise, early signal is recall on the permanent confirmed-incident replay set, abuse is convenience sampling quietly hiding rare, severe categories, and decision is a hard release-blocking bar tied to that one number.

And if you want to be sure it really works, try it somewhere elseSame four letters, a pharmacy chain instead of a SOC. A different kind of rare, severe case entirely.

Ashgrove Pharmacy Group uses an AI tool to flag dangerous drug interactions before a prescription is filled. Mapped onto LEAD: link is patients never receiving a genuinely dangerous combination, without every routine, harmless interaction getting flagged and burning out the pharmacists reviewing them. Early signal is recall on a permanent replay set of every confirmed serious interaction the chain has ever caught or missed, not the model's overall flag-accuracy rate, which stays comfortably high because most flagged interactions are common and mild. Abuse looks the same as Wraithbind's: a convenience sample built from recent prescriptions will be full of common drug pairs and thin on the rare, dangerous ones, since those are exactly the cases too infrequent to show up much in any recent window. A pharmacist named Grace Odhiambo caught a near miss when a rare anticoagulant interaction, confirmed once three years earlier, scored as low-risk, because that case had long since aged out of the convenience sample behind the model's eval. Decision: below 95 percent recall on the confirmed-interaction replay set, the release doesn't ship, full stop, regardless of how good the overall flag-accuracy number looks.

A different kind of incident, the same shaped gap: convenience sampling quietly forgetting the cases that were rare because they were serious.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "lock in every past confirmed real incident as a permanent replay set, and watch recall on that set, not aggregate accuracy," and stop.
Cost: no time to build a full stratified eval set before the next release. Say so honestly, and ship advisory-only on any category with thin coverage, rather than shipping fully autonomous on an untested gap.
The model got better, for real: if a newer model version genuinely closes the gap on rare cases, that's a legitimate reason to loosen the advisory-only gate, not a shortcut being taken to skip building the replay set.

Where people run it wrong.
They treat a large eval set as inherently rigorous, without checking whether it actually contains the rare, high-stakes cases that matter most.
They watch aggregate accuracy release over release and assume a flat, healthy number means nothing has changed underneath it.
They let convenience sampling quietly redefine what "good" means, one release at a time, without ever deciding to.

How to use it live. The moment an interviewer asks about a minimum eval set, ask yourself: which handful of past real cases would I never forgive myself for letting a model fail on, and is there a guarantee those cases are permanently in the set? Answer that, and the rest of the eval design follows.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0