Explain how you would build an eval set that includes adversarial cases.

How to answer this, stage by stage

Nobody is grading whether you land on exactly 350. They're grading whether you actually define what "adversarial" means before sizing it, whether the costliest patterns get more than an equal share, and whether the slice survives a check against the whole eval set. Seven moves get you there.

Let's learn

Every Monday morning, a trust and safety team checked one number: 97 percent.

The product is a safety model that reads every message and photo the moment someone tries to send it, and decides whether it's safe to go through to the other person.

Before there was any real eval set, the team tested new model versions against about 2,000 examples of real content that users had reported over the past year: the everyday stuff, obvious nudity, obvious slurs, obvious spam links. Against that set, the model caught 97 out of every 100.

Now here's what a scripted romance scam actually looks like. It never mentions money in the first two weeks. It builds trust first, then asks to move the conversation off the app, then, only after a real bond exists, brings up "this investment app I use." None of that trips a filter trained to catch nudity, slurs, and spam links, because none of it looks like anything the filter had ever been shown as bad.

Five hundred and forty scripted messages a week did not show up as a drop in the 97 percent, because the eval set only ever tested content the model had already learned to catch.

At its worst, that gap costs a real person their savings before anyone at the company even knows the pattern exists, because the only signal an untested attack ever produces is the report that comes in after it already worked.

The choice I would take back. The eval set was built entirely from real, historical, user-reported content, with no category written specifically to test evasion. That was fine before launch, when nobody had a known attack pattern to write test cases against yet. It stopped being fine the day scam rings started rewriting their opener every couple of weeks specifically to dodge whatever got caught last time.

What I would leave alone. Photo nudity detection doesn't need this same treatment. People occasionally crop or blur a photo to sneak one past it, but they're not doing it in an organized, evolving way, and a second static rule already catches almost all of it. Not every category is worth a maintained red-team budget, only the ones facing someone who adapts.

The lesson. An eval set built only from cases you've already caught tells you how good you used to be. It says nothing about whether you can catch what someone is building right now to get past you.

Now here is the same thing as a story

Skip this if you already believe a reported-content sample and an adversarial one aren't the same thing. Read on if you want to feel why they aren't.

Marlowe Estrada can read a scam opener in one line, before it even finishes loading. Two years running trust and safety at Emberlane will do that.

Marlowe built the first real eval set the week Emberlane crossed a million messages a day: 2,000 examples pulled from a year of real user reports, weighted toward whatever categories got reported most. It felt like the honest choice. Nobody had cherry-picked a single example.

For most of a year, that felt like enough. Every Monday, Marlowe pulled the eval score for whatever model version had shipped that week, and it held at 97 percent release after release. Early on, Marlowe would also hand-read a sample of borderline messages, just to see how the near-misses actually looked. They usually looked fine. So the hand-read got smaller. Then it stopped, replaced entirely by one habit: check the number, ship if it holds.

Then came a Thursday in the fourth quarter, a routine trust and safety audit, not a crisis. Someone above Marlowe pulled thirty confirmed scam accounts from the last three months and re-ran their entire message history back through the safety model, retroactively, to see what it would have caught.

The same eval set, filled two different ways. One only holds what already went wrong. The other holds what someone is trying to make go wrong next.

The model had scored those thirty accounts' messages clean for a median of eleven days before a user ever filed a report.

One of the thirty was a man who had wired $42,000 over six weeks to someone he had never met, believing they were building toward a shared future once "this one investment" paid out. Every message in that exchange had scored clean.

Marlowe pulled the eval set that had passed the release those messages shipped under. Two thousand examples, all of them real, historical, already-reported content. Zero of them were a case someone had written on purpose to get past the model. The romance-scam pattern that had just cost a real user $42,000 had no representative in the test at all.

Marlowe rebuilt the eval that month: five named attack patterns, a floor of 50 red-team-written cases each, doubled to 100 for the two costliest, romance-scam scripts and minor-safety evasion, for 350 adversarial cases layered on top of the existing 2,500. Run against the model as it stood, the romance-scam slice alone scored 61 percent clean, well under the bar Marlowe set for it. A fix that flagged off-platform redirect language on its own, regardless of the exact wording around it, took eleven days to ship. Rerun, the slice cleared 94 percent. Over the following month, confirmed scam accounts on Emberlane got caught after a median of 1.5 days instead of eleven.

The thing I'd tell myself, the week I built that first 2,000-example sample: a test built only from what already happened can only ever catch what already happened again. It will never catch what someone is building right now, specifically, to get past it.

BOUND, sized against someone actually trying to beat it

This is a sizing question about how many red-team cases to write and for which named attacks, not a person's trust flipping between two settings, so BOUND fits and FLIPS doesn't.

B, break it down. The adversarial slice equals the number of known attack patterns, times a minimum floor of red-team-written cases per pattern, plus extra for the patterns already known to cost more than the rest. Emberlane's trust and safety team tracks five named patterns.
O, own the numbers. A floor of 50 cases per pattern, since under 50 a real evasion pattern can't be told apart from a few lucky guesses. 5 times 50 is 250. Two patterns, romance-scam scripts and minor-safety evasion, get that floor doubled to 100, adding 100 more. Total: 350.
U, use a range. With a narrower list, 3 highest-volume patterns at the same floors, the total drops to about 200. Splitting patterns more finely, say 8 patterns once regional scam-script variants get tracked separately, pushes it toward 550 once the costliest ones stay doubled. Start at 350, the plain five-pattern split, and only add a new pattern once real reports show it's actually happening.
N, nail the sanity check. 350 adversarial cases sitting inside a full eval set of about 2,500 everyday examples is roughly 12 percent of the total. High enough that a real evasion pattern shows up and can be tracked release over release. Nowhere near high enough to turn the whole eval set into a test of nothing but people trying to break the product, when most real traffic is two people just talking.
D, direction. A new named attack pattern moves the total more than the floor does. Naming a sixth pattern, high-risk, adds 100 on its own. Raising the floor from 50 to 65 across all five only adds 75. If the total needs to shrink, that's a reason to check whether two patterns actually overlap before thinning the floor, since a thin floor is exactly what let the romance-scam pattern hide the first time.

And if you want to be sure it really works, try it somewhere else

A resume-screening tool ranks incoming applications for a recruiting team, reading each resume and scoring it against a role's requirements.

B, break it down. The adversarial slice equals the number of known manipulation patterns times a floor of red-team-written resumes per pattern. A typical set of named patterns runs to three: hidden prompt-injection text meant for the model instead of a human reader, invisible keyword-stuffing, and fabricated but verifiable-sounding credentials.
O, own the numbers. A floor of 30 resumes per pattern, 3 times 30 is 90. Prompt injection gets that floor doubled to 60, since it can flip a ranking outright rather than just nudging it, adding 30 more. Total: 120.
U, use a range. Splitting keyword-stuffing by field, technical roles versus everything else, pushes the total toward 180. Testing only the single riskiest pattern, prompt injection alone at a floor of 60, brings it down to 60.
N, nail the sanity check. 120 adversarial resumes inside a full eval set of about 1,200 real applications is 10 percent of the total, close to the 12 percent that held up for Emberlane. A number well under that would mean the tool's score says almost nothing about whether someone can game it.
D, direction. Here the lever isn't a new pattern so much as a new delivery method for the same one. Prompt injection hidden in a PDF's metadata instead of its visible text is still the same pattern, but it needs its own written cases, since a filter tuned to catch visible white-on-white text will miss it completely.

Swap the trigger and it still runs.
Speed: an interviewer caps the answer at 90 seconds. Skip straight to the split: 50 per pattern, doubled to 100 for the two costliest, 350 total, about 12 percent of the eval set. The build-up backs it up if they ask.
Cost: instead of asking for a sample size, a manager caps the red-team writing budget at 40 hours. Work backward: at roughly 20 minutes to write and verify one solid adversarial case, that's about 120 cases. Cover the two costliest patterns first and treat the rest as next quarter's work.
The model got better: a new version almost never falls for a generic scam message anymore in general use. The floor doesn't drop on its own. Rerun the romance-scam slice specifically, since a clean overall number says nothing about the one pattern that used to fail.

Where people run it wrong.
They size the adversarial slice by what sounds thorough, "a few hundred hard examples," instead of a floor times a named pattern count.
They write the adversarial cases once at launch and never revisit them once a real attacker starts adapting weekly.
They let the adversarial slice grow until it's most of the eval set, so the score stops reflecting what the product does for the ordinary majority who aren't trying to break it.

How to use it live. Say the equation before any number: "the adversarial slice is attack-pattern count times a per-pattern floor, doubled wherever a miss actually costs someone something." That buys the time to name real patterns instead of guessing a round number that sounds thorough.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0