Describe the red-team exercise you would run before a consumer launch.

Tallgrass Mobile is a phone carrier. The Tallgrass Assistant answers billing questions, changes plans, and activates roaming, by voice or by chat. Callum Bretz leads trust and safety for the assistant, and owns whether it's ready to talk to real customers.

How to answer this, stage by stage

Seven moves. This one runs a little longer because a design answer needs its risk and its limits stated out loud, not just its idea.

Let's learn

Say we build a voice and chat assistant for a phone carrier. It answers billing questions, changes a customer's plan, and turns on roaming before a trip.

Before this design existed, getting ready for a release meant two engineers spending an afternoon typing hostile prompts into the chat window: rude language, a fake account number, a request phrased to sound urgent. If nothing obviously broke, the release shipped. It cost about four hours and it caught the obvious stuff.

Then a rival carrier's assistant made the news. A caller had phoned in pretending to be a customer's distressed spouse, said the customer had lost their phone and needed the number moved to a new SIM right away, and the assistant walked the caller through it. No prompt injection, no strange syntax. Just a calm, plausible story, told well.

Here is the turn. The problem was never that the ad hoc afternoon of testing missed a clever prompt. The problem is that it was never actually testing the thing that mattered: a patient, well-spoken person telling a normal-sounding lie. Nobody had scripted that scenario, because nobody had thought to. An afternoon of typing scary questions catches scary questions. It doesn't catch a calm one.

At its worst: a calm, convincing caller talks the assistant into a SIM swap or an account change it should never have approved, and the first anyone hears about it is a customer complaint or a fraud report, not a test that caught it early.

What I would leave alone: a read-only feature, like checking a bill balance, doesn't need this level of scripted adversarial testing. There's no action to trick the assistant into taking, so the cost of a rare bad answer stays small no matter how creatively someone phrases the question.

The lesson: a red-team exercise that only tries to sound scary will only catch things that sound scary. The real risk almost always sounds completely reasonable.

Now here is the same thing as a story

The short version above is the plan. Read this one for how Callum actually got there.

Callum Bretz has spent six years in trust and safety roles, three of them at Tallgrass. He is the person people send a strange customer transcript to when nobody else can explain what happened.

This was the entire process, for two years, before anyone wrote any of it down.

For most of that time, it was enough. The assistant only answered questions, so a bad answer cost a customer a follow-up call, nothing more. The afternoon of ad hoc testing caught rude language and obvious tricks, and that was the whole threat model anybody had in mind.

Then a rival, a carrier three states over, had its own assistant fooled by exactly the scenario nobody at Tallgrass had ever scripted: a caller claiming to be a customer's spouse, saying the phone was lost, asking for the SIM to be moved. Calm, patient, entirely plausible. It worked. The story made the local news within a day.

The middle milestone is the one that actually changed anything. Nobody built the real suite until watching it happen to someone else.

Callum pulled together a small group the next morning. Not to panic. To write down, for the first time, exactly what kinds of people might try to trick the assistant, and to give each one a script instead of a vague worry.

Four pieces, and the release gate is the one that actually has teeth. Nothing ships past a failed persona.

The first release candidate run against the new suite failed on persona seven, the impersonated spouse. It failed on purpose, that first time, because the team ran the exact scenario the rival had missed to make sure the suite actually caught it. It did. The release stayed home for four extra days while the fix went in.

We did not build the suite to catch clever attackers. We built it to catch calm, ordinary-sounding ones, since those were the ones that had actually worked.

The suite was never going to be complete on day one. The 48-hour rule is what keeps it from staying incomplete.

Not every new case blocks a release. Severity decides that. But every single one gets written down somewhere.

Three months later, when Tallgrass Assistant finally launched to every customer, the suite had grown from ten personas to fourteen, four of them added because of live misses caught during the beta. Nobody remembers the exact wording of the first afternoon's ad hoc questions anymore. Everybody can still recite persona seven.

What I would tell myself, back when the plan was still "two engineers, one afternoon": the goal was never to sound clever in the test room. It was to sound exactly like the kind of person who actually calls in.

SPARK, the exercise Callum actually ranNot a checklist. SPARK forces the anchor to be one decision you could point at and defend.

The recap, one line per letter: situation is an afternoon of unwritten ad hoc testing, payoff is standing coverage that grows instead of resetting, anchor is ten scripted personas as a hard gate, risk is a live miss becoming a script inside 48 hours, and keep out is automated fuzzing, deliberately held for later.

And if you want to be sure it really works, try it somewhere elseSame five letters, a hospital scheduling assistant instead of a phone carrier. A completely different field.

Milbrook Clinic Group is piloting an AI assistant that reschedules patient appointments and answers insurance questions over the phone. Noor Sabbagh runs the pilot and owns whether it's ready for real patients.

Mapped onto SPARK: situation is that today, a single nurse spends an hour before each update calling the assistant herself with a handful of odd requests. Payoff is a habit of catching a category of failure, not just a handful of individual bugs, before it ever reaches a patient. Anchor is a standing set of eight personas built around insurance and identity confusion specifically: the caller with the wrong birth date, the caller trying to reschedule someone else's appointment, the caller reciting insurance details that don't match on file. Risk is a real patient mix-up slipping through; the fix is the same 48-hour rule, turning any live miss into a ninth persona. Keep out is full clinical triage testing, since this assistant only schedules, it never gives medical advice, so testing it against clinical judgment questions would be solving a problem this product doesn't actually have.

Same four pieces, completely different personas. The shape of the anchor travels even though the content never does.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "ten scripted personas, a hard release gate, and any live miss becomes an eleventh script within 48 hours," and stop.
Cost: there's no headcount to write ten scripts by launch. Say so honestly, and start with the three highest-severity personas only, adding the rest after launch instead of delaying it.
The model gets better, for real: if the assistant's overall accuracy improves, that's still not a reason to shrink the suite. A better model can still be talked into a SIM swap by a calm, patient caller; the suite exists for that specific behavior, not for accuracy in general.

Where people run it wrong.
They test for rude language and call it done, missing the calm, plausible attacks that actually work.
They run red-teaming once before launch and never again, so the suite never grows past its first day.
They let the gate become advisory under deadline pressure, which quietly turns a hard gate into a suggestion.

How to use it live. If an interviewer asks what your red-team exercise actually tests, don't answer with a category like "safety" or "bias." Name one specific persona and the exact line they'd say. It proves you've actually thought about a person, not a policy.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0