What is the argument for maintaining separate golden sets per customer segment?

The four moves for splitting a golden set

A tradeoff question wants a pick, not a checklist of eval fields, so PICK does the work here.

One more thing before the walkthrough moves on: this pick covers one market at a time, not the whole golden set forever. Most candidates hear "test the risky one more" and answer by making everything bigger. Say which market earns the segmented pass and which ones stay on the shared floor, and you've shown real judgment instead of reciting a checklist.

Let's learn

Fieldcopy is a box on a marketer's screen: type in a product and a campaign brief, get back three headlines and two lines of body copy, ready to paste into an ad.

Before Fieldcopy, someone wrote every regional ad by hand, one market at a time: US English one week, Mexico Spanish the next, India English the week after, each one rewritten from scratch to fit that market's own tone, about six hours a week just on copy.

One rulebook, checking three very different markets

With Fieldcopy drafting from the same brief, a marketer gets three headline options and two lines of body copy back in under a minute, for any market. Most weeks that cuts copy time from six hours down to about forty minutes.

To ship Fieldcopy, the team built a golden set: 150 real ads, graded by hand as "good enough to publish" or not, checked before any draft ships. It took about three weeks to build, almost all of it against US campaigns, since the US was the only market live when the set was built.

Here is the turn. The set answered "is this good ad copy" just fine, for a long stretch. But it only ever asked that question in one market's shape. When Mexico and India launched six months later, the same 150-ad set kept gating drafts in both, since building two more sets sounded like triple the review work for a two-person team.

At its worst, that looks like this: a Mexico ad says a camp stove "cures the mountain cold in seconds," a claim that reads fine to a US eye, because nothing in the shared set was ever graded for a claim shaped like that. Fieldcopy scores it clean. A marketer approves it in under a minute, the way she approves forty ads a week.

What I would leave alone. The shared set, for anything not yet spending real money in a market: internal drafts, new ad formats being tried out, and generic checks like grammar or a weak call to action that don't change by region. US and English Canada also stay on one set. They're close enough that a second one wouldn't catch anything new.

The lesson. We built the golden set to answer "is this good ad copy." It answered that well. We never built a separate answer to "is this ad copy legal and appropriate in the specific market about to run it," and only the second question could actually stop a campaign.

The claim nobody wrote for the US

You don't need this to answer the question. Read it if you want to feel why the segmented pass has to happen before a market goes live, not after.

The dashboard Esperanza Duval opens every Monday has three tabs, one per market, and for a long stretch of last year, all three said the same thing: green.

She's run growth marketing at Solstice Outfitters for four years, the only person on the team who watched all three markets grow from week one. Every Monday at nine she'd open the tabs, skim the week's drafts, tap approve on whatever cleared the check, and be back on her actual job, growing the channels themselves, by half past nine.

First she stopped reading every headline word for word once Fieldcopy kept getting them right. Then she stopped opening the Mexico and India tabs on their own, just scanning the combined pass rate across all three. By month nine, if the number was green, she moved on without opening a single ad.

Then, on a Thursday call about something else entirely, the Mexico operator said it in passing: "oh, is the cold-weather stove line okay to call it a cure? Our old agency always got twitchy about that word."

Same tool, two very different ways to be wrong

Esperanza laughed it off for about four seconds. Then she pulled up the live campaign.

The ad had been running six days: "cura el frío de la montaña en segundos," a headline Fieldcopy had drafted, scored clean against the shared set, and she'd approved without a second look, the way she approved forty other ads that same week. Nothing in the 150-ad golden set had ever included a claim shaped like this one, so the claim had never actually been tested, only assumed fine because nothing like it had ever failed before.

I want to say the problem is that Fieldcopy scored the ad clean. It's not really that. A claim that was never tested isn't wrong, exactly. It's just unknown, and Fieldcopy reported its confidence in a number nobody had ever checked against a claim like this one.

Months before, when the golden set first got built, the team spent maybe fifteen minutes on the question of one set or three. Building three felt like triple the review work for two people already stretched thin. One set, one bar, felt clean. Nobody sat down and asked whether "good ad copy" meant the same thing in Guadalajara as it did in Ohio.

Esperanza pulled a year of Mexico's own flagged and rejected ads the week after and built a segmented set for that market: sixty real ads, including the kind of claim that had just cost them two weeks. Run the same brief against it now, and Fieldcopy flags the curative-sounding claim itself, before anyone ever taps approve. The ad never goes live. Mexico's push keeps its two weeks.

One design hands Esperanza a green light she has no way to double check. The other hands her the one flag that actually mattered.

And the thing I'd tell myself, if I could go back: we built a golden set to answer "is this a good ad." We never asked whether a good ad in one market could be an ad we're not allowed to run at all in another, and only the second question could actually stop a campaign.

PICK, run against one region

This is a tradeoff about where to spend eval hours, not a checklist for building any old test set, so PICK is the tool.

Run PICK again, at an insurance desk

ClaimSort reads an incoming insurance claim, the policy, the damage report, the photos, and drafts a settlement recommendation across every claim type a mid-size insurer handles: fender-benders, kitchen fires, and further down the list, flood damage on a total loss. Same shape of question, a different kind of harm.

P. Segment ClaimSort's golden set for catastrophic and flood claims before its recommendation touches a real payout, off the shared floor used for routine auto and homeowner claims.
I. A false "needs a person to look" flag on a routine fender-bender costs an adjuster maybe fifteen minutes of double-checking, felt the same day. An underpaid flood claim built on a golden set that never had enough real flood cases doesn't surface until a policyholder appeals or a regulator audits, months later: a corrected payout, and real penalty risk on top of it.
C. Catastrophic claim patterns are rare in any small, evenly-spread sample, and a shared set will always be dominated by the routine claims that make up most of the volume. The miss doesn't cost anyone until the payout's already sent, or worse, already disputed.
K. Drop the segmented pass for a claim type once it settles into a couple of rules that don't interact with anything else, the same line Fieldcopy's team uses.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0