Explain the difference between a golden set, a regression suite and a benchmark.

How to answer this, stage by stage

Seven moves. The trap in this question is answering it like a glossary entry, so the middle stages force the definitions to earn their keep against one real product.

Let's learn

Whyline is a screen on a picker's handheld, next to the route Dockrow's warehouse app already gives. It doesn't choose the route. It just says why, something like "go by aisle 9, dock 3 is holding for a charge."

When Whyline launched, Noelle Kirkbride built three checks. A golden set: thirty-five hand-picked routes with the exact right reason written next to each. A regression suite: five hundred real explanations pulled every week and auto-graded for format, right length, names an aisle, no contradictions. A spot in a public warehouse-AI benchmark, scored on how clear and useful the explanations read next to other vendors' systems. All three came back green, and stayed there. The golden set held near 97 percent. The regression suite held in the mid-90s. The benchmark score climbed from 84 to 89 over five months.

Here is the turn. Those three green scores were never the problem. The real story is a kind of route none of the three tests were built to check: one where the app now holds back for a charging slot, and the stated reason never caught up.

At its worst, this costs the floor real minutes on a busy night. Battery-reserved routes were now more than a quarter of everything the floor walked, and three in four of those explanations named the wrong reason. Pickers who trusted the stated reason took the same wrong shortcut, over and over, night after night.

The gap between three green tests and the change none of them were built to see

The choice I would take back. Building three real tests at launch was the sensible call. Back then, the app had never given a wrong reason, so nobody thought to check for one. I would take back treating those three tests as finished once they passed, with nothing that forced a re-check the day the optimizer's own reasoning changed underneath them.

What I would leave alone. The regression suite's format checks don't need touching. They still catch a route explanation that's too long, or doesn't name an aisle, at close to the same rate they always did. Rewriting checks that already work spends time the real gap needs.

The lesson. A test can be real, and still be answering a question nobody is asking. Three green scores told us the explanations were well formed, well scored, and matched a fixed set of known cases. None of them ever asked if the stated reason was the true one.

The night Rowan Ibarra stopped trusting the reason on the screen

Read the short version above if you're short on time. This is the long version, for when you want to feel exactly where the six months went.

Noelle Kirkbride has run eval for Dockrow's warehouse software for four years. Before Whyline, her job was rating routes: was it short, was it safe, did it clear the floor fast. She built the first golden set herself, thirty-five routes nobody on the team could argue with.

For the first five months after Whyline launched, its dashboard was the easiest read in her week. Monday, she'd check the regression suite: green. She'd glance at the golden set: 97, same as always. Once a quarter she'd check the benchmark leaderboard: 84, then 86, then 89, climbing.

Early on, she also pulled twenty real explanations herself every Friday and read them the way a picker would, line by line. By month three, she was down to five. By month five, she'd stopped. The dashboards had never once told her something was wrong, so there was nothing left to check for.

Then, at a Tuesday shift huddle in month nine, a picker named Rowan Ibarra mentioned it in passing. "The app keeps telling me it's the short way," he said. "That's not why aisle 14 is slow. It's the charging queue. It's always the charging queue now." Nobody wrote it up. It wasn't a complaint. It was barely a sentence.

Noelle almost let it go. Her first thought was to add five new cases to the golden set and call it handled. But "always" stuck with her, so instead she pulled the decision log for aisle 14 and matched it against fifty real explanations from that same week.

Twelve of the fifty named the actual reason. The other thirty-eight all said some version of "shortest path," while the app itself had really held the route back to protect a charging slot.

I want to say the app got worse at writing explanations. It did not get worse. It got fluent enough to sound sure of a reason it had stopped checking, and none of the three tests had ever been built to notice.

So here is the decision I would take back.

Ten months earlier, in the room where they'd greenlit Whyline, someone had asked whether the golden set and the regression suite would need touching once the optimizer's own logic changed. The room's answer was no. "The tests check the output, not the model," someone said, and it sounded right at the time, because back then the output had never lied.

I would put a fourth check in from month six, the day the charging-slot rule shipped: pull fifty real explanations, tagged by route type, and check each one against the decision log by hand. On the battery-reserved routes, that check would have caught the gap three months earlier, on the first bad batch, not the four-hundredth.

And the part I'd want to tell myself, if I could go back: we built three real tests and never once asked, out loud, which question none of them were built to answer.

What the segment split actually showed

Before blaming coverage, Noelle's team checked whether the regression suite's own scoring was even right. Two people hand-checked 20 flagged battery-reserved explanations against what the automatic suite said. They agreed on 18 of 20. The grading itself was not the problem. That left the coverage.

Three reasons three green tests can all miss the same lie

Not because anyone cut a corner on purpose. Each of the three tools earns its own kind of trust, and that trust is exactly what lets a mistake none of them were built for slip through all three at once.

Three separate, checkable reasons the three tools get treated as interchangeable

Running TRACE on an explanation nobody fact-checked

This reads like a question about definitions, but the real job is diagnosis: work out why three real, passing tests all missed the same kind of lie. GUARD would fit if the harm landed unevenly across a protected group; here the harm is a whole kind of mistake none of the three tools ever learned existed.

Same shape, a permit reason nobody had double-checked either

A city permits office runs an AI tool that drafts the first reason a building application gets flagged as incomplete, for a clerk to review before it's sent. Elsbeth Rourke, the product lead, built a golden set and a regression suite eight months before the office added a new zoning rule for backyard units, plus a spot in a public civic-tech benchmark.

T. All three stayed green across three releases, including the month the zoning rule shipped. Applicant complaints about a flagged reason that didn't match the actual case file held steady around one in twenty for months, then started climbing five weeks after the rule went live.
R. Split by whether the flag touches the new zoning rule. Non-zoning flags, 74 percent of volume: complaint rate barely moved. Zoning-linked flags, 26 percent of volume, up from 5 percent before the rule: 51 percent of them named a reason the case file didn't actually support, up from 4 percent.
A. Before blaming coverage, staff hand-check 15 flagged applications against the regression suite's own verdict. They agree on 14 of 15. The scoring checks out.
C. Three candidates: the regression suite feels like a quality gate because it fires on every release, though it only ever checked format and tone; the benchmark feels authoritative as a shared civic-tech standard, though it never checked whether a flagged reason matched the file; and the golden set felt redundant once the other two existed, so nobody added a single zoning-rule case to it.
E. Pull 40 real zoning-linked flags sent after the rule change, and check each against the actual case file. Suite pass rate: 93 percent. Real match to the file: 46 percent.

Flashcards (click a card to flip it)

This is a diagnosis question wearing a definitions question's clothes, so these eight test the TRACE moves and the real figures behind them, not just the three words.

Check yourself Score: 0 / 0