What happens to your golden set when the product's scope expands?

How to answer this, stage by stage

Eight moves. The timeline itself carries half the argument here, so this one spends two full stages just pinning down the gap between when the roads changed and when anyone touched the test.

Let's learn

Pathwise is a screen a dispatcher already has open before the first van leaves the lot. Every morning it looks at that day's deliveries and works out the order a driver should hit them in.

Before anyone tuned anything, Pathwise's route-match score, how closely its stop order matched what a senior dispatcher would have picked, was rough. It matched only sixty-six out of every hundred of Zanyar's hand-built downtown examples. That felt right. Nobody trusted it yet.

Ten months and five retrains later, that same score sits at ninety-seven. Every retrain beat the one before it. Every retrain replaced what was already live.

Here is the turn. Those thirty-one extra points are not the real story. The real story is what never showed up anywhere on that climbing line: how Pathwise plans a route it was never built to test. That split sat underneath the whole ten months, quietly getting worse as regional routes grew from a sliver of the day's work to more than a third of it.

At its worst, this costs Cinderpath drivers stranded on roads nobody built a test for. Regional routes, the ones with a far zone, an electric van, or a bundle of stops, only land on time sixty-one percent of the time. And regional routes are no longer rare. They are thirty-seven percent of everything Pathwise plans on a given day.

The gap between a climbing score and the road it was never shown

The choice I would take back. Building the golden set only from downtown routes, ten months ago, was the sensible call. Regional, electric, and bundled routes did not exist yet. There was nothing else to build it from. I would take back leaving the set frozen there with no trigger tied to scope changes. I would tie a golden-set refresh to every scope launch, the same way a new feature gets a rollout plan.

What I would leave alone. The downtown segment does not need touching. It lands ninety-four percent of deliveries on time, same as before the expansion, because that is exactly the job the golden set was built to test. More review there would only slow down the part that already works.

The lesson. A golden set is a snapshot of the world on the day you built it. The day the product's job changes, that snapshot does not know it is out of date, and it keeps grading you as if nothing changed, unless someone tells it otherwise.

A frozen night on a farm road fifty-two miles out

Read the short version above if you are short on time. This is the long version, for when you want to feel exactly where the ten months went.

The dispatch floor at Cinderpath Delivery gets loud around 6am, once the regional runs start rolling out behind the downtown ones. Zanyar Rahimi has run eval and ops scoring for Pathwise for two years. He can look at a week's match score and tell, inside a minute, whether a retrain earned it or got lucky on a slow week.

He built the golden set himself, with two senior dispatchers, the month Pathwise first launched. A hundred and eighty routes pulled from real downtown deliveries, each one checked against what a senior dispatcher would have planned by hand. Fast to build. Easy to trust, because everyone on the team had driven those exact streets.

For the first four months, watching the match score climb felt like the model getting sharper. Sixty-six. Seventy-eight. Eighty-five. Every point felt earned. Somewhere in there, without anyone deciding it on purpose, the team quietly stopped doing something it used to do every few weeks: pull a handful of that week's real routes and check them by hand against what Pathwise had picked. Once the golden score was climbing fast on its own, the manual check started to feel like double work.

Then Cinderpath signed two grocery accounts that needed same-day delivery past the ring road, up to fifty-five miles out. Ops swapped half the exurban fleet to electric vans to meet the contract's terms, and let Pathwise bundle up to twenty stops on one run instead of six, to make the long drive worth it.

Nobody touched the golden set. Why would they. It kept scoring in the nineties.

Three weeks before the holidays, a Pathwise-planned regional route sent a driver fifty-two miles out on a bundled eighteen-stop run, including a charging stop the route had placed wrong. The van sat at four percent battery on a farm road with no signal, for almost three hours. Eleven of the eighteen deliveries on that run were frozen groceries. All eleven had to be written off.

Zanyar's first thought, and everyone's first thought, was that this was a fluke. One bad electric route on one bad day. Then Marisha Tan, the regional account manager, pulled a month of regional deliveries for a client call and found the near miss was not rare at all. In a smaller way, without a stranded van attached to it, it was happening on nearly four regional routes out of every ten.

I want to say the model got worse at picking routes. It did not get worse. It just never got asked, not once in a hundred and eighty examples, to plan a route with a charging stop in it, or a bundle past six stops, or a zone past the ring road.

So here is the decision I would take back.

Ten months earlier, building the set only from downtown routes was the sensible call. Regional, electric, and bundled work did not exist yet to pull examples from. I would put a refreshed, tagged sample back alongside it: real regional routes, real bundled routes, checked every time scope changes, not just at launch. Not because the downtown set is wrong. Because of what it would have shown, months earlier: ninety-four percent on time for the job it was built to test, sixty-one percent for the job nobody ever wrote a single example of.

That is the whole difference. One process trusts a score that only ever answered for the roads it already knew. The other checks that score against the roads drivers are actually on tonight.

And the part I would want to tell myself, if I could go back: we built a test that only ever answered for the map we already had. Nobody decided that on purpose. We just never redrew it when the map changed.

What the recut actually showed

Before blaming the golden set's coverage, Zanyar's team checked whether the score was even being read right. Two dispatchers hand-checked a sample of real routes against what the golden set's own rubric said they should score. They agreed on nearly every one. The grading was not the problem. That left the coverage.

Three ways a golden set stops matching the job

Not because anyone cut a corner on purpose. A set built entirely from the old scope can be met honestly, retrain after retrain, and still stop meaning anything the day the job changes.

Three separate, checkable ways a set can pass and still miss the new job

TRACE, for a score that graded the wrong map

This reads like a question about a document, but the real job is diagnosis: work out why a set the team trusted for ten months quietly stopped matching the job. GUARD would fit if the harm were one group losing a fair shot at an appeal; here the harm is a whole kind of route the set never learned existed.

Same shape, a permit tool that only ever saw houses

Thistlewick County's building department runs PermitLens, a tool that pre-screens permit applications for missing documents before a human reviewer sees them. Ilhan Botros, the model owner, built the acceptance set two years ago: a hundred and thirty applications, all single-family home renovations, scored against a benchmark called PermitCheck. Every quarterly retrain has had to beat the last version's score on that set before it replaces the one screening live applications.

T. PermitCheck's score climbed from seventy-two to ninety-one over six quarterly retrains. The share of commercial and multi-unit applications correctly flagged for missing documents never moved off roughly one in three, the whole time.
R. Split by application type. Residential, sixty-eight percent of volume: PermitLens flags correctly eighty-eight percent of the time. Commercial and multi-unit, thirty-two percent of volume: thirty-four percent.
A. Before blaming coverage, two reviewers blind-recheck thirty flagged applications against PermitLens's own output. They agree on twenty-eight of thirty. The scoring checks out.
C. Three candidates: the acceptance set's hundred and thirty examples are all residential, none commercial; the rubric checks for residential-only documents, a single owner's deed, homeowner's insurance, that a commercial application does not even use, so the few commercial cases in the set get graded on the wrong checklist; and the county's digital-services group added commercial permitting nine months ago without telling the team that owns the acceptance set.
E. Compare the share of commercial and multi-unit applications in the acceptance set. Three of a hundred and thirty, about two percent. Thirty-two percent of real applications now. PermitLens barely tests the application type that is now a third of what it reviews.

Flashcards (click a card to flip it)

This is a diagnosis question about a test set outliving its own scope, not a habit changing, so these eight test the TRACE moves and the real numbers behind them.

Check yourself Score: 0 / 0