What success criteria should be agreed before a pilot begins?

How to answer this, stage by stage

Eight moves. Name what the numbers can and can't settle before the story, or the answer sounds like project management instead of a decision.

Let's learn

Every month, Grovemill Crate's retention team called the subscribers who had already paused or skipped that week's box. That was the whole method. Wait for someone to act like they were leaving, then try to talk them out of it.

Over a normal eight-week stretch, that reactive method saved about nine subscribers. Not nine percent. Nine people, total, out of thirty-eight thousand, because by the time someone skips a box, they've usually already decided.

Anchorlight's pilot changed the shape of the problem. Every week, it scored all thirty-eight thousand subscribers for risk of canceling in the next thirty days, before anyone skipped anything. It flagged about fifteen hundred people a week as high risk. Grovemill's retention team, three people, could realistically call or email about five hundred. So the team worked the top five hundred scores each week and left the rest.

Here's the turn. Fifteen hundred flags against a five-hundred-person capacity was not really the problem either, though it should have been fixed. The real problem showed up eight weeks later, at the readout meeting, when the numbers came in and nobody in the room had agreed, before that morning, what a good number even looked like.

At its worst, that kind of gap doesn't just waste a meeting. A pilot that genuinely worked gets shelved because the loudest objection in the room happened to land after the fact, with nothing written down earlier to weigh it against. Or a pilot that genuinely didn't work gets waved through, because the one flattering number got said first and nobody had a pre-agreed number to hold it against.

The false-alarm side of the same pilot told the opposite story. Sixty-nine out of every hundred people the model flagged would have stayed even if nobody called them. The retention team spent three of their five days a week on people who were never leaving.

Same eight weeks. Same report. Two people circled two different numbers and both were right.

What I'd leave alone. Nobody needed to pre-agree the exact wording of the retention offer email, or which color the dashboard uses for a high-risk score, or how many people get called first each morning. Those can flex all pilot long without changing whether the pilot's own question gets answered honestly.

The lesson. A pilot without a number agreed in advance doesn't fail to produce an answer. It produces two answers, both true, and lets whoever argues harder pick which one counts.

Now here is the same thing as a story

Use this version when you've got a few minutes. The short version is above. This is for when the decision actually needs to survive a room full of people who disagree.

Every Friday afternoon, before the call with Grovemill, Priyanka pulled up the week's model scores and read them the way other people read a weather report. Two years running product for Anchorlight had taught her to check the shape of the numbers before she checked whether they were good.

The first month of the pilot was quiet in the best way. The scores climbed steadily, week over week, nothing dramatic, no red flags in the pipeline. Grovemill's retention team worked their five hundred calls, reported back what worked, and the Friday call ran fifteen minutes. Priyanka liked those calls. Everyone seemed to be rowing the same direction.

By week six, the numbers were genuinely good. Forty-four percent of the people who actually canceled had been flagged in advance. Priyanka started drafting the readout deck early, the way you do when you're fairly sure how a thing is going to land.

Then week eight arrived, and with it the two final numbers. Fifty-one percent caught. Sixty-nine percent false alarm rate. Priyanka put both on the same slide, because both were true, and walked into the readout expecting a conversation about what to do next.

She got an argument instead.

Anchorlight's VP of sales spoke first. "We caught half the people who would have quit, before they quit. Nobody in retail loyalty gets that number. This is a clear yes." Grovemill's retention lead spoke right after him, and she wasn't wrong either. "My team spent three days a week for two months calling people who were never leaving. That's not a rounding error, that's most of their week." Neither one raised their voice. Neither one was lying. They were reading the same eight weeks and landing in different countries.

Priyanka sat through forty minutes of a meeting that was supposed to take ten. Nobody moved. The VP had the win-number memorized. The retention lead had the wasted-hours number memorized. Both of them had shown up ready to argue a case, because that's what an unwritten pass bar turns a readout into: a trial, with everybody as their own witness.

Two months earlier, at kickoff, someone from Grovemill had actually asked, "so what number are we looking for here?" Priyanka remembers answering, "let's run it for eight weeks and see, and we'll figure out together what counts as good," and the room nodded, because it sounded reasonable, even generous. Nobody circled back to pin it down. It just quietly became the plan.

I would take that back. I'd have said, in that same kickoff meeting, "before we start the clock, I want two numbers on paper: the lowest catch rate that makes this worth Grovemill's time, and the most false alarms your team can absorb without it costing them. Let's agree those today, and agree who says yes or no at the end, before either number exists to argue about."

Here's the replay. Same eight weeks, same fifty-one percent, same sixty-nine percent, but this time the kickoff page says: catch at least forty percent, keep false alarms under seventy-five percent of the flagged list, Grovemill's VP of ops signs off within five business days of the final number. The readout meeting still happens. It runs twelve minutes instead of forty, because the room isn't deciding what the numbers mean anymore, only reading them against a bar that was already agreed two months earlier.

One version hands the room two true facts and lets them fight. The other hands the room a yes or a no that was already decided, and just needs reading out loud.

And the thing I'd tell myself, back in that kickoff meeting: "we'll figure it out together once we see it" isn't collaboration. It's postponing the hardest conversation to the one moment everyone has the least reason to agree.

LEAD, for a pilot that has no number yet

This sounds like a question about what to measure. Underneath, it's still asking whether a team can act on a decision made in advance, instead of a fight made after the fact. That's LEAD, run on a pilot's own ending instead of a dashboard.

And if you want to be sure it really works, try it somewhere else

Northfell Freight runs a hundred and twenty trucks out of three regional yards. Bertholt Aldercroft leads product for a predictive-maintenance tool that flags which trucks are likely to break down before they actually do, and Northfell agreed to a ninety-day pilot across the fleet.

Bertholt pulled the pilot's own logs the week after the highway incident: eleven of fifteen real breakdowns caught, with an average lead time of six days. He printed both numbers next to the seventy-percent floor Northfell had agreed to at kickoff, and the follow-up meeting, expected to run an hour, lasted nine minutes.

Waiting for the breakdown tells you last. The pre-agreed warning sign was already ringing days earlier.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0