What is the right pilot length and success criteria for an AI vendor evaluation?

Alderwick Stage Collective is a small nonprofit consortium of five theaters. Corwin Delacroix manages the box office and set pricing by hand for years. Seatwise is the vendor pitching AI dynamic pricing to recommend seat prices show by show. Priyasha Nayar, a marketing coordinator three weeks into the job, asked the question that changed the pilot plan.

How to answer this, stage by stage

Nobody is scoring whether you can say "run a pilot first." They're scoring whether you can name the one number that would have warned you early, and the one that wouldn't.

Let's learn

Say a nonprofit theater consortium wants an AI tool that sets ticket prices show by show, instead of a box office manager guessing at a fair tier from memory.

Before Seatwise, Corwin Delacroix set prices by hand, checking last season's attendance and picking a tier that felt fair, about three hours of work per show. Later analysis showed that manual guess left about twelve percent of possible revenue unrealized most seasons, mostly on shows that sold out faster than expected.

This is the whole job Seatwise is meant to sharpen. Nothing about it names a real subscriber yet.

Seatwise proposed a twelve month pilot, run across all five venues at once, with one success measure: at least ten percent revenue lift and no reported season-holder cancellations by the end of the year.

At its worst, trusting the vendor's twelve-month, revenue-only plan risks running a subscriber-alienating pricing pattern across all five venues for nearly a full year before anyone found out, by which point a whole season of loyal subscribers could have quietly moved on. Watching a weekly leading signal instead cost Corwin about two extra hours a week of tracking. Against a year of unnoticed damage across the whole consortium, that trade is an easy one to accept.

What I would leave alone: a low-stakes tool, like an AI assistant that drafts internal show-schedule emails, doesn't need this level of pilot design. A short trial and one end-of-month read is genuinely enough there, since nothing loyalty-related is on the line.

The lesson: a good pilot isn't measured by how long it runs. It's measured by whether its success number would have warned you weeks before the number the vendor wanted you to wait for.

Now here is the same thing as a story

The short version above is what you'd say defending this pilot design to Alderwick's board. Read this one for how the real signal actually got found.

Corwin Delacroix had priced Alderwick's five stages by hand for eleven years, and could guess a fair tier for any show within a few dollars just from remembering how fast last year's version sold.

Seatwise's proposal was the left side of this picture. Alderwick shipped the right side instead.

Seatwise's sales team pitched the twelve-month, all-venue rollout as the fastest way to see a real number. Corwin nearly signed it as written. It read like due diligence: a whole year of data, a clean pass or fail at the end.

Priyasha Nayar, three weeks into her marketing coordinator job, asked one question in the planning meeting: "What happens to our season holders if this is wrong for a whole year before we find out?" Nobody in the room had an answer.

The vendor's proposed metric sat in exactly the wrong corner of this map.

Corwin had considered simply asking Seatwise for weekly revenue reports instead of building anything new. He dropped that idea. Revenue reports come from the vendor's own system, and they wouldn't catch a subscriber behavior shift the vendor's own model wasn't tracking as a metric at all.

Instead, Alderwick bounded the pilot to its smallest venue and eight weeks, and Corwin added one number of his own: how many season-ticket holders bought a single ticket instead of using their season seat, checked every week, against a normal baseline of about three percent a month.

These thresholds were set before the pilot started, not argued over once the number looked bad.

By week three, the switch rate had already climbed to nine percent. By week four it crossed the ten percent pause line Corwin had set in advance. Seatwise's model, it turned out, was quietly raising prices fastest on exactly the shows season holders always claimed early, since to the model that early claiming looked like strong demand from new buyers, not loyalty from people who already had a seat.

Alderwick paused the pilot at week six, added a season-holder carve-out to Seatwise's pricing rules, and re-ran the same eight weeks. The switch rate settled back near its normal three percent, and revenue still rose, just nine percent instead of eleven, a smaller gain the consortium was glad to take in exchange for keeping its subscribers.

LEAD, in one screenNot a lecture on picking north star metrics. LEAD is what tells you which number would have warned you first.

Seatwise's original plan had none of these four. That absence was the whole warning.

The recap, one line per letter: link is naming what the theater actually can't afford to lose, early signal is the season-holder switch rate moving weeks before renewal data ever could, abuse is revenue lift looking healthy while that switch rate climbs underneath it, and decision is the pause and rollback thresholds set before the pilot ever started.

Eight weeks, one venue, a real decision at the end. Not twelve months across everything Alderwick runs.

And if you want to be sure it really works, try it somewhere elseSame four letters, a funeral home network instead of a theater consortium. A different early signal breaks the second pilot.

Grayfield Funeral Alliance piloted Rememberly, a vendor offering AI-drafted tribute page content for families to personalize. The vendor's proposed success measure was "fewer than five formal complaints over six months." Mapped onto LEAD: link is the real outcome, families feeling the tribute page truly represented their person, not merely the absence of a complaint. Early signal is the rate of families requesting a full manual rewrite instead of lightly editing the AI draft, which climbed weeks before any family would file a formal complaint, since most grieving families quietly redo the page rather than complain about it. Abuse is that "complaints" as a metric rewards families staying silent, not the tool actually working. Decision is treating a manual-rewrite rate above twenty percent in any two-week window as an automatic pause, set before the pilot began.

Grayfield's early signal wasn't a number on a dashboard. It was families quietly asking for a redo.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "bound the pilot, name the number that moves before the damage does, and set the pause line up front," and stop.
Cost: there's no staff time to track a new weekly number by hand. Say so honestly, and pull the smallest possible proxy from data you already have, rather than skipping the leading signal entirely.
The model gets better, for real: if Seatwise's next version genuinely stops over-pricing season-holder seats, that's still worth a fresh short pilot, since a fixed model deserves the same scrutiny as a first one.

Where people run it wrong.
They accept a vendor's proposed pilot length and success metric without asking who wrote it and what it's convenient for them to measure.
They pick a single lagging number and wait the whole pilot to check it, instead of watching something weekly.
They never set the pause threshold in advance, so it gets negotiated away in the moment by whoever wants the pilot to keep running.

How to use it live. The moment someone proposes a pilot's length and success metric, ask: what would move first if this goes wrong, and can we watch that instead of waiting for the number that only shows up at the end? Let that answer set the real plan.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0