Design the pilot you would run to evaluate two competing AI vendors.

Greymarsh Terminal handles inbound shipping containers for a mid-size port. Iris Halvorsen runs container inspection there. Portalis and Quaydeck both sell AI that reads camera footage of a container and flags likely shipping damage before a truck driver takes possession.

How to answer this, stage by stage

Nobody is scoring whether you can name two vendors. They're scoring whether your pilot design stops either vendor from controlling its own evidence.

Let's learn

Here is what happens when a team lets a fast, exciting demo stand in for a real evaluation.

Before either vendor, Greymarsh's inspectors checked every inbound container by eye, about four minutes each, roughly two hundred a day across the team, catching around eight of every ten billable damage cases before a truck drove off with it. An AI read on the same camera footage could flag likely damage in under a second, at every gate, all day.

This is the job neither vendor has touched yet. The anchor only makes sense next to this.

Here's the turn: speed was never the risk in this design. The risk was that Greymarsh's original evaluation plan merged two separate questions, does this even work, and which of the two is better, into a single same-day bake-off, using whatever containers each vendor's own rep chose to show. That felt efficient with peak season two weeks out. It also meant each vendor got to pick its own best evidence.

At its worst, this doesn't just risk picking a slightly worse vendor. It risks signing a multi-year contract based on a reel that was never a fair test, then discovering the gap only once real containers, and real disputed claims, are already on the line.

What I would leave alone: the terminal's basic gate-camera hardware doesn't need re-evaluating alongside the vendors. Both companies work with the same existing camera feed, so the hardware itself was never the variable in question.

The lesson: a fast bake-off feels efficient right up until you realize you let the two companies being judged also choose the evidence they're judged on. The fix isn't more time. It's who controls the sample.

Now here is the same thing as a story

The short version above is what you'd say defending the pilot design to the terminal's operations committee. Read this one for how close the reel came to deciding it instead.

Iris Halvorsen had run container inspection at Greymarsh for nine years, sharp enough to spot a hairline dent through a truck's side mirror at thirty feet. When both vendors came calling with pilot offers, she scheduled what felt like the obvious next step: a live side-by-side demo, one afternoon, both systems watching the same gate.

The demo day never tested either side of this picture. It just showed easy containers where nobody was ever wrong.

Portalis's sales rep arrived with a reel: forty containers, hand-picked from a customer reference site, each with an obvious dent or a torn seal. Quaydeck's rep, seeing the bar had been set, brought a similar reel of his own. Both systems called every container correctly. Iris nearly signed a letter of intent with Portalis that same week, since it had called its forty a beat faster.

A dockworker named Nils Aabo, restacking pallets nearby, glanced at the monitor mid-demo and said something that stuck with Iris: "These are all the easy ones. Where's the container from Tuesday, the one with the scuff that took three of us arguing to call?" Nobody had an answer.

Iris pulled the letter of intent and redesigned the evaluation from scratch: six weeks, both vendors reading the same live, unfiltered gate stream, scored against her own team's independent call on every container, released to blind review so nobody knew which vendor's flag they were checking.

This is what replaced the reel. Every part of it exists to take control away from whoever's selling.

The near miss happened in week five of the redesigned pilot, not week one of the sales demo. That's the whole point of shadow mode.

SPARK, in one screenNot a longer bake-off. SPARK is what stops a vendor demo from quietly writing its own grade.

None of these are missing by accident. Each one is a decision the pilot deliberately postpones.

The recap, one line per letter: situation is the manual inspection process both vendors are meant to change, payoff is inspectors focusing only on escalated cases, anchor is the blind, parallel, shadow-mode design, risk is inspectors behaving differently if they know which containers are being watched, and keep out is withholding auto-release authority from both vendors for the entire pilot.

Portalis caught more damage. It also called more false alarms. Neither number alone would have told Iris that.

And if you want to be sure it really works, try it somewhere elseSame five letters, a recycling plant's contamination-sorting cameras instead of a shipping port. A reporting decision breaks the second story, not a sales reel.

Gullwash Recycling piloted Clearbin and Binscope, two vendors selling AI cameras that spot contaminants on a sorting line before they reach a bale. Mapped onto SPARK: situation is line workers manually pulling contaminants off a moving belt today, payoff is workers focusing only on the items either camera flags as unclear, anchor is the same shadow-mode design, both cameras watching the same belt for six weeks, scored blind against a worker's own call, risk is lighting changing sharply between day and night shifts in ways a vendor's daytime demo would never reveal, and keep out is neither camera getting authority to auto-reject a bale during the pilot.

The old decision here isn't a sales reel, it's a reporting one: the plant's original test setup had each vendor's dashboard report a single end-of-day accuracy number, with no breakdown by shift. That made sense, since daily summaries were how every other machine on the line got tracked. It stopped making sense once the night shift's dim lighting turned out to be exactly where both systems struggled most, hidden completely inside a daily average that looked fine.

This is what the shift breakdown made possible. A single daily average could never have produced this rule.

Swap the trigger and it still runs.
Speed: an interviewer caps you at a minute. Say "shadow mode, same stream, blind scoring, no auto-release," and stop.
Cost: there's no budget to run both vendors for six full weeks. Say so honestly, and shrink the window rather than the sample, testing fewer weeks on the same real, unfiltered stream instead of a shorter test on a curated one.
The vendor gets better, for real: if a vendor's next model version genuinely narrows the gap with a competitor, that's the blind scoring doing its job, telling you the decision is closer than the last quarter's numbers suggested.

Where people run it wrong.
They let each vendor supply its own demo sample instead of using one shared, unfiltered stream.
They score against a vendor's own reported accuracy instead of a blind, independent human call.
They report a single average number instead of breaking results out by the conditions that actually vary, like shift, lighting, or container type.

How to use it live. If an interviewer pushes for more detail, name the one design choice that would have caught the near miss: whoever controls the sample controls the result, so never let the vendor choose it.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0