Build the vendor scorecard you would take to a procurement committee.

Kestrel Systems builds internal tools for a mid-size logistics software company. Priya Nandakumar runs the developer platform team and is choosing an AI code-review vendor for two hundred engineers. She's building the scorecard that goes to the procurement committee before anyone signs anything.

How to answer this, stage by stage

Nobody in the room is grading your PowerPoint skills. They're grading whether the tool you hand them would have caught the vendor that looked perfect on stage and fell apart on week three.

Let's learn

Before the scorecard existed, here is how Kestrel picked a vendor.

A platform engineer would sit through a thirty-minute demo, skim a feature comparison deck, and guess at fit based on a gut read of the sales call. That process took about two hours per vendor and produced a recommendation nobody could really defend later, because nobody had written down why one vendor beat another.

Two hours per vendor, and the weakest step was the one nobody wrote down.

Here's the turn: a written scorecard doesn't just make the choice faster. It moves the whole decision from "who pitched best" to "who held up on our own code, under our own contract terms." That's not a small change in paperwork. It's a change in what gets to count as evidence.

Five lines, one number. Not five separate green checkmarks that never get added up.

At its worst, skipping the pilot line costs Kestrel a year-long contract with a vendor whose review quality never improves past a coin flip, on code two hundred engineers touch every day.

Same vendor, same week. One score comes from a script. The other comes from ten years of real code.

The choice I would take back: an earlier draft of the scorecard folded "pilot results" and "demo quality" into one shared column called "product quality," each worth half. That made sense when the team assumed a good demo and a good pilot would usually agree. It stopped making sense the day a vendor scored a 9 on the demo half and a 4 on the pilot half, and the shared column quietly averaged them into a 6.5 that told the committee nothing true.

What I would leave alone: the scorecard doesn't need a line for every minor IDE plugin or a comparison of every pricing tier Kestrel will never reach at two hundred seats. Grading those to the same depth as security and pilot performance would just bury the two lines that actually decide the vendor.

Naming what's out of scope is part of the design, not a gap in it.

The lesson: a scorecard is not a record of what a vendor showed you. It's a bet on what a vendor will do once nobody from their sales team is in the room anymore.

Now here is the same thing as a story

The short version above is what you'd walk the committee through in the meeting. Read this one for how close Kestrel actually came to getting it wrong.

Priya Nandakumar had run Kestrel's developer platform team for four years and had sat through more vendor pitches than she could count. She could tell within ten minutes of a sales call whether a rep was overselling, and her team trusted her read on almost everything.

Two vendors made the shortlist that quarter: Fenwick AI and Northlane AI. Both demoed well. Both had glossy accuracy numbers on their own benchmark decks. For the first few weeks of the evaluation, the informal read inside the team leaned toward Northlane, its demo had felt slightly slicker, its sales engineer slightly sharper on follow-up questions.

Then the pilot started. Both vendors got two weeks, unpaid trial access, running live against Kestrel's actual pull request queue, not a curated sample. Fenwick's suggestion-acceptance rate climbed steadily as engineers got used to its style. Northlane's stayed flat, and then started slipping, as its suggestions kept flagging patterns that were common, harmless quirks in Kestrel's own fifteen-year-old billing module.

Northlane's demo was the test drive. The billing module was the pothole nobody drove over during the demo.

By week six, Fenwick's engineers were accepting its suggestions on nearly two out of three pull requests. Northlane's number had drifted under forty percent and wasn't moving. If Priya's team had signed off the informal, demo-driven read from week one, Kestrel would have committed a year of licensing fees to the vendor whose real-world number was going the wrong way.

The old, undocumented process would have signed Northlane. The scorecard, with the pilot weighted above the demo by design, caught the split before procurement ever wrote a check.

Fenwick moved from the middle branch to Buy after six weeks. Northlane never left it.

SPARK, in one screenNot a checklist of nice-to-haves. SPARK is what tells you which one design decision the whole scorecard has to protect.

The recap, one line per letter: situation is the two-hour, gut-feel process the scorecard replaces, payoff is trading "best pitch" for "best six weeks on our own code," anchor is weighting the live pilot above the demo, risk is protecting that anchor with a minimum pilot window, and keep out is naming three things left ungraded on purpose.

And if you want to be sure it really works, try it somewhere elseSame five letters, a grocery chain's AI restocking vendor instead of a code-review tool. A different keep-out line breaks the second story.

Pinegrove Market is a regional grocery chain choosing an AI shelf-restocking vendor, Shelfline AI, to predict which items run low before a store associate notices. Esther Adeyemi, Head of Store Operations, is building that scorecard for her own procurement committee. Mapped onto SPARK: situation is a store manager currently reordering by habit and a weekly paper count; payoff is replacing "reorder what usually runs low" with "reorder what this specific store's real sales pattern says will run low this week"; anchor is the same shape as Kestrel's, weighting a live pilot in three real stores above Shelfline's own case-study numbers from other chains' stores.

The keep-out line is where this story breaks differently. Kestrel left minor integrations and unused pricing tiers ungraded. Pinegrove's early scorecard draft instead left seasonal and holiday demand ungraded entirely, reasoning that a four-week pilot couldn't capture a pattern that only shows up once a year anyway. That made sense as a scoping decision. It stopped making sense once the pilot happened to run through a regional harvest festival week, and Shelfline's restocking suggestions, tuned on a chain with no such event, badly underordered produce for three straight days, a gap the scorecard had no line built to catch, because nobody had asked what happens in the one week the model has never seen.

Pinegrove's pothole was a festival week. Nobody had asked the scorecard to watch for it.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "weight a real pilot on our own data above the demo, and name what you're deliberately not grading yet," and stop.
Cost: there's no budget for a paid pilot before signing. Say so honestly, and shrink the pilot to the cheapest slice that still touches real, messy data, rather than skipping straight to a demo-only score.
The model gets better, for real: if a vendor's next release genuinely closes the gap a pilot once caught, that's the pilot number doing its job again, telling you it's safe to re-score, not proof the pilot step is no longer needed.

Where people run it wrong.
They let the demo and the pilot share one averaged column instead of scoring them separately.
They try to grade everything, including details that don't decide anything, and bury the two lines that do.
They never name what's deliberately out of scope, so a gap like a festival week reads as a scorecard failure instead of a scoping choice made in the open.

How to use it live. When someone asks you to build an evaluation tool, ask back: what's the one thing a slick pitch could fake, and what's the one thing it can't? Build the scorecard so only the second thing can move the vendor down.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0