Build a scoring rubric for ranking ten candidate AI features.

Haventhorne Hotels runs 34 boutique properties and ships one new AI feature every quarter, guest-facing or back-of-house. One spring, ten teams pitched ten different candidates at once: a concierge chat, a pricing engine, a voice assistant, a maintenance sensor, and six more, all competing for the same engineering slot. VP of Product Dagmara Grimshaw had to rank all ten before the room handed the slot to whoever pitched hardest, which is exactly what it was about to do.

How to answer this, stage by stage

Nobody's grading whether Dagmara can make dynamic pricing sound exciting for five minutes. They're grading whether the rubric she names would actually have caught six rooms sitting under floor before they went live everywhere.

Let's learn

Haventhorne Hotels runs thirty-four boutique properties. Every quarter, the product team ships one new AI feature, something a guest touches or something that runs behind the front desk.

Before Dagmara Grimshaw built a rubric, ten AI ideas got pitched in one two-hour meeting every spring. Whoever had the best deck and the loudest sponsor in the room walked out with the quarter's one engineering slot. Nobody wrote a real number down.

Two ways to fill one engineering slot. Only one of them can be checked afterward.

Now, all ten candidates get the same six-line scorecard first, about ninety minutes of real scoring using numbers already sitting in Haventhorne's own booking and reservation systems, before anyone argues for a favorite out loud.

Guest-facing and back-of-house, hardware and pricing logic, all competing for one slot.

The rubric gets built in this order, every quarter, before a single candidate gets scored.

Here is the turn. Ranking these ten wrong isn't really about picking a slightly worse feature first. It's about which mistake you can quietly fix later and which one you can't take back once it's live on every guest's screen, or every booking channel, at once.

At its worst, this costs weeks, not hours. A pricing bug that goes live on every OTA channel at every property doesn't get quietly fixed the next morning. Some of Haventhorne's contracts need twenty-one days' notice before a live rate can be corrected back up, so an underpriced room stays bookable long after anyone's found the mistake.

What I would leave alone: a small wording change to a guest confirmation email, or a minor color tweak to the booking screen, doesn't need this rubric. Run the full six-line score only on candidates that are genuinely new AI capability, competing for the same engineering quarter.

The lesson: a rubric's real job isn't to make a ranking look scientific. It's to stop a room from mistaking confidence for evidence.

Now here is the same thing as a story

What you'd actually say sits above. Read this one for the ninety minutes it took to find out the loudest pitch in the room was also the most dangerous one.

Dagmara Grimshaw has run product at Haventhorne Hotels for five years. Ask her which guest complaint costs the most staff time and she'll tell you before you finish the question: a room that isn't ready at 3pm, every time.

Every spring, ten teams pitch their AI idea for the year's one open engineering slot. That spring, dynamic rate pricing had the loudest room. Bohumil Ravenshaw, the CFO, walked in with a slide that promised a real revenue lift across all thirty-four properties, and he'd been talking about it in the hallway for a month before the meeting even started.

Nobody else in the room had numbers that confident. The concierge chat idea had a modest, believable case. The voice assistant idea barely had a case yet at all. By the time the meeting ended, dynamic pricing had the room's vote, the way the loudest idea always had the room's vote.

Engineering built it over six weeks. It tested clean in every simulation the team ran on old data. So it went into a live night-audit test run across all thirty-four properties, scheduled to go fully live to every OTA channel at 6:14am.

Ninety-four minutes between the first wrong price and the trace that caught it.

At 12:00am, the test run started fine. By 1:40am, six room types, mostly the garden suites nobody watches closely, had quietly dropped under the floor rate written into Haventhorne's own OTA contracts, the minimum price the hotel had promised each channel it would never charge less than. The model had never been shown that floor. It only knew how to fill rooms, and empty garden suites were the cheapest rooms left to discount.

Nobody caught it at 1:40am. Dagmara caught it at 3:14am, running the same pre-launch trace she ran on every feature before it touched a real guest, out of habit more than suspicion.

Six rooms, under floor, on a system about to go live everywhere in three hours.

She paused the launch at 3:20am. Nobody outside the product team ever saw those six wrong prices. But if the trace had run an hour later, or if she hadn't run it at all, those prices would have gone live on every OTA at 6:14am, and some of Haventhorne's contracts required twenty-one days' written notice before a live rate could be raised back up. A model that had never been shown its own contract's floor would have stayed live, discounting rooms nobody meant to discount, for three weeks minimum.

Bohumil wasn't careless. His revenue case was real; dynamic pricing genuinely could lift revenue once it worked. He'd simply never been asked the question the rubric now asks every candidate: does real, usable signal already exist to build and check this, and what's cheap to test before betting the whole quarter on it. Nobody had asked, because nobody had a rubric yet. The room just voted for the best pitch.

So here is the decision Dagmara took back. Haventhorne used to let the loudest pitch and the biggest sponsor decide the quarter's feature, with nothing written down to compare one candidate against another on the same terms. She built the six-line scorecard the next week: guest and revenue lift, reversibility, dependency, evidence cost, detection cost, blast radius, each weighted, each scored the same way for every candidate before anyone pitched a word.

Run against that scorecard, dynamic pricing scores low, not because it's a bad idea, but because it hadn't earned the right to run unattended yet. It goes back to engineering with the floor built in as a hard constraint, and it gets scored again once there's a real, checkable test behind it.

And the thing I'd tell myself, standing in that room the day the loudest pitch won: confidence is not evidence, and a rubric's whole job is to make you check the difference before the room forgets to.

ORDER, turned into the six columns Haventhorne scores every candidate on

This isn't a straight two-way tradeoff. Haventhorne had ten genuinely different candidates competing for one engineering quarter, guest-facing and back-of-house, hardware and pricing logic alike. That's what ORDER ranks. PICK only picks a side.

Six lines, one weight each, scored the same way for every candidate.

Reversibility isn't a reason to avoid the bold idea. It's a reason to check the evidence before shipping it.

One alternative is worth naming and rejecting directly: scoring every candidate purely on projected revenue impact, and ranking by that single number. It lost, because two of Haventhorne's highest-revenue candidates, dynamic pricing and no-show scoring, also carried the least usable signal and the hardest-to-catch failures; a revenue-only rubric would have ranked exactly what nearly went live under floor as the top pick. The AI-specific failure worth naming here is a missing constraint mistaken for a pricing problem: the model wasn't wrong about demand, it had simply never been given the contract floor as a hard rule, so it optimized cleanly toward a number nobody had told it mattered. The guardrail is the dependency and evidence lines themselves: no candidate ships past a low dependency score without a cheap, real check run against it first, the same trace that caught the six rooms at 3:14am. And the trade-off is accepted on purpose: a slower path to the most exciting-sounding revenue feature, in exchange for catching a three-week, contractually-stuck mistake before it ever reached a single OTA channel.

And if you want to be sure it really works, try it somewhere else

Same six lines, a farm equipment rental cooperative instead of a hotel chain, and the honest ranking doesn't change shape.

Cottersgate Equipment Cooperative rents tractors, balers, and irrigation gear to about 400 member farms. Ops lead Domenica Trevanion faced the same fork: three AI candidates for one open engineering slot. A bilingual chat assistant for the rental desk, since half the co-op's seasonal renters speak a first language the desk staff don't. A predictive maintenance system for the tractor fleet, using sensors that don't exist yet. And a late-return risk score, flagging which renters are likely to bring equipment back late, so the desk can double-book shared inventory with more confidence.

The item in the top-right corner is the one that needs a silent test before it tells anyone anything.

Same steps, mapped onto Cottersgate. Outcome: protect whether a member actually trusts what the co-op tells them, not whether Cottersgate looks advanced at the annual meeting. Reversibility: a late-return score that wrongly flags a longtime member as high-risk is hard to walk back inside a co-op where everyone knows everyone; a chat assistant that mistranslates once is a quiet, private fix. Dependency: the chat assistant can lean on years of rental-desk scripts and FAQs already written down; predictive maintenance needs sensors Cottersgate hasn't installed yet, so its dependency score stays low until that hardware exists. Evidence: the chat assistant is cheap to check, real bilingual staff can review a sample of its answers in an afternoon; the late-return score is not cheap to check quietly, since testing it for real means actually flagging a member. Rank: the bilingual chat assistant ships first. Predictive maintenance gets queued behind the sensor rollout it depends on, not killed. The late-return score waits for a silent test, scored against real outcomes with no member ever seeing a flag, before it tells the desk anything.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to Rank, name the pick and why, everything else is support.
Cost: there's no budget for a full sensor rollout before the meeting. Say so plainly, and score predictive maintenance honestly low on dependency rather than pretending the sensors already exist.
The model got better, for real: a newer translation model needs far less review before it's trustworthy. Say that too, and move the chat assistant's evidence cost down, which raises its total.

Where people run it wrong.
They skip Dependency because "the model is smart enough to figure it out," when the real gap is that no usable signal exists yet, not that the model is weak.
They score every candidate on the same weights every quarter without ever checking whether guest impact still deserves the heaviest line.
They let the rubric's own tie-break rule get argued case by case instead of fixed in advance, which quietly turns it back into "whoever's louder wins."

How to use it live. Before scoring anything, ask out loud: "does real, usable signal already exist for this, or would we be starting from nothing?" If the honest answer is "starting from nothing," that's the whole Dependency check, and it's reason enough to queue the feature instead of rank it as ready.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0