Write the one-page memo recommending against an AI initiative your CEO championed.

Skyloom sells B2B software. Julienne Desroches, its CEO, championed QuotaSense, an AI model meant to replace the sales team's simple weighted-pipeline forecast. Anneke Vosloo is the AI PM who backtested it before the company-wide rollout, and has to write the memo saying it isn't ready.

How to answer this, stage by stage

Nobody is scoring whether you can write a firm memo. They're scoring whether the memo would actually survive being read by the person who championed the thing it's about.

Let's learn

QuotaSense is a model meant to predict how much revenue Skyloom will close each quarter, replacing a spreadsheet where sales ops manually weights each open deal by its stage's historical close rate.

Four steps, none of them exotic. The backtest just asked the model to predict something Skyloom already knew the real answer to.

Julienne championed QuotaSense after seeing a competitor's case study, and the instinct behind it was reasonable: a company Skyloom's size shouldn't still be running its forecast off a hand-tuned spreadsheet. Before rolling it out company-wide, Anneke's team ran the backtest that should happen before any forecasting tool ships.

Averaged across all six quarters, QuotaSense's forecast error came out to 34 percent. The existing pipeline method, the one it was meant to replace, averaged 31 percent over the same period. The replacement was measurably worse than what it was replacing.

Here's the turn: the problem was never that QuotaSense was poorly built. The turn is that Skyloom's real sales data, roughly 340 closed deals across three years, lumpy and seasonal, simply isn't enough for a model to reliably out-learn a method sales ops had already hand-tuned against the same real patterns for years.

At its worst, shipping QuotaSense as planned would have cost Skyloom real money for a measurably worse forecast, plus real trust with a sales team that would have quickly noticed their new tool was wrong more often than the spreadsheet they'd used for years.

What I would leave alone: Skyloom's existing weighted-pipeline method stays exactly as it is for now, it's simple, explainable, and currently more accurate than its proposed replacement. The goal was never protecting it out of habit, the backtest just happened to show it was still winning.

The lesson: a competitor's case study is evidence about their data, not yours. Before replacing a working method with an AI model, backtest the model against your own real history and let that number, not the pitch, make the call.

Now here is the same thing as a story

The short version above is what you'd say out loud in the room. Read this one for what it actually felt like to write a memo you knew your CEO wouldn't want to read.

Anneke Vosloo had been Skyloom's AI PM for two years, and she genuinely liked working with Julienne. Julienne moved fast, trusted her team, and rarely needed convincing on good ideas. QuotaSense was the first time Anneke had real evidence Julienne's instinct was ahead of the evidence.

The backtest wasn't dramatic to run. Anneke pulled six quarters of real deal data, closed and open, and asked QuotaSense to predict what sales ops had already lived through. Then she ran the exact same six quarters through the existing pipeline method, the one everyone already trusted.

Not a story about AI being wrong in general. A story about this specific model, on this specific data, this specific quarter.

The numbers came back plainer than she expected. 34 percent average error for QuotaSense. 31 percent for the spreadsheet. Not a close call dressed up as one, a real, checkable gap in the wrong direction.

She sat with it for a day before writing anything. The easy version of this memo buried the number in three paragraphs of context and hedging. The other easy version was blunt enough to read as "your idea doesn't work," with nothing offered in its place. Neither one would actually survive being read by someone who'd championed this publicly.

The actual writing challenge wasn't finding the courage to say no. It was finding the version that would actually get read and acted on.

She wrote the real version: the number first, the recommendation second, a concrete next step third, all inside one page. No apology for the finding, no attack on the idea's premise either.

Julienne read it in the hallway, not a scheduled meeting, which Anneke had half-dreaded. "Show me the backtest," was the first thing she said. Not defensive, just direct. Anneke had the six-quarter chart ready.

The part of the pitch that never made it into the original case study Julienne had seen.

Anneke never had a fixed rule for when a CEO's pet initiative should get killed outright versus redirected. She had a feeling with two settings: the evidence says the underlying goal is wrong, or it says only this specific bet is wrong. This was clearly the second. Better forecasting was still worth wanting. This model, on this data, right now, wasn't the way to get it.

The fork the memo actually ran, stated plainly instead of buried in hedges.

Back when the rollout timeline first got scoped, moving fast on a promising case study wasn't an unreasonable call. It stopped being reasonable the moment a real backtest was possible and nobody had insisted on running it before the launch date got set.

Here's the replay: same model, same real data, but with the backtest run before the timeline instead of after. QuotaSense ships as a secondary input alongside the pipeline method, not a replacement, while two more quarters of real data accumulate. No wasted rollout, no sales team trust lost to a tool that measurably underperformed what it replaced.

One version of this story ships a worse forecast to the whole sales org and finds out from complaints. The other spends one afternoon on a backtest and turns a public commitment into a smaller, honest next step instead.

What I'd tell myself, staring at that 34 percent number the night before writing the memo: a good instinct and a ready model are two different things, and the only way to tell them apart is to actually check.

LEAD, run on a forecast that lost to the spreadsheet it was meant to replaceNot a script for softening bad news until it's harmless. LEAD is what makes a hard recommendation survive being read by the person who won't want to hear it.

The recap, one line per letter: link is the real backtested error rate the whole memo hangs on, early signal is running that backtest before the rollout instead of after, abuse is avoiding both the over-hedged and the over-blunt failure modes, and decision is the actual one-page memo, numbers first, a real next step, and the door left open.

And if you want to be sure it really works, try it somewhere elseSame four letters, a customer-support triage tool instead of a sales forecast. This time the early signal comes from a pilot team's real complaints, not a formal backtest.

Tobias Renquist is the AI PM at Freightloom, a logistics software company. Freightloom's CEO championed AutoTriage, a model meant to auto-route every support ticket to the right team without human triage. Mapped onto LEAD: link is the real metric, percent of tickets correctly routed on the first try, backtested against 90 days of real tickets: AutoTriage hit 71 percent, the existing human triager hit 89 percent. Early signal came from a two-week pilot with one support pod before full rollout, where misrouted tickets visibly piled up in the wrong queues. Abuse means avoiding a memo that either buries the 71-versus-89 gap in soft language, or one that dismisses the CEO's underlying goal, faster ticket routing, which was still a real problem worth solving. Decision is a memo recommending AutoTriage handle only the clearest, highest-confidence 40 percent of tickets automatically, routing the rest to the human triager, instead of replacing the role outright.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Skip straight to "the model backtested worse than what it replaces, here's the number, here's the smaller next step," and stop.
Cost: no time to run a full backtest before the memo is due. Say so honestly, and recommend delaying the rollout decision until one exists, rather than guessing at a verdict.
The model got better, for real: say a later backtest shows QuotaSense finally beating the pipeline method. Write the follow-up memo just as plainly, with the new number leading it, the same way the first one did.

Where people run it wrong.
They write the memo as a personal objection instead of anchoring it in a real, checkable number.
They soften the recommendation so much the actual "don't roll this out yet" gets lost in the caveats.
They frame it as killing the initiative entirely, instead of pausing one specific rollout decision on today's evidence.

How to use it live. The moment an interviewer asks you to push back on a leader's pet initiative, ask yourself first: what's the one real number this recommendation would stand or fall on, and do I actually have it. That question buys real thinking time, and it's usually exactly what turns an opinion into a memo someone can act on.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0