What do you do when leadership wants an AI feature primarily for the press release?

Corvenna reads a brand's social accounts and tells its marketing team which content angles are about to take off. Nowcast is the feature that predicts trends before they peak. Palmira Vosberg owns Nowcast's roadmap. Three weeks before Corvenna's Series B announcement, her CEO wants Nowcast as the centerpiece slide, and the number it was actually built on turns out to be three accounts, not fourteen hundred.

How to answer this, stage by stage

Nobody is grading whether you can describe a polite way to slow leadership down. They are grading whether you'll notice that "make it announceable" quietly ate the whole spec, and catch it before a demo that worked on three accounts ships confidently wrong to the other fourteen hundred.

Let's learn

Corvenna is a dashboard that watches a brand's social accounts and tells its marketing team what to post next. Nowcast is the feature inside it: three predicted trend windows a day, tap to add one straight to the content calendar. Before Nowcast, a marketing team found trends by scrolling their own feed mid-morning and guessing. After Nowcast, it's a card with a confidence-sounding badge on it.

Nothing about the ask itself changed. What Palmira required before treating it as scoped changed completely.

In the internal demo, built on Corvenna's three best-instrumented flagship accounts, Nowcast called nine of the next ten trend windows correctly. Leadership loved it enough to put it on the Series B deck. Palmira shipped it to all 1,400 accounts the same week the demo passed, because Corvenna's launch checklist had one gate: the demo works, ship it everywhere.

Here's the turn. Those misses weren't spread evenly. When Corvenna finally stratified that 180-account sample by follower count, accounts under 4,000 followers came back at 21 percent. Accounts over 50,000 followers, the same size as the demo accounts, came back at 61 percent. Nowcast wasn't a little worse everywhere. It was excellent exactly where the demo happened to look, and thin everywhere else, because smaller accounts simply don't have enough posting history for the model to have real signal to work from.

At its worst, Orla Duchamps, the marketing manager at Petalcombe, a skincare brand with about 2,800 followers, bets the brand's full week of paid-boost spend, $16,000, on a Nowcast call recommending a specific content angle. It flops. Engagement lands well under Petalcombe's own baseline. Orla wasn't careless. The card looked exactly as confident as it did for the flagship accounts in the demo, because it was built from the same badge, the same UI, no visible difference at all.

What I would leave alone: a caption-suggestion tweak nobody's betting a press cycle on doesn't need this ritual. If it gets the tone wrong 1 time in 10, someone edits it, costs a few seconds. Being wrong there is cheap. The two-bar gate earns its keep on anything a customer might act on with real money or real trust behind it, not on everything with "AI" in its name.

The lesson: a demo proves a feature can be right. It never proves how often. Those are two different questions, and for two years Corvenna only ever asked the first one, because asking the second one used to be free: nothing had ever been expensive enough to need it.

Now here is the same thing as a story

The short version above is what you'd actually say in the room. Read this one when you want to feel exactly what $16,000 and one bad week costs, not just hear the number.

Palmira Vosberg has run Nowcast's roadmap at Corvenna for two years, and before that she shipped the caption-suggestion tool and the best-time-to-post nudge, the two features that made her the person leadership handed vague ideas to. She was good at it: turn a hallway comment into a build, get a demo in front of the room fast, ship it when the room nodded. For two years, nodding was enough. A good demo and a good feature had, in her experience, always been the same thing.

Then Godfrey Winterbach, Corvenna's CEO, asked for something with a bigger name attached: Nowcast, an AI that predicts trends before they peak, as the centerpiece of the Series B announcement, three weeks out. Palmira did what always worked. She scoped it to whatever would demo best, fast: three flagship accounts with the richest posting history Corvenna had, the ones every internal tool already ran cleanest on. In the room, Nowcast called a specific meme format two days before it peaked, live, on screen. People actually stood up. Nobody, including Palmira, asked whether it did that for the other 1,397 accounts, because in two years, a demo that good had never once turned out to be hollow.

Nobody decided to stop asking. It stopped being asked over two good years, the way most habits actually do.

Corvenna's launch checklist had exactly one gate for a press-tied AI feature: the demo passes, engineering flips it on for every account. That gate had never once been wrong before, so Nowcast rolled out to all 1,400 accounts the same week the room stood up.

For the flagship-sized accounts, the good months held. Then, five weeks in, a routine account-health call surfaced something odd: Petalcombe, a skincare brand with 2,800 followers, had spent its entire week's paid-boost budget, $16,000, chasing a Nowcast call that never landed. Orla Duchamps, who runs their marketing, hadn't filed a complaint. She'd just quietly asked her account manager, on a call about something else entirely, whether Nowcast's calls were "actually tested on brands our size, or just the big ones."

Nobody at Corvenna could answer that question with a number. That was the whole problem in one sentence.

The whole answer, in one picture. Nobody designed a dial. Everybody got a switch, and for two years nobody had to notice.

Palmira built the real eval set that week: 180 accounts, stratified across follower bands and industries, the population Nowcast actually served instead of the three that had impressed a room. It came back at 38 percent overall. Broken out by follower band, accounts under 4,000 sat at 21 percent. Accounts over 50,000, the size of every demo account, sat at 61 percent. Nowcast hadn't gotten a little worse for smaller accounts. It had never really been tested on them at all.

The easy fix would have been a disclaimer: add a line under the badge saying results may vary by account size. Palmira actually drafted one. She killed it within a day. Orla hadn't read fine print before betting a week's budget. She'd read a badge that looked exactly as confident as the one Corvenna's CEO had watched call a meme format two days early. A caveat doesn't change what the badge looks like at 9am on a Monday. It just moves the blame after the money's already spent.

Here's the decision I'd take back instead, and it isn't really Palmira's. Back when Corvenna's launch checklist was written, every AI feature it covered was small: caption tone, posting time, things where a wrong call cost a person a re-type. One gate, demo passes, ship it, was a sensible rule for that population of features. Nobody ever split "the demo is ready" from "the feature is ready for everyone," because nothing before Nowcast had been expensive enough to make the difference visible. The same shortcut that kept two years of small features shipping fast is what let a three-account demo stand in for fourteen hundred real ones.

Run the same kind of ask again, eight weeks later. Godfrey wants Nowcast Live, real-time competitor benchmarking, unveiled at a marketing conference keynote, six weeks out. This time Palmira splits the gate in two before a single line of the brief gets written. An announce track: the same three flagship accounts, honestly labeled a curated preview, ready in time for the keynote. An eval track, running in parallel: a fresh stratified sample of 220 accounts, with a real pass bar, 65 percent overall and 50 percent within the smallest follower band, before anything reaches every customer. Two weeks before the keynote, the eval track flags the same weak spot: accounts under 4,000 followers and under 90 days of history. This time it's caught before a customer ever sees it. The team ships a guardrail, those accounts get "still gathering signal" instead of a confident call, and the keynote goes ahead on schedule with the honest preview. General availability follows three weeks later, once the stratified hit rate clears 71 percent overall and 58 percent in the smallest band. Zero accounts get a wrong, confident call during the gap.

What I'd tell the version of myself who wrote that first launch checklist, back when every AI feature Corvenna shipped was small enough that being wrong cost an afternoon: the checklist that gets you through two good years doesn't announce when it's stopped fitting the feature you're about to ship through it. Somebody has to go looking for the moment it stopped fitting, on purpose, because a demo that works will never once raise its own hand and say it isn't enough.

FLIPS, or the two bars a press date almost skipped

Not a trick to sound structured. It's the difference between a feature that survives contact with 1,400 real accounts and one that only ever had to survive a room of people who wanted to believe it.

Four setup and payoff letters, and one question that only had to be asked once it actually mattered.

The announcement never got smaller. It just stopped being the only thing standing between a demo and 1,400 real accounts.

The AI-specific failure worth naming plainly is eval-set representativeness: Nowcast wasn't wrong so much as untested outside the population that happened to impress a room, so a model that looked calibrated on three rich-history accounts turned out to have almost no real signal for accounts without that history. The guardrail is the follower-count and history-length threshold: thin accounts get an honest "still gathering signal" state instead of a confident call, checked against a real stratified sample before every general release, not promised to be right, held to a target rate on purpose. There's a real trade-off, accepted on purpose: the two-track gate costs two to three weeks of delay and real eval-labeling effort before general availability, in exchange for never shipping a confidently wrong call to an account that never had enough data to earn one. For a caption-tone tweak nobody's betting a press cycle on, that overhead costs more than it's worth, which is exactly why it's a gate for expensive, announceable asks, not a rule for every feature with a model in it. One alternative got a real look and got rejected: a disclaimer under the badge, "results may vary by account size." It never shipped, because it doesn't change what the badge looks like at 9am on a Monday, it just moves the blame after the money's already spent.

And if you want to be sure it really works, try it somewhere else

Same five letters, a completely different flip family this time. Nobody scopes to a good-looking demo here. A check just quietly stops seeing the cases that would embarrass it.

Sedgemoor sells farm-advisory software, and Bloom Forecast is its AI feature: it predicts the best harvest window for each of Sedgemoor's roughly 600 partner farms. Eamon Cuthbertson owns Bloom Forecast's roadmap. When leadership wanted it as the headline feature for a regional ag-tech conference, Eamon didn't scope around a flashy demo. The trap here was quieter: his data team had, over months, learned which three or four farms had the cleanest sensor telemetry, no gaps, no missed readings, and had started pulling only those farms into every pre-release check, without ever deciding to.

Same five questions, a completely different way the flip hides. This time the check itself quietly stopped seeing the mess.

It wasn't a decision anyone made in a meeting. A messy farm's data dragged the pre-release number down and slowed the demo prep, so whichever engineer was running the check that week would quietly swap it for a cleaner one. Within four months, the same three farms had been in every single check, and the messy 90 percent of Sedgemoor's real farms had been in none.

Both stories run F through S. Only one letter, the I, tells you which way the habit actually broke.

F · Eamon Cuthbertson, who owns Bloom Forecast's roadmap at Sedgemoor.
L · The data team stopped pulling whichever farms were due for a check and started pre-selecting the three or four with the cleanest telemetry, because a messy farm always dragged the number down.
I · The pre-editing flip, a different shape from Palmira's. Old setting: any farm due for a check gets pulled in, including the messy ones, and the check logs what actually breaks. New setting: the population gets filtered before the check ever runs, so a "check" always meant a check of the easy cases, and the number stopped measuring anything real. Nothing in between: either the check sees the real population, or it sees a hand-picked one.
P · Sedgemoor's validation pipeline never had a fixed, mandatory sample. Any engineer could pick which farms to include. That was fine while Bloom Forecast was a 40-farm internal beta and all 40 looked fairly similar. Nobody ever built a fixed roster, because the population itself used to be uniform.
S · Six weeks before a regional keynote, Eamon locks a fixed, stratified 85-farm sample, drawn once and reused for every release, spanning every sensor-quality band Sedgemoor actually has. Nineteen of those 85 farms have sensor readings that arrive more than six hours late at least once a week, and the fixed sample catches something the cherry-picked one never could: late readings were being silently treated as "no frost risk" instead of flagged unknown. The team ships a freshness guardrail nine days before the conference. Bloom Forecast's new feature, Frost Alert, launches on the fixed sample's real number, not a hand-picked one, and it holds in the field afterward.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: an AI ask scoped purely to look good in a demo will always find three accounts willing to make it look good, no matter how good the model actually is.
Cost: no budget to build a full stratified eval pipeline from scratch. Reuse the customer population itself; sampling 180 existing accounts costs almost nothing extra, it's refusing to skip the sampling that costs something.
The model got better, for real: say Nowcast's overall accuracy climbs on its own next release. Doesn't matter, maybe matters more. A model getting quietly better is exactly when nobody thinks to check whether the improvement reached the accounts that needed it most.

Where people run it wrong.
They treat "we have an eval set" as proof it's representative, instead of checking whether it actually mirrors who'll use the feature.
They let a genuinely impressive demo substitute for the second bar, instead of treating the demo and the eval as two separate questions.
They wait for a customer's bad week to force the second bar into existence, when the whole point of setting it early is that you don't need one to show up first.

How to use it live. If an interviewer asks how you'd handle a press-driven AI ask, ask yourself one thing before answering: if this shipped to every account tomorrow morning, could I already say what number would prove it was working? If the honest answer is no, that's the whole question, answered.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0