Explain how you would handle an ROI model whose assumptions your CFO rejects.

Snagline is Nettlebrook Systems' AI agent that plays a studio's game build the way a tireless tester would, hunting for crashes, softlocks, and broken quest states before a human ever touches it. Odilia Winterscale owns its ROI story. Three days before a quarterly investment review, Ignatius Thackerston, Nettlebrook's CFO, pulls Cavernlight Games' own numbers on their open-world RPG Emberreach, and finds a catch rate nothing like the 85 percent sitting in Odilia's deck.

How to answer this, stage by stage

Nobody is grading whether you can stay calm when a number gets challenged. They're grading whether you check it against real data before you decide whether to fold or hold.

Let's learn

For two years, every ROI deck Odilia Winterscale built for Nettlebrook Systems said the same number: eighty-five percent.

Snagline is the AI agent Nettlebrook sells to game studios. It plays a build of a game for hours at a time, the way a tireless QA tester would, and files a bug report the moment something breaks.

Five steps, and the middle one is the model checking its own confidence, calibrated against 2,300 human-confirmed bugs across 60 shipped titles, mostly linear and mobile games.

Before Snagline, a studio the size of Cavernlight Games ran a contract QA team through a title's last six months, at roughly 1.3 million dollars for a project the size of Emberreach. Snagline runs the same build every night, unpaid overtime a human team can't match, and never gets bored on hour six of clicking the same door.

Averaged across every genre Nettlebrook has ever shipped against, mostly linear levels, puzzle titles, mobile games, Snagline catches 85 percent of the bugs a matched human QA team would find, in the same window. That's the number in Odilia's deck. That's the number in every deck.

Same nightly test budget, about 40 agent-hours either way. A linear level's whole map fits inside that budget. An open world's quest branches don't.

Here's the turn. The 85 percent isn't wrong. It's an average, and an average is quietly promising that every game tests the same way. Emberreach doesn't. It's an open-world RPG with something like forty hours of branching quest content, and Snagline's real catch rate there, the genre-segmented number nobody put in the deck, is 61 percent.

At its worst, that gap doesn't just make one slide wrong. Cavernlight's own QA lead had already told their account manager Snagline "missed a ton of stuff in the questline branches." If Odilia's next answer to Ignatius is a shrug, or a promise to check and get back to him, every number she's ever brought him gets quietly discounted from that afternoon on, whether or not it deserved to be.

Not a bug in the calculator. A default that made sense the year it was built, and nobody's job to come back and check.

What I would leave alone: the blended number itself is fine, left alone, for the deals it was always honest about. A puzzle-game studio or a mobile-game studio gets a catch rate within a point or two of 85 percent either way. Building genre-segmented reporting for them would be paperwork nobody asked for.

The lesson: an ROI model is only as honest as its most averaged number. The day your product's accuracy stops being uniform across the customers you sell it to is the day defending that average, in front of someone whose whole job is checking averages, becomes a losing move. Segment it before someone makes you.

Now here is the same thing as a story

Read the short version above when you're in the room. Read this one when you want to feel why nine words in a Slack channel did more damage than the actual gap in the number.

Odilia Winterscale can read a genre-segmented eval table the way some people read a box score: straight to the row that's actually moving, past the average sitting on top of it.

She built Snagline's confidence-check step herself, three years ago, back when Nettlebrook had four customers and all four of them shipped tight, linear levels. The system was simple then. Play the build, flag anything odd, check how sure you are, and hand anything under the line to a person. It worked, because at four customers, 85 percent meant 85 percent for everyone.

For two years, that number carried every deck Odilia built. New studios signed. Old studios renewed. The board slide never needed a second line.

Nettlebrook's pipeline changed slowly enough that nobody marked the day it happened. First it was one open-world deal, small, a proof of concept nobody expected to renew. Then two more, bigger. By the time Cavernlight Games signed for Emberreach, an open-world RPG with something like forty hours of branching quest content, open-world titles were a third of the pipeline, and the ROI calculator still had exactly one output field.

The trigger was nine words in a Slack channel nobody thought mattered. A solutions engineer, closing out the Cavernlight deal retro, wrote: "their QA lead says we missed a ton of stuff in the questline branches." Nobody replied. It sat there for four days.

Neither of them was being unreasonable. One of them just checked first.

Ignatius Thackerston reads deal retros the way other people read the news, out of habit, three days before he has to sign off on anything. He read that line on a Tuesday. Wednesday, he pulled Cavernlight's actual bug counts himself, matched them against the human QA team's own findings for the same build, and got 61 percent. Not 85.

He didn't wait for the quarterly review to ask about it. He forwarded Odilia the comparison at 6:40 that evening with one line: "Walk me through this before Thursday."

The whole answer to this question, in one picture. One dial can't describe a world that keeps changing shape underneath it.

Thursday came fast. Odilia had two days, and the easy version of Thursday was the one where she asked for more time. "Let me dig into that and get back to you" would have bought her a week, and cost her something she couldn't get back: the next time she said a number in that room, Ignatius would check it before he believed it.

So she didn't ask for the week. She opened Snagline's own eval dashboard, live, in the meeting, and did the thing the blended number had let everyone skip for two years: filtered by genre. Open-world titles, 61 percent, not 85. She said so before he could. "You're right. The 85 is real, it's just an average, and it was never Emberreach's number. Here's Emberreach's number."

The decision she'd take back happened three years earlier, in a fifteen-minute stretch of a roadmap meeting nobody remembers clearly. Someone had asked whether the ROI calculator should report catch rate by genre from day one. The room said no, there wasn't enough variance yet to bother, every customer at the time shipped roughly the same shape of game. That was true, then. Nobody wrote down when to come back and check whether it was still true.

Run Thursday's meeting again, with that one thing fixed. Ignatius still finds Cavernlight's real number before the meeting, the same way, the same nine words in the same Slack channel. But now the deck he's holding already has a line for it: open-world, 61 percent, Emberreach's actual figure, not the blended one. He doesn't have to catch anything. He just checks a number that was already honest. The meeting runs eleven minutes instead of forty, and it ends with him asking about the traversal module budget, not about whether he can trust the deck.

One version of that Thursday spends forty minutes rebuilding trust. The other spends eleven minutes talking about what to build next.

What Odilia would tell her past self, back in that fifteen-minute roadmap meeting: an average is a promise that the thing behind it doesn't change shape. The day Nettlebrook started selling into open worlds, that promise stopped being true, and nobody came back to check whether the calculator still knew it.

PICK, or how an average survives being checked

Not a script for staying calm when a CFO pushes back. PICK forces a real commitment about which way you move when the pushback lands, then makes you say, out loud, which mistake actually costs more.

One of these mistakes gets caught in the room. The other one gets caught two years later, spread across every number after it.

And if you want to be sure it really works, try it somewhere else

Same four letters, a garment mill instead of a game studio, and this time the average was hiding a print, not a genre.

Flawtrace is Hallowick & Combe's vision system for the last stop on a fabric line: a camera reads every roll for broken threads, dye spots, and weave slubs before it ships to a cutting floor. Perdita Casterbrook owns its ROI story. Blended across the whole product line, mostly solid and lightly textured fabrics, Flawtrace catches 91 percent of the defects a trained human inspector would find. On patterned fabric, busy prints that visually hide a flaw inside the pattern's own noise, the real number is 68 percent.

Same shape of gap as Snagline's. The camera never gets less accurate. The pattern just gives it more places to hide a flaw.

Gwyneira Rookwood, Hallowick & Combe's finance lead, rejected the blended 91 percent the same way Ignatius rejected Snagline's 85: a new upholstery contract was entirely patterned fabric, and the 91 percent in Perdita's case had nothing to do with what that specific line would actually catch.

Same rank, different lever: Perdita conceded the blended number live, in the room, and rebuilt with the segmented 68 percent. At 650,000 dollars a year in inspection labor for that line against a 180,000 dollar Flawtrace license, 68 percent still nets Hallowick & Combe about 262,000 dollars a year, a smaller number than the 411,000 the blended figure implied, and a real one.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: concede fast when the data backs the objection, defend fast when it doesn't, and never leave without saying what would change your mind.
Cost: no time this quarter to build genre-segmented reporting company-wide. Ship the cheap version: flag just the disputed segment with its own honest number, expand the rest later.
The model got better, for real: say Flawtrace's pattern-matching zone improves and patterned-fabric catch rate jumps to 80 percent overnight. The position doesn't change. A bigger number still gets defended with the same segmented honesty, not folded back into one blended figure.

Where people run it wrong.
They defend the average instead of conceding fast, and lose credibility for every number that comes after.
They quietly lower every deal's assumption to the worst-case number to avoid ever being caught again, which underprices the honest deals right along with the risky one.
They wait to bring the real number until after the meeting, which reads exactly like getting caught, not like preparation.

How to use it live. Ask which number the objection is actually about before answering: "Is this pushback on the average, or on this specific segment?" That question alone tells you whether to defend or concede, and it buys a few real seconds to check.

Three things worth stating directly, since this is where the real judgment sits. The alternative Nettlebrook considered, and rejected, was locking the ROI calculator to always quote the conservative 61 percent floor, for every deal, regardless of genre. It lost, because it would underprice the real, defensible 90-plus percent number on most of the customer base, puzzle titles, mobile games, tightly bounded levels, punishing the honest deals to protect against the one that wasn't. The AI-specific failure worth naming is a coverage gap that tracks genre, not accuracy: Snagline explores each nightly build under a fixed budget, about 40 agent-hours regardless of genre, and a linear level's reachable-state graph gets fully walked inside that budget in a handful of passes, while an open-world quest graph with branching flags does not, so the same 40 hours buys a much smaller slice of coverage without the model ever reporting itself as less sure. The guardrail is genre-tagged eval reporting as the calculator's default output, plus a coverage-confidence flag on any build where exploration didn't reach a set fraction of known reachable states. And the trade-off is real: Nettlebrook could raise the open-world exploration budget to close the gap faster, but more agent-hours per nightly build means a slower nightly turnaround and a higher compute bill per run, and the team accepts a lower catch rate on purpose rather than make studios wait past sunrise for a build that used to be ready by 7am.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0