How much of an AI roadmap should be eval and infrastructure work?

Every roadmap has a line for new features and, if there's room, a line for eval and infrastructure work. Haldenport Telecom ships TriageAssist, a tool that reads incoming support tickets and routes them to the right queue with a drafted reply. Wren Kastellan is the AI PM who has to tell leadership how much of the next four quarters should go to something that produces no new feature at all.

How to answer this, stage by stage

Nobody is scoring whether you know what an eval set is. They're scoring whether you'll actually fund it before a feature exists to defend it.

Let's learn

Here is what happens when a roadmap treats eval and infrastructure work as a task you get to once the real features are shipped.

Before TriageAssist, a Haldenport support agent read each incoming ticket, decided billing or technical or retention, and typed a reply from scratch, about six minutes a ticket. With TriageAssist, a route and a drafted reply come back in under two seconds. The tool launched handling one simple flow, and a shift lead could eyeball a handful of routes each morning and trust the rest.

Every box after the second one depends on the second one existing at all.

Here's the turn: two years in, Haldenport's roadmap added a new language model, then a sentiment-escalation feature, then a routing tweak for high-value accounts, each one a fresh prompt change shipped straight to production. None of it is the problem by itself. The problem is that not one of those changes had anything to compare itself against, so nobody could tell a real improvement from a quiet regression until a person noticed the queue looked wrong.

At its worst, a roadmap with no standing eval floor doesn't fail loudly. It fails as a slow climb nobody is watching, until the cost of catching up is far bigger than the eval work would ever have cost.

What I would leave alone: I wouldn't put an eval floor under a purely cosmetic change, like reordering which fields show on the agent's screen. Nothing about routing accuracy depends on that, so the thirty percent doesn't need to cover it.

The lesson: eval and infrastructure work isn't the thing you fund after the roadmap is real. It's the thing that decides whether anything else on the roadmap can be trusted at all.

Now here is the same thing as a story

The short version above is what you'd say defending next quarter's roadmap to leadership. Read this one for how a small, sensible decision turned into three hours nobody had.

The night desk at Haldenport's contact center runs from six to two. Marisol Trench has worked it for four years, and part of the job, before TriageAssist, was reading the first line of a ticket and knowing within a breath which queue it belonged in.

For the first eighteen months after TriageAssist launched, Marisol pulled ten routes at random each shift and checked them against what she'd have called herself. They matched, almost every time. She stopped pulling ten. She pulled two. Eventually she stopped pulling any at all, because the tool had never once been wrong in a way that mattered.

Everything in the top right has to exist before anything else is safe to ship.

Then the roadmap moved fast for two quarters straight: a language-model update, a sentiment-escalation feature, a routing tweak for high-value accounts. Each one shipped behind a launch date. None shipped behind a regression check, because no one had ever built one, and nobody had put "build one" on a roadmap that already felt full.

Five weeks after the language update shipped, Marisol noticed the billing queue looked heavier than usual. She pulled a sample. Almost a quarter of billing-dispute tickets were routed wrong, most of them landing in technical support instead, where nobody could actually resolve a billing hold.

Marisol went back to checking every route by hand, the way she had at launch, except now the volume was three times higher. What used to be a five-minute spot-check at the start of shift became three hours a night, every night, until someone finally rebuilt the harness that should have existed from the start.

Four small parts. None of them is a new feature. All four are the reason a feature is safe to ship.

ORDER, the number that decides what ships firstNot a wishlist ranked by excitement. ORDER is what tells you which quiet decision every other decision depends on.

The recap, one line per letter: outcome is correct routing without a silent new failure, reversibility is that skipped eval work compounds and can't be undone after the fact, dependency is that eval work sits underneath every feature rather than beside it, evidence is a two-week golden-set spike before committing a full quarter, and rank is a standing third of every quarter going to eval and infrastructure, permanently.

And if you want to be sure it really works, try it somewhere elseSame five letters, a warehouse's pallet-scanning model instead of a support queue. A different kind of decision gets taken back this time.

Corvale Freight runs a model that reads photos of pallets at intake and flags damage before a shipment gets accepted. Mapped onto ORDER: outcome is catching real damage without slowing intake down. Reversibility is highest for the model's confidence calibration, since a badly calibrated model that's been trusted for months is far harder to walk back than a feature that just ships late. Dependency runs the same way, nothing about adding a new damage category is safe until the existing thresholds are known to hold. Evidence is a one-week sample of manually re-inspected pallets against the model's calls. Rank puts calibration work first, ahead of the two new damage categories requested for next quarter. The reversal here isn't an absent golden set, it's a merged step: Corvale had combined "model flags damage" and "shipment accepted" into one automatic action with no pause for a human glance, because pausing felt like it defeated the point of automating intake at all. Once the model's confidence drifted slightly high, that merged step meant nobody caught anything until a damaged pallet was already three states away.

The same four things fund both a support queue and a pallet scanner.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "fund eval work as a standing third of every quarter, because it's the one thing every feature after it depends on," and stop.
Cost: leadership says there's no headcount for a dedicated eval owner this quarter. Say so honestly, and start with the golden set alone, since a comparison point costs almost nothing and everything else builds on it.
The model gets better, for real: if a new model version genuinely routes better across the board, that's still a change that needs the same golden set to confirm it, not a reason to skip the check because the news is good.

Where people run it wrong.
They treat eval work as a one-time launch task instead of a standing percentage of every quarter after.
They let the loudest feature request set the ranking instead of asking what it actually depends on.
They wait for a person to notice the queue looks wrong instead of building the alarm that would have caught it in week two.

How to use it live. The moment someone asks how much of a roadmap should be eval work, ask yourself: which of these items becomes impossible to trust if I skip it. Rank that one first, and the percentage follows.

Eval work isn't squeezed in before features. Features get sequenced behind it.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0