What does no-regret roadmap work look like for an AI product?

Corrigan Waste Services runs a camera system on its recycling trucks that flags contamination, food waste, plastic bags, the wrong kind of glass, mixed into a bin before it gets emptied. Yolanda Presek leads the product team behind it, and the question in front of her is what roadmap work is actually safe to build regardless of how the detection model underneath it changes.

How to answer this, stage by stage

Nobody is scoring whether you can list "good infrastructure practices." They're scoring whether you can name the one number that would have warned you before the penalty hit.

Let's learn

What would still be worth having built, even if the camera vendor changed tomorrow?

Corrigan Waste Services mounts a small camera above each truck's hopper, and a model flags contamination as bins get emptied, food waste in the recycling, the wrong kind of glass, a plastic bag wrapped around cans. Before the system existed, contamination got caught only at the sorting facility, days later, with no way to trace it back to a specific route or a specific bin. With the cameras, flags happen in real time, at the truck.

Three tests, and the vendor's own polished dashboard doesn't clear any of them.

Here's the turn: Corrigan's driver-incentive system only ever credited a flag if it matched an "obvious" contamination category the vendor's model was confident about. Ambiguous cases, a partly-rinsed container, a mixed-material item, earned a driver nothing, even when catching those was exactly what the municipal contract's contamination-rate penalty actually depended on.

At its worst, an incentive system nobody meant to design badly can quietly stop a whole category of catches, weeks before it shows up anywhere leadership is looking.

What I would leave alone: I wouldn't rebuild the driver mobile app's basic screen. It's a fine interface regardless of which camera feed sits behind it; the problem was never the screen, it was what got credited underneath it.

The lesson: the roadmap work that survives a model change isn't about the model at all. It's about whatever number would have told you something was wrong before the model ever needed to be blamed for it.

Now here is the same thing as a story

The short version above is what you'd say defending a leading-indicator dashboard to a compliance officer. Read this one for how a single incentive decision quietly changed forty drivers' habits at once.

Baird Kessling has driven a Corrigan recycling route for eleven years, and can tell a mixed-material yogurt cup from a clean one from the truck's cab without slowing down.

For the first several months after the cameras launched, Baird flagged whatever looked wrong to him, obvious or not, since the tool's dashboard showed every flag the same way and paid no attention to which kind it was.

One clock was already telling the truth. Nobody was looking at it yet.

Then dispatch rolled out a new weekly leaderboard, ranking drivers by confirmed contamination catches. Confirmed meant matched to the vendor model's own "obvious" category. Ambiguous flags, the ones that needed someone at the sorting facility to actually go check, never made it onto the board at all, because they took days to resolve and the leaderboard reset weekly.

The number on the leaderboard looked fine. The judgment call it was supposed to reward had already left the building.

Baird noticed, the way any driver watching a leaderboard notices, that ambiguous flags never moved his rank. So, without anyone telling him to, he quietly stopped bothering with them. Why spend the extra ten seconds squinting at a half-rinsed container when it earns nothing and slows down the route?

The quarterly contamination report stayed close to normal for a while, seven point two percent, comfortably under the eight percent penalty line, because obvious contamination was still getting caught fine. Then, five weeks after the ambiguous-flag rate had already cratered to near zero, the report jumped to eleven point four percent, over the line, triggering a real financial penalty on the municipal contract.

What I'd tell myself, watching that report land: nobody at Corrigan decided to stop catching ambiguous contamination. A leaderboard just quietly told forty drivers, all at once, that it wasn't worth their ten seconds.

LEAD, the number that doesn't care which camera is runningNot a plea to "watch your metrics." LEAD is what tells you which number would have rung first.

The recap, one line per letter: link is staying under the contract's penalty threshold, early signal is the ambiguous-flag rate falling weeks before the contamination rate itself, abuse is dispatch centers reclassifying flags just to hit a mandated minimum, and decision is a review of the incentive system, not the model, once the rate drops below a set floor.

And if you want to be sure it really works, try it somewhere elseSame four letters, an HVAC service company instead of a waste hauler. Different flip family entirely, the same regretted vendor bet underneath it.

Coalridge HVAC built a predictive-maintenance dashboard tightly wired to one sensor vendor's proprietary data format, flagging compressors likely to fail before a service call was ever placed. Mapped onto LEAD: link is avoiding unplanned compressor failures across serviced buildings. Early signal would have been the same shape of thing, the share of buildings getting a full nightly scan versus a partial one, moving before failure rates did. Abuse would be technicians marking partial scans as "complete" to avoid flags about incomplete coverage. Decision would trigger a review of scan scope, not the sensor model, once full-coverage scans dropped below a set floor. The flip here is different: scope, not substitution. When the sensor vendor introduced a per-scan licensing fee, Coalridge's nightly full-building scans shrank overnight to the highest-risk units only. Thoren Vasek, a field technician who used to review every unit's overnight data each morning, found himself reviewing only the flagged top-risk subset, and a mid-tier compressor with no history of problems failed without ever having been scanned in the new, narrower nightly pass.

The same missing layer that would have caught Corrigan's collapsing flag rate would have caught Coalridge's shrinking scope too.

This layer is what makes Corrigan's fix and Coalridge's fix the same fix, wearing two different uniforms.

The attribution layer and the vendor's own dashboard sit in exactly opposite corners.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "no-regret work produces a number that warns you early and survives a vendor swap, here that's a shared attribution layer, not the vendor's dashboard," and stop.
Cost: no budget to build the full attribution system this quarter. Say so honestly, and start by just tracking the ambiguous-flag rate manually, since visibility alone is most of the value.
The vendor relationship holds for years, for real: if Corrigan genuinely never needs to switch camera vendors, the honest move is to say the attribution layer still paid off, since it also caught the incentive-gaming problem that had nothing to do with the vendor at all.

Where people run it wrong.
They build the polished, vendor-specific dashboard first because it's the visible, demoable piece, and treat the incentive layer underneath as an afterthought.
They track a leading number once, as an audit, instead of as a standing check with a real threshold and a real action.
They assume a metric that's easy to move is automatically informative, without asking whether it can be gamed instead of genuinely improved.

How to use it live. The moment someone asks what no-regret work looks like, ask yourself: what number would tell me something's wrong weeks before the official report does, and would that number still matter if we swapped the model tomorrow? Build whatever produces that number first.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0