Critique an ROI model that assumes every AI-completed task would otherwise have been done by a human.

Corbelwatch watches every active camera on a Fennborough Construction Group job site for a missing hard hat, a missing hi-vis vest, missing eye protection, or a harness that isn't clipped in at height. Malachi Ekwueme owns what its ROI actually claims. For two years the slide said the same thing: Corbelwatch avoids nine million dollars a year in EHS coordinator labor, one dollar for every flag it raises. Then Graciela Basurto, Fennborough's CFO, went looking for where that nine million had actually shown up in the safety budget, and couldn't find it.

How to answer this, stage by stage

Nobody is grading whether you can say "not every task would have been done by hand." They're grading whether you can turn that into a real number, and whether the reason you give for the discount is about the model, not just about people being slow.

Let's learn

For two years, Malachi Ekwueme's quarterly slide said the same thing: Corbelwatch avoids nine million dollars a year in EHS coordinator labor. Nobody had thought to ask what fraction of that labor a coordinator would actually have spent.

Corbelwatch is Fennborough Construction Group's camera system. It watches every active job site, all shift, for a missing hard hat, a missing hi-vis vest, missing eye protection, or a harness that isn't clipped in above six feet.

Before Corbelwatch, Fennborough ran four EHS coordinators across fourteen sites, each site getting three to four timed walking rounds a shift. Nobody had ever measured exactly what fraction of real violations those rounds actually caught. The best independent guess, from a safety consultant's estimate two years back: about three in ten.

Before Corbelwatch, catching a real violation meant a coordinator's round happening to line up with the moment it was visible. Most of the time, it didn't.

With Corbelwatch running, every camera stays on for the whole shift, at every site. It doesn't blink between rounds. Fennborough's cameras log about 180,000 flagged violations a year, at roughly a cent and a half each to run through the model.

Cameras don't choose when to look. A round does, four times a shift, down one path.

Here's the turn. The extra flags aren't the problem, more visibility was the whole point of building Corbelwatch. The problem is what the ROI slide did with them next: it took all 180,000 and multiplied by the value of a human catch, $50 each, as if a walking round would eventually have found every single one. It wouldn't have. Rounds happen every couple hours. A harness clip fused from two camera angles through scaffold framing was never going to be caught by someone standing in one spot.

What it costs at its worst: this spring, while planning two new site openings, Fennborough's regional leadership nearly held EHS coordinator headcount flat instead of adding two more, reasoning that Corbelwatch had already absorbed most of a coordinator's job. Graciela Basurto, reviewing the safety budget line by line before signing off, noticed something that didn't add up: if Corbelwatch had genuinely avoided nine million dollars in coordinator labor, overtime pay should have fallen by a lot more than the $180,000 it actually had. She paused the headcount decision and asked Malachi a plain question: "Whose number is this? Where did it actually show up?"

Two real sites, two real catch rates. The naive model's 100% never sat anywhere near either one.

What I would leave alone: the small set of flags for a hard hat missing at ground level for ten minutes or more, in plain, unobstructed view. Those really would almost always get caught by a round eventually, they aren't time-window sensitive or occlusion-dependent, so there's no reason to discount them the way the scaffold-level flags need discounting.

The lesson: a substitution rate isn't a constant you set once. It's a coverage claim, and it needs to be measured again every time the thing doing the coverage, four coordinators on rounds, or a camera's confidence threshold, actually changes.

Now here is the same thing as a story

The short version is above, for the room. Read this one when you want to feel why a number that assumes a person was always going to be there is the whole mistake.

Malachi Ekwueme has owned Corbelwatch's numbers since the pilot, back when it watched one site, Vesper Point, sixty workers, one EHS coordinator who was on that site nearly every working hour. The ROI math from that spring was simple: value avoided, divided by what it cost to build. $9,000 in avoided coordinator overtime that first quarter, divided by a $6,000 pilot build. 150%. Genuinely close to true, at that scale, with that coordinator standing right there most of the day.

For a year, nothing about that math needed to change. Fennborough had one site running Corbelwatch. The pilot coordinator really did catch almost everything the cameras also caught, because she really was almost always present. The number held up because the assumption behind it, that a person would have been there anyway, happened to be nearly true.

It stopped being true in three ordinary steps, and none of them looked like a mistake. Fennborough rolled Corbelwatch out to a second site, then a fifth, then all fourteen, and staffed EHS coverage the only sane way a construction company staffs it: four coordinators, sharing fourteen sites, each site getting three or four timed rounds instead of one person standing there the whole day. Flagged violations climbed from a few hundred a month to 180,000 a year. Every quarter, someone updated the flag count on the slide. Nobody ever updated the assumption sitting underneath it: that each of those flags was still a catch a person would otherwise have made, the way the Vesper Point coordinator once nearly did.

Some of what Corbelwatch flags was never a question of how often a round ran. It's a detection no single person, standing anywhere, could have made alone.

Then came an ordinary Tuesday in a budget meeting, not a crisis.

Graciela Basurto was signing off on two new site openings, and the plan in front of her held EHS coordinator headcount flat instead of adding the two coordinators those sites would normally need. The logic on the page was Corbelwatch's own ROI slide: nine million dollars in avoided labor implied coordinators had plenty of slack to absorb more sites. Graciela does this for a living, though, and she does one thing before signing anything: she checks whether a claimed saving shows up anywhere real. Coordinator overtime had dropped, but only by $180,000. Not the kind of drop nine million dollars in avoided labor should have left behind.

"Whose number is this," she asked Malachi, in the flat tone that isn't an accusation yet. "This one's been on the slide for two years. Has anyone ever actually measured it?"

Malachi didn't have a real answer, and that was the actual problem, not the size of the number itself.

Four points on one line. Three of them came from an actual audit. One of them, the one on the slide for two years, never did.

What Malachi built over the next two weeks was the audit that should have run the day Corbelwatch left its pilot site. He and Emre Yilmaz, Fennborough's site EHS lead, ran two weeks of shadow coverage at two sites chosen on purpose: one scaffold-heavy multi-story build, one open single-level slab job, the two ends of how hard a site is to see across. Coordinators ran their normal rounds. Cameras logged everything in parallel, unfiltered. Then Malachi compared the two lists.

At the open slab site, coordinators independently caught 38% of what the cameras also flagged. At the scaffold-heavy site, harder sightlines, more occlusion, they caught 22%. Neither number was close to a hundred. Neither was a coincidence either: it lined up almost exactly with the old three-in-ten guess for how much of the real violation rate a walking round had ever caught, back before Corbelwatch existed at all. A person can only be in one place at a time, whether or not there's a camera watching too.

The decision Malachi traced it back to sat in a single pricing meeting the week the pilot launched. Someone asked whether the ROI's substitution assumption should be a number that gets re-measured, or a fixed hundred percent baked into the formula. A hundred was faster to ship, and at Vesper Point's scale, with one coordinator on site nearly full time, it was close enough to true that nobody pushed back. Nobody wrote a date to come check it again.

The whole answer to this question in one picture. A flag was never one thing. It was always three, and only one of the three was ever a labor number.

Run that pricing meeting again, with one line added: revisit the substitution assumption every time the site count or the model's confidence threshold changes, not just once at launch. Same rollout to fourteen sites, same budget meeting, same sharp question from Graciela. This time Malachi's answer is immediate: "22 to 38 percent, measured two months ago, here's the audit." The headcount plan doesn't get cut. The two new sites get their coordinators.

One design let "the ROI" mean whatever number a pilot site happened to produce. The other lets it mean whatever the current coverage actually is.

What Malachi would tell himself, back in that first pricing meeting: a hundred percent substitution wasn't a lie. It was just a measurement of one site, with one coordinator standing there nearly full time, that got treated like a law of nature instead of a fact that would stop being true the moment the company grew.

BOUND, or the range a launch-day pilot never earned

Not a story wearing a framework's clothes. This is a substitution-rate estimate that got frozen at a hundred percent when it was almost true once, and BOUND is what turns "not every task transfers" into a number a CFO can actually check.

Three things worth stating directly, since this is where the real judgment sits. The trade-off being accepted: loosening Corbelwatch's confidence threshold buys real safety coverage, catching borderline cases a tighter threshold would miss, but it buys that recall by pulling in exactly the class of flags a labor-substitution number can't explain, so the honest substitution share keeps shrinking even as the flag count climbs. The AI-specific failure mode worth naming separately is false-positive drift: a rolled-up sleeve or a shadow across a hard hat's brim can read as a violation under low winter light the way it wouldn't in summer glare, and the guardrail is a precision audit against a golden set stratified by season and site type, not one blended precision number, so a seasonal false-positive creep doesn't hide inside an average that still looks fine. And the alternative genuinely considered and rejected: freezing Corbelwatch's threshold at its pilot-era setting to keep the substitution math simple, turned down because a frozen threshold misses exactly the borderline, high-consequence cases, an unclipped harness at height, that the looser setting exists to catch.

And if you want to be sure it really works, try it somewhere else

Same five letters, a trucking fleet instead of a job site, and this time the thing no one vantage point could see wasn't a harness clip. It was a driver's eyes.

Wakewatch is Thistlegate Freight Co's in-cab driver-monitoring system. It watches for eyes-closed events, head-nods, and phone glances across the fleet, all shift, on every truck. Thistlegate runs it across 620 trucks, logging about 46,000 flagged fatigue and distraction events a year.

Same BOUND, a different fleet, a different thing no single human vantage point could have caught in time.

Radim Tavarez owns Wakewatch's numbers, and Thistlegate's original ROI sheet had the same shape as Corbelwatch's: every flagged event valued as a catch a dispatcher would otherwise have made by reviewing footage. The reality is starker here than at Fennborough. A dispatcher reviewing live feeds can realistically watch a handful of trucks at a time out of 620, and most fatigue events last under three seconds, well under what a spot-check schedule could ever intersect. Radim's own shadow measurement put dispatcher substitution at just 8% to 15%, far lower than Corbelwatch's 22% to 38%, because a truck cab has no equivalent of a walking round at all.

Same rank, different lever: Thistlegate's fix isn't a bigger dispatch team. It's the same substitution habit, run on Wakewatch's own numbers, with the lower 8 to 15 percent range driving a much smaller labor-value claim and a much larger share of the flags valued instead as crash-avoidance value, priced against Thistlegate's own claims history.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: don't count every flag as a human catch, measure the real substitution rate, and value the rest as injury or incident avoidance instead of zero.
Cost: there's no time this quarter to run a real shadow audit. Ship the cheap version first, a documented estimate built from last year's near-miss logs, revisited next quarter with real audit data.
The model got better, for real: say Corbelwatch's false-positive rate got cut in half. The corrected ROI improves, but the substitution problem doesn't go away, a more accurate model still isn't the same thing as a model whose every flag was a guaranteed human catch.

Where people run it wrong.
They assume the substitution rate is a hundred percent because "the AI did the work," without ever measuring what a person would have caught in parallel.
They swing too far the other way and price the human-uncatchable flags at zero, when they still carry real, separately priceable safety value.
They set the substitution rate once and never revisit it as the model's threshold, or the scale of the rollout, changes.

How to use it live. Ask one clarifying question before any arithmetic: "Has anyone actually measured what a human process would have caught in parallel, or is this substitution rate just assumed?" That question alone usually tells you whether you're about to defend a real number or a guess wearing one.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0