What is the difference between model quality and product quality? Give an example where they diverge.

OpenBay is Curbwise's live map for the city of Bellmoor. It tells a driver, block by block, whether a curb spot is likely to be open right now. Ceridwen Stroud owns OpenBay's metrics. Marnix Nettlefold's team just retrained the prediction model and wants to call it a win.

How to answer this, stage by stage

Nobody's grading whether you can define two vocabulary words. They're grading whether you'll go check a real, live, segmented number before agreeing that a rising benchmark means the product got better.

Let's learn

The scorecard is one sheet of paper, printed every Monday, pinned above Ceridwen Stroud's desk with last week's number circled in pencil.

OpenBay is Curbwise's live map. It looks at a block in Bellmoor and tells a driver, right now, whether a spot is actually open.

These are two different report cards. Nothing says they have to agree with each other.

Before the retrain, OpenBay's ten-minute-ahead accuracy sat at 84 percent, measured against a year of logged curb-sensor readings. Downtown, where Ceridwen watches closest, the confirmed-spot rate ran 68 percent: a driver who followed OpenBay's suggestion found the spot open, no reroute, about two times out of three.

Marnix's team retrained the model with weather and the city's event calendar folded in. Ten-minute-ahead accuracy on that same historical test climbed to 91 percent, a real seven-point jump. Citywide, the confirmed-spot rate even ticked up, 74 to 78 percent. Downtown, on its own, it fell to 54 percent.

Here's the turn. The new model is heavier, so it takes longer to run, and the live map only redraws every 95 seconds now instead of 20. Downtown, a spot flips every three and a half minutes, so 95 seconds of staleness eats a third of the whole window before a driver even arrives. Outer neighborhoods, where a spot sits open for 22 minutes, never feel it. Blend both into one citywide number and the outer gains bury the downtown loss so completely that the citywide line still reads as a win.

What it costs at its worst: downtown drivers stop trusting the map exactly where trust mattered most. They go back to circling the block, checking two or three other apps, or driving straight past the block OpenBay just sent them to, because they've already been burned once. That defeats the entire reason Curbwise built OpenBay for downtown in the first place.

What I would leave alone: the outer ring. Its 89 percent confirmed-spot rate is a real, honest gain, not a number hiding a loss somewhere else, because staleness genuinely doesn't matter when a spot sits open for 22 minutes. Rolling the whole retrain back everywhere would throw away a real improvement to fix a problem that only lives in one part of town.

The lesson: a model's report card and a driver's actual 90 seconds are two different tests, graded on two different clocks. When they disagree, believe the second one until someone proves otherwise, because the second one is the only test the driver ever takes.

Now here is the same thing as a story

The short version is above, for when you're in the room. This one is for feeling why a rising benchmark and a real product win are not automatically the same claim.

Ceridwen Stroud can look at a single week's OpenBay numbers and tell you, before she's finished her coffee, which neighborhood in Bellmoor is about to become a problem. Two years running the scorecard will do that to a person.

Every Monday she prints the same sheet: OpenBay's confirmed-spot rate, citywide, broken out by the six curb zones Curbwise tracks. She pins it above her desk and circles whatever moved. It isn't fancy. It works.

Marnix Nettlefold's retrain shipped on a Tuesday in March. His team had spent two months on it, folding in weather and the city's event calendar, and the historical test came back at 91 percent, up from 84. Ceridwen watched the citywide confirmed-spot rate for the first two Mondays after launch, and it did exactly what a good retrain should do. It went up. 74 to 76. Then 76 to 78. She circled it both times, in green.

For a while, she stopped pulling the segmented view. The blended number was supposed to be the whole scorecard's job, that was the point of building one number in the first place, and the blended number kept saying the same good thing. She had four other launches to watch that quarter. Something had to give, and the six-zone breakdown, which had never once disagreed with the blended number before, was the thing that gave.

Nothing dramatic happened after that. No single Tuesday where a number cratered. Downtown's confirmed-spot rate slid a point most weeks, sometimes two, buried inside a citywide average that kept climbing because the outer ring kept getting better and better. Nobody flagged it, because nobody was looking at downtown on its own.

What finally made her look again wasn't the scorecard at all. A support lead mentioned, almost in passing, in a hallway, that downtown reroute complaints were up. Not a spike. Just a mention. Ceridwen pulled the six-zone view for the first time in five weeks.

Both of these were true at the same time. Only one of them was the one Marnix's deck led with.

Downtown's confirmed-spot rate had fallen from 68 to 54. Reroutes, drivers told mid-drive that the block they were headed to had already filled, had gone from 9 percent of downtown trips to 22. The outer ring, meanwhile, had climbed from 76 to 89, comfortably enough to hide the whole thing inside a citywide number that had read as a win the entire time.

The choice Ceridwen would take back happened in a planning meeting, months earlier, when the team set the ship gate for every future retrain: validation accuracy has to improve on the historical test set, measured as one blended number. Nobody argued. It was simple, it was fast to check, and back then turnover was roughly even across Bellmoor, so a blended number and a segmented one told the same story anyway. That stopped being true the day a retrain started trading downtown speed for outer-neighborhood gains, and the gate never noticed, because the gate was never built to look.

She dug into why the new model was slower. It wasn't the weather feature or the event calendar, those were cheap to compute. It was the heavier ensemble underneath both of them, which pushed the live map's refresh cycle from 20 seconds to 95. Downtown, where a spot flips every three and a half minutes, 95 seconds of staleness eats a third of the whole window. Outer neighborhoods, where a spot sits open 22 minutes, never feel 95 seconds at all.

Run the same five weeks again, with staleness at arrival tracked by zone from day one instead of a blended accuracy score. It starts climbing in week one, 25 seconds to 41. By week three it's past 70, well before the confirmed-spot rate has moved anywhere Ceridwen would notice on the blended view. She holds the new model downtown, ships it everywhere else, and the reroute number never gets past 11 percent.

One version of that scorecard hides five weeks of downtown losing trust inside a number that's technically true. The other one catches it in the first one.

What I'd tell my past self, the one who wrote "validation accuracy improves" into the ship gate and moved on to the next launch: a number that only checks whether the model got better at its own test will never once tell you whether it got better at the driver's.

LEAD, or the two clocks a launch review never compares

Not a way to make "model quality" and "product quality" sound like the same idea with different words. LEAD is what forces you to name the second clock and check it before anyone signs off on the first.

Four checks, run on one retrain. Skip any one of them and a real regression can travel all the way to a leadership deck.

The early signal (E) moved in week one. The outcome everyone was actually watching (L) didn't move on the blended view until week five.

Same 95 seconds of staleness, three different costs, depending entirely on how fast the spot underneath it turns over.

Three things worth naming directly, since this is where the real judgment sits. The AI-specific failure mode is a quiet distribution shift the historical validation set was never built to catch: it measures how right the model is about a block, not how long that rightness survives before the block changes underneath it, so a heavier, more accurate model can pass its own test while getting slower in a way the test never scores. The guardrail is the staleness-at-arrival number itself, tracked by zone, because it moves before the outcome does. There's a real trade-off too: the fix isn't free. A faster update cycle downtown likely means a lighter model there, which gives back some of the seven accuracy points the retrain earned. Curbwise is choosing a smaller, faster-refreshing win over a bigger, staler one, and that's a real cost, not a free upgrade.

One number, staleness, three facts about the city it lands on. The number alone never tells you which cost it's about to cause.

And if you want to be sure it really works, try it somewhere else

Same four letters, a different curb entirely. This time the model isn't predicting where a car can park. It's predicting which machine is about to break.

Frostwell HVAC Networks runs PulseCheck, a model that reads a unit's sensor history and flags which ones are likely to fail soon, so a dispatcher can send a technician out before a business opens to a dead system. PulseCheck's failure-prediction score, measured against a decade of maintenance logs, climbed from 0.81 to 0.89 after a retrain. A real gain, on paper.

A different mechanism, the same shape of divergence: the benchmark can't see the thing that actually breaks the product.

Run the same four letters. Link: not the offline failure-prediction score, but breakdowns actually prevented, a technician arriving before the unit fails, not after. Early signal: the fraction of flags dispatchers actually act on, split by thermostat generation. For older analog units, that act-rate held steady at 82 percent. For newer smart thermostats, which send richer but differently formatted telemetry, it fell from 74 to 41 percent over three weeks, because PulseCheck's own confidence output got noisier on that telemetry, and dispatchers learned to distrust flags on those units specifically, even the correct ones.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: model quality is graded against old logs, product quality is graded against the next real moment, and a benchmark can rise while a segment of the product quietly falls if nobody checks by segment.
Cost: no budget this quarter to retrain PulseCheck on new telemetry. Ship advisory-only flags on new units, hold auto-dispatch, and leave the old-unit path exactly as it is, since that's the segment already paying for itself.
The model got better, for real: say OpenBay's next retrain also gets faster, cutting the update cycle back under 15 seconds citywide. Downtown's confirmed-spot rate should recover, but only if someone actually checks the downtown number instead of assuming a faster model automatically means a faster map everywhere it's deployed.

Where people run it wrong.
They report one blended number because a segmented one looks like they're not confident the launch worked.
They test speed and accuracy on separate dashboards and never ask whether the two trade against each other inside the same retrain.
They treat a rising offline benchmark and a healthy live product as the same fact, instead of two separate claims that usually, but not always, travel together.

How to use it live. Ask one question before trusting any reported win: "what does this look like split by the segment where it's hardest, not averaged across the segment where it was always going to be easy?" That question alone usually tells you whether a number is measuring real product quality or just the parts of the product that were never going to be a problem.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0