What are the five dimensions a PM should compare models on before a product decision?

Every evening for six years, Camila Restrepo pulled one bracket in twenty off the stamping line at Stancroft Metal Works and checked it by hand for hairline cracks. The plant makes suspension brackets for a mid-size auto parts supplier. GaugeSight is the vision model Stancroft bought to check every single bracket instead of one in twenty.

How to answer this, stage by stage

Nobody is scoring whether you can list five words. They're scoring whether you can rank them, defend the order, and say what changes the order.

Let's learn

Every evening, Camila Restrepo pulled one bracket in twenty off the line and checked it by hand.

Stancroft Metal Works stamps suspension brackets for an auto parts supplier. Before GaugeSight, Camila and two techs sampled one bracket in twenty, about 15 minutes of hands-on checking per hour of production, and still missed some hairline cracks that were hard to see under the shop's overhead lighting.

With GaugeSight, a camera above the line photographs every single bracket and flags cracks in real time, at whatever pace the vendor's demo ran at, which was a slow one.

Four manual tasks, all riding on one person's eyes under one kind of light.

Here's the turn: the real problem was never picking the most accurate model. It was that nobody had ranked "keeps up with the real belt" above "wins the benchmark," so the plant bought the second thing thinking it was buying the first.

The five things worth ranking. Speed under real load is the one most comparison sheets skip.

At its worst, a plant pays for a 100 percent inspection tool and quietly runs it as a 25 percent inspection tool, without anyone deciding that on purpose.

What I would leave alone: on Stancroft's slower aerospace-bracket line, 40 units an hour, this same ranking wouldn't change anything, since 1,400 milliseconds is nothing against a line running that slow.

The lesson: the five dimensions aren't a checklist you fill in once and forget. They're a ranking, and the ranking changes with your own volume, not the vendor's demo traffic.

Now here is the same thing as a story

The short version above is what you'd say in the room. Read this one for what it felt like the week a sister plant's comment made Camila go back and check her own numbers.

Camila Restrepo could spot a hairline crack under bad light faster than anyone else on her shift.

When GaugeSight went live, the first months were good. The camera caught cracks Camila had been missing for years, and for a while the plant ran at its slower ramp-up pace, comfortably inside GaugeSight's 1,400-millisecond scoring time.

The habit thinned in three beats. At first, Camila still spot-checked a handful of GaugeSight's "clear" calls herself, out of old habit. Within a month, she trusted the flags enough to stop double-checking clears entirely. By the time the line ramped to full speed, she wasn't watching the queue at all, she was watching the flags.

You can't judge cost or speed until you know your own real volume first. Order matters here too.

At a quarterly ops call, a quality engineer from Stancroft's sister plant mentioned, almost in passing, that their vision model "barely breaks a sweat, even during ramp week." Camila went back and pulled her own timing logs that night.

The logs showed GaugeSight taking 1,400 milliseconds a frame against a line now moving a bracket every 900. The gap wasn't hurting individual calls, it was piling up, frame after frame, all day.

Two mistakes, very different to undo. Only one of them quietly changes how people work.

Camila realized the real question was never whether GaugeSight was accurate enough. It was whether an accurate-but-slow answer that arrives after the next bracket has already passed the camera is worth anything at all.

When the original comparison sheet was built, someone in the room said, "let's weight this mostly on accuracy, since that's what actually matters," and it sounded right, since nobody had yet measured what the real line speed would ramp up to.

Rerun the same shift with the ranking fixed: Candidate C scores three points lower on the vendor's benchmark, 94 versus 97, and clears every single frame at 220 milliseconds. Zero backlog. And two weeks later, it flags a real hairline crack in a bracket that would have sat in GaugeSight's unchecked queue.

What I'd tell myself, hearing that sister-plant comment land at the ops call: the five dimensions were never a list to recite. They were a ranking, and I'd ranked them by the demo's pace instead of my own.

ORDER, applied to ranking the five dimensions themselvesNot a script for always picking the fastest model. ORDER is what tells you which dimension actually decides your case.

And if you want to be sure it really works, try it somewhere elseSame five letters, a grain co-op instead of a stamping plant. The ranking logic doesn't care what the belt is carrying.

Delphine Okoro runs operations for Prairiehatch Grain Co-op, which uses SiloWatch, a model that scans incoming grain loads for mold and insect damage before they're binned. Mapped onto ORDER: outcome is keeping contaminated grain out of a shared silo, at whatever truck-arrival rate harvest season actually brings. Reversibility: a wrong quality call is reversible, retrain or swap later, but once trucks start backing up at the scale house because scoring is too slow, drivers start skipping the scan on the busiest days, and that habit is hard to undo once it forms. Dependency: you can't price the model's cost per truck until you know harvest-week arrival rate, which is nothing like the co-op's average week. Evidence: run the finalist for one real harvest-week morning, trucks lined up for real, before signing a contract. Rank: quality on the co-op's own grain samples first, then speed against harvest-week arrival rate, then cost at that volume, then how many photos per truck it can take in at once, then how it handles a grain type it's never scored before. Delphine found the co-op had ranked cost first, since the vendor's per-scan price looked cheapest of three options, without weighting how the cheap price required capping input to one photo per truck bed. On a genuinely mixed load, one photo missed contamination pooled at the back, a failure mode nobody saw until a buyer rejected a shipment two weeks later.

Plotted together, the trade a benchmark ranking hides becomes obvious in one picture.

A five-step plan that costs one real shift of testing and saves a quarter of backlog nobody notices until it's huge.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "rank quality as the bar, then speed under your real load, then cost, then input size, then failure behavior, and test all five at your real peak before signing," and stop.
Cost: no budget for a full real-load test before deciding. Say so honestly, and commit to testing the two closest finalists for even one real shift before choosing between them.
The model got better, for real: if a new candidate claims better accuracy than your current pick, that's still worth testing against your own real load before switching, since "more accurate" and "still fast enough for your belt" aren't the same claim.

Where people run it wrong.
They copy the vendor's own benchmark ranking instead of building one against their own eval set and their own real load.
They test finalists in a quiet demo environment and never once at real peak volume.
They rank all five dimensions equally instead of admitting one of them decides the case.

How to use it live. The moment an interviewer asks for criteria to compare models on, name the five, then immediately ask yourself which one would actually break the product first at this company's real volume. Say that one out loud before the others; that's the part that shows judgment instead of memory.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0