What should you spike first when both data quality and model capability are in doubt?

Cornbelt Growers Cooperative is piloting RowSense AI, a computer-vision tool that reads drone and phone photos of corn leaves for early signs of disease. Tunde Okonkwo is the AI PM who has to decide what to spike first when nobody can say yet whether the problem is the photos or the model.

How to answer this, stage by stage

Nobody is scoring whether you know that both data and model can be bad. They are scoring whether you can say, concretely, which one to check first and why getting that order wrong is the expensive mistake.

Let's learn

The phone in Colette Vranken's hand held two hundred photos of the same rows of corn, taken across three mornings, meant to teach a tool the difference between a healthy leaf and one showing the first signs of blight.

Before RowSense AI, Colette walked the rows herself every Tuesday, checking maybe 40 plants by eye across two hours, catching disease early enough to treat about 85 percent of the time. The tool promised to scan every plant a drone could photograph, thousands per hour, and flag the ones that needed a second look.

Four steps, in order. Skip the first one and the third step has nothing real to be scored against.

Here's the turn: when the first spike came back with mixed results, the instinct in the room was to ask "is the model good enough." That is the wrong question to ask first. Nobody yet knew whether the two hundred photos themselves were even reliable enough for any model, good or bad, to learn something true from.

At its worst, engineers spend three months fine-tuning a model against labels that two human experts couldn't even agree on themselves, produce a model that scores well on paper against those same shaky labels, and ship a tool that's confidently wrong on exactly the blurry, real-world photos a drone actually produces.

What I would leave alone: I wouldn't run this same two-step data-then-model order for every future model update. Once the data pipeline and label agreement are established and stable, a routine model refresh can be spiked directly against that known-good baseline.

The lesson: we asked whether the model was ready, when the real first question was always whether the data even had a stable, agreed-upon answer to be ready against.

Now here is the same thing as a story

The short version above is what you'd say defending the spike order in a five-minute stand-up. Read this one for what it looked like the week two bad readings in a row nearly sent the team straight to blaming the model.

Every Tuesday, Colette printed the week's photo batch and walked out to the yard with it, checking the physical plants against what the drone had caught, because nine years of doing this by eye had taught her that some things a photo just doesn't show.

RowSense's first pilot batch looked promising on the surface: the model flagged 34 plants as showing early blight symptoms out of 2,000 scanned. Tunde and the engineering team were pleased. Then Colette walked two of the flagged plants herself and found nothing there at all, just normal leaf shadow that the model had mistaken for a lesion.

Four honest signs it was never really about the model's accuracy score.

Two in a row, both false alarms, both traced back to the same shadow pattern under low morning light. The easy read in that room was "the model needs more training." Tunde didn't take that read.

He pulled two of Cornbelt's own agronomists, Colette and a colleague from the co-op's second site, and had them independently label the same 200 photos the model had been trained on, without telling either of them what the model had said. On the clean, well-lit half of the sample, they agreed 94 percent of the time. On the blurrier, real-world half, shot at odd angles and low light, they agreed only 61 percent of the time.

The audit that answered the real question before the model ever got blamed for anything.

The real question was never whether the model could tell a shadow from a lesion. It was whether the photos themselves carried a clear enough signal for anyone, human or model, to tell the difference reliably in the first place.

The cheapest test to run also turned out to be the one blocking every other test downstream of it.

Once the audit named the real gap, the fix wasn't a bigger model. It was a stricter field-photo protocol: a minimum lighting threshold and a required angle range before any photo entered the training set at all. Re-run against 200 new photos collected under that protocol, agreement between the two agronomists rose to 89 percent, and the model's flagged false-alarm rate on real field conditions dropped from roughly 1 in 15 to about 1 in 60.

The asymmetry that decided the order: one mistake compounds quietly for months, the other one is a Tuesday afternoon fix.

What I'd tell myself, watching those two false alarms almost get pinned on the model: the model was never lying. It was answering exactly the question the data gave it, and nobody had checked yet whether that was a fair question to ask.

ORDER, the rank that survives the follow-up questionNot a script for auditing data before every single model change. ORDER is what tells you exactly which risk is hardest to undo, and why that one goes first.

The recap, one line per letter: outcome is cheap confidence before committing real engineering time, reversibility is a bad model swap versus a compounding bad-data problem, dependency is that model testing has no honest ground truth until labels are trustworthy, evidence is a two-day agreement audit before touching the model, and rank is data first, model second, defended by the fourfold drop in false alarms once the field-photo protocol shipped.

And if you want to be sure it really works, try it somewhere elseSame five letters, a fish-quality grading tool instead of a corn field. Different flip family entirely, the same audit-first order.

Niamh Colby runs quality control at Saltbourne Fisheries, piloting a tool that grades fish freshness from photos taken dockside by crew as catches come in. Mapped onto ORDER: outcome is cheap confidence before investing in a full grading-line rollout. Reversibility favors the data question again: a bad model swap is cheap, but a bad labeling protocol, disagreement between graders on what "grade A" even looks like, compounds across every catch already logged. Dependency is that model accuracy against inconsistent human grades measures nothing real. The flip here is different from Cornbelt's: it's a concealment flip, crew who worry the tool will replace them start quietly using their own private grading system and only reporting the tool's number when it happens to match, which hides the real disagreement rate from management entirely. Evidence is a blind, independent regrading of 150 recent catches by two senior graders. Rank still lands data first, though the reason getting there runs through a different, more human problem.

The same asymmetry holds in a completely different industry: the data mistake is the one that keeps costing you.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "spike the data first, because a bad model swap is reversible and a bad data decision compounds," and stop.
Cost: there's no budget for a full two-agronomist relabeling exercise. Say so honestly, and run the smallest real version, twenty photos, two reviewers, rather than skipping the check entirely.
The model tests better than expected, for real: even a strong early model score is worth auditing the data behind, since a model can score well against labels that are consistently wrong in the same direction.

Where people run it wrong.
They treat "the model isn't good enough yet" as the default explanation before checking whether the data even has a stable answer.
They standardize test conditions so cleanly that the pilot never meets the messy real-world inputs it will actually face.
They let one clean spike result convince them the whole pipeline is solid, instead of testing the next batch of real conditions too.

How to use it live. The moment an interviewer asks what to spike first, ask yourself: if I test the wrong one first and I'm wrong, which mistake can I still undo cheaply next week, and which one have I already baked in? Whichever one you can't undo goes first.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0