What is your policy for adopting a newly released model into a production feature?

Cornflint AgTech sells CropVigil, an app that reads a phone photo of a crop leaf and flags likely disease before it spreads. Thiago Bandeira is the AI PM who decides whether, and how, a newly released model ever gets to touch a real field.

How to answer this, stage by stage

Nobody is scoring whether you'll say "test before you ship." They're scoring whether you can name the actual order of steps, and which one you'd never let anyone skip.

Let's learn

What happens the first time a new model release goes straight into production with nobody checking it first?

Before CropVigil, a field agronomist inspected leaves by hand, about 10 minutes a field, catching disease reliably but slowly, one field at a time. With CropVigil's first model, a photo comes back with a disease flag in about 3 seconds, at 89 percent accuracy across Cornflint's roughly 40,000 subscribed fields.

The five letters, held up as one page. Reversibility is the one most adoption policies never write down.

Here's the turn: newer models kept coming out, each claiming higher accuracy. Cornflint's team stopped trying every single one of them, for fourteen straight months, after hearing what happened to a peer company's rollout. That wasn't caution. It was the same all-or-nothing choice running in reverse: instead of risking everything on day one, they risked nothing, forever.

At its worst, farmers start comparing CropVigil's flags against a competitor's app on their own, notice the gap, and quietly switch, while Cornflint's own dashboard shows nothing more alarming than a slow, familiar decline in renewals.

What I would leave alone: I wouldn't run offline replay or a canary for a copy-only change, like rewording the disease alert message, since nothing about the model's actual prediction changed.

The lesson: a policy that only knows "adopt everything" or "adopt nothing" isn't a policy. It's a coin flip wearing a caution sign.

Now here is the same thing as a story

The short version above is what you'd say defending the policy in a planning meeting. Read this one for what it felt like the morning a peer company's near miss became everyone's excuse.

Cornflint's field team logs on before sunrise, before the dew burns off the leaves, to catch blight while a spray can still save the row.

Thiago had run model rollouts at two companies before Cornflint, and he was good at reading a benchmark for what it actually promised, not what it claimed. In CropVigil's first year, a new model release came out every few months, and Thiago's team tried each one within two weeks of announcement, comparing it against the current model on a batch of recent photos before deciding.

Each step only exists because the one before it got done first, not because someone remembered to be careful.

Then word reached Thiago about Oakspire Analytics, a peer agtech company that had cut over a new vision model to all of its fields on release day. The new model had reclassified early blight as healthy on a rare cultivar that wasn't well represented in its training data. A farmer skipped treatment on that field for a week, trusting the all-clear, and lost 20 percent of it before anyone noticed the pattern.

Thiago's team didn't decide, in any one meeting, to stop adopting new models. Nobody wrote that policy down. It just built up slowly: the next release came out, and nobody proposed testing it. Then the one after that. By the time anyone noticed, fourteen months had passed and three better models had come and gone untried.

Oakspire's mistake wasn't trying a new model. It was only ever having the third branch to choose from.

The real question was never whether to trust a new model. It was whether the rollout had any small, reversible steps at all, or only one giant, irreversible one, which is exactly what made "never adopt" feel like the only safe choice.

The actual design decision, close enough to argue with: small, cheap steps before the one expensive one.

When CropVigil's original deploy pipeline was first built, someone said, "the model barely changes release to release, let's keep the pipeline simple," and it sounded reasonable, since at the time, it was true.

Four checks, and Oakspire's rollout, by its own account, had skipped straight past all four.

Rerun the same new model release with the staged policy in place: offline replay against 1,200 labeled photos from last season shows 94.2 percent accuracy versus the current model's 89.1, with no regression on the rare cultivar. A 48-hour canary on 800 opted-in fields confirms it. Staged expansion follows over two weeks, full rollout by day 16, with nobody's crop ever put at risk to find out.

Sixteen days, and at every one of them, there's still a way to stop.

What I'd tell myself, hearing about Oakspire's lost field: the failure wasn't trusting a new model. It was never building a way to trust it a little at a time.

ORDER, the policy that turns "adopt or don't" into a real sequenceNot a script for adopting every release blindly. ORDER is what tells you exactly how much of a new model to trust, and when.

The recap, one line per letter: outcome is disease caught early without false-flag fatigue, reversibility is the ladder from harmless replay to irreversible cutover, dependency is the golden set and opted-in fields that have to exist first, evidence is the free offline replay that catches most regressions, and rank is the fixed order: replay, canary, stage, expand.

And if you want to be sure it really works, try it somewhere elseSame five letters, a city permits office instead of a farm. The field changes, the fourteen months of standing still doesn't.

Rosalind Okonjo runs product at Foxglen City Permits, where PermitScan reads a submitted building plan and classifies which review track it needs. Mapped onto ORDER: outcome is a permit routed to the right reviewer the first time, without adding delay for straightforward applications. Reversibility runs the same ladder, offline replay against last year's decided permits, then a canary on a small share of new submissions with a human reviewer double-checking, then staged expansion. Dependency is a labeled set of last year's permits with known outcomes, and reviewer capacity lined up to check the canary. Evidence is the free offline replay against those known outcomes. Rank is the same fixed order regardless of how good a new model's vendor claims it is.

Different building entirely, same fixed order, same chances to stop before the switch is final.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "replay, canary, stage, expand, always in that order, never skipped," and stop.
Cost: no budget for a proper canary infrastructure. Say so honestly, and start with offline replay alone rather than skipping straight to a full cutover to save the canary's cost.
The new model is a clear, obvious improvement: that's still not a reason to skip straight to full rollout, since "clearly better on average" is exactly the claim that hid Oakspire's rare-cultivar gap.

Where people run it wrong.
They treat "we tested it" as one binary step instead of an ordered sequence with real gates.
They let one bad experience turn into a policy of never adopting anything new, which is just the same all-or-nothing risk running backward.
They set the rollback bar after a bad week, when it's already too generous to matter.

How to use it live. The moment an interviewer asks for your model-adoption policy, ask yourself: what's the cheapest, most reversible step I could take first, and what has to exist before I can even take it? Name that first step, and the rest of the order follows on its own.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0