Give an example of an output that is accurate but not useful.

An accurate number that answers the wrong question is not a small miss. It is a miss with a straight face, and it looks exactly like success on a dashboard.

How to answer this, stage by stage

Nobody is grading whether you can define "accurate but not useful." They are grading whether you can name a real example and explain, out loud, the actual mechanism behind it. Seven moves get you there.

Let's learn

What happens when a number keeps getting more correct, checked against real weather stations every single night, while the thing it is supposed to cause, a farmer covering his vines on a cold night, keeps happening a little less often?

Fieldstead is an app that tells a farmer, for one specific field, whether frost is coming tonight, rain tomorrow, and when a safe window opens to spray without the wind or the wet ruining it.

Before this kind of tool existed, most growers leaned on a county-wide forecast that covered forty thousand acres at once. About one call in four was wrong for their own field, because a county has hills, low pockets, and open ground all mixed together, and the forecast could only speak for the middle of it. A wrong call the safe way, covering when nothing needed it, cost a grower about three hours standing in the cold for nothing. A wrong call the other way, not covering when he should have, could cost him a whole block of fruit.

Fieldstead's field-level model got the frost call right on 94 percent of nights, checked against each grower's own on-field weather station. That is a real number, and it climbed steadily for a year.

Here is the turn. The extra correct nights were not the problem. The problem is what growers did with an alert that kept saying, in effect, "the field is fine." Fieldstead's own logs showed the share of frost alerts that a grower actually acted on, cover the vines, run the wind machine, call a neighbor for help, held flat near 40 percent even as the model's accuracy climbed from 81 to 94 percent. A more correct number was not turning into more of the right action.

At its worst, this gap can sit invisible for a whole season, because "the number matched the station" and "the farmer did the right thing" are two totally different tests, and a team that only measures the first one has no way to see the second one slipping.

What I would leave alone: the app's seven-day outlook, the one growers use to plan labor and spray supplies for the week, does not need block-level precision. A field-average number is fine there. Nobody is standing in the cold at midnight deciding whether to cover based on next Thursday's outlook. The mistake was never the field average itself. The mistake was using it for a decision that needed to be made block by block, tonight.

The lesson: an accurate number and a useful number get graded by two different tests. Build an eval for only the first one, and the second gap stays invisible right up until a real cold night finds it.

Now here is the same thing as a story

The short version sits above. Read on for the April morning Sorina drove out to a vineyard at 5am with a flashlight, trying to work out why the app had said everything was fine.

Sorina Coulter has run alert quality at Fieldstead for three years. She built the first version of the field-station network herself, driving out to enrolled farms on weekends to set the little metal boxes in the ground. Ask her which of Fieldstead's twelve hundred fields sit in a valley and which sit on open ridge, and she can tell you most of them from memory.

Garrick Aubrecht grows wine grapes on a slope two hours from the coast. He enrolled his vineyard in Fieldstead's second season, mostly because a neighbor swore by it. For a year and a half it was, in his words, "the best five minutes of my evening." He'd check the app at dinner, see the night's risk number, and know within a minute whether he needed his crew or could sit down.

He used to walk the low rows himself most cold nights anyway, just to check the app against his own eyes. By the second spring, he had stopped. The app had been right every time he checked it, and walking three rows in the dark at 9pm got old.

The habit that held Fieldstead's own quality bar together thinned in three beats nobody noticed at the time. First, Sorina had once called five or six growers a week after every frost alert, just to ask "did you cover, and did it matter." She stopped once the dashboard's accuracy number climbed past 90 and the calls started to feel like busywork nobody asked for. Second, a plan to put a dedicated sensor in every enrolled field's lowest corner got pushed twice in planning, because the field-average number never gave anyone urgency to fund it. Third, Garrick's vineyard had been onboarded without its own low-corner sensor, so the model estimated that block's temperature from the field-average station near his barn, a few hundred yards uphill from the actual cold pocket.

The trigger wasn't a storm. It was one ordinary Tuesday in April. At 9pm, Garrick's phone read "31°F, moderate risk." He had covered on nights colder than that before with no trouble. He read it, felt nothing new, and went to bed.

His three lowest rows, the ones nearest the creek at the bottom of the slope, hit 25 degrees that night. Five degrees colder than the app's number, and three degrees below the point where new grape buds take real damage.

Sorina drove out at 5am once the frost-damage report came in from the field team. She wasn't checking whether the model was wrong, exactly. The station reading that night, the one near the barn, was right. She was checking why an accurate number had sat calm the whole time a real problem was three degrees away.

The old decision went back to a planning meeting in Fieldstead's first year, when Sorina and two engineers built the very first version of the accuracy eval. Someone said, reasonably, "let's grade every alert against the nearest weather station, that's the one number we can actually check." Nobody else could argue with it. It was the only ground truth they had.

Nobody in that meeting decided that field-average and block-actual were the same question. It just quietly became the only question the eval ever asked.

TRACE, run on the six degrees nobody measured

This isn't a diagnosis of one bad night. It's TRACE run on the general question this page opened with, using Garrick's near miss as the ruler for what "useful" actually has to mean.

The three candidates side by side. Two ruled out on evidence, one confirmed: the model answered the wrong question, a field's average instead of one block's actual number.

Three things worth saying plainly, since this is where the real judgment sits. The rejected alternative: running full block-level modeling on every enrolled field, every night, no matter how flat the terrain. That would have caught Garrick's block too. It would have also multiplied the overnight compute bill across eleven hundred fields that have no cold pocket at all, and added real minutes to how long the nightly alert batch takes to finish for every grower, not just the ones who need it. Fieldstead rejected that and instead triggers block-level modeling only for fields sitting in the top quarter of measured elevation variance, and even then only on nights where the field-average risk crosses 25 percent, a band that covers about one night in six during frost season. Slower and pricier for the fields that actually need it, untouched for the ones that don't.

The AI-specific failure worth naming by name: this is a form of cold start. Garrick's block had no sensor of its own, so the model estimated it from the nearest station, a guess that works fine on flat ground and badly on a slope with a known cold pocket the station has never actually measured. The guardrail Fieldstead ships now: any block estimate built this way gets labeled "modeled, not measured" in the grower's app, and it doesn't get treated as a trusted number for a cover decision until the block has built up twenty nights of real agreement with a nearby sensor. And the bar for trusting a block estimate isn't "the model is never wrong." It's calibrated and stated as one: a block's modeled low counts as reliable once it lands within 2 degrees of that block's own sensor reading on at least 90 percent of the nights in its own validation set, not on every single night, on nine nights out of ten, by design.

And if you want to be sure it really works, try it somewhere else

Same five letters, an AI radiology report drafting tool instead of a weather app, and the exact same shape shows up with no field or vine anywhere in sight.

Radiovue drafts written summaries of chest X-rays for referring physicians, using the radiologist's own annotated findings as its source. Lyall Brix runs product for the report-drafting feature there.

T, timeline. Radiovue's draft-to-expert match score had held near 97 percent for months, essentially perfect agreement with what a human radiologist would have written. One emergency physician's delayed follow-up on a real finding, caught in a routine chart review, is what first surfaced a gap the match score never showed.
R, recut. Recut the metric from "did the draft mention every finding" to "how many items down the list does the most urgent finding sit." Draft reports that read clean and complete on the match score were burying a single urgent line behind six or more routine ones, listed by anatomical order instead of by what needed a call today.
A, assume nothing. Lyall's team checked first that the report had actually reached the physician's inbox and been opened. It had, within four minutes of being signed. The gap wasn't delivery. The team's harder assumption to give up: a 97 percent match score had felt like proof the drafts were doing their job.
C, cause candidates. No instruction: ruled out, Radiovue's drafts do flag findings as "urgent" in bold. Wrong question answered: ruled out, the finding itself was correct and matched the right anatomy. Buried in detail: confirmed. The urgent line sat fourteenth of nineteen findings, true and bolded, under thirteen lines of accurate, ordinary detail a tired physician skims past at 2am.
E, evidence test. Physicians who acted on an urgent finding within the target window had it appear in the first three lines of the draft. Physicians who didn't had it appear past line ten, on every case Lyall's team pulled for review.

Every line in the real report was true. The one that needed a same-day call sat last, under five lines of correct, ordinary findings that answered questions nobody in the emergency room was actually asking.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to the example and name the mechanism in one breath: "a correct number that answers a different question than the one the person actually needed."
Cost: engineering says running block-level or urgency-sorted models on every case is too slow to ship this sprint. Don't drop the fix, narrow the trigger, apply it only where the risk band or the terrain variance says it's worth the extra compute.
The model got better, for real: say the underlying model's match score genuinely climbed another few points that quarter. That's still not the same claim as "every case is useful now." A model that gets more accurate on average can still be quietly wrong, in the same useless way, for the one case type it was never actually being measured on.

Where people run it wrong.
They fix the recut, then still glance at the old blended accuracy tile out of habit, because that's the number they've trusted for years.
They see accuracy climbing and treat that as proof the eval is fine, instead of checking whether action-taken is climbing along with it.
They respond to one bad case by adding a fourth or fifth eval dimension instead of asking whether the real fix is a guardrail on the one segment, unsensored fields, unranked reports, that's actually thin.

How to use it live. Open with the mechanism, not the story: "Here's an example: a number that's checked, correct, and still answers a question the person didn't ask." That buys you the room to defend the distinction on its own terms before the interviewer pushes you toward a vaguer answer.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0