How do you measure whether users over-trust your AI feature?

Not a values statement about trust. A way to turn a habit nobody can see themselves losing into one real number, built around the person who never knew a model was even in the room.

How to answer this, stage by stage

Nobody's grading whether you know over-trust is bad. They're grading whether you can turn it into something you could put a number on and actually act on. Seven moves get you there.

Let's learn

Before any of this, fixing a small bug meant forty minutes: an engineer wrote it by hand, ran the tests, and read the diff twice before opening a pull request. Slow, but every line had been read by the person shipping it.

Driftpoint sits inside a junior engineer's code editor. It reads a bug report or a failing test, looks at the file, and writes a fix, not just the next few words, a whole patch, ready to commit.

With Driftpoint, that same fix takes about six minutes. The tool writes it, the existing tests pass, and the engineer opens a pull request. On a monthly audit that re-checks a sample of 200 merged Driftpoint fixes against a set of 600 known tricky patterns, race conditions, broken null checks, weakened auth logic, Driftpoint's suggestions come back correct about 94 times out of 100.

Here's the turn. Those six wrong suggestions out of a hundred are not the real problem. The real problem is what a person does with a tool that's right 94 percent of the time. They stop checking. Not because anyone told them to. Checking a fix that's almost always right starts to feel like wasted effort, until it's easier to just accept it and move to the next ticket.

At its worst, that habit costs more than one wrong line of code. It costs the one thing every reviewer, every on-call engineer, every person downstream was counting on without knowing they were counting on it: that somebody actually looked at this before it shipped.

What I would leave alone: the everyday completions, a finished variable name, a closed bracket, an import line. If Driftpoint gets one of those wrong, the code doesn't compile and the engineer notices on the next line. That mistake costs a keystroke to fix. Flagging those as unverified would just be noise. The flag only earns its place on the categories the golden set actually catches people missing: auth, money, anything touching more than one organization's data.

The lesson: the safer a tool feels, the less anyone checks it, and the less anyone checks it, the harder its mistakes are to catch, because by then nobody's looking. A model that's right 94 percent of the time isn't 94 percent safe. It's exactly as dangerous as whatever happens on the other 6, multiplied by how sure everyone's gotten that it won't.

Now here is the same thing as a story

The short version sits above. Read on for the six hours in March that Talon spent chasing a bug that traced back to fourteen seconds of Lucca's Friday.

Lucca can spot a race condition by the shape of a stack trace before the debugger even finishes loading. Two years into the job, that's already a small reputation. Driftpoint arrived on his team in his first month, and for a long stretch it was the best part of his day: a bug report would land, he'd open the file, and by the time he'd finished reading it, a fix was already sitting there, correct enough that reading it twice felt almost silly.

He read it twice anyway, at first. Every fix, line by line, checked against the bug report the way he'd been taught. Driftpoint kept being right. Not every time, but close enough that a second read almost never turned anything up.

The habit thinned in three beats nobody would have called a decision. First he stopped reading fixes for the small stuff, formatting, renamed variables, anything a compiler would catch on its own if it were wrong. Then he stopped reading the medium fixes too, the ones touching two or three files, because Driftpoint's tests always passed and the diff always looked like something he'd have written himself. By his second year, he was reading closely only when a fix touched a file he already didn't trust, payments, or anything with the word auth in its name. Everything else, he glanced at for a few seconds and accepted.

The trigger wasn't a bad review or a shouting match. It was ordinary: a bug report asking why users on the shared enterprise plan sometimes saw a loading spinner that never resolved on the order history page. Small. Annoying. Not the kind of ticket that makes anyone nervous.

Driftpoint suggested a fix that resolved the spinner by loosening a session check that had, it turned out, been doing two jobs at once: keeping the page fast, and keeping one organization's orders separate from another's on that shared plan. The fix solved the spinner. Lucca opened the diff, saw it touched a file called session_scope.py, and for a second some part of him thought about reading it closely. Then he saw the tests were green, the file was three lines shorter, and it was 5:40 on a Friday. He accepted it. Fourteen seconds passed between the suggestion appearing and the commit going out.

For most of the weekend nothing happened, because most users never hit the rare condition behind it, being logged into two linked organizations on that shared plan at once. On Monday morning, forty-one of them did. For about six hours, before anyone noticed, some of those forty-one users could see order history that belonged to a different organization on their own plan.

Talon was on call that week. The page didn't say unverified AI fix, fourteen seconds, no review. It said auth anomaly, order-history endpoint. Talon spent most of the morning where any engineer starts: reading the endpoint's own code, which looked fine, then working backward through every service that endpoint called, because nothing in the ticket, the pull request, or the commit told him where to look first. The pull request said fix loading spinner on shared-plan orders, reviewed and approved. Nothing on that page said a model had written the fix, or that nobody had read it.

I want to say the problem was that Driftpoint got it wrong. It did, this once. But that's not really the story, because Driftpoint being right 94 times out of 100 is exactly what built Lucca's habit in the first place. He never had a number in his head telling him ninety-four percent, keep checking. He had a feeling, and the feeling only had two settings: this is basically always right, or this needs a real look. Enough good weeks flipped it to the first setting, and once it flipped, a green test suite and a Friday evening weren't going to flip it back.

So here is the decision I would take back.

When Driftpoint first shipped, the team built it to insert a suggestion into the editor exactly the way a keystroke lands: no marker, no tag, nothing in the diff or the pull request saying this line came from a model, let alone whether anyone had read it before it shipped. That made sense back when Driftpoint only finished variable names and closed brackets. Nobody sat down and decided it should still work that way once it was writing whole fixes to files named session_scope.py.

I would put a flag back. Every suggestion Lucca accepts gets tagged unverified until something proves someone looked: a test that touches the changed lines gets run, or the diff stays open long enough that a fourteen-second accept simply can't happen on a listed file. The flag doesn't stop Lucca from accepting fast when the fix is genuinely small. It stops the pull request from lying to Talon by omission.

Run the Friday replay with the flag in place. Lucca accepts the same fix in the same fourteen seconds. The pull request now shows unverified next to it, because session_scope.py sits on the list of files that always need a real check. Talon's review screen shows the flag before he clicks approve, and the review takes him four extra minutes instead of the six hours he actually spent. Forty-one users never see the spinner turn into something worse.

What I'd tell myself, back when we shipped that first version: the day a tool starts writing code good enough that people stop reading it, that's the day it needs a way to say, out loud, nobody's read this yet. We waited for an incident to teach us that. We didn't have to.

GUARD, run against fourteen seconds and a green test suite

This isn't a policy question. It's GUARD run on one decision: what a junior engineer's editor does the moment a fix it wrote gets accepted with nobody reading it.

One of them holds a real choice each time. The other one just lives with whichever choice got made, and has no way to see which.

The gap sits in the middle box, on purpose. Nothing in the pull request template ever asked whether this fix had been read.

Three things worth saying plainly, since this is where the real judgment sits. The alternative I'd turn down first is a quarterly survey asking engineers to rate how carefully they check Driftpoint's suggestions. Over-trust is exactly the kind of thing a person can't see from inside it: Lucca would have rated himself careful, and the tag on his own commit says fourteen seconds. The failure worth naming by name is confident wrongness: Driftpoint's fixes read exactly like something a careful senior engineer would write, clean, idiomatic, passing every test that already existed, which is precisely why the usual tell, code that looks off, never fires. The guardrail is the tag plus the golden-set audit plus the seeded checks, none of which depend on the fix looking suspicious in the first place. There's a real cost to running this, not a free one: gating a listed high-risk file adds a real four minutes, on average, to a review that used to take thirty seconds, for every fix that touches that file, whether or not that particular fix was actually wrong. We accepted slower reviews on a short list of high-risk files in exchange for not finding out about the next session_scope.py from a page at 2am. And the bar for treating the unverified rate as a real signal, not noise, isn't one bad Friday. It's a group's zero-verification-on-wrong rate sitting more than fifteen points above its trailing three-month average for two audit cycles running, before anyone gets paged about the pattern itself.

And if you want to be sure it really works, try it somewhere else

Same five letters, Kettleridge County's permit office instead of a code editor, and the same shape shows up with no engineer anywhere in sight.

Kettleridge County runs a pre-screen tool on every building-permit application: it reads the submitted plans and flags each one likely compliant or likely non-compliant before a human clerk ever opens the file. Zanya Loresh is one of three clerks who process that flagged queue.

G, groups. Zanya, who decides whether to open a flagged file and check it herself or clear it on the tool's flag. And every applicant behind that file, mostly small contractors and homeowners, who never see the flag at all, only a letter saying approved or denied.
U, unequal. Zanya's cost for trusting a wrong flag is small: her queue clears faster, and most weeks the flag was right. The applicant's cost is not small. A wrongly denied permit means a stalled renovation, a missed contractor booking, sometimes a re-application fee, and they pay it whether or not anyone at the county ever double-checked the tool.
A, ability to contest. An applicant who gets denied has no way to ask whether a person actually looked at their file or the tool cleared it in six seconds, because the denial letter doesn't say which one happened. The county never built a field for it.
R, reduce. Tag every clearance the same way: verified only once a clerk has spent real time on the file, not just clicked past it. Track the share of denials overturned on appeal that carried the unverified tag all the way to the letter going out.
D, detect. Twice a quarter, run a sample of applications the county's own inspectors have already fully checked back through the tool as a seeded test, and see how often a clerk clears a wrong flag with no real look. A real number, checked on a schedule, not a wait for the next appeal to surface it.

Different building, same missing field: nothing in the letter says whether a person actually looked.

Swap the trigger and it still runs.
Speed: an interviewer gives you thirty seconds. Skip straight to the one number, the share of wrong outputs shipped with zero verification, and say plainly why it beats a trust survey.
Cost: legal says a real per-file audit trail is a full quarter of engineering work. Don't use that as a reason to ship with no tag at all, log the six-second clears by hand in the meantime and watch that number weekly until the real system lands.
The model got better, for real: say the pre-screen tool's accuracy climbs from 90 to 97 percent this year. That's not the same claim as "clerks can trust it more." A model that's right more often gives people even less reason to look, which is exactly the condition this whole answer exists to catch.

Where people run it wrong.
They build the verification tag, then let the clerk mark her own file verified with one click, which measures nothing but how fast someone can click.
They treat a quiet appeals queue as proof the tool's fine, instead of asking whether people who don't know a model was involved even know they can appeal.
They respond to a bad number by demanding every file get a full manual check, which kills the tool's entire purpose instead of fixing the one flag that was missing.

How to use it live. Open with the two people, not the metric: "I'd name who's deciding and who's stuck living with the decision, then build the number around the second person, not the first." That's the sentence that tells an interviewer you found the actual risk, before you've said a single number out loud.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0