A traditional feature either works or has a bug. Explain why that framing breaks for an LLM feature.

Slipstream is Bellwright's AI pair-programmer. It drafts the next few lines of code as a developer types, right inside the editor. Tamsyn Ezenwa handles every completion a developer reports as wrong. Zaneta Perowne owns Slipstream's roadmap and has to answer for it every quarter. Eight weeks after Bellwright swapped in a newer base model, Callaghan Endres, the VP of Engineering, asked Zaneta a question that should have been simple: how many Slipstream bugs are still open?

How to answer this, stage by stage

Nobody is grading whether you can say "AI is different." They're grading whether you can name the exact measurement that replaces a bug count, and defend it when someone asks what a single wrong output should actually trigger.

Let's learn

A normal bug has a fingerprint. Same input, same wrong output, every single time, until someone changes the code. That fingerprint is the whole reason "works or has a bug" makes sense for ordinary software.

Slipstream is Bellwright's AI pair-programmer. It reads the code a developer is typing and drafts the next few lines for them, right there in the editor.

Before Slipstream, nothing suggested anything. There was no ambiguity to manage, because there was nothing in between "the developer typed it" and "it shipped."

Slipstream drafts a completion in under a second, and developers accept it about 44% of the time. On a 6,000-prompt eval set Bellwright built to test it, checking whether the code compiles and passes the tests a human's version would need to pass, Slipstream gets it right 91.2% of the time.

Here's the turn. Slipstream getting a completion wrong is not, by itself, news. Every model that ships is going to be wrong sometimes, on purpose, because of how it's built. The real question is what Bellwright's engineers do the moment a wrong completion shows up. Right now, every wrong completion becomes a ticket, and every ticket gets a fix, the same way a null-pointer bug gets a fix. Once merged, it's marked Fixed and closed.

What it costs at its worst: Tamsyn's team spends about 30 hours a week working these tickets, roughly three-quarters of two engineers' time, chasing individual reports one at a time. Under a category-level eval review instead, the same team spends about 6 hours a week, and catches a real regression the ticket queue never would have.

Not a bigger bug tracker. A different kind of report entirely, built around a rate instead of a checkbox.

What I would leave alone: the simplest completions, finishing a `for` loop, naming a variable, closing a bracket, sit at 99% correct on the eval set and nobody has ever filed a ticket about one. The bug-or-noise question barely matters there. Building category monitoring for completions that are already this reliable would just be more dashboard for no reason.

The lesson: a feature that's allowed to be wrong sometimes needs a different kind of done. Build that ruler on purpose, or you find out the hard way, one closed ticket at a time.

Now here is the same thing as a story

Read the short version above when you're in the room. Read this one when you want to feel why a resample, not a bad quarter, is what actually cracked the tracker open.

Tamsyn Ezenwa can smell a bad completion before she's finished reading it. Two years on Bellwright's completion-quality team taught her that much. A stray argument order, a method that doesn't exist on that class, a bare `.sort()` where a comparator clearly matters, she catches it in the time it takes to scroll past it.

For the first few months after Slipstream shipped, her job was simple, the way she liked it. A developer would file a ticket: this completion was wrong, here's the prompt, here's what it should have said. Tamsyn read it, found the pattern, wrote a fix, and watched the count in the corner of the tracker tick down by one. Forty tickets a week in, thirty-nine closed by Friday. She was good at this the same way she'd always been good at bugs: find the exact input, find the exact line, change it, done.

It thinned out in three ordinary-looking steps. First, Bellwright's engineering org grew, and the ticket count climbed with it, forty a week toward eighty. Second, Tamsyn stopped reading every report end to end before patching it. She'd learned the shapes by then, a wrong sort here, a hallucinated method there, and she could write the narrow fix from the ticket title alone. Third, closing a ticket stopped meaning she'd checked whether the underlying problem was actually gone. It meant she'd made that one reported case stop happening, which is not the same thing, though for a long time nobody had reason to tell the two apart.

Nothing looked wrong in any single ticket. It took a resample, weeks later, to show what had actually been happening the whole time.

The trigger wasn't a new hire, and it wasn't an angry customer. It was Tamsyn's own teammate glancing at her screen on a Thursday and saying, "Isn't that the same fix as ticket 402?" It wasn't. It was close enough that she went back and pulled sixty of her own closed tickets, thirty days old, and reran the original prompt ten times each at Slipstream's normal settings, the way a developer would actually hit it.

Twenty five of the sixty failed again, at least once, in that resample.

I want to say the problem was Slipstream getting worse. It did get worse, somewhere. Eight weeks earlier, Bellwright had swapped in a newer base model, and one category, error handling around async and await, had quietly slid from 93% correct to 81% over three weeks. But that's not really the story either. The tracker never had a setting for "this rate moved." It only had a checkbox.

Three weeks after Tamsyn's resample, Zaneta Perowne walked into the quarterly roadmap review with the same slide she always brought: tickets opened, tickets closed, a number that had looked fine every single quarter since launch. Callaghan Endres, the VP of Engineering, asked the question he always asked: "How many Slipstream bugs are still open? When do we call code completion done?"

Zaneta didn't have a real answer, not in the tracker's own language. Slipstream was never going to hit zero open bugs, not because the team was behind, but because a completion that's right 91% of the time is still going to be wrong on the other 9%, forever, by design. Asking when it would be "done" was asking a question that didn't fit the thing being measured.

This is what Zaneta and Tamsyn built instead of a bigger tracker. A real check, before anything gets called a bug.

The decision Zaneta would take back traced to the meeting, two years earlier, where Bellwright set up its one shared issue tracker for every team. Someone asked whether AI features needed their own workflow. The honest answer at the time was that nobody had built one yet, so Slipstream got folded into the same open-or-fixed system as everything else. It made sense when Slipstream was new and nobody had learned its shape yet.

Run that quarterly review again, with the category-eval report instead of the ticket count. Callaghan asks the same question. This time Zaneta has a real one: Slipstream is running at 90.4% correct against an 88% floor, one category needed real attention this quarter, and the eval report caught it by day 12 of the drift instead of week 8. Tamsyn's team spends 6 hours a week watching for the next one instead of 30 hours patching noise.

One design let "fixed" mean whatever the last patch happened to touch. The other lets it mean the rate actually moved.

What Tamsyn would tell herself, back when she was closing forty tickets a week and feeling good about it: the question was never how fast can you close it. It was always what does "closed" even mean for something you never expect to reach zero, and for two years nobody had written down an answer.

PICK: the rule that decides which wrong answer gets a patch

Not a way to dress up "the model is sometimes wrong" so it sounds acceptable. PICK forces a real measurement to replace the bug count, then makes you say, out loud, which kind of mistake actually costs more.

Same size on a ticket dashboard. Nowhere near the same size in what they actually cost Bellwright.

And if you want to be sure it really works, try it somewhere else

Same four letters, a field of corn instead of a codebase, and this time the wrong output isn't a bad sort call. It's a bad irrigation call that quietly stresses a crop.

Petrichor is Greenswale Farms Cooperative's irrigation-scheduling AI. It reads soil-moisture sensors, the weather forecast, and each field's crop stage, and tells a farm manager how much water to run and when, across roughly 2,100 field blocks for the co-op's 340 member farms. Yashvi Sowden owns Petrichor's recommendation quality.

Same shape of mistake in a completely different field, literally. A single reported miss gets treated like a defect, instead of checked against the category it belongs to.

Same rank, different lever, mapped onto PICK: the position is the same, grade against an acceptable error rate on a held-out eval set, segmented by the thing that actually varies, sensor hardware batch instead of completion category. The impact lands on Yashvi, who chases one-off override requests, and on the co-op's agronomy lead, who reports a fine-looking average while a real subset of farms quietly under-waters. The cost asymmetry holds too: a single farmer's report is cheap and visible, a hardware-wide calibration drift is hidden and expensive. And the kill criteria are the same shape: does the same wrong call reproduce on other blocks sharing that sensor batch, or is this one field's own local issue, shade, drainage, a loose sensor mount.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: grade against an eval-set error rate, not a bug count, and only chase a wrong output individually when it reproduces and its category breaks its own noise band.
Cost: no budget this quarter to build full category-level eval monitoring. Ship the cheap version first: resample every "Fixed" ticket once, thirty days out, and treat a repeat failure as the real signal instead of a full nightly pipeline.
The model got better, for real: say Slipstream's aggregate score jumps from 91% to 97% overnight. The habit doesn't change. A higher bar just means the acceptable floor moves up too, not that Slipstream graduates to "bug-free."

Where people run it wrong.
They let every developer complaint become an individually-owned ticket, and burn a team's whole week chasing outputs that were never going to repeat.
They watch only the aggregate number and miss a real regression hiding inside one small category.
They quietly turn a narrow patch into a broad rule, and never check what else it silently suppressed.

How to use it live. Ask the resample question before naming anything a bug: "Does this fail again if I run it ten more times, and does its whole category look sick on the eval set, or is this the only place it shows up?" That question alone tells you whether you're looking at a real regression or normal variance.

The whole answer to this question, in one picture. A checkbox has two settings. A rate you manage never stops moving, and that's the point.

Three things worth stating directly, since this is where the real judgment sits. Bellwright's team considered forcing Slipstream to decode at temperature zero everywhere, so the same prompt always produced the same completion, reproducible the way normal software is. They rejected it, because removing sampling variance across the board also removes the variety that makes many completions good in the first place, trading a real, ongoing quality loss for the comfort of a fake bug-or-not binary. The AI-specific failure worth naming is silent distribution drift after a model version swap: Slipstream's underlying model changed, one category's behavior shifted with it, and nothing about that shift looked like a bug in any single ticket. The guardrail is the nightly, category-level eval run against a stated noise band, with automatic escalation the moment a category breaks its own floor, instead of waiting for the aggregate to notice. And the trade-off is real: narrowing Tamsyn's overbroad `.sort()` rule back down cost her about four hours to properly scope, a cost Bellwright accepted on purpose, because the overbroad version had been quietly suppressing roughly 800 legitimate sort completions a week to close one reported case.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0