How does non-determinism change your regression testing strategy?

Quillhatch sits inside Marlstrand's support inbox and drafts the reply before an agent even opens the ticket. Marcellus Rutherglen owns how its regression suite decides a build is safe to ship. Three weeks ago, engineer Emeric Carrow trimmed a slow, wordy line out of Quillhatch's prompt. The nightly test passed. Two days later, support agent Osbert Duvalier caught the wrong refund date sitting in a live draft, seconds before it would have gone out.

How to answer this, stage by stage

Nobody is grading whether you can say "we'd add more tests." They're grading whether you understand why a single pass, from a model that samples, proves almost nothing.

Let's learn

Quillhatch drafts the reply. A person reads it, fixes anything off, and hits send. Most days, that's the whole product, and it works exactly like that.

Before Quillhatch, an agent on Marlstrand's support team wrote every billing or account reply from scratch, about six minutes each: read the ticket, check the account, type the answer. A queue of forty emails ate most of a morning.

With Quillhatch, an agent opens a ticket and a full draft is already sitting there, built from the account and the question. Most mornings, reading and fixing a draft takes under two minutes instead of six.

This is what Quillhatch's suite checked every night, right up until the week this happened.

Here's the turn. Quillhatch's mistakes were never really the risk. The risk was that Marcellus's team had a test built to answer a question the model had already stopped answering the same way twice.

Every night, the suite picked one golden case, refund window on an annual plan cancelled mid-term, sampled Quillhatch's draft once, and diffed it against the saved, approved answer: no refund after the first 30 days, prorated inside them. When Quillhatch gets that fact right, it almost always writes it the same way the approved paragraph does. When it gets it wrong, it swaps in a different, equally fixed, wrong paragraph, borrowed from the monthly plan's 14-day policy instead.

Five steps, and the whole outcome turns on step three, which nobody controls and the old test never checked twice.

On a Monday, Emeric Carrow trimmed a slow, wordy clause out of Quillhatch's prompt, the one that spelled out the refund policy explicitly, to shave latency from 2.4 seconds to 1.6 and cut token cost by about 15 percent. After the trim, that one case came out right about 65 percent of the time and wrong, stating a 14-day window instead of 30, about 35 percent of the time.

The nightly suite ran that Monday night and sampled the correct branch, the 65 percent chance, so it showed a clean pass. The build merged Tuesday morning. On Wednesday, Osbert Duvalier, reviewing a live draft before sending it to a customer cancelling an annual plan, caught the 14-day figure sitting in the text and stopped it. On Thursday, Marcellus's team reran the exact same case, same code, nothing changed, twenty times: 7 of 20 came back wrong. Not a fluke in the code. A coin that had been landing wrong more than a third of the time, and the suite had only ever flipped it once.

At its worst, this costs more than one wrong email. A prompt regression this size, caught by luck once, ships again the next time nobody happens to open the right draft. And a graded suite that flags every ordinary paraphrase as broken teaches the team to stop reading its own alerts, which is its own way of going blind, just slower.

What I would leave alone: the sign-off boilerplate at the bottom of every draft, the greeting line, the "let us know if you need anything else." Those come from a fixed template with no fact in them at all. Testing them for exact wording was never buying Marlstrand anything, and it still isn't.

The lesson: a test that checks one sample against one saved string only proves the model can be right once. It never proves the model is right often enough to trust with a real customer's inbox.

Now here is the same thing as a story

Read the short version above when you're in the room. Read this one when you want to feel why a green checkmark almost wasn't enough.

The support queue at Marlstrand fills up fast right after nine, when overnight billing questions land all at once.

Osbert Duvalier has worked that queue for three years. He's the one people ask when a ticket looks routine but reads a little off, a cancellation with a strange date, a refund that doesn't match the plan. He reads carefully, out of habit more than instruction.

Quillhatch arrived a year ago, and for months it was the best part of his shift. He'd open a ticket, a full draft was already there, and nine times out of ten it needed nothing more than a name check before he sent it. He read every draft closely at first, the way he read everything. Then, slowly, the way anyone would, he started reading faster. The drafts kept being right, so there was less to catch.

Nothing about that was careless. It was what a good tool earns.

On a Monday, three weeks ago, Emeric Carrow shipped a small change on his side of the building. Quillhatch's prompt had a long clause spelling out the refund policy word for word, and it was making every draft a little slower and a little more expensive to generate. He trimmed it, trusting the model to still know the policy from the rest of the prompt around it. The nightly suite ran that night. It passed.

Not a bigger test. A different question asked twenty times instead of once.

By Wednesday, a customer named nothing in particular to Osbert, just another annual plan, cancelled seven days after paying, wrote in asking what she'd get back. Quillhatch's draft said fourteen days. Osbert didn't know that number was new. He knew it was wrong, because he'd read the actual policy a hundred times and it had never once said fourteen. He deleted the draft and wrote the real answer by hand, then flagged it to Marcellus before his coffee was cold.

Marcellus's first question wasn't "how did this get past the test." It was "how many times has this happened and gone out." Nobody could answer that, which was its own answer. On Thursday, her team reran the exact case Emeric had touched, the same code, nothing changed since Tuesday's merge, twenty separate times. Seven came back wrong. The bug hadn't been fixed and rediscovered. It had never gone anywhere. The test had simply never asked twice.

Marcellus's team tried both wrong versions before landing on the right one, and both failures taught them something the story alone wouldn't have.

The decision Marcellus traced it back to wasn't Emeric's prompt edit. It was older than that: the day the suite was built, someone chose one sample and an exact string match, because it was cheap, and because Quillhatch's earliest replies were short enough that the model barely had room to vary. That was a sensible call for the product Quillhatch was then. Nobody ever came back to ask whether it still fit the product Quillhatch had become.

Run the same Monday again, with the new suite already live. Emeric trims the same clause. That night, the suite samples the case twenty times instead of once: 13 pass, 7 fail, a 65 percent rate, well under the 95 percent bar. The build is blocked automatically before it ever reaches a customer's inbox, three days before Osbert would have had to catch it by hand, and before anyone can ever ask how many times it already went out.

The whole answer to this question, in one picture. A single sample tells you about the night. Twenty samples tell you about the model.

One design left the truth up to which branch got sampled on a Monday night. The other one asks the same question often enough that luck stops mattering.

What Marcellus would tell herself, back on the day the first suite got built: a test that only ever asks once was never really testing the model. It was testing the night.

SPARK: teaching the suite to stop trusting a lucky roll

Not a way to dress up "add more tests" in five letters. SPARK forces you to say what the old test actually assumed, then prove the new one survives being wrong in both directions.

Three things Marcellus's team is deliberately not automating yet, and why none of them are the fact that almost went out wrong.

And if you want to be sure it really works, try it somewhere else

Same five letters, a real estate follow-up assistant instead of a support inbox, and this time the fact that drifted wasn't a refund window. It was a monthly fee.

Kerbside, built by Osterholt, drafts the follow-up email an agent sends after a showing: the price, the square footage, the monthly fee, how long the listing has been on the market. Caspian Merrivale owns its regression suite. Osterholt runs 150 golden follow-ups through it nightly, 25 of them fact-critical, price and fee figures a buyer could reasonably rely on.

Different product, different fact, same shape of failure: a single nightly sample that can't tell lucky from right.

A provider-side model version bump, not anything Osterholt's own team touched, quietly changed how Kerbside handled co-op listings with a maintenance fee that's separate from the HOA fee. After the bump, about 70 percent of samples on that case still stated the fee correctly, and about 30 percent conflated the two figures, understating what a buyer would actually owe every month. The nightly suite, one sample, exact match, happened to land on the correct branch the night of the bump. It passed. Days later, an agent caught the wrong fee in a live follow-up before it reached a buyer.

Same rank, different fact: Caspian's team built the identical fix, 20 samples a night on the 25 fact-critical cases, graded against the real fee and square-footage figures rather than the exact wording, a 95 percent bar. Their own shadow trial with one shared bar across all 150 cases false-alarmed 6 nights out of 10; splitting the bar by category brought that down to almost none.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: twenty samples, grade the fact not the wording, a high bar for facts and a lower one for phrasing.
Cost: no budget this quarter to run 20 samples on every case. Start with only the handful of fields a customer could actually be harmed by, price and fee, and widen the graded set later.
The model got better, for real: say the provider's next version drops the conflation entirely. The pass rate on that case climbs to 100 percent and the bar stops flagging it, exactly as it should. A better model doesn't change the design, it just clears the bar the design was already checking.

Where people run it wrong.
They keep a single exact-match sample and just run it more often, without changing what "pass" actually checks.
They set one shared bar for every kind of case, and either miss real regressions or drown the team in noise.
They treat a human catching it once, by luck, as proof the process is working, instead of proof it isn't.

How to use it live. Ask this before naming a fix: "does this test's pass or fail actually mean the same thing every time it runs, or could the exact same code pass tonight and fail tomorrow?" That question alone usually finds the flaky spot before you have to guess at one.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0