What do you do when engineering and research disagree about the cause of a quality regression?

Marlspire writes ad headlines, descriptions, and hero images for small brands, through a model called Glaze. Corisande Kastelic owns whether those drafts are good enough to ship untouched. This is the week her approval rate cratered, her data engineer named one cause, her researcher named a different one, and both of them turned out to be right about something and wrong about how much of it.

How to answer this, stage by stage

Nobody is grading whether you know what a checkpoint is. They're grading whether you'll let a three-week fix get built on whichever explanation sounded more confident in the room.

Let's learn

Marlspire takes what a small brand is running, a product name, a price, a few photos, a short brief, and writes the ad: three headline options, two lines of description, and a hero image variant, ready to load into a campaign. A model called Glaze does the actual writing and image work.

For four months running, marketers shipped about seventy one of every hundred Glaze drafts with no real rewrite, just a click to launch. A boot brand's ad might read "Terracotta suede, cognac stitching, built for wet leaves." That number held steady enough, week after week, that Corisande stopped checking the dashboard daily and moved to once a week.

Three weeks ago it dropped. Fifty eight one week, fifty two the next. The same boots started coming back "Great boots for the season. Shop now." Marketers filed the same kind of complaint, over and over: the ad still says the right things about the product, and it still reads like it could be selling anything.

Here is the turn. Corisande's team could have just picked a fix and moved on. The real problem showed up two hours into the meeting about it, when her data engineer and her researcher each had a clean, confident, opposite explanation, and neither one had anything visibly wrong with it.

Same boots, same brief, same model. The only thing that changed between these two drafts was the week they got written in.

What it costs at its worst: Marlspire spends a sprint rebuilding the wrong half of the system, ships a fix that barely moves the number, and by the time anyone checks the actual data, three more weeks of every-other-ad going out sounding like nobody in particular have already reached real customers spending real budgets.

What I would leave alone: Marlspire's legal-disclaimer checker, the small tool that just confirms the required "results may vary" line is present in every ad, doesn't care which model wrote the copy or which checkpoint served it. It's checking for a fixed string, not for how the writing sounds, so it never needed touching.

The lesson: a schema bug that drops real data and a model that quietly changed behavior can produce the exact same complaint, from the exact same marketers, in the exact same week. My job isn't to trust whichever explanation gets delivered with more confidence. It's to make both sides say something specific enough to be wrong about, then go check.

Now here is the same thing as a story

Say the short version out loud in an interview. Read this one when you want to feel exactly how two honest, well-built cases can point in opposite directions and both be partly true.

Every Friday at four, Corisande pulls the week's ad-approval number before anyone else gets to it, because bad news lands softer on a Friday than a Monday.

For four months the number barely moved. Seventy, seventy two, sixty nine, seventy one. Marlspire's marketers, mostly small brand owners running their own campaigns, kept shipping Glaze's drafts almost as fast as it wrote them. Glaze would pull the product name, the price, and a few real details from the listing, color, material, what makes it different, and hand back a headline, a description, and a hero image variant. Most weeks, about seven in ten went out exactly as written.

Three weeks ago the pipeline changed. Baldemar Cruz, who'd built and rebuilt Marlspire's catalog feed for eight years, shipped a schema migration meant to be a quiet cleanup, three scattered attribute tables folded into one clean table Glaze's prompt-builder could read from directly. He tested it against a sample of listings before shipping. It looked fine.

It wasn't. A mapping bug in the new schema meant that for about four in ten catalog items, mostly apparel and home goods, the exact categories where color and material carry the ad, those fields came back empty when the prompt-builder went looking for them. Nothing crashed. Nothing errored. The prompt just quietly asked Glaze to write about a product with no color, no material, and no size, the way it always had for products that genuinely didn't have those fields.

A week after that, something else changed, and nobody on Marlspire's side did it. The vendor whose model sits underneath Glaze rotated the version behind an alias called "creative-default," the one Marlspire's code had always called instead of naming an exact version. It happens every few months, usually makes things a little better, and usually nobody notices. This time nobody noticed either. There was no changelog entry on Marlspire's side to notice, because nothing had ever been set up to record it.

Only one of these two changes showed up in a pull request. The metric didn't crater the week of that one, it cratered the week after the one nobody tracked.

Corisande's Friday number held at seventy the week the schema shipped. It read fifty eight the week after. Fifty two the week after that. By the third week, the same complaint was showing up from marketers who'd never had a bad word to say about Glaze: the ad still says the right things, and it still reads like nobody in particular wrote it.

Corisande called a meeting Tuesday morning. Baldemar walked in with the schema diff already pulled up. "I can show you exactly which items lost which fields," he said. "Four in ten of them, right in the categories getting the worst complaints. This is the migration." Fjola Dumitrescu, who'd been quietly tracking how varied Glaze's output language was, for an unrelated project on repetition, had her own chart ready. "Output diversity cratered exactly the week the checkpoint rotated," she said. "Not the week of the migration. A week later. This is the model."

Both charts were real. Both people had built their case honestly, from data they'd actually looked at. Corisande's first instinct was to trust Baldemar, eight years on this exact pipeline against Fjola's two. She caught that instinct before she acted on it. Her second instinct, patch the schema and quietly retrain a little too, just to cover both bases, she caught that one as well. Patching both without testing which one actually mattered meant spending a sprint of engineering time and still not knowing, next time this happened, which explanation to trust first.

So instead she asked both of them the same question. "What would you expect to see, specifically, if you're right, that the other explanation wouldn't predict?"

Baldemar answered first. "If it's the schema, items that still have full attribute data should look close to normal. The drop should live entirely in the four in ten that lost fields." Fjola answered next. "If it's the model, it shouldn't matter whether the item has full data or not. Every item should be down by roughly the same amount, because the thing that changed sits inside the model, not the input."

Marlspire already had exactly what the test needed: two hundred frozen prompts in its monthly regression set, built years earlier from items with complete attribute data, never touched by anyone's schema. Corisande ran those two hundred prompts, unchanged, through the model serving live traffic that same afternoon.

They scored fifty five. Down from seventy one. Prompts that had never lost a single field failed almost as badly as the rest of the catalog.

Then she recut live traffic itself by attribute completeness. Items with full data: fifty eight. Items with dropped data: forty three.

Neither theory was the whole story. Fjola's explained most of it, the checkpoint change alone was enough to drag even untouched items down thirteen points. But Baldemar's explained a second, real, additional drop, fifteen more points, sitting on top of that, exactly where the missing fields were. Nobody had called that mix going in. Both of them had been arguing for one clean cause, because that's what a confident diagnosis usually sounds like.

The decision Corisande would take back sits further back than either Tuesday. It's the day, over a year earlier, when someone wired Marlspire's code to call the model by the alias "creative-default" instead of pinning an exact version, because pinning felt like extra work for no visible benefit at the time. That decision made sense when the model barely changed and nobody was watching closely. It stopped making sense the day a vendor's routine, invisible version bump could sit one week away from a real engineering change and both look, from a marketer's dashboard, exactly the same.

What I'd tell myself, in that Tuesday meeting: the fastest way through two confident, opposite stories was never to pick one. It was to ask each of them what a wrong answer would have looked like, and then go find out.

TRACE, so two confident explanations don't get averaged into a guess

Not a way to prove Baldemar or Fjola wrong. TRACE is what stops a real, partial answer on each side from turning into either a coin flip or a lazy fifty-fifty patch.

The whole method in one picture. Before trusting either explanation as the full story, ask which half of the pipeline that person can even see.

Two claims that cannot both be true in the same way. That's what makes them worth testing instead of arguing about.

No new experiment. Just an old, untouched batch of prompts run back through today's system, and a number that already had a baseline to compare against.

Three things worth saying plainly, since interviewers push here. Corisande considered a third option before landing on the test: just believe whoever presented more confidently in the room, since a decision had to happen fast either way. She rejected it, because Tuesday's meeting proved confidence tracks how well someone can read their own data, not how much of the total problem their data actually explains. The AI-specific failure worth naming by name is silent checkpoint drift: a model reachable through an alias instead of a pinned version can change behavior on a date nobody chose to record, and it will look, on a dashboard, exactly like nothing happened that week. The guardrail: pin an exact version, and route any change to it, vendor-initiated or not, through the same eval suite a code change would have to clear. And the trade-off, accepted on purpose: pinning means Marlspire stops getting the vendor's improvements for free the moment they ship, someone now has to test and approve every version bump by hand before it goes live. Corisande took that trade, because the alternative was finding out about the next change from a Friday number instead of a changelog.

And if you want to be sure it really works, try it somewhere else

Same five letters, a veterinary aftercare tool instead of an ad generator, and this time the discriminating test is a stack of old paper templates nobody had thrown away.

Brindlecare drafts the aftercare instructions a pet goes home with after a clinic visit, medication doses, wound care, when to call back, built from the vet's own visit notes and reviewed by a vet tech before anything leaves the building. Tanwen Anagnos runs product there, and hit a version of Corisande's exact Tuesday four months into a push to cut how often techs had to substantially rewrite a draft before it could go home with an owner.

Different product, same shape of pipeline: notes go in one end, a person checks the output before anything reaches a pet owner.

Brindlecare's rewrite rate, how often a tech changed more than a phrase, had sat near eighteen percent for months. It jumped to forty one percent over ten days. The clinic's data engineer pointed at a recent change to how visit-note audio got transcribed into structured fields, dosage and frequency sometimes landing in the wrong field after a vendor update to the speech-to-text step. Brindlecare's own ML lead pointed at the drafting model itself, quietly swapped to a cheaper, faster version two weeks earlier to cut response time, now writing vaguer aftercare language across the board.

Mapped onto TRACE: the timeline showed the transcription vendor's update landing five days before the model swap, both inside the same two-week window the rewrite rate started climbing in. The recut split cleanly along what each side could actually see, the engineer watched the structured fields, the ML lead watched the drafted language. The two predictions: the engineer's said only visits with a garbled dosage field would need heavy rewrites; the ML lead's said every visit would read vaguer, dosage field intact or not. The assumption Tanwen refused to make: that the transcription bug's five-day head start made it the obvious first suspect, or that the safe move was patching both without checking which one explained the forty one percent.

Old paper aftercare templates from before Brindlecare existed, still scanned and filed for every visit type as a fallback reference, gave Tanwen a frozen, unaffected comparison. She ran this month's visit notes against last year's already-approved template language and checked which parts still matched. Dosage language matched the old templates almost exactly, wrong field or not, ruling out the transcription bug as a widespread cause. General care language didn't match at all, on any visit, transcription problem or none. The cheaper model was the larger cause; the transcription bug added a smaller, real problem on top, concentrated on exactly the visits where a field actually got scrambled.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: make each explanation say something the other wouldn't predict, then run the cheapest test that tells them apart.
Cost: no time to trace a real incident. Ask one question instead: what would each side expect to see if they're right, that the other side's story wouldn't produce?
The model got better, for real: say the vendor's new checkpoint genuinely improved something else, latency, cost, whatever else it touches. A real improvement in one place doesn't rule it out as the cause of a regression somewhere else, someone still has to check both.

Where people run it wrong.
They treat "engineering versus research" as a turf fight to referee, instead of two data sources that each cover half the picture.
They average the two theories into a patch-both plan without ever finding out how much of the drop either one actually explains.
They stop looking the moment one test result confirms a favorite theory, instead of checking whether it explains the whole gap or just part of it.

How to use it live. When an interviewer throws two competing explanations at you cold, buy two seconds by asking one thing back: "what would each side expect to see if they're right, that the other wouldn't?" It's a short question, and it's usually the one the interviewer was hoping you'd ask.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0