Explain how you would run a bug triage meeting where half the bugs are model behaviour, not code.

Fellwick Outfitters rents expedition gear on a subscription: tents, packs, cold-weather kit, swapped in and out as a member's plans change. Tideglass is the model that scores every active membership's chance of cancelling in the next 30 days, so customer success knows who is worth a call. Melisande Kestrideth has run Tideglass's bug triage for about three years. This is the meeting where a new hire asks the one question the tracker was never built to answer.

How to answer this, stage by stage

Nobody is grading whether you can describe a tidy meeting agenda. They are grading whether you will notice the moment a ticket stops meaning what your tracker was built to assume it means.

Let's learn

What happens when half of what lands in triage isn't really a bug, not the way the tracker was built to understand one?

Say a subscription business rents outdoor gear: tents, packs, stoves, swapped in and out as a member's plans change. A model called Tideglass looks at every active membership once a week and scores how likely that member is to cancel in the next 30 days, so customer success knows who is worth a call. Before Tideglass's first real upgrade, virtually everything reported about it in the weekly bug meeting really was a code bug: a broken field, a job that crashed overnight, a score that came back blank. Triage closed those in about four days on average, the same script every time: reproduce it, assign it, fix it, verify it, close it.

Then Tideglass got a real upgrade, more behavioral signals folded into the weekly score, and the ticket queue didn't just grow. It changed shape. Six weeks in, open tickets had tripled, from twelve to forty-two. And roughly half of them, twenty-one, weren't code bugs at all. They were the model being confidently, systematically wrong about one kind of member: freelance-tier accounts, paying month to month, flagged as high cancel-risk about three times as often as they actually left.

The old script never broke on purpose. It just met a kind of ticket it was never built to sort.

Here's the turn. Those twenty-one tickets are not the real problem. The real problem is what Melisande does with a ticket that doesn't fit the only script the tracker has ever offered her.

At its worst, a real code bug sits one ticket away from getting waved off with the same private guess that correctly caught twenty other tickets that really were the model. A member pauses their membership over the off-season, and a mapping bug scores them as if they had already cancelled. That ticket almost gets closed "probably the model, table it," the same note used a dozen times that month, before a colleague checks the raw logs and finds an actual, ordinary, fixable bug.

What I would leave alone: a ticket with a clean stack trace and an obvious null-handling bug does not need a debate about which lane it belongs in. Keep those on the fast, four-step script exactly as they are. The split only earns its keep on the ones that don't have an obvious answer yet.

The lesson: a bug tracker built for a feature with no model in it does not grow a second lane by itself. Somebody has to decide, on purpose, that "is this code or is this the model" is a real question with two real answers, before the meeting where half the room can't tell which one they're looking at.

Now here is the same thing as a story

The short version above is what you'd actually say in the room. Read this one when you want to feel exactly what three weeks of guessing cost, ticket by ticket.

Melisande can read a bug report and tell, inside a minute, whether it's a broken pipe or a broken argument. She has run bug triage for Tideglass, Fellwick Outfitters' churn model, for three years, since before the company had a customer-success team big enough to need one.

For the first year, that skill was basically all triage needed. Tideglass launched as a simple weighted score wired to a handful of clear signals: days since a member's last rental, days until their plan renews, whether a support ticket got filed last month. Almost everything reported about it really was a code bug: a null field, a job that died overnight, a score that came back the same for every member on a Tuesday because a cron job hadn't run. Melisande's Thursday meeting ran the same four steps on every single ticket: reproduce it, assign it, fix it, verify it, close it. Average time to close, about four days. Nobody argued with the process, because the process had never once been wrong about what kind of thing it was looking at.

Then, fourteen months in, Tideglass got a real upgrade: more behavioral signals folded into the weekly score, tuned on a much bigger set of past outcomes. The upgrade shipped on a Monday. By the following Thursday, three new tickets had landed that didn't look like anything from the old script.

The first one, Melisande just closed herself, quietly, with a note: "checked, model called it right, not a bug." Nobody in the meeting even saw it happen.

The second week, four more came in shaped the same way, and she started sorting them before the meeting even started, skimming the list over coffee and deciding in her head which ones would get the real four-step treatment and which ones she'd wave through with a similar note.

Nobody decided, on any single day, to stop running the old script. It thinned out in three quiet steps, until the fourth one had nothing left to explain.

By the fourth week, that was just how Thursdays worked. She never announced it. Nobody voted on it. It was simply the fastest way to keep the meeting moving once a third of what showed up did not fit the only script the tracker had ever offered anyone.

Then, in the sixth week, Tybalt Vancewood, three weeks into the job, sat in his second triage meeting and asked the obvious question nobody else had asked out loud: "Wait, why are we asking for a repro step on this one? The model gave exactly the score the code told it to compute. What are we actually testing?"

Nobody had a clean answer. Melisande didn't either, not really, not one she could say in one sentence.

By then the queue told its own story. Open Tideglass tickets had tripled since the upgrade, from twelve to forty-two, and by Melisande's own rough count, about half of them, twenty-one, weren't code bugs at all. They were Tideglass being confidently wrong about one kind of member: freelance-tier accounts, paying month to month, flagged as high cancel-risk about three times as often as they actually left.

The same pattern showed up three separate times that month, filed by three different customer-success reps who had no way of knowing they had already reported it. Melisande's private sorting handled all three differently. The first, she closed as "model behavior, not a bug." The second, she assigned to an engineer, who spent four days combing the scoring code for a defect that did not exist before giving up. The third sat marked "needs more info" for five weeks, because by then she had stopped being sure which bucket it belonged in.

And it very nearly cost more than four days. Buried in that same pile was a ticket about members who had paused their subscription over the off-season, gear rentals dip hard in winter, getting called by a worried customer-success rep who had seen their risk score spike. Melisande almost closed it with the same note she had used a dozen times that month: "probably the model, table it." A colleague checked the raw event logs first. A pause event was firing a webhook that Tideglass's own intake code was quietly mapping to the same field as a cancellation. Every paused member was being scored as if they had already said they were leaving. That was a real, ordinary, findable code bug, sitting one ticket away from getting waved off with the same gut call that had correctly caught twenty other tickets that really were the model.

Here's the decision I'd take back, and it isn't Melisande's private sorting, and it isn't "hire someone to keep the queue smaller." It goes back to the week Tideglass first launched, when the team set up the bug tracker with exactly one flow: report it, reproduce it, link a code diff, mark it fixed, verify, close. That was the entire quality bar the tracker knew how to hold. It was a completely reasonable choice at the time, because for that whole first year, everything reported about Tideglass really was a code defect. Nobody built a second lane, because there was nothing yet to put in it.

We considered the obvious fix first: just tell customer-success reps to flag anything score-related as "model issue" before it ever hits the tracker, and route those straight to Melisande without a formal ticket at all. Rejected. That just moves the private judgment call one step earlier and hands it to people with even less view into what Tideglass is actually doing, and it still leaves no record anywhere of what pattern got reported, by whom, or how often.

Run the same kind of upgrade again, five months later, with the tracker's intake redesigned in between. New question, first thing, on every single ticket: did the code do the wrong thing, or did the model decide the wrong thing. Code answers keep the old four-step script exactly as it was. Model answers go into their own lane: log the specific pattern, check it against a labeled eval set of past outcomes for that segment, and hold it against a target rate, freelance-tier false high-risk flags under 8 percent of that segment, checked weekly, not a box that says fixed or doesn't.

Tideglass gets its next real upgrade, a new signal drawn from support-ticket sentiment this time. Reports come in fast again. But the freelance-tier pattern gets tagged into the eval lane the same day it is reported the third time, merged into one tracked case instead of three separate guesses. Six weeks later, open tickets sit at nineteen, not forty-two. And this time, a real code bug, a currency field mismapped for annual-tier renewal reminders, gets caught inside two days, because it lands with a stack trace attached and never has to survive anyone's gut call about which lane it belongs in.

That's the whole difference. One design hands Melisande a script. The other hands her a switch she has to throw herself, silently, forty-two times a month.

What I'd tell my past self, the one who set up that tracker back when Tideglass was simple enough that every ticket really was a code bug: a bug tracker doesn't grow a second lane on its own just because the thing behind it got smarter. Somebody has to decide, on purpose, that a wrong answer and a wrong number are two different problems, before the week a new hire asks why nobody in the room can tell them apart.

FLIPS, or the question Tybalt actually asked

Not a trick to sound structured. FLIPS is what makes you notice that a new hire's plain question was the whole audit, three weeks before anyone ran one on purpose.

Four setup and payoff letters, and one hard question sitting in the middle of all of them.

The whole S step, drawn as the one question that now runs before anyone picks an owner.

The whole answer, in one picture. Nobody designed a dial. Everybody got a switch, and for a year nobody had to throw it themselves.

Same kind of forced model upgrade. The only thing that changed is what stood between the ticket and Melisande's private guess.

The AI-specific failure worth naming plainly is silent segment-level miscalibration: no crash, no error, no code diff a normal review would flag, just a model that is confidently wrong about one kind of member and no code defect anywhere in sight. The guardrail is a segment-scored eval lane that checks for that pattern on purpose, not a person's private feel for which tickets "smell like" the model. There's a real trade-off, accepted on purpose: a model-behavior ticket no longer gets an instant fixed-or-not answer. It gets checked against the eval set on a schedule, which is slower for any one ticket and costs someone real time keeping that labeled set current, in exchange for actually knowing whether the pattern improved instead of a false "fixed" checkbox nobody ever verifies against a real outcome.

And if you want to be sure it really works, try it somewhere else

Same five letters, a completely different flip. This time nobody's private guess is even wrong on purpose, it's just working from input that's already been quietly cleaned.

Crestwick Software runs support across nine languages through Bridgetongue, a tool that translates an incoming ticket into English before an agent ever reads it. Ottalie Merricks has owned Bridgetongue's bug triage for about eighteen months, the same weekly meeting, the same tracker, the same one script.

Same five questions, a completely different way the flip hides. This time the mismatch never even reaches the meeting.

F · Ottalie Merricks, who has owned Bridgetongue's bug triage at Crestwick Software for about eighteen months.
L · She stopped requiring agents to submit tickets exactly as customers wrote them, typos, mixed languages and all, once a few badly mangled translations had already gone out and become embarrassing.
I · The pre-editing flip, running in a different shape from Melisande's. Old setting: agents feed Bridgetongue the raw ticket, however messy. New setting: agents quietly clean up anything that looks hard first, stripping slang, code-switched phrases, typos, before it ever hits translate, so the tool almost never sees the input it was actually bought to handle. Nothing in between: either the ticket goes in as written, or somebody has already smoothed the one part that made it hard.
P · Bridgetongue's tickets could only close against a linked code change, the same as everything else in the tracker, so a translation-quality complaint had nowhere to go except a code-bug ticket that usually got closed "cannot reproduce, translated correctly on the cleaned text we tested."
S · The redesigned intake splits every ticket: did the code call Bridgetongue correctly, or did Bridgetongue's own output miss something. Translation misses get logged against an eval set built specifically from raw, uncleaned tickets, and Bridgetongue's real weak spot, tickets that mix two languages in the same sentence, finally shows up in the numbers instead of being filtered out by agents' own tidy habit before anyone could measure it.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: a tracker that can only close against a code diff will always end up hiding a churn score, a fraud flag, or a translation, behind "cannot reproduce."
Cost: no budget for a second eval set. Reusing yesterday's model-behavior tickets as tomorrow's eval cases costs nothing extra, it's just deciding to keep them somewhere instead of closing and forgetting them.
The model got better, for real: say Bridgetongue's own benchmark score jumps on the next release. Doesn't matter. A model that's mostly excellent is exactly the one nobody thinks to keep checking on its worst ten percent.

Where people run it wrong.
They treat a clean eval score as proof the split isn't needed yet, without asking whether the eval set still contains the hard cases at all.
They let "the last few tickets were fine" become a reason to stop watching, instead of a reason to check the same way one more time.
They wait for a customer complaint before treating silence as a signal, when the whole point of this flip is that agents' own tidy habit means the complaint may never come.

How to use it live. When an interviewer asks how you'd run a triage meeting with a model behind half the tickets, ask yourself one thing before answering: if the exact same wrong pattern got reported by three different people this week, would your tracker even notice it was the same pattern. If the honest answer is "only if someone happens to remember," that's the whole question, answered.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0