What does a healthy retry rate look like and what does an unhealthy one indicate?

How to answer this, stage by stage

Nobody is grading whether you can say "it depends." They are grading whether you can take one ambiguous number apart, in front of them, and come out the other side with a number that actually means something. Seven moves get you there.

Let's learn

Every evening, before he had ever heard of Castloom, Onur used to sit with headphones on and type up forty five minutes of interview tape by hand, word for word. It took him close to an hour, most nights.

Castloom is a tool a reporter drops an interview recording into. It hands back a clean transcript, the false starts and filler words stripped out, and three pull quotes ready to paste into a story. With it, that same forty five minute interview took Onur about four minutes to review.

For five months, Castloom's overall retry rate, the share of sessions where someone pressed Run it again, sat close to nine percent. Nobody thought much about it. Then, in week six, the team shipped a faster model under the hood, twenty seconds to process a recording instead of ninety. By week ten, retries had climbed to twenty four percent.

Here is the turn. That climbing line is not one story. Split it by whether the person changed anything before pressing the button again, and it becomes two lines. The exploratory slice, someone tightening a pull quote's tone or naming a speaker, held flat near eight or nine percent the entire time. The correction slice, the exact same recording resubmitted with nothing changed, climbed from one percent to fifteen.

The fast model was genuinely better on clean studio audio, one speaker, quiet room, its retry rate there actually fell, from seven percent down to five. On phone calls and cross talk, two people talking over each other, it got much worse, correction retries there ran up to forty one percent. Blended into one company wide average, the win on clean audio quietly covered for the loss on messy audio.

All three were true at once. Only the first one was the actual cause of the climb. The other two are why nobody caught it sooner.

At its worst, this cost more than a slightly busier support inbox. In week nine, on a phone interview with a city contractor, Castloom's pull quote attributed a specific dollar figure to him, forty thousand dollars, worded clean and confident. The contractor never actually said that number. It came within one editing pass of running in print under his real name.

The choice I would take back is not the fast model itself. It's that when Run it again first shipped, the team logged one number, retries per session, and decided not to tag whether the resubmitted input had actually changed. That felt like extra plumbing for a debugging case nobody was worried about yet. It stopped being a debugging case the day a faster model started guessing on the audio it had always struggled with most.

What I would leave alone: the tone only version of the button, the one that only reruns the pull quote wording and never touches the transcript itself. Nothing factual can go wrong there. Gating it would just slow down people doing the job right.

The lesson: a number that is climbing is not evidence of anything by itself. It is a question you haven't split into two questions yet.

Now here is the same thing as a story

The short version sits above. Read on for the ordinary Tuesday this almost went out under a real person's name.

The Run it again button sits under every transcript Castloom hands back. One word: rerun. It has been there since the first version of the product, and almost nobody who built it thought twice about what it actually meant when someone pressed it.

Onur has covered city government for six years, mostly phone interviews he does from wherever he happens to be, a courthouse hallway, his car, more often than not the back booth of a diner near the paper's office, working the tape into a story after everyone else has gone home. He is good at hearing what a source actually means underneath what they actually say. He does not miss much.

For most of a year, Castloom was the best part of his night. He'd finish a call at nine, drop the recording in, and by nine-twenty he'd have a clean transcript and three pull quotes sitting in front of him, work that used to eat his whole evening. He'd skim it, maybe swap one quote for a sharper one, and be home by ten.

He stopped double checking pull quotes against the raw tape somewhere around month four. They kept being right. He had a paper to get out.

Ren runs product for Castloom. Ren is the kind of person who reads a metrics review line by line and actually stops on the number that moved, not just the ones that moved the most. The morning retries crossed twenty percent, the instinct in the room was almost celebratory: people were getting more out of every recording, using the tool harder, look at that engagement. Someone had a draft of an internal update half written, framing it exactly that way.

What stopped that update from going out was a habit, not a hunch: before calling any number good, split it by what actually changed underneath it. Ren pulled the retry logs and, for the ones that had a settings change attached, tagged them exploratory. For the rest, the ones where the resubmitted audio and settings were identical to the first attempt, tagged them correction. The exploratory line had barely moved all quarter. The correction line had gone from nearly nothing to fifteen percent of all sessions.

That same morning, a note from the newsroom's audio lead landed in the same inbox. Onur had flagged a near miss. Nothing dramatic, no alarm went off. He'd been reviewing a pull quote from the city contractor interview, forty thousand dollars, worded a little too smoothly for a guy who'd been talking fast and cutting himself off all call. He pressed Run it again out of habit more than suspicion. It came back different. So he did something he hadn't done in months: he put the raw audio on and listened to the whole exchange himself. The contractor never said a dollar figure at all.

Onur caught it with an hour to spare before file time. From that night on, he stopped trusting any pull quote Castloom handed him without listening to the raw clip first, on every interview, not just the shaky ones. The four minutes Castloom used to save him quietly went back to something closer to forty.

We did not just lose an hour of Onur's night. We lost the reason the product existed in the first place.

The old decision, told plainly: back when Run it again first shipped, in a sprint focused on getting the button out fast, someone raised whether to log "resubmitted with no change" as its own event, separate from a normal retry. It got cut. Extra plumbing, for a debugging case that felt theoretical at the time. Nobody in that meeting was picturing a faster model that would one day need exactly that tag to be caught.

What Ren actually did: shipped the confidence gate two weeks later. Below about sixty five percent confidence on a segment, Castloom now shows the raw words with a flag, "listen to this part yourself," instead of a smoothed out quote. Three weeks after it shipped, the correction retry rate on phone and cross talk audio, the segment that had spiked to forty one percent, was back down near twelve. Onur still double checks anything the flag touches. Everything else, he trusts again, and Castloom is back to costing him four minutes a night instead of forty.

What I would tell myself, before any of this: a number that only tracks how often people press a button was never going to tell you why they pressed it. You have to go build the reason into the number on purpose.

TRACE, or how to stop reading one number two ways

This is a diagnosis question, a number moved and you have to work out what it actually means, so TRACE fits. Not a habit with two settings, and not a ranked list of what to build first.

Two things worth naming directly, since this is where the AI specific judgment actually lives. The alternative most people reach for first is capping or rate limiting retries, forcing people to accept whatever the first pass gave them. That got rejected on purpose: it punishes the reporters doing legitimate refinement, and it hides the real defect rather than fixing it, since a journalist who gives up instead of retrying a third time doesn't show up as a retry at all, they show up as silence. The failure mode worth naming by name is confident wrongness, a model producing a fluent, clean sounding answer on a stretch of audio it was actually unsure about, with nothing in the output signaling that uncertainty. The guardrail is the per segment confidence score the pipeline already produced, gating whether a pull quote gets auto generated at all. The bar is not a fixed rule, it's a threshold checked against a rolling set of manually verified cross talk clips, recalibrated roughly every two weeks, since a fixed number drifts out of true the moment the diarization model underneath it changes. And the trade is real: the confidence aware second pass adds a few seconds and more inference cost on flagged clips. That is a cheap price next to a fabricated quote reaching print under a real name.

And if you want to be sure it really works, try it somewhere else

Same five letters, an online marketplace instead of a newsroom, so the method proves itself instead of repeating a story I happened to prepare.

Shelfnote writes product listings, a title, feature bullets, a description, from a seller's photo and spec sheet. Iker Vask runs a small home goods shop through it. The button there is called Rewrite this listing.

T, timeline. Regenerate rate held near six percent for months, then climbed to nineteen percent over five weeks after a model swap aimed at punchier listing copy.
R, recut. Exploratory retries, a seller tweaking keywords or tone, stayed near six percent. Correction retries, the same photo and spec sheet resubmitted unchanged, climbed from under one percent to twelve.
A, assume nothing. The first read was that sellers loved the punchier copy. Ruled out, because the exploratory slice never moved.
C, cause candidates. The model stating a material or a dimension that never appeared on the spec sheet, worded like a fact. The upload form having no way for a seller to say "no measurements given," so the model guesses instead of saying so. A spec rich category, electronics, genuinely improving and falling, hiding a spec poor category, handmade goods, where the model had to invent detail to sound finished.
E, evidence test. Diff the new listing text against the source spec sheet on identical input retries. A dimension appearing in the copy that exists nowhere in the source is the fingerprint of a confident guess, not a rewrite.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to the split: don't say whether a rising retry rate is good or bad until you know whether the input changed before the retry.
Cost: engineering says the confidence pipeline can't ship for six weeks, not two. Don't leave the plain retry button as the whole plan in the meantime. At minimum, show a visible flag, "this one ran on a tricky recording," until the real fix lands.
The model got better, for real: say the next diarization model gets meaningfully better at overlapping speech. Retry rate should fall. If it doesn't fall specifically on the segment that used to be hardest, that is itself worth chasing. A better model on paper does not always mean better on the exact slice your users actually depend on it for.

Where people run it wrong.
They treat a climbing retry number as automatically bad, or automatically good, without ever splitting it by whether the input changed.
They fix the blended average and call it done, missing that one segment is still on fire underneath a healthy looking overall line.
They put the fix entirely in a person's judgment, "just double check everything," instead of a number the product surfaces at the exact moment the risk shows up.

How to use it live. Say the split before you say anything about whether the number is good: "Before I call a rising retry rate healthy or not, I need to know if people are changing something before they retry, or resubmitting the exact same thing hoping for a different answer, because those are opposite stories wearing the same number." That buys you room to give the real diagnosis instead of reciting "watch your metrics closely" on reflex.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0