Examine an AI feature that failed publicly and identify the product decision behind it.

GleamSort scans historical photographs, generates a descriptive tag and caption for each one, and publishes it straight to a public digital catalog. Selassie Bekele is a digital archivist at a city library system and was on shift the week one batch of captions went badly wrong, publicly, on the library's own site.

How to answer this, stage by stage

Six moves. The interviewer doesn't want a play-by-play of the bad tweet. They want the decision underneath it.

Let's learn

Picture a library's entire photo archive going live to the public with one caption nobody, human or otherwise, ever looked at first.

GleamSort scans a library's historical photographs and generates a tag and short caption for each one, describing what's in the photo so the public catalog is searchable.

For most of the archive, modern, well-photographed, richly represented collections, GleamSort's captions are genuinely good, and the public catalog grew fast without anyone flagging a problem.

Then, in week seven, a batch of Reconstruction-era photographs went live with a caption misidentifying the people in one image, a mistake serious enough that a local news outlet picked it up within days.

What I would leave alone: modern, well-photographed collections don't need the same scrutiny. Routing every single photo through manual review, including the 97-percent-accurate modern ones, would slow the whole pipeline down for a risk that mostly isn't there.

The lesson: a public failure usually isn't a new mistake. It's an old, sensible-sounding decision, made under real pressure, meeting the one case it was never actually tested against.

Now here is the same thing as a story

The short version above is what you'd say defending this diagnosis in a postmortem. Read this one for how the pattern actually got found.

Selassie's library had a backlog of forty thousand unscanned photographs sitting in boxes when GleamSort first rolled out, tagging each one and dropping it into a review queue before anything went public.

The actual decision sits at milestone two. Milestone four is just where the rest of the world finally noticed it.

Three months in, with the backlog still enormous and the review queue the clear bottleneck, GleamSort's team merged tagging and publishing into one automated step. The backlog cleared in weeks instead of years, and it looked, on every dashboard anyone checked, like an unqualified win.

One pipeline has a pause built into it. The other one doesn't, and nothing on a dashboard shows you that difference.

Seven weeks after the merge, a set of Reconstruction-era portraits went live. One caption misidentified the people in a well-known historical photograph, a mistake that would have been obvious to almost any human reviewer on sight.

Four steps, and there used to be a fifth one, a review step, sitting right between the second and third box.

A local reporter found it within three days, and it ran as a story about the library "letting AI write history," which is a much bigger headline than "a batch job skipped a checkpoint."

The dangerous corner is top left: thin training data and a high cost if wrong. That's exactly where the viral photo came from.

Three explanations were on the table once the library actually looked into it: a single rogue mistake in an otherwise fine system, a genuine model weakness on a specific kind of photo, or a missing safety net that used to catch exactly this.

It wasn't one of these three. It was the second and third, together, which is the far more common shape a real failure actually takes.

The evidence test confirmed it: nearly every photo published in that viral batch belonged to the thin-data segment, and its real accuracy, 58 percent, had been sitting there the entire time, hidden inside a 90-plus percent overall number nobody had ever bothered to slice.

Someone at GleamSort merged tagging and publishing to clear a genuinely enormous backlog, under real pressure from a library board asking why digitization was taking years. That was a sensible call, made honestly, under real constraints. I would take it back the moment the backlog pressure eased, and route the thin-data segment specifically back through review, instead of removing review for everyone.

We merged the steps because the backlog was the visible, urgent problem, and the review queue was the thing standing between us and clearing it. It took a viral news story, and a public apology from the library board, to see that the queue wasn't just slowing us down. It was the only thing keeping our weakest segment out of public view.

TRACE, run on one viral captionNot a story about a bad caption. TRACE exists to find the decision sitting underneath one.

The recap, one line per letter: timeline is the seven-week gap between the merge and the viral photo; recut is the 58-percent segment hiding inside a 90-plus-percent average; assume nothing is ruling out a one-off glitch by checking the whole batch; cause candidates are rogue mistake, model weakness, and missing review; evidence test is the batch-level segment check.

And if you want to be sure it really works, try it somewhere elseSame five letters, a sports highlight feed instead of a library archive. This time the public failure is a wrong name attached to an injury clip.

Rallytag is an AI feature for a regional sports network that auto-tags player names in highlight clips and publishes them straight to a public social feed, no human review between the two steps.

Mapped onto TRACE: the timeline shows the auto-publish merge shipped a full season before the failure, with aggregate player-ID accuracy sitting near 94 percent the entire time. Recut shows the real weak spot: reserve and recently-traded players, who appear far less often in the training footage than starters, sit closer to 61 percent accuracy. Assume nothing rules out a one-off glitch by checking whether the error repeated across multiple clips of the same player, and it did. Cause candidates are a rogue mistake, a model weak spot on less-seen players, or the missing review step, and the evidence test, checking every clip from the same broadcast against player roster tenure, confirms it's the same combination as GleamSort: a real model weakness that used to get caught by a step that no longer exists.

A sports feed instead of a library archive, and the same missing piece: nothing sits between generating a label and putting it in front of the public.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "the caption wasn't the failure, removing the review step that used to catch this exact weak spot was" and stop.
Cost: no budget to rebuild full manual review this quarter. Say so, and start by routing only the identified weak segment back through review, which is far cheaper than reviewing everything.
The model gets better, for real: even if overall tagging accuracy improves to 96 percent, the thin-data segment can still sit at 60 percent underneath it. A better average doesn't fix an uneven one.

Where people run it wrong.
They treat the viral mistake as the problem to fix, patching that one caption instead of the pipeline decision behind it.
They trust an aggregate accuracy number without ever asking what it's averaging over.
They assume a public failure means the model is broken everywhere, when it's usually broken badly in one specific, findable place.

How to use it live. When asked to examine a public AI failure, ask yourself one question before anything else: what changed in the pipeline, not the model, before this became possible? That question almost always points at the real decision, not the caption everyone's talking about.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0