Explain the risk of a feedback loop where the model's outputs become its own training data.

Here is what happens when a mistake gets a passing grade once, and then keeps getting graded off that same passing grade forever. Solstice Translation Co. runs a machine-assisted pipeline for translating technical equipment manuals. Ines Vartan, a senior localization engineer, is the one who has to explain, to a new hire who noticed something odd, why a slightly wrong phrase had been "approved" by nine different people.

How to answer this, stage by stage

Nobody is scoring whether you've heard the phrase "model collapse." They're scoring whether you can actually diagnose one small case of it, out loud, with a real test.

Let's learn

The five letters, held up as one page. Evidence test is the step this question is really testing.

Solstice Translation Co.'s tool suggests a machine translation for each line of a technical manual, and once a person marks a suggestion approved, it's stored as a trusted match the system will offer again for similar lines. A year ago, translators hand-corrected about 38 percent of the tool's suggestions. After a real upgrade to the translation engine, that correction rate is down to 6 percent now.

The third step is where a real correction and a rubber stamp produce the exact same word on the record.

Here's the turn: the extra trust the improved engine earned was not the problem by itself. Say plainly what happened next: segments approved with barely a glance started carrying small drifted phrasings a real read would have caught, and they got stored as "approved" all the same, indistinguishable from a segment someone genuinely corrected.

What I would leave alone: the custom, one-off sections of each manual rarely reuse an old translation-memory match, so there was never a loop there to begin with, and treating them with the same suspicion would just slow down work that was never at risk.

The lesson: being reviewed once is not the same as staying correct forever. The moment a model's own output can re-enter its own training or reference data, you need a way to tell a real judgment from a passed-through one, or the system will happily teach itself the same small mistake, over and over, and call it consensus.

Now here is the same thing as a story

The short version above is what you'd say diagnosing this out loud in an interview. Read this one for how the phrase actually spread, one quiet approval at a time, before anyone thought to ask why.

Ines Vartan has managed Solstice's translation memory for five years, and can usually spot a machine-suggested line from a human-written one within a sentence.

Nobody decided, on any single day, to stop reading the suggestions closely. It happened one confident, correct-sounding line at a time.

Around month ten, Solstice's machine translation engine got a real upgrade, and suggestions started matching what a human translator would have written almost every time. Reviewers, who'd been correcting more than a third of every batch, started trusting the tool more, and fairly, since it had earned it. By month fourteen, a reviewer approved a line reading "the device must get warmed before use" without a second look, an oddly built phrase, not quite wrong, not quite right, and it went into translation memory as a trusted match.

Same button, same log entry, two entirely different things actually happened before it got pressed.

Over the next fourteen months, that one phrase resurfaced as a high-match suggestion for similar warning text across twenty seven different manuals, and every time, a reviewer, often a different one, approved it in seconds, since it read as fluent and confident, exactly like every other phrase the improved engine now produced. By month twenty eight, a new hire reviewing safety-warning sections for the first time noticed the same odd phrase in her fourth manual that week, and asked Ines why nine different senior translators had all approved the same slightly wrong construction, independently, apparently, over more than a year.

Three honest suspects, and only one of them survives a real lineage check.

Ines didn't have an answer that morning. She pulled the lineage on every one of the 340 segments carrying that phrase, across all twenty seven manuals, and found all 340 traced back to the exact same original translation-memory entry, the one rubber-stamped in at month fourteen, regardless of which of nine different reviewers was credited with approving each later instance.

When the fast-approval workflow was first built, someone said, "the engine's good now, let's not slow reviewers down on lines it clearly nails," and it sounded reasonable, since the correction rate really had dropped for good reason, most of the time.

Rerun the same fourteen months with lineage tracking built in: the month-fourteen approval gets logged as a pass-through, not a verified correction, since the reviewer spent under two seconds on it. The system flags it before it can auto-populate as a high-match suggestion at scale, and a person takes a real second look. The phrase either gets fixed on the spot or, at worst, spreads to a handful of segments instead of 340, caught within weeks instead of over a year.

What I'd tell myself, listening to a new hire ask a question nobody senior had thought to ask: the tool never lied. It just kept doing exactly what nine different people, in nine different rushed moments, had told it was fine.

TRACE, the check that told nine reviewers apart from one loopNot a script for assuming every AI system will silently corrupt itself. TRACE is what tells you whether this one already has, and where.

The recap, one line per letter: timeline is fourteen months between the first rubber-stamped approval and discovery, recut is boilerplate sections carrying nearly all the drift while looking fine in the blended average, assume nothing rules out a style choice and a tooling bug, cause candidates names coincidence, personal style, and the feedback loop, and the evidence test, a shared segment ID across nine different credited reviewers, confirms the loop.

And if you want to be sure it really works, try it somewhere elseSame five letters, a funeral home's AI drafting tool instead of a translation pipeline. Different flip family entirely, the same review that quietly stopped happening.

Genevieve Okonkwo manages operations at Harrow and Vale Funeral Care, where a drafting tool suggests service program text and obituary language from a family's intake notes, and staff-approved drafts feed the tool's reference library for future families. Mapped onto TRACE: timeline is when standard service-program language started reappearing nearly unchanged across unrelated families. Recut shows the drift living almost entirely in the standard program templates, reused 68 percent of the time, while personal life-story sections, written fresh for each family, stayed clean. Assume nothing rules out a genuine template update, the wording never changed in the official style file. Cause candidates are coincidence, one staff member's habitual phrasing, or reused approved drafts feeding back into the reference library faster than anyone reviewed them. The evidence test traces the phrasing back to a single overworked week where a part-time staffer approved twelve drafts in ten minutes each, all now sitting in the reference library as though a careful review had happened every time. The flip here is substitution, not over-trust: as approval time got scarce, review effort got rationed toward the personal, emotionally sensitive sections, and the standard sections, the ones that quietly kept teaching the tool, got the least scrutiny of all.

A different flip entirely: not a rubber stamp on one document, but care rationed away from the exact sections quietly feeding the tool.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "any output that gets reused as training data needs to know whether it was really reviewed or just approved, that's the whole risk," and stop.
Cost: there's no time to build lineage tracking before the next release. Say so honestly, and start by capping how much of any retraining batch can come from the tool's own prior approved output, even without full tracing.
The engine got better, for real: if a genuine model upgrade explains a falling correction rate, that's good news worth confirming, not automatically evidence of a hidden loop, and the evidence test is what tells the two apart.

Where people run it wrong.
They treat "a human approved it" as proof of quality forever, with no record of how carefully.
They watch one blended quality number and miss that the drift is concentrated entirely in the reused, low-scrutiny slice.
They assume a feedback loop only happens with obviously bad output, when the whole danger is that it's fluent and confident enough that nobody objects.

How to use it live. The moment an interviewer asks about a feedback loop, ask yourself what happens to output after a person signs off on it. If the answer is "it can come back as training data with no record of how carefully it was checked," you've found the risk before you've said a single word about the model itself.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0