Describe the smallest test that would tell you whether retrieval quality is the blocker.

Meridian Freight Exchange runs TariffLink, which answers a freight forwarder's question about tariff classification by pulling relevant customs rulings and writing an answer. Rafael Quintanilha is the AI PM deciding, before spending a single week tuning anything, whether the model's answers are wrong because it can't reason, or because it was never shown the right ruling to reason about.

How to answer this, stage by stage

Nobody is scoring whether you know that RAG has a retrieval half and a generation half. They're scoring whether you can isolate which half is actually broken in an afternoon.

Let's learn

Say we build a tool that answers a freight forwarder's question about which tariff code and duty rate applies to a shipment, by finding the relevant customs ruling and writing an answer from it.

Before TariffLink, a compliance specialist looked up the right ruling by hand, about 15 minutes per query, across roughly 30 queries a day, over seven hours daily just on lookups.

Two very different jobs hide inside one pipeline. An overall accuracy number can't tell you which one failed.

Here's the turn: the wrong answers weren't really the problem to chase first. The problem was that nobody had checked whether TariffLink's retrieval step was even handing the model the right ruling to work from, before spending weeks assuming the model itself needed to get smarter.

At its worst, a freight forwarder files a customs declaration based on a confidently wrong duty rate, faces a penalty or a shipment delay at the border, and the whole tool gets pulled just as Yewande Adisa's did at a different company, after weeks of tuning a model that was never actually the problem.

What I would leave alone: for queries where the correct ruling is unambiguous and always the top retrieved result, like standard container shipping rates, I wouldn't spend more time hunting for a retrieval problem that clearly isn't there.

The lesson: an overall accuracy number tells you something is wrong. It never tells you which half of the pipeline to fix first.

Now here is the same thing as a story

The short version above is what you'd say proposing the golden-set check in a debugging meeting. Read this one for what Yewande's six weeks actually cost before Rafael heard about it.

Rafael Quintanilha had shipped two smaller features at Meridian before TariffLink, both simple enough that "make the model better" had always been the right instinct.

TariffLink launched at 61 percent accuracy against expert judgment, and stayed there, week after week, no matter how the team adjusted the prompt. Nobody had checked what the model was actually being shown before it wrote each answer.

A peer's six weeks, and Rafael's two days, spent on the exact same underlying question.

Yewande Adisa, a PM at a different customs brokerage, had built something similar the year before. Her team spent six weeks tuning prompts and trying larger models, and only afterward discovered retrieval had been returning outdated tariff-schedule sections 40 percent of the time, a corpus problem no amount of prompt tuning could ever have touched.

Hearing Yewande's story, Rafael refused to start tuning TariffLink's prompts until he'd checked the same thing directly. He froze 40 real past queries with a known correct ruling and checked one thing only: did that ruling appear anywhere in the top 5 passages retrieval actually pulled.

Four things, and none of them involve reading a single generated answer yet.

Retrieval found the correct ruling in only 58 percent of the 40 queries. On those, TariffLink's final answer was right 88 percent of the time. On the other 42 percent, where the correct ruling never reached the model at all, accuracy was only 19 percent.

The real question was never whether TariffLink's model could reason about tariffs well enough. It was whether the model had ever actually been handed the ruling a correct answer would require, and for four out of ten queries, it hadn't.

One measurement, three possible readings. Only one of them matched what Rafael actually found.

When the first debugging meeting happened, someone said, "let's try a stronger model, the current one might just not be reasoning well enough about tariff rules," and it sounded reasonable, since a smarter model has fixed real problems before.

Three real suspects, named before the test ran. Only the golden set could say which one actually did it.

Rerun the six weeks with the golden-set check run first: the 58 percent retrieval hit rate surfaces on day one, at a cost of a single afternoon, instead of surfacing after six weeks and no improvement to a prompt that was never the actual problem.

Eight hours, one afternoon. The whole diagnosis fit inside a single working day.

What I'd tell myself, hearing how close Rafael came to spending six weeks the same way Yewande had: a model that answers confidently from the wrong passage will never tell you it was never shown the right one. You have to go check.

TRACE, the recut that finds the real suspectNot a script for distrusting every model. TRACE is what tells you which half of a RAG pipeline actually earned the blame.

The recap, one line per letter: timeline is a flat 58 to 64 percent since launch, recut is 88 versus 19 percent split by retrieval success, assume nothing is checking corpus freshness first, cause candidates are retrieval, generation, and a stale corpus, and evidence test is the 40-query hit-rate check that confirmed retrieval as the blocker.

And if you want to be sure it really works, try it somewhere elseSame five letters, a translation agency instead of a customs desk. The same half of the pipeline turns out to be the real suspect.

Katarzyna Wrobel runs product at Linden & Vos Translation Partners, testing TermLock, a tool meant to look up the correct legal or medical term for a translator working on a document. Mapped onto TRACE: timeline is accuracy sitting flat since launch, not drifting, which rules out a recent regression. Recut is splitting accuracy by whether the correct glossary entry was retrieved at all, the same split that mattered for TariffLink. Assume nothing means checking whether the glossary itself has been updated with recent regulatory terminology before blaming the retriever. Cause candidates are retrieval missing the right entry, the model ignoring a correct entry it was given, or a glossary that's simply out of date. Evidence test is a 25-query golden set, checking retrieval hit rate on known correct terms before ever looking at a single translated sentence.

A different document type entirely, and the same shape holds: low retrieval, low accuracy, moving together.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "check retrieval hit rate on a small golden set before touching the model," and stop.
Cost: no time to build a full golden set. Say so honestly, and hand-check even 10 real queries rather than skip the retrieval check entirely.
The model really is the problem, for real: if retrieval hit rate is high and accuracy is still low, that's a genuine generation problem, and saying so plainly is what makes the test trustworthy instead of a way to always blame the same thing.

Where people run it wrong.
They tune prompts and swap models before ever checking what the model was actually shown.
They read one overall accuracy number and never split it by whether retrieval succeeded.
They blame the retrieval algorithm without first checking whether the underlying corpus is even current.

How to use it live. The moment you're asked to diagnose a disappointing RAG answer, ask yourself: did the model ever actually see the right passage? Build the smallest test that answers just that question, before touching anything downstream of it.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0