How would you eval a retrieval system separately from the generation on top of it?

Score retrieval and generation apart, or every wrong answer looks like the model's fault, even on the day it never got the page at all.

How to answer this, stage by stage

Nobody is grading whether you can define recall and precision. They are grading whether you can tell two different bugs apart without guessing.

Let's learn

Fernbank is a chat bot on the company Slack at Aleutia Systems. Employees type a question about an internal page, a spec sheet, a safety rule, a hardware manual, and it reads the wiki and answers instead of making them dig for it.

Before Fernbank, employees searched the wiki search box by hand, or messaged the docs team directly. When it worked, it took about eleven minutes on average. About one question in five never got answered at all, because the person asking gave up and just guessed.

Fernbank answers in under ten seconds. Checked every week against a three hundred question golden set, a real person's answer key held back and never used to train anything, it gets the right answer about ninety one percent of the time, company wide.

Then the company moved its wiki into a new system. The blended accuracy score barely moved, ninety one down to about eighty nine. Nine in ten questions were still landing fine. But one slice, questions about engineering hardware specs, torque values, clearance numbers, wiring limits, quietly fell from ninety percent right to forty three percent right, and the blended number never showed it, because that slice was a small share of the total questions Fernbank answered every day.

The team's first guess was the obvious one. The model must be making things up. That is a real failure mode, it has a real name, and a real fix, a stricter prompt, a lower temperature, a rule telling the model to only answer from the page in front of it. They spent a week on that fix. The engineering spec category did not move.

Reginald Cobham, who runs quality on Fernbank, split the numbers by stage instead of by guess. He built a small golden set of forty questions just for the engineering spec category, with the correct page marked next to each one, and checked whether that page even came back in the top five results. It did, only forty one percent of the time. Then he took the same failing questions and force fed the correct page straight into generation, skipping retrieval on purpose, and graded only the answer. It came back right eighty eight percent of the time, close to Fernbank's normal rate.

The gap between those two numbers, forty seven points, is what proved it. If generation had also been near forty percent, the bug would have been in the model. Because generation stayed near normal, the bug had to be upstream of it. He checked the page loading job next and found it. Fifty eight of the two hundred forty engineering pages had a slightly different page format after the move, and the job that reads pages into Fernbank's search list quietly skipped anything it could not read, with no error, no warning, nothing.

At its worst, a chat bot that quietly hands out the wrong torque spec on real hardware is worse than no bot at all. The old way, asking a person, was slow, but it got the number right eventually. A confident wrong number from Fernbank looked exactly like a right one.

What I'd leave alone: the HR policy category. Those pages kept their old format through the move, so retrieval there was never at risk, and a wrong answer about leave days costs someone a reread, not a cracked part. Spending audit time re-checking retrieval there would take time away from the category where a miss actually costs something.

The lesson: a blended score can be completely honest and still hide the one slice where being wrong actually hurts someone. Ninety one percent right told the team Fernbank was fine. It never told them that the confident, hidden number was ninety one percent right on nine tenths of the questions and forty three percent right on the tenth that involved real hardware.

Now here is the same thing as a story

Read the long version below when you want to feel why the guess cost a week, not just be told which stage was actually broken.

Before Fernbank existed, Reginald Cobham was the person engineers pinged directly when they needed a spec fast. Six years at Aleutia Systems had put most of the wiki in his head. Ask him the torque value for a bracket on the X-42 mount and he could quote it back before you'd finished the question.

Fernbank launched and, for the first few months, that habit quietly ended. Engineers asked the bot instead of pinging Reginald, and it answered in seconds, correctly, almost every time. He didn't mind. That was the whole point of building it. He still ran the weekly golden set check out of habit, ninety, ninety one, ninety percent, week after week, and moved on.

The wiki move happened on a Tuesday in the second week of the quarter. Nobody announced it as risky. Pages got copied into a new system with a cleaner layout. The blended weekly score that Friday read eighty nine. A two point dip. Reginald noted it and moved on, same as always.

Nothing dramatic happened next. No single bad Tuesday. Just, over the following weeks, a few engineers mentioned in passing that Fernbank had given them a number that "felt off" on a hardware question, and they'd double checked it themselves and moved on, the way you do with something you mostly trust.

Then a technician on the assembly floor almost torqued a mount bolt to the number Fernbank gave him. A second engineer glanced at the screen before he turned the wrench and said the number looked wrong, it matched an older, retired version of the mount, not the current one. It was wrong. Fifteen minutes of checking would have caught it after the fact. The second glance caught it before.

Half the team wanted to retrain the model that afternoon, or at least tighten the prompt so it would hedge more on hardware questions. Reginald asked for one hour instead. He pulled the last three weeks of low confidence answers and split them by document category, not by guess.

He built a forty question golden set just for that category and checked recall at five: forty one percent. The right page was mostly not even coming back. Before touching anything else, he ran the one test that actually separates the two stages. He took the same forty questions, force fed the correct page straight into generation, and graded the answers by hand.

Eighty eight percent came back right. Close to Fernbank's normal rate. The model was fine. It had simply never been handed the page.

The decision that opened the door went back to the day the page loading job was first built. It was written to skip, quietly, any page it couldn't read cleanly, rather than stop the whole load and page someone at two in the morning over one bad file. That made sense back then; the wiki barely changed shape, and a hard stop for one odd page would have cost more sleep than it saved. Nobody ever added a check comparing how many pages got moved to how many actually made it into Fernbank's search list, because for two years those numbers always matched anyway.

Run the same Tuesday again with one change: a small, cheap recheck, fifty questions per document category, runs automatically the moment new pages get loaded in. The wiki move ships the same way. Within a day, the engineering category's recall at five reads forty something instead of ninety, the check fails, and the deploy gets flagged before a single technician ever sees a wrong torque number on a real bolt.

One design trusted a single company wide number to speak for every category underneath it. The other design asks each category the question its own stakes actually deserve.

What I'd tell myself, back on the day that loading job was first written: the moment a job is allowed to skip something silently instead of saying so out loud, ask what silently means a year from now, when the thing it's skipping might matter. Nobody asked. That's on the room, not on the bot.

TRACE, run backward from one wrong torque spec

Not five guesses in a row. TRACE rules candidates out on purpose, until one test actually separates the two stages left standing.

The page was gone from the search list before generation ever got a turn at it.

Three things worth stating directly, since this is where the real judgment sits. The rejected alternative was retuning the generation prompt first, before running the split at all, stricter grounding language, a lower temperature, a rule to always hedge on hardware questions. It lost because it treats every wrong answer as a model problem by default, and it cost a full week with zero change in the one category that actually mattered. The AI specific failure worth naming by name is a silent index gap, a document that drops out of what the model can even find, with no error thrown anywhere to say so. The guardrail is two part: a check comparing pages moved to pages actually added to the search list on every load, and a small, cheap recheck, fifty questions per category, run on every deploy that touches documents or the search index. That guardrail isn't free. A full rebuild of the whole four thousand page search list costs about forty dollars in compute and twenty minutes end to end, so it only runs once a week. The fifty question recheck costs under a dollar and ninety seconds, so it runs on every single deploy, a real trade accepted on purpose, cheap and frequent over complete and slow. And the bar it enforces was never zero misses, a search system running thousands of questions a day can't promise that. It's a probability bar, checked against the golden set: the recheck fails and blocks the deploy when recall at five on any one category drops below eighty percent, not a promise that the top result is always right.

And if you want to be sure it really works, try it somewhere else

Same five letters, a different industry, and this time retrieval is innocent and generation is the actual bug.

Corvina is a tablet app field technicians use at Ridgemont HVAC. Mid repair, a technician asks it for a spec instead of flipping through a printed manual. Doru Vela runs quality on it.

A technician asked Corvina for the mounting bolt torque on a compressor, and it answered with a number that turned out to belong to a different, similarly numbered unit, the A250 line instead of the A400 line the job actually used. Doru recut the failing questions by product line and found it mostly happened when a job involved two units with near identical model numbers.

Retrieval looked healthy: recall at five for those exact questions checked out at ninety five percent, both manuals were coming back in the top five results almost every time. Doru ran the same evidence test anyway. Force the correct A400 page in as the only source, skip retrieval, grade the answer alone. It came back right only about forty percent of the time. The model kept quoting the A250 number even with the right page sitting right in front of it.

Retrieval did its job both times. The model just could not tell two manuals apart.

Same method, opposite result: label every retrieved chunk with its equipment line before it reaches the model, and when a question names one model by name, pass only that model's top chunk instead of all five blended together. Retrieval and generation can look identical from the outside, wrong answer, high recall, and still be broken in completely different places.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to the one test, force the right document in and see if the answer turns correct, that single line answers the question on its own.
Cost: there's no budget this quarter for a full weekly rebuild of the whole search index. Run the cheap per category recheck instead, it catches the same collapse for a fraction of the price.
The model got better, for real: say the generation model itself gets upgraded to a newer version. That's not proof retrieval survived the swap unchanged. A new model can read the same chunks differently, so rerun the split before trusting the blended number again.

Where people run it wrong.
They read a healthy blended score as proof nothing anywhere is broken, and never check a single category on its own.
They fix a wrong answer by retuning the generation prompt, when the model never had the right document in front of it to begin with.
They treat "the page came back in the top five" as good enough, without checking whether the model actually used it once it arrived.

How to use it live. Say the two stage split out loud before guessing at a fix: "before I answer that, let me ask whether the right page even came back, or came back and got ignored, because those need two different fixes." That buys a beat to think instead of guessing which stage broke in front of the interviewer.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0