Explain how source attribution should work in a RAG product.

Trestlewood Systems runs cloud infrastructure for other companies. Halden Priest is a senior site reliability engineer there, and he's been on the payments on-call rotation since before Trestlewood adopted Docktree.

How to answer this, stage by stage

Nobody's grading whether you can name every RAG citation pattern. They're grading whether you can commit to one and defend the asymmetry underneath it.

Let's learn

Here's the sentence nobody could trace, and why it almost cost a payments team a queue full of in-flight messages.

Docktree reads Trestlewood's runbooks, Confluence pages, and past incident postmortems, and answers an on-call engineer's question directly instead of making them search four tabs.

Before Docktree, Halden could grep the internal wiki himself and know, in about two minutes, which runbook applied to a given symptom, mostly from memory since he'd read most of them at some point. After Docktree, that same lookup takes about ten seconds, one question typed into a chat box.

The turn: the danger was never that Docktree occasionally retrieves a stale document. The danger is that when it does, one combined link at the end gave no way to tell which sentence in the merged answer came from that stale document at all.

What I would leave alone: a question like "who's on call this week," pulled from one live scheduling doc with nothing blended in, is completely fine with a single link at the end. There's no disagreement between sources to guard against when everything came from one place.

The lesson: attribution in a retrieval product isn't decoration on top of the answer. It's the thing that tells a person exactly where their trust is allowed to stop and their own judgment has to start.

The citation step sits between the merge and the trust. Skip it and the merge becomes invisible.

Now here is the same thing as a story

The short version above is what you'd say defending this design in a review. Read this one for the night it actually mattered.

The object in this story is a chat window bolted into Trestlewood's on-call paging tool, the same one Halden has open on his laptop at 3am more nights than he'd like.

Halden's the kind of engineer who reads postmortems for fun. Ask him about a payments-worker queue backing up and he'll usually already know which of three runbooks applies before you finish the sentence.

For the first several months after Docktree launched, it was the best change to his on-call life in years. He'd type a question, get a clean paragraph back, glance at the "Sources (4)" link, and move on. The answers were always right, or right enough, and after a while he stopped clicking that link at all. Why would he. It had never once mattered.

One of these costs a reader a few seconds. The other one costs a person their ability to check anything at all.

Then came a 3am page for a payments-worker queue climbing past threshold. Halden asked Docktree what to do. It answered smoothly: check the consumer lag, confirm the dead-letter count, then restart the queue consumer to clear the backlog.

That last step was true in 2022. It was deprecated eight months ago, after the team learned a consumer restart drops in-flight payment messages instead of clearing them. Docktree had pulled from the old runbook and the new one, blended them into a single confident paragraph, and put one link at the bottom covering all four sources equally.

A single end-of-answer link can't carry any of these four things for one specific sentence. Only per-claim attribution can.

Halden's cursor was over the restart button. A teammate on the incident call, half a beat before he clicked it, said, "wait, isn't that the step we killed after the September thing?" That's the whole story. One remembered detail, seconds before it mattered.

With per-claim attribution shipped since, that same blended answer now shows the restart line tagged to the 2022 runbook, dated, next to a second line tagged to the current one, dated three months ago. Halden doesn't need a teammate's memory anymore. He can see the age difference himself, in the same five seconds it used to take to read the sentence.

Plenty of Docktree's answers still get one simple link. The risk only shows up when sources of different ages get blended together.

I shipped the single link because inline citations tested as "cluttered" in a demo with clean, single-source answers. Nobody in that demo ever asked a question that blended two runbooks of different ages. It took a teammate's memory standing between Halden and a real mistake to see that the demo had never actually tested the failure that mattered.

PICK, in one screenNot a style guide. PICK is what forces you to name the asymmetry instead of listing pros and cons forever.

The recap, one line per letter: position is per-claim attribution over a single end link, impact is seconds of skimming against minutes of unnoticed risk, cost asymmetry is optimizing against the hidden one, and kill criteria is the click-through floor that would prove the design wrong.

And if you want to be sure it really works, try it somewhere elseSame four letters, a sales call-notes tool instead of an incident bot. A different industry, and this time the asymmetry runs through a quote instead of a runbook step.

Meridian Notes is an AI meeting-notes product for sales teams. Soraya Bakhtiar runs sales operations at a company that uses it to auto-summarize every customer call.

Mapped onto PICK: position is quote the customer's exact words for anything resembling a commitment, a price, or a date, and paraphrase freely for general sentiment. Impact is a rep who trusts a paraphrased "customer agreed to the renewal price" when the customer actually said "I'd need to check with finance," which shows up months later as a mis-forecasted deal. Cost asymmetry is that exact quotes cost a rep a little extra reading now, visible and cheap, while a wrongly paraphrased commitment costs real forecasted revenue later, hidden until the renewal date arrives and the money doesn't show up. Kill criteria: if reps say the quoted lines clutter their notes and they stop reading summaries at all, fall back to a paraphrase with a confidence flag instead.

The top-right corner, high stakes and high ambiguity, is exactly where an exact quote earns its place over a paraphrase.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "cite per claim, not per answer, because the risk lives in disagreement between sources," and stop.
Cost: per-claim attribution adds a few hundred tokens and a bit of latency to every answer. Accept it. Correctness during an incident is worth more than a slightly cheaper, slightly faster response.
The model gets better, for real: even if Docktree's retrieval accuracy improves next quarter, that changes nothing about whether two genuinely current, genuinely correct sources might still disagree in a way only per-claim attribution can show.

Where people run it wrong.
They treat attribution as one design choice for the whole product, when low-stakes and blended-source answers need different treatments.
They add citations everywhere by default and then watch people stop reading them entirely, which is its own kind of failure.
They never define a kill criteria, so a design nobody's checking on quietly becomes permanent regardless of whether it's still earning its cost.

How to use it live. When asked how attribution should work, ask yourself first: what happens the one time two retrieved sources disagree with each other. Design for that moment, not for the average one.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0