What does the PRD need to say about logging and observability?

How to answer this, stage by stage

Eight moves. Say what the log has to prove before you list what it has to store, or the requirement reads like a checkbox instead of a decision.

Let's learn

Every year, during a lawsuit, someone has to read through a mountain of email and memos and work out which ones a judge is never allowed to see, because a lawyer's private advice is written inside them. That job is called discovery review, and it used to move slowly on purpose, because getting it wrong is expensive in a way most mistakes aren't.

Before the tool, a team of thirty contract attorneys read a mid-size case's documents by hand, about sixty an hour each. A quarter of a million documents took them close to three weeks and cost the client about 1.4 million dollars in review time.

Now Docket Sense, a classifier built by Casewell AI, tags all quarter of a million documents overnight. Reviewers only have to check the roughly nine thousand it flags as close calls. Three weeks becomes four days.

Here's the turn. A few wrong tags out of a quarter million barely move an accuracy number. That was never the real risk. The real risk is what happens when one specific tag turns out to be wrong on a document that should never have left the building, and somebody has fourteen days to explain why it did.

At its worst, that's exactly what happened. A memo forwarded between a company executive and in-house counsel got tagged not privileged and went out to the other side's lawyers in a contract dispute with a company called Bramholt. Someone on Bramholt's team spotted the words "attorney client, please keep confidential" halfway down the page and sent a clawback letter: give it back and prove it was a mistake, or the whole executive's file could lose its privilege too.

Once the letter landed, the team went back and checked something nobody had been watching: how many of Docket Sense's tags actually had enough attached to explain a call, not just to record one.

The log said "handled" the whole time this was happening

The number that finally moved was how long it took to answer a challenge like Bramholt's.

What I'd leave alone. Documents Docket Sense tags not responsive because they're clearly nothing, calendar invites, out-of-office replies, don't need this much logging. Nobody is ever going to send a clawback letter over a lunch invite, so a tag and a timestamp are enough there.

The lesson. A log that only says what happened isn't a log of why it happened. If it can't answer that on its own, it isn't protecting anyone. It's just proof something ran.

Now here is the same thing as a story

Use this version when you've got the time. Nobody argues with a Tuesday nine days into a deadline.

Ifeoma Uchendu can read a privilege log the way most people skim a grocery list, and she's the one people at Casewell AI call when a tag looks wrong. She wrote Docket Sense's original logging spec four years ago, back when the whole tool was still running on one law firm's test data.

It shipped to Straithorn LLP two summers ago. For most of that first year, review teams loved it. An associate would kick off an overnight run, come in the next morning to a stack of maybe two hundred close calls instead of a quarter million documents, and spend the day on the ones that actually needed a human eye. Partners started saying the tool had given them back a week of every case.

The habit faded in three small steps, none of them looking wrong on its own. First, as the case grew, someone on the platform team switched the logging pipeline to a cheaper storage tier to keep the bill from climbing, and quietly capped how much of each passage got saved. Second, nobody re-checked the completeness number afterward, because nobody had ever been asked to watch it in the first place. Third, reviewers stopped even glancing at "why" a tag came out the way it did, because the tags themselves kept looking fine on their face.

Then, on a Tuesday in the Bramholt matter, an associate on the other side read a produced memo and found the words "attorney client, please keep confidential" sitting in it. She sent a letter. Ifeoma got one question from Straithorn's partner she couldn't answer: "Why did the tool call this one not privileged?"

It would be easy to say the model got a call wrong, and it did. But that's not really the story, because a single wrong tag out of a quarter million was always going to happen. The real problem was that nobody could tell whether it was one mistake or a pattern, because the one thing that would have answered that, the actual passage the model saw, had never been kept.

So here's the decision I would take back.

Two years earlier, in the meeting where the logging spec got trimmed, someone had asked whether they really needed the whole passage saved with every tag. Storage costs were climbing fast as the case grew, and the confidence score already looked like it was doing the job. Ifeoma remembers agreeing. It seemed like a fair trade at the time.

I would put the passage back. Compressed if it needs to be, but attached to every tag, along with the document's metadata and the model version that made the call.

Here's the replay. Same storage bill, same pressure to cut it, new spec. The passage and the metadata stay attached to every tag instead of getting dropped. When Bramholt's letter arrives, Ifeoma pulls the exact sentence Docket Sense saw, the executive's name, the date, and the model version, straight from the log. She has the real answer back to outside counsel in under four hours. No twelve thousand four hundred documents re-checked by hand. No nine days.

One design makes you re-read the whole case to answer a question. The other makes the log answer it for you.

And the thing I'd tell myself, back in that meeting: I traded away the one thing nobody could rebuild later, to save space on the thing everybody could rebuild anytime.

What LEAD looks like when the metric is "can you explain yourself"

This is a concept question about writing a PRD, but the shape underneath it is a metric question: what's the one number that tells you a log is actually useful, before somebody needs it to be. That's LEAD, not GUARD. GUARD is for who gets treated unfairly. Here the harm is that a decision can't be explained at all, to anyone, fair or not.

And if you want to be sure it really works, try it somewhere else

A clinic called Willowmere Family Health uses a similar tool to sort patient portal messages into urgent and routine, so a nurse's queue surfaces the ones that can't wait. Same shape of question. "Log every triage decision" survives an audit the same way "log every classification" did, and explains just as little.

Same idea, a different desk

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0