What logging do you need to make AI incidents investigable at all?

How to answer this, stage by stage

Nobody is grading whether you can name four field types. They are grading whether you can size what logging them actually costs, and say plainly what you would leave out. Six moves get you there.

Let's learn

Every day, Bexley Mutual's underwriting team used to write about 40 adverse-action letters by hand, one at a time, each one explaining in a line or two why an application got declined or rated up. Slow work, and every letter came out a little different depending on who wrote it. But there was always a person in the building who could tell you exactly why they wrote what they wrote.

Then Bexley built Undertow. It scores every one of the roughly 14,000 applications that arrive each day, and for the roughly 4,900 that get declined or rated up, it drafts the explanation letter itself, pulling from the applicant's file and the company's own underwriting rules. Underwriters still sign off, but most letters go out close to what Undertow wrote first.

None of that speed is the problem. Undertow is faster than any team of people, and on the whole its scores hold up fine. The problem shows up the day someone asks why. A state examiner opens a file on one specific decline. A policyholder's lawyer asks Bexley to explain, in writing, exactly why their client's premium tripled. And a finished letter is not an answer to that question. It is the output of an answer nobody kept the working for.

At its worst, Bexley can't answer the question at all. The letter is on file. The score is on file. But nobody kept the exact context Undertow scored against, which version ran that day, or whether an underwriter changed a word before it went out. The company ends up defending a decision it genuinely cannot reconstruct, which in front of a state examiner is close to the worst place a business can stand.

The choice I would take back. When Undertow's letter-writing feature first shipped, it was treated as a drafting aid for a human who would rewrite most of it, so its calls got logged the same lean way as any internal draft tool, a timestamp and a pass or fail flag, nothing about what it actually saw. That made sense at first, because underwriters rewrote nearly every letter by hand. It stopped making sense once the acceptance rate climbed past 80 percent and most letters started going out close to verbatim.

What I would leave alone. The roughly 65 percent of applications that get approved with no adverse letter at all don't need the expensive per-token confidence trace. A structured scoring event is enough there, because nothing about a routine approval is under the kind of scrutiny a decline is.

The lesson. Logs built so a team can improve a model are a different thing from logs built so a stranger outside the team can rebuild one decision. Bexley had built the first kind. The question it got asked was the second kind.

Now here is the same thing as a story

The short version is above. Read on if you want to feel what it's like to search a log for something that was never kept.

Petrina Vasek has run Bexley Mutual's underwriting platform for three years, and spent four years as a line underwriter before that. Ask her what belongs in a defensible file and she can tell you before you finish the question, because she used to build those files herself, one document at a time.

Undertow's first year was a good one. Scoring that used to take a team most of a morning finished before the coffee did. Letters went out clean and on time. Petrina's team watched the acceptance rate climb, quarter over quarter, until underwriters were editing maybe one letter in five instead of nearly all of them.

Nobody decided to stop paying attention to the logging behind it. It just thinned, in three ordinary steps. First, the team stopped spot-checking whether a sampled letter's reasoning matched its retrieved context, because the letters kept reading fine. Then, when the infrastructure team went looking for somewhere to trim storage costs during a budget review, Undertow's narrative logs looked like the obvious, safe place to cut, since nobody had touched them in months. Finally, a platform migration quietly dropped the retrieved-context links entirely, and nobody flagged it, because nothing downstream had broken yet.

Then a new compliance analyst joined the team. Reviewing a routine monthly sample of ten closed declines, she pulled one at random and asked Petrina a plain question: which exact version of Undertow wrote this letter, and what did it actually see when it wrote it.

What followed took three days. Deployment records placed roughly which week the model version changed, but not which exact build handled that one call. The applicant's file still existed, but the specific claims-history snapshot Undertow had actually retrieved at scoring time had since been overwritten by newer data in the same database row, because nothing had captured it as its own artifact. The team pieced together a plausible account from adjacent records and old Slack threads. Plausible was the best word anyone could honestly use for it.

The three lost days weren't the real cost. The real cost was what Petrina had to write in the response to the examiner: that Bexley could not fully substantiate the letter's reasoning against what the model had actually seen. An ordinary spot-check turned into a formal finding, the kind that shows up in next year's exam with extra scrutiny attached.

What stayed with Petrina wasn't the three days. It was that investigable turned out to have no partial credit. Either the chain reconstructs or it doesn't, and by the time someone asks, it's too late to go back and log the thing you should have kept.

The old decision, remembered as a meeting: eighteen months earlier, when the letter-writing feature was still a beta nobody fully trusted, someone on the platform team asked whether it was worth capturing the full retrieved context on every call. The answer at the time was no, it would double the storage footprint of a feature that was still mostly getting rewritten by hand anyway. Nobody revisited that call as the feature matured into the thing that actually produced most of Bexley's adverse-action letters.

The redesigned pipeline logs all four artifacts, tied by one request ID, on every scored application. Run the same compliance sample through that design and the answer takes Petrina about twenty minutes: pull the request ID, and the exact input, model version, output, confidence trace, and any override are all sitting under it, waiting.

The thing Petrina would tell her past self, back in that beta-stage meeting: a feature that's mostly getting rewritten by hand today is exactly the feature that's about to stop being rewritten by hand, and the logging decision you make while nobody's watching is the one an examiner reads eighteen months later.

BOUND, spelled out: the four artifacts and what logging them actually costs

This is a sizing question, how much evidence has to be kept, and at what cost, before an incident becomes investigable, not a person's habit switching between two settings, so BOUND fits and FLIPS doesn't.

B, break it down. Total daily log volume equals scoring events times their size, plus narrative events times their size. Each event carries four linked parts: input plus retrieved context, model version plus settings, output plus confidence trace, and any human override, all under one request ID.
O, own the numbers. Bexley scores about 14,000 applications a day, at roughly 2 kilobytes logged per scoring event, about 27 megabytes a day. About 4,900 of those trigger a generated letter. A lean confidence trace, the model's top pick per output token, runs about 12 kilobytes per letter. A full trace, the top five candidates per token, runs about 34 kilobytes, because most of the size in a narrative event is the confidence data, not the text itself.
U, use a range. Lean logging: 27 plus about 57 megabytes, roughly 85 megabytes a day, about 30 gigabytes a year. Full logging: 27 plus about 163 megabytes, roughly 190 megabytes a day, about 68 gigabytes a year. Over the seven years Bexley's underwriting files must be kept by regulation, that's a standing total of roughly 210 to 480 gigabytes.
N, nail the sanity check. At a blended storage rate of about 0.6 cents per gigabyte per month, weighted toward cold archival storage after the first ninety days, the full seven-year archive costs 15 to 35 dollars a year, under 40 dollars even at the high end. That's less than one seat of most compliance software for a single month. Storage was never the real constraint.
D, direction. The single assumption worth arguing over is the confidence-trace granularity, lean versus full, because it's the one lever Bexley actually controls; it swings the narrative side of the estimate by about 2.8 times. Retention length is set by regulation, not judgment, so debating it wastes a meeting. Debating logprob granularity doesn't.

One thing the arithmetic alone doesn't show: the 2.8 times gap only matters on calls that become the record of decision. An underwriter typically cycles through two or three drafts before accepting a letter, and logging full confidence detail on every one of those drafts, not just the final one, would nearly triple the compute and latency cost of the feature for detail nobody will ever need to reconstruct. So the full trace runs only on the call whose output actually got sent. Drafts still get a lean fingerprint, timestamp, prompt version, and output hash, cheap enough to prove how many drafts existed without paying for the full detail on each.

And if you want to be sure it really works, try it somewhere else

Windlow Veterinary Group runs an AI tool called Tarragon that reads X-rays taken at its clinics and flags likely fractures before a vet reviews the case. The group is scaling it across forty locations.

B, break it down. Same four-part shape: the input image plus the animal's structured history, the exact model version that scored the image, the flagged regions plus a confidence score per region, and whatever the reviewing vet confirmed, dismissed, or added.
O, own the numbers. Windlow's forty clinics run about 900 X-ray scans a day. A compressed copy of the image plus structured metadata runs about 400 kilobytes logged per scan, about 360 megabytes a day. About 18 percent of scans get at least one flagged region needing a vet's review, roughly 160 a day; logging the flagged-region overlay plus its confidence map adds about 80 kilobytes per flagged scan, another 13 megabytes a day.
U, use a range. Total volume runs a tight range, about 370 to 390 megabytes a day, because unlike Bexley's text-generation case, there's no confidence-trace granularity choice driving a wide spread; an image classifier's confidence output is a small, fixed-size number per region regardless of how detailed the review gets.
N, nail the sanity check. At about 140 gigabytes a year and a shorter, three-year veterinary record retention window typical for the group's insurance underwriter, that's roughly 420 gigabytes standing at steady state, costing under 30 dollars a year on the same blended storage rate. Same conclusion as Bexley: storage was never the constraint.
D, direction. Here the lever that matters most isn't a logging design choice, it's the retention window itself, since Windlow's insurer, not Windlow, sets how long a flagged case has to stay provable. That's worth confirming in writing before building anything, because guessing wrong on a fixed regulatory number is a rebuild, not a tuning knob.

Swap the trigger and it still runs.
Speed: an interviewer caps the answer at ninety seconds. Skip straight to the four artifacts under one request ID, then the range, then the one deliberate gap.
Cost: the platform team says storage budget is tight this quarter. Don't drop the retrieved-context logging to save money; drop the full confidence trace on draft calls instead, since that's the expensive part that buys the least.
The model got better: Undertow's decline rate drops and letters need almost no editing. That doesn't make logging less necessary, it makes each individual letter more likely to be trusted at face value, which is exactly when an unreconstructable one becomes the most damaging.

Where people run it wrong.
They log the final output and call it done, without the retrieved context that would show whether a wrong answer came from bad retrieval or from the model inventing something on its own.
They treat "storage is cheap" as a reason to log raw free text everywhere, when the real cost of over-logging PII is who can see it, not what it costs to store.
They promise every incident will be reconstructable, then quietly narrow what counts as an incident once that promise turns out to be expensive to keep.

How to use it live. Say the four artifacts and the tied ID before any number: "investigable means I can pull one request ID and see what it saw, what ran, what it said, and what a person did about it, all four, or it isn't really investigable." That buys the room to talk about the one deliberate gap, instead of reciting a generic logging checklist.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0