How would you document an AI system for an external audit?

Solmara Port Authority uses Quaymark to score incoming shipping containers for customs inspection risk, flagging the ones worth a human officer's closer look. Renzo Bastian leads compliance and product for Quaymark, and keeps every past audit's request list pinned above his desk.

How to answer this, stage by stage

Nobody is grading whether you can list five document types from memory. They're grading whether you can show which one is missing and why that's the one that matters.

Let's learn

Quaymark reads a container's manifest, origin, and shipping history, and scores how worth a closer customs look it is, before it ever reaches a human inspector's queue.

Before any real audit pressure, Quaymark's documentation lived wherever it was easiest to write it: eval results in a shared spreadsheet, model updates announced in an engineering Slack channel, limitations known only informally, by whoever had been on the team longest.

Now, the turn: the real risk isn't that Quaymark's documentation is thin in places. It's that "we have a document about this" and "we can answer a specific question about this" quietly became the same claim, when they had never actually meant the same thing.

What I would leave alone: Quaymark's data-lineage documentation, already scoring a full 2, doesn't need more investment right now. It's honestly the strongest category, and spending more effort polishing it further would come straight out of the time the version-history gap actually needs.

The lesson: documentation for an audit isn't measured by how many folders exist. It's measured by how fast a real question, asked cold, gets a real answer.

Now here is the same thing as a story

The short version above is what you'd say defending Quaymark's readiness to Solmara's board ahead of the audit. Read this one for how the gap actually got found.

Renzo Bastian has led compliance for Quaymark for three years. He can usually tell how a request will go just from how specific the first question is.

A shipper filed a formal appeal after a container got flagged and delayed for inspection, on solid grounds: their cargo had cleared customs at four other ports without incident that same month. The appeal's first question was simple: which version of Quaymark scored this container, and on what evidence.

Five categories. Renzo could answer four of them from memory. The fifth took a search party.

Nobody could say immediately. Quaymark had shipped three model updates in the two months before that container was flagged, and none of them had been tied, in any searchable record, to the specific decisions made under each one.

Three steps that shouldn't have existed at all. The honest answer to a simple question took two full weeks to assemble.

It took Renzo's team two weeks of cross-referencing deployment logs, engineer memories, and a half-updated changelog to reconstruct which version had actually scored that container.

One category sat in the single worst corner: the thing an auditor cares about most, and the thing the team had documented least.

The shipper's appeal was eventually resolved in their favor, the flag had been a false positive from a since-corrected model bug. But the two-week delay to even answer the question was itself flagged, separately, as a governance concern by Solmara's own internal review board, months before any external audit ever arrived.

The missing version pin took the worst branch. Everything else the team was missing was recoverable by comparison.

Renzo's team spent the following year building a version-pin log directly into Quaymark's deployment pipeline, so every scoring decision automatically recorded its model version, timestamp, and linked eval snapshot the moment it happened.

Four fields, captured automatically, not four fields someone has to remember to fill in after the fact.

A full year, in this order, because the version-pin log was the one thing everything else depended on being trustworthy.

The old process asked engineers to remember which version did what, well after the fact, if anyone ever asked. The new one records it automatically, the moment a decision is made, whether anyone asks or not.

I let model updates live in an engineering Slack channel because it was the fastest way to keep the team moving, and for most of a year, updates were rare enough that nobody needed more. It took a shipper's appeal, and a genuinely embarrassing two-week wait for a simple factual question, to see that "fast to write" and "fast to retrieve later" had never been the same requirement.

BOUND, the readiness broken openNot a single grade. BOUND is what forces every category in that score to say where the evidence actually is.

The recap, one line per letter: break it down is the five categories, own numbers is real evidence for the strong ones and a named gap for the weak one, use a range is the seven-out-of-ten spread, nail the sanity check is the five-minute lookup test, and direction is the missing version-pin log as the fastest way to sink the audit.

And if you want to be sure it really works, try it somewhere elseSame five letters, a university admissions office instead of a port authority. This time the missing category isn't version history at all.

Amsel College uses an AI tool to pre-screen scholarship applications before a human committee reviews the shortlist. Colette Furlan runs that admissions product, and prepared for her own external accreditation review the same way Renzo prepared for his.

Break it down: the same five categories, but Amsel's weakest one turned out to be known limitations, not version history. Own numbers: their model version log was solid, every scoring run tagged and timestamped from day one, but nobody had ever written down which applicant profiles the model consistently under-scored, first-generation applicants with nontraditional transcripts, until a committee member noticed the pattern by hand. Use a range: Amsel scored 8 out of 10 overall, stronger than Quaymark's readiness, with the limitations category sitting at a bare 1. Nail the sanity check: could an accreditor ask "what kinds of applicants does this system score worst" and get a real answer? Before the fix, no, only an impression from one committee member's memory. Direction: writing the known-limitations document, backed by an actual audit of scoring patterns by applicant type, was the single fix that mattered most, since version history was never actually the risk there.

The same before-and-after shape showed up at Amsel too, just anchored to a different missing category than Quaymark's.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "five categories, scored on a real scale, and the version-pin gap is what sinks an audit fastest," and stop.
Cost: there's no engineering budget this quarter for an automated version-pin pipeline. Say so honestly, and start with a manual log, one spreadsheet row per model deployment, since even a manual record beats a Slack channel nobody can search.
The model gets better, for real: if Quaymark's scoring accuracy improves, that says nothing about whether a specific past decision can still be traced to its model version. A better model with no version history is still a documentation gap.

Where people run it wrong.
They treat "a document exists" as the same claim as "a specific question can be answered fast," when the two are not the same thing at all.
They score readiness as pass-fail instead of a real maturity spread, hiding exactly which category needs the most attention.
They wait for the real external audit to find the gap instead of running a mock audit first, with someone playing the skeptical outsider on purpose.

How to use it live. When asked how to document an AI system for an audit, don't list the five categories and stop. Say which one is weakest today, and why that specific one is the one most likely to sink the audit first.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0