What should be in your record-keeping from day one to survive a future audit?

Northgate Hire scores and ranks job applications for corporate hiring clients. Esteban Ruiz is the product manager for the scoring model. For over a year, he kept a shared log on a printed sheet clipped to his monitor, where he jotted down anything his weekly spot-check turned up.

How to answer this, stage by stage

Six stages. Enough to show the reasoning, short enough that nobody loses the thread.

Let's learn

Say a company builds a tool that scores job applications for other companies to use in hiring, thousands of applications a week, across dozens of clients.

For its first year, the tool logged whatever the current system happened to keep: the latest model, the latest score. Esteban Ruiz, the product manager, ran his own weekly check on top of that. Every Friday he pulled twenty scored applications, compared the score against the resume by eye, and jotted anything odd on a printed sheet clipped to his monitor.

The checks kept coming back clean. By month nine, Esteban was pulling five applications instead of twenty. By month fourteen, the printed sheet mostly sat blank.

At its worst: a client's legal team, working a discrimination complaint from a rejected applicant, asks Northgate to produce the exact scoring record from fourteen months back. Nobody can. The model's been updated twice since. The exact inputs from that day were never kept anywhere permanent.

What I would leave alone: the tool's day-to-day scoring dashboard, the live view recruiters actually use, didn't need to change at all. The gap was entirely in what got kept behind the scenes, not in what anyone saw on screen.

The lesson: record-keeping that feels like overhead on a small team is really just a decision, made quietly, about which future questions you'll be able to answer and which ones you'll have to admit you can't.

Now here is the same thing as a story

The short version above is what you'd say defending the ranking. Read this one for how Esteban actually found the gap.

Esteban Ruiz built Northgate's scoring model from its first client. For a long while, his Friday check was the whole safety net, twenty pulled applications, a resume in one hand, a score on the screen, a printed sheet for anything that looked off.

It never turned up much. Which is exactly why, over about a year, he pulled fewer and fewer applications to check.

Then came the audit. Not a routine one. One client's outside counsel, working a discrimination complaint from a rejected applicant, sent a formal request: produce the exact model version, inputs, and score behind that applicant's rejection, fourteen months earlier.

The first box was never built, so nothing after it could actually be trusted.

Esteban spent three weeks trying. The model had shipped two newer versions since. The exact inputs from that week had been overwritten by the current system's normal behavior. His own override log, the thing he'd once filled in every Friday, had gone quiet ten months earlier. At the end of three weeks, the honest answer to outside counsel was: we can't reconstruct this.

One of these gaps you can fix next sprint. The other one you can only ever fix going forward.

That distinction was the whole lesson. Esteban went back and sorted every record type by one question: if we skipped this today, could we ever add it back for a past decision, or is that day simply gone.

The cheapest fix, version pinning, turned out to be one of the two most valuable ones too.

He built the ranked list from that, and mapped what one fully audit-ready record actually needed to contain.

Four parts. Northgate had reliably kept exactly one of them.

The first two weren't really a choice. Everything after them was.

Ten months after the fix shipped, a second, much smaller inquiry came in on a recent applicant. This time Esteban pulled the exact version, inputs, score, and reviewer note in about forty minutes, without touching outside counsel at all.

The order matters as much as the list. Skip the top one and nothing below it can be trusted either.

The old approach asked what was easy to log this sprint. The new one asks what can never be recovered if it isn't logged today, and puts that first, no matter how unglamorous it looks on a roadmap.

I kept only the latest state because a small team building fast doesn't feel like it needs a permanent record of every version. It took watching a real legal request fail, in slow motion, over three weeks, to see that the record-keeping decision had already been made the day we chose not to snapshot anything.

ORDER, for ranking what to keepNot a wish list. ORDER is what forces you to rank record types by what breaks first if you skip them.

The recap, one line per letter: outcome is the ability to reconstruct any decision on demand, reversibility is what separates version pinning from a retention policy, dependency is why the version ID has to come first, evidence is checking what past audits actually cited, and rank is the five-item order that follows from all of it.

And if you want to be sure it really works, try it somewhere elseSame five letters, a self-storage company instead of a hiring tool. This time it's a pricing model, not a hiring model, and the audit comes from a different direction entirely.

Keepwell Storage uses a model to set dynamic prices on its storage units by demand, location, and unit size. Wren Ashby manages that pricing model.

Mapped onto ORDER: outcome is being able to show, for any customer at any past date, exactly what price they were shown and why. Reversibility says the price a specific customer saw on a specific day is gone forever if it wasn't logged that day, while a pricing-rule change can always be documented going forward. Dependency says you need a rule-version ID before a shown-price snapshot means anything, the same shape as Northgate's problem. Evidence is checking whether past pricing-discrimination complaints in this industry turned on missing shown-price records specifically. Rank puts shown-price snapshots first, ahead of even the pricing rules themselves, since a customer complaint is about what they personally saw, not the general policy.

A different product, the same four missing parts, just wearing a price tag instead of a score.

Swap the trigger and it still runs.
Speed: an interviewer caps you at a minute. Say "pin the version, snapshot every decision, log every override, in that order," and stop.
Cost: engineering says full snapshotting is too expensive to build this quarter. Start with version pinning alone, since it's nearly free and unlocks everything that comes after it.
The model gets better, for real: if the scoring model's accuracy improves, the record-keeping gap doesn't close on its own. A better model that still can't show its work fourteen months later has the exact same audit problem.

Where people run it wrong.
They build retention policy first because it sounds the most official, and skip the version pinning that makes retention meaningful at all.
They assume logging can always be added later, when for any past decision, it can't.
They rank by engineering convenience instead of by what an actual audit would ask for first.

How to use it live. When this question comes up, ask yourself which record type, if missing, can never be recovered for a decision already made. Rank from there, not from what's cheapest to build this sprint.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0