Examine an enterprise AI product's trust and security page as a product artifact.

Halcyon Ridge sells an AI copilot that reviews loan documents for banks and credit unions, flagging clauses a human should double check. Vashti Orenthal does vendor risk review for Northscale Credit Union, and her job this quarter is deciding whether Northscale signs with them.

How to answer this, stage by stage

An interviewer handing you a trust page wants to see you read past the design. Six moves get you there.

Let's learn

Halcyon Ridge's copilot reads a loan file and flags the three or four clauses inside it that a human reviewer should look at twice.

Before the copilot, Northscale's own review team read every file cover to cover: about eleven pages per file, forty files a week, catching almost everything that mattered because a trained eye reads everything, slowly.

With the copilot, that same team now reads Halcyon Ridge's flags first: three or four clause snippets per file, in under two minutes, and moves on to the next file.

The turn: the copilot's flags are not the problem. Vashti's actual job right now is not judging whether the tool works. It's judging whether the page in front of her gives her any way to check.

At its worst: Northscale signs a two-year contract on the strength of a badge wall and a single unlabeled percentage, and the seventeen-point gap only surfaces the first time a flagged clause turns out to be wrong on a loan that already closed.

What I would leave alone: the badges themselves are still worth something. SOC 2 genuinely tells you the vendor has a real security process. Don't throw that signal out just because it can't tell you whether the model is accurate too.

The lesson: a trust page with no version pin and no eval set isn't halfway to a real audit. It's a promise with none of the parts that would let anyone check it, and the badge wall is what makes that easy to miss.

Now here is the same thing as a story

The short version above is what you'd say in the room. Read this one for how Vashti actually caught the gap.

Vashti has reviewed vendor contracts for six years, and she can usually tell within ten minutes of a sales call which claims are going to survive a follow-up question.

Two sides of the same page, and only one of them is written to be checked.

The first read went well. Halcyon Ridge's page had a clean layout, three compliance badges, a quote from a bank CISO, and a single bold percentage: 98% flag accuracy.

Then a new hire on Vashti's team, doing a practice run before her own first vendor call, asked a question nobody on the team had asked out loud in months: "wait, accuracy on what, exactly?"

Four things a real trust page names. Halcyon Ridge's page named none of them.

Vashti went back to the page with that question in hand and found nothing answering it: no eval set named, no date, no version number, no failure category, nothing.

The evidence step, the one in the middle, is the one Halcyon Ridge's page skips straight past.

She emailed Halcyon Ridge asking for the eval set and the version pin behind the 98% figure. Three days passed before a reply came back: a restated version of the same marketing sentence, no set, no date, no version number.

Vashti later found the same 14-month-old version pin quoted in five other credit unions' vendor files. Nobody had ever asked for a newer one.

She ran her own forty real loan files through the copilot before writing her recommendation. Its flags matched a human reviewer's judgment on 81 of them, not 98.

Only one claim on the whole page landed in the provable, exact corner. Most of it lived in the take-our-word zone.

The old process asked whether a page looked serious. The new one asks whether a page's claims could survive Vashti asking one specific follow-up question about each of them.

I used to treat a badge wall as most of the score on a vendor review, because for years it covered most of the real risk on the vendors we bought from. It took a new hire's plain question, the kind she asked because she hadn't yet learned to skip it, to show me the badges had stopped covering the part of the risk that actually mattered now.

AUDIT, five checks for a page that wants your trustNot a vibe check. AUDIT is what forces a marketing page to answer for its own claims.

Five checks, in order. Halcyon Ridge's page fails four of the five before you even open a follow-up email.

The recap, one line per letter: ask who paid for it means every figure traced back to Halcyon Ridge alone, uncover the eval set means no named set existed behind the 98%, demand the version pin means a 14-month-old pin nobody had refreshed, isolate what's missing means no failure rate anywhere on the page, and test it yourself means Vashti's own 81% replication was the real finding.

And if you want to be sure it really works, try it somewhere elseSame five checks, a municipal permits office instead of a credit union.

Milnrace sells an AI copilot that reviews building-permit applications for cities, flagging incomplete or non-compliant submissions before a human clerk looks at them. Reuben Kastanidis handles vendor procurement for a mid-size city's permits office.

Mapped onto AUDIT: ask who paid for it finds every claim on Milnrace's trust page sourced to Milnrace's own case-study team, with a named client city quoted but no independent reviewer anywhere. Uncover the eval set finds a real one this time, a described set of 500 real historical permit applications, though Milnrace won't say which city they came from. Demand the version pin finds an actual date, refreshed twice in the last year, a genuine point in Milnrace's favor. Isolate what's missing finds no false-negative rate anywhere, only the flag-accuracy number, meaning nobody knows how many bad applications the tool waves through uncaught. Test it yourself means Reuben runs 25 of his own city's past applications through it, before signing, specifically checking how many known-bad ones it missed rather than just how many flags matched.

Same five checks. This vendor passes two of them and still fails on the one number that matters most: what it misses.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "no eval set, no version pin, no failure rate, so I'd test it myself before signing," and stop.
Cost: there's no time or budget to run your own forty-file test before a renewal deadline. Say so honestly, and ask for the vendor's own false-negative rate in writing instead, since a written number you can hold them to still beats a page with none at all.
The model gets better, for real: if Halcyon Ridge's next version genuinely improves, that's still not a reason to skip the eval-set question, a better model with no way to verify it is just a more convincing unverifiable claim.

Where people run it wrong.
They score a trust page on how many badges it has instead of on what those badges actually certify.
They accept a percentage with no eval set as if a number alone were proof.
They stop at reading the page instead of testing the product on their own real data before signing.

How to use it live. When someone hands you a trust page, ask yourself the version-pin question first, out loud if you can: which build, on what date. If the page can't answer that in one line, nothing else on it needs to be argued with.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0