CaseIntermediateAI Opportunity & Model Strategy / Evaluating AI vendors as a buyer / #6
Describe the data handling questions you would put to a vendor on behalf of your security team.
PICK the intern's question that found six weeks of a confidential interview sitting on someone else's server
The Harlow Falls Courier is a small-town newspaper covering local government and courts. Adaeze Okonjo is the managing editor, working off a laptop with a cracked hinge she's carried through a decade of council meetings. Inkwell sells an AI that transcribes recordings and analyzes documents for newsrooms.
The direct answer
Hold a hard line on one category only: anything that could identify a confidential source never touches Inkwell's standard pipeline, no retention, no training use, no reporter-linked logging, verified in writing, even if it costs more and takes longer to set up. Everything else, routine council-meeting tapes, public filings, can run on Inkwell's normal terms without a second thought. Treating every file with the same caution wastes time on the files that don't matter and quietly excuses the one category that does.
Do this, in order
Ask whether they retain audio and transcripts after processing, and for how long.Why: retention is what turns a one-time upload into an ongoing exposure.
Ask whether they train on your uploads, and whether the opt-out is actually enforced, not just offered.Why: a toggle that silently resets on a plan change is the same as no opt-out at all.
Ask whether uploads are logged against the individual reporter's account.Why: that metadata alone can tie a specific investigation to a specific byline, even if the recording itself never leaks.
Ask what happens to your data under a subpoena or a breach.Why: this is the scenario ordinary vendor due diligence skips, and the one that matters most for a newsroom.
Ask if sensitive files can run through a separate, verified, zero-retention path.Why: without this option, you're stuck choosing between full caution everywhere or none at all.
How to answer this, stage by stage
Nobody is scoring whether you can recite a data-handling checklist. They're scoring whether you know which one file type earns the hard questions and which ones don't need them at all.
Stage 1
Scope it to one real newsroom
Say it like this
"I'll answer this for a small newsroom deciding how strict to be with an AI transcription vendor, on behalf of a security-minded editor."
Why this works
Keeps a broad security question from turning into a generic list of vendor-due-diligence boilerplate.
Stage 2
Say the structure out loud
Say it like this
"I'll use PICK. Position, my actual stance, before any reasoning. Impact, who feels each kind of mistake. Cost asymmetry, which one is worse. Kill criteria, what would change my mind."
Why this works
Shows this is a real tradeoff decision, not just a list of questions memorized from a template.
Stage 3
Give the position, before any reasoning
Say it like this
"Be strict only where a source could be identified. Be normal everywhere else. Treating every file the same either slows the whole newsroom down or, worse, quietly relaxes the one category that actually matters."
Why this works
This is the direct answer, stated first, exactly what a tradeoff question is testing for.
Stage 4
Name who feels each kind of miss
Say it like this
"Being too strict on a routine council tape costs a reporter a day's wait on a story nobody's protecting. Being too loose on a confidential interview costs a source their safety, and there's no version of that mistake you can walk back."
Why this works
Names both sides in real, felt terms instead of an abstract policy statement.
Stage 5
Prove it with the near miss
Say it like this
"A new hire, Junot Ferreira, asked Adaeze a plain question: does Inkwell keep our audio after it transcribes it? Nobody had actually checked. A confidential interview on a corruption story had been sitting on Inkwell's standard servers for six weeks, logged under the reporter's own account, with the training opt-out silently reset."
Why this works
Turns "ask good security questions" from a lecture into a specific, ordinary Tuesday that nearly went wrong.
Stage 6
Close on the one line
Say it like this
"Ask every data-handling question you can think of, but hold the hard line in exactly one place: anything that could name a source. Everywhere else, move at normal speed."
Why this works
Restates the direct answer in one breath, ready for a live follow-up.
Let's learn
Here is what happens when the same data-handling caution gets applied everywhere, or nowhere, instead of exactly where it's needed.
Before Inkwell, Harlow Falls Courier reporters transcribed recordings by hand, about ninety minutes for a one-hour council meeting, catching quotes accurately but eating an evening per story. Inkwell transcribed the same hour in under two minutes, at a fraction of the cost, freeing reporters to spend that evening actually reporting instead of typing.
This is what every upload went through, sensitive or not, until someone asked.
Here's the turn: speed was never the risk. The risk was that every recording, a routine zoning-board tape and a confidential whistleblower interview alike, ran through the exact same standard pipeline: logged to the uploading reporter's account, retained for ninety days, with a training opt-out that existed on paper but had quietly reset itself after a plan change nobody noticed.
Turnaround time, in days, for a routine request versus a source-sensitive one
The extra caution costs real time, but only on the category where it actually matters. Applying nine days to every file would have made the tool useless for daily reporting.
At its worst, treating a confidential interview like a routine tape doesn't just risk an awkward audit. It risks a source's safety, since a leaked identity, a breach, or a subpoena can't be undone once the data ever left a controlled path.
The choice I would take back
Inkwell's default account setup logged every upload against the individual reporter's name, a decision made for simple internal cost-tracking on routine transcription. That made sense for the newsroom's everyday workflow. It stopped making sense the moment a confidential interview ran through the same account-logged pipeline, since the metadata alone, which reporter uploaded which file and when, could tie a specific investigation to a specific byline.
What I would leave alone: transcribing a public city-council meeting doesn't need any of this scrutiny. That recording is already a public record the moment it's made; there's no source to protect and no reason to slow it down.
The lesson: a data-handling policy that treats every file the same either drowns the newsroom in unnecessary caution or quietly forgives the one file that needed it most. The fix isn't a stricter default. It's a second, separate path for the files that actually deserve one.
Now here is the same thing as a story
The short version above is what you'd say defending the new policy to the publisher. Read this one for how an intern's plain question found the gap.
Adaeze Okonjo had run the newsroom for seven years, sharp enough to catch a misquote from a council transcript on a first read. When Inkwell went live, it handled routine transcription so well that within a couple of months, uploading a recording had become as automatic as hitting send on an email, sensitive interviews included.
The newsroom's habits treated both sides of this picture the exact same way.
Knowledge spark: why would a training opt-out silently stop working?
Vendor accounts often carry settings tied to a specific plan tier. When a plan changes, some settings reset to that tier's default instead of carrying over automatically, and a toggle that used to say "off" can quietly say "on" again without anyone getting a notification about it.
Junot Ferreira, a new hire three weeks into the job, was setting up their own account when they asked Adaeze a question that felt almost too basic to ask out loud: "Does Inkwell keep our audio after it transcribes it?" Adaeze realized she didn't actually know, six months into using the tool daily.
She pulled the account settings that afternoon and found a confidential interview, recorded for an ongoing corruption investigation, sitting on Inkwell's standard servers under the normal ninety-day retention, logged to the reporter's own account, with the training opt-out toggled back on after a plan upgrade three weeks earlier. Six weeks had passed since the upload.
Nobody had done anything careless. The newsroom had simply never asked the one question that separates a routine recording from a dangerous one: does this file need a different path than all the others?
The interview hadn't leaked. No breach, no subpoena, no story about it. But it had sat exposed for six weeks on a system nobody had verified, reachable by a training pipeline nobody had confirmed was actually off, and identifiable to a specific investigation through nothing more than an account log.
This is the new path built the same week, just for the files that need it.
An intern's plain question did more than six months of normal vendor management had.
PICK, in one screenNot a blanket policy for every file. PICK is what tells you exactly which one category earns the hard questions.
P
Position. The stance, before any reasoning.
Strict, verified handling only for anything that could identify a source. Normal vendor terms for everything else.
Interviewers are testing whether you can commit to a real line instead of saying "it depends" about every file.
I
Impact. Who feels each kind of miss.
A reporter waits an extra day on a routine tape under too much caution. A source's safety is permanently at risk under too little.
Naming both sides in real terms keeps this from becoming an abstract security lecture.
C
Cost asymmetry. Which one is actually worse.
A slow routine request is visible and gets fixed with a complaint. A leaked source's identity is invisible until a breach or a subpoena, and by then it's permanent.
This is the hardest step, and the whole position turns on getting this asymmetry right.
K
Kill criteria. What would change the pick.
If Inkwell can prove, with an actual audit trail, true zero-retention processing for sensitive audio, the strict path could extend to more file types with confidence, not less caution overall.
Naming this in advance is what separates a real decision from a fixed, permanent rule that never updates.
Five questions. Only the answers to the last two decide which path a file takes.
The recap, one line per letter: position is strict handling for source-identifying files only, impact is a reporter's wait against a source's safety, cost asymmetry is the leaked-source case being the one you can't undo, and kill criteria is watching for real, auditable proof of zero retention before loosening anything.
Only one item in this picture needs the nine-day path. Everything else should keep moving at normal speed.
And if you want to be sure it really works, try it somewhere elseSame four letters, a compounding pharmacy chain's prescription-verification vendor instead of a newsroom. A merged workflow breaks the second story, not a logging default.
Caldwell Apothecary Group runs ScriptGuard, an AI tool that verifies prescriptions before they're filled. Mapped onto PICK: position is strict, human-reviewed handling for anything involving controlled substances or mental-health medications, normal automated handling for routine refills, impact is a pharmacist's extra minute checking a routine refill against a patient possibly receiving an unreviewed, incorrectly auto-approved controlled-substance refill, cost asymmetry is the same shape, a slow refill gets noticed and fixed fast, a wrongly auto-approved controlled substance can cause real harm before anyone catches it, and kill criteria is the same test, proof of a genuinely reliable automated check before loosening the human-review requirement.
The old decision here isn't a logging default, it's a merged one: Caldwell's initial ScriptGuard rollout combined routine refill verification and controlled-substance verification into one automated approval queue, with no separate pause for the controlled-substance category. That made sense to keep the rollout simple and avoid friction on day one. It stopped making sense the moment a controlled-substance refill needed exactly the kind of human pause the merged queue had quietly designed away.
Caldwell picked the left side by default. Nobody had decided that on purpose.
Inkwell's demonstrated zero-retention compliance rate, by quarter, against the kill threshold
Only once Inkwell crossed the kill line, real audited proof, not a claim, did the strict path get considered for a second file type.
Swap the trigger and it still runs.
Speed: an interviewer caps you at a minute. Say "strict only for source-identifying files, normal everywhere else," and stop.
Cost: there's no budget for a separate verified pipeline right now. Say so honestly, and hold the confidential files out of the vendor entirely until the budget exists, rather than lowering the bar to fit the price.
The vendor gets better, for real: if Inkwell later offers a genuinely audited zero-retention mode as a standard feature, that's the kill criteria being met, and it's the moment to extend the strict path further, not a reason to relax it.
Where people run it wrong.
They apply the same caution to every file, which either slows the newsroom to a crawl or, more often, quietly erodes over time until sensitive files get treated like routine ones.
They accept a vendor's opt-out toggle as permanent instead of verifying it after every plan or account change.
They wait for a breach or a subpoena to discover a gap instead of asking a plain question about retention on day one.
How to use it live. If an interviewer asks for the full list of questions, name the one that actually decides everything else first: does this file identify a source? Let the answer to that single question route every other data-handling decision.
Flashcards (tap any card to flip it)
1 · THE FRAMEWORK
What framework fits an "A or B" tradeoff question like how strict to be with a vendor?
Tap to flip
ANSWER
PICK: position, impact, cost asymmetry, kill criteria. It forces a committed line instead of "it depends."
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Adaeze Okonjo, managing editor of the Harlow Falls Courier for seven years, who found the gap after an intern's plain question.
3 · THE POSITION
What's the actual position, in one line?
Tap to flip
ANSWER
Strict, verified handling only for files that could identify a source. Normal vendor terms for everything else.
4 · THE COST ASYMMETRY
Which kind of miss is actually worse, and why?
Tap to flip
ANSWER
A leaked source's identity. A slow routine request is visible and fixable; a leaked identity is invisible until it's already permanent.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Letting Inkwell log every upload against the individual reporter's account by default, for simple cost-tracking, with no separate path for sensitive files.
6 · THE NUMBER
Fill in the blank: the source-sensitive verified path takes ___ days, against 1 day for a routine council tape.
Tap to flip
ANSWER
9 days, the real cost of caution, but paid only where it actually matters.
7 · THE REPLAY
Same confidential interview, same reporter, but the verified path already exists. What changes?
Tap to flip
ANSWER
The recording never touches Inkwell's standard servers at all. It runs through the zero-retention path, with no reporter tag and no training exposure, confirmed by audit, not by trust.
8 · CROSS PRODUCT TRANSFER
Section 4 answers this same question again for a different product. Which product, and what old decision gets taken back?
Tap to flip
ANSWER
Caldwell Apothecary Group's ScriptGuard rollout. The reversal is a merged workflow, one automated queue with no separate human pause for controlled substances.
Check yourself Score: 0 / 0
Multiple choice
1. Why does this answer treat routine council-meeting tapes differently from confidential interviews?
A. Council tapes are shorter and cheaper to process.
B. A council tape is already a public record with no source to protect; an interview can identify someone whose safety depends on secrecy.
C. Inkwell only offers strict handling for audio under ten minutes.
D. Reporters prefer faster turnaround on council tapes.
Show hint
Look at the position step and "what I would leave alone."
Show answer
B. The whole position rests on which files can actually identify a source, not on file length or cost.
True or false
2. True or false: the confidential interview at Harlow Falls Courier actually leaked before anyone noticed the gap.
True
False
Show hint
Look at what happened after the audit found the file.
Show answer
False. It never leaked. The near miss was that it sat exposed for six weeks with no verification, not that harm actually occurred.
Fill in the blank
3. Fill in the blank: the confidential interview had been sitting on Inkwell's standard servers for ___ weeks before the audit found it.
Show hint
Look at the story, right after Junot's question.
Show answer
Six weeks. Long enough that a breach or subpoena during that window would have had real material to expose.
Short answer, where it wouldn't matter
4. Name a part of the newsroom's Inkwell use where this extra scrutiny genuinely doesn't apply.
Show hint
Look at "what I would leave alone."
Show answer
Model answer: Transcribing a public city-council meeting. It's already a public record with no source to protect, so there's no reason to slow it down.
Short answer, apply it yourself
5. Think of a tool at your own job that handles at least one category of sensitive information. What's the one question you'd ask that would actually separate the safe files from the risky ones?
Show hint
Look for the single distinguishing fact about a file that changes how risky it is, not a blanket rule for every file.
Show answer
Model answer: Most workplaces have one clear line, like whether a file names a person, a patient, or a source, that should route it to stricter handling automatically.
Short answer, name the reversal
6. What old decision does this answer take back, and why did it make sense when it was made?
Show hint
Look at "the choice I would take back."
Show answer
Model answer: Logging every upload against the individual reporter's account by default, which made sense for simple cost-tracking before any sensitive file ever ran through it.
Before you close the answer
Why this works
Tests whether you can draw one real line between routine and sensitive data instead of treating every file the same, and whether you know what actually matters to a security team beyond a generic checklist.
Follow-up traps
"Isn't a separate verified path just extra cost for something that might never happen?" Response: the cost is small and bounded, nine days on a handful of files a year; a leaked source is unbounded and permanent, which is exactly the asymmetry this position is built on.
"What if you can't tell in advance whether a file is sensitive?" Response: default to the strict path whenever a reporter isn't certain, since the cost of caution on an ordinary file is a delay, not a disaster.
If pressed
The Courier's new verified path routes sensitive audio through on-device transcription first, sending only a redacted, source-anonymized text summary to Inkwell for any further analysis, so the raw audio never leaves the newsroom's own hardware at all.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
