The direct answer
Replace it with a number. Each week, pull 50 factual claims at random from real outputs, open the document each one cites, and count how many of those documents actually say what the sentence says. Ship the assistant while that score holds at 98 percent or higher, and pull it off any story that names a real person the moment it drops under 90.
Do this, in order
Rewrite the criterion into a weekly sampled check with a hard number.Why: a sentence nobody can test can never fail a build, so it never actually stops anything from shipping.
Build the sampling harness before the assistant reaches a single reporter, not after a near miss finds it for you.Why: a leading signal is only worth having if it moves before someone gets hurt by what it would have caught.
Score whether the citation actually supports the exact sentence, not just whether it's a real, clickable document.Why: a dead link is an easy bug to spot. A real document that says something else is the dangerous case, and only a person reading both catches it.
Set a harder floor for claims that name a real person than for the average.Why: the overall score can sit at 98 percent while the ten riskiest sentences, the ones that could get someone sued, are exactly the ones failing.
Count a missing citation as an automatic miss, never a free pass.Why: otherwise the cheapest way to raise the score is to stop making checkable claims at all.
Post the weekly score where the newsroom can see it, not only in an engineering channel.Why: a number nobody outside the build team ever looks at is a chart on a wall, not a criterion.
How to answer this, stage by stage
Seven moves. Say the number out loud before you defend it, or it sounds like a guess dressed up as a rule.
1
Say what's actually wrong with the sentence
Say it like this
"The problem with 'the model should not hallucinate' isn't that it's wrong. It's that nobody can fail a build on it. There's no number in that sentence, no sample size, no way to check it on a Tuesday afternoon. Before I rewrite it, I want to name that, because that's really what the question is asking me to fix."
Why this works
Naming the actual flaw first tells the interviewer you're not about to bolt a number onto the same vague sentence. You're going to change its shape.
2
Put one real person under it
Say it like this
"Say this is a research tool for a reporter, someone like Noor, who covers courts and public records at a small newsroom. She pastes in a case or a name, and the tool hands back a short summary with a citation on every factual sentence, a filing, an old article, a public record."
Why this works
A criterion about "the model" is empty until someone would act on what it says. Naming the reader turns "hallucinate" into a question about what happens if she believes something untrue.
3
Ask what the sentence is actually standing in for
Say it like this
"What we really want isn't 'the model is never wrong about anything, ever.' Every model gets things wrong sometimes. What we want is that Noor can trust a citation enough to write it into a story without re-reading the whole document herself. That's the thing 'don't hallucinate' is really protecting."
Why this works
This is the outcome step. Say it plainly, or the number you write next has nothing real to aim at.
4
Write the number, before you defend it
Say it like this
"Here's what I'd actually check. Every week, pull 50 factual claims at random from real outputs. For each one, open the document it cites and read the sentence next to it. Does that document actually say what the claim says, not just something nearby? Score the percent that pass. That's the criterion: 98 percent of a weekly sample, checked by a person against the real source, not 'no hallucinations.'"
Why this works
This is the direct answer, said with the mechanics attached. A number with no sampling method behind it is just a nicer-sounding guess.
5
Say how someone would cheat that number
Say it like this
"And I'd say straight away how this gets gamed. The easy way to hit 98 percent is to only sample the boring claims, dates, filing numbers, things that were already easy to check, and never sample the sentences where the model stitches two documents into a conclusion. Those stitched sentences are where the real risk sits, and an average score can hide them completely."
Why this works
Naming the abuse case before anyone else does is what separates someone who understands metrics from someone who just picked one off a list.
6
Attach an action to every score, not just the good one
Say it like this
"So I'd score the stitched, inferred claims separately, on a tighter bar, 99.5 percent, not 98. Above that, ship it wide. Between 90 and 98, a person reads every flagged story before it runs. Below 90, on any claim naming a real person, the assistant stops making that kind of claim at all and just shows the search results instead."
Why this works
A metric nobody acts on is decoration on a dashboard. This line proves the number actually changes what the team does.
7
Close on the line that would have caught it early
Say it like this
"The whole reason to do this is that a score drifting from 99 to 92 over six weeks is something you catch on a Tuesday afternoon with a spreadsheet. A reporter catching it by luck at midnight, right before a story about a real person runs, is the version where you got away with it once."
Why this works
This ties the metric back to the actual stakes, and it's the sentence worth leaving the room with.
If you only get through two stages
Stages 3 and 4 are the answer. Say what "don't hallucinate" is really protecting, then say the number that checks it. Everything else in this list is how you defend that number under follow-up.
Let's learn
Say we build a tool that reads news archives, court filings, and public records, then writes a short summary with a link to where each fact came from. A reporter pastes in a name or a case, and gets a page back in under a minute.
Before a tool like this exists, pulling every filing and cross-checking every date for one story takes a reporter about five hours. With the tool, that same pull takes about 25 minutes. That's the whole pitch, and it's a real one.
Here's the catch. About two claims out of every hundred cite a document that doesn't actually say what the sentence says. The citation is real. It's just pointing at the wrong thing, a case about a different dispute, a filing that mentions the person but not the fact in question.
Two wrong claims in a hundred is not what breaks this tool. What breaks it is what a reporter does the first time she catches one. She stops opening citations to check them, and starts opening every single one, on every story, always.
The sentence didn't fail because a model got something wrong. It failed because nobody could point to the day it started going wrong.
Not every wrong claim in the tool costs the same
What that means at its worst: the tool ends up slower than working by hand. She reads the summary, then does the same five hours of checking anyway. Worse than if we'd never built it, because now the reading time is pure add-on.
Needs the hard floor
Claims that could end up in print
- Anything naming a real, identifiable person
- A criminal charge, a lawsuit, or a finding of fault
- A claim built by combining two or more documents into one conclusion
- Anything likely to end up as a pull-quote or a headline
98 percent is already plenty
Everything else the tool writes
- Suggested related articles or search terms
- Background like a court's location or a filing date range
- Word choice and phrasing in the summary itself
Knowledge spark: what counts as a hallucination here
Not a typo, and not a missing citation. It's when the model states a fact with a citation attached, and the document it cites does not actually say that. It looks exactly as confident as a true claim. That's the whole problem with it.
The choice I'd take back
We signed off on "the model should not hallucinate" as the acceptance criterion, nodded, and moved to the next line item. Nobody could test that sentence, so nobody built a way to check it. I would take that back and build the weekly sample, 50 claims checked against their real sources, before the tool ever reached a reporter's desk.
What I'd leave alone. The related-articles list and the general background lines. A wrong suggestion there costs somebody a click, not a correction. Fixing the wording of a summary wouldn't have stopped a single bad citation, and chasing it would have used up time that belongs on the hard floor instead.
The lesson. If you can't say the number that would make you pull a feature, you haven't written a real requirement. You've written a hope wearing a checkbox. Every acceptance criterion needs a person with a sample and a cut-off point next to it, or it isn't actually a criterion yet.
One midnight paragraph
You don't need this to answer the question. It's here so "98 percent" stops being an abstraction.
Noor can read a court docket the way some people read a recipe, straight down, no wasted motion, and she already knows which paragraph the real story is hiding in. She's covered courts and public records for six years at the Fernbridge Record, a three-person investigations desk that runs on grants and a patient board.
The tool, which the newsroom calls Pallas, arrived in January. For months it was the best part of her evening. She'd paste in a case number at ten at night and have a clean summary with links by 10:01. In February she opened two or three of those citations a story, just to see if it was behaving. It always was.
By April she opened one, the one behind the line she was actually going to quote. By June, most nights, she opened none at all unless a sentence read a little too neat.
Then came a Tuesday in July. Nothing dramatic. She was closing a piece about a local contractor, due before a 6am hearing, and Pallas had cited a specific prior lawsuit to back one line: that the contractor had faced the same complaint once before, in another county. She needed the exact paragraph for a pull-quote, so she opened the filing herself for the first time in weeks.
The case was real. The docket number was real. It just wasn't about the same thing. It was a fence dispute between two other people who happened to share a similar name.
We didn't lose one wrong paragraph. We lost every night she used to get back.
She didn't publish that line. She also didn't go back to trusting Pallas. She reopened every citation in the piece by hand, one at a time, and that took her the rest of the night. She filed as the sun came up.
Every story since, she opens every citation. The same five hours it used to take before Pallas existed. Now she also reads Pallas's summary first, so it's five hours plus 25 minutes she'll never get back.
She never had a percentage in her head. She had a feeling, and it only had two settings: grab the citation and move on, or go find the real document herself. One midnight paragraph flipped it, and nothing small was going to flip it back.
She never had a dial. She had a switch, and it only flips one way on its own.
Eight months earlier, in the kickoff meeting for Pallas, the requirements doc had one line under quality: the model should not hallucinate. Everyone nodded. Nobody put a number next to it, because nobody in the room could think of one that didn't sound made up on the spot. So the box got checked, and the meeting moved on.
Here's the replay. If a weekly sample of 50 claims had existed from week one, the score would have told the real story long before Noor needed a lucky click. Ninety-nine in week one. Still 97 by week three. Down to 92 by week six, the exact week a new county's e-filing system, with messier scans, got added as a source.
What the weekly sample would have shown, week by week
92%
Week 6, the near miss
98 percent or higher, ship as is
90 to 97, a person reads every flagged story
under 90, real-person claims turn off
The score never even reached the hard floor. It crossed into the amber zone in week three, three weeks before Noor's midnight paragraph, and nobody was watching it, because nobody had built anything to watch.
With that chart running, the fix would have been an afternoon: pull the new county's filings out of the source list, or add a human check on that source specifically, the moment the score dipped under 98. Instead it was a near miss at midnight and a habit that never came back.
What I'd tell myself, back in that kickoff meeting: I let a sentence I couldn't test into a document everyone signed. I should have said so out loud, in that room, instead of nodding and moving to the next line.
LEAD, the four questions that write the criterion for you
This is a metric question wearing an editing task's clothes, so the framework is LEAD, not FLIPS. FLIPS finds the moment someone's behavior snaps after a change. Here nothing has changed yet. The whole question is what number would have caught the change before Noor did.
LEAD, for a question about how you'd measure something
Knowledge spark: what makes a signal "leading"
A trailing signal tells you about the damage after it's already done, a lawsuit, a correction printed on page two. A leading signal moves first, while the damage is still small enough to fix in an afternoon.
L, link. The real outcome, not the model's own score. Here, it's whether Noor can trust a citation enough to write it into a story without re-reading the whole document herself.
E, early signal. The number that moves before that trust breaks. Here, the percent of a weekly sample of 50 real claims whose cited document actually says what the sentence says.
A, abuse. How the number gets hit without the real problem going away. Sample only the easy, already-public claims, and the stitched, inferred sentences, the ones actually doing the risky work, never get checked.
D, decision. What changes at each score. Above 98, ship wide. 90 to 98, a person reads every flagged story first. Under 90, on claims naming a real person, the assistant stops making that kind of claim at all.
The check that makes LEAD honest
Try swapping the outcome in the L step. Change it from "she can trust a citation without opening it" to "the model is never wrong about anything, ever." The number changes completely, and it stops being something you could build. If your metric survives any outcome you plug into it, you never really tied it to one. You just picked a percentage that sounded rigorous.
And if you want to be sure it really works, try it somewhere else
A four-vet clinic wants an assistant that drafts a one-page summary of a pet's labs and history before the vet walks into the exam room. Same shape of question: "the model should not hallucinate a lab value" is just as untestable as the newsroom version.
L. A vet can walk into the room and treat the drafted summary as the labs, instead of re-reading four PDFs first.
E. Each week, pull 30 drafted summaries at random. For every specific lab value the draft states, check it against the source PDF. Score the percent that match exactly.
A. The model rounds values into safe-sounding bands, "normal," "slightly elevated," which are technically defensible but useless for dosing. The score looks fine while every real number underneath it is fuzzed.
D. Above the bar, keep auto-drafting for every patient. Below it, any value outside the normal range gets shown as the raw PDF number inline, not a paraphrase, so the vet is reading the real figure either way.
Same switch, a different desk
Swap the trigger and it still runs
- It gets slower. Doesn't matter. Sampling 30 summaries a week takes the same afternoon whether the model answers in one second or ten.
- It gets more expensive. If the price per call triples, a working weekly sample lets the clinic swap to a cheaper model in a day and prove it's still accurate, instead of guessing and hoping.
- It gets better than planned. If the model starts reading messy handwritten notes reliably, the same weekly check just gets rerun against a wider set of documents. The process doesn't change. Only what's being sampled does.
Where people run it wrong
- Sampling only the clean, typed lab reports, never the faxed or handwritten ones, so the score looks great on the documents that were never the problem.
- Treating the sample as a one-time certification instead of a weekly habit, so a new document type slips in unwatched.
- Averaging every claim into one score instead of scoring the high-stakes ones, dosing, allergies, separately, so a bad hallucination hides inside a good overall number.
If you're asked this cold
Say the reframe out loud before you say a number. "Let me say what 'don't hallucinate' is actually protecting, then I'll give you the number that checks it." That's not stalling. It's the L step, and it gives you somewhere to stand while you work out the real figure.
Flashcards (click a card to flip it)
1 · THE FRAMEWORK
Which framework fits this question, and why not FLIPS?
Tap to flip
ANSWER
LEAD, for a metric question. FLIPS finds the moment someone's behavior snaps after a change. Here nothing has changed yet, the whole question is what number would catch the change before a person does.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Noor Vaziri, six years on the courts and public-records beat at the Fernbridge Record, a three-person investigations desk.
3 · THE HABIT
What did she stop doing because it kept being right?
Tap to flip
ANSWER
Opening the document behind a citation to check it. She went from checking two or three a story, to one, to none, over about five months.
4 · THE SWITCH
What's the two-setting switch in this story?
Tap to flip
ANSWER
Trust a citation on sight, or open every one by hand. Once she'd been burned once, there was no setting in between.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Signing off on "the model should not hallucinate" as the acceptance criterion with no number attached, so nobody built a way to check it before it mattered.
6 · THE NUMBER
The rewritten criterion checks ______ claims a week, and needs ______ percent of them to have a citation that actually backs the sentence.
Tap to flip
ANSWER
Fifty claims. Ninety-eight percent. Below 90, on any claim naming a real person, the assistant stops making that kind of claim at all.
7 · THE REPLAY
Same near miss, new design, what changes?
Tap to flip
ANSWER
The weekly score shows 99 percent in week one and 92 by week six, right when a new document source got added. A person reading that chart pulls the source out on a Tuesday afternoon, weeks before Noor needs a lucky click.
8 · THE TRANSFER
Section four runs LEAD on a different product. Which one, and what's its early signal?
Tap to flip
ANSWER
A vet clinic's lab-summary drafting tool. Its early signal is the percent of a weekly sample of drafted summaries whose stated lab values match the source PDF exactly.
Check yourself Score: 0 / 0
Fill in the blank
1. The rewritten criterion samples ______ claims a week, and needs ______ percent of them to have a citation that actually backs the sentence.
Show hint
Both numbers are in the direct answer at the top of the page.
Show answer
Fifty. Ninety-eight. Below that, and below 90 specifically, the assistant should stop making claims that name a real person until the source is fixed.
Multiple choice
2. Which of these would count as a genuine pass under the rewritten criterion?
- A. The citation is a real, working link that opens.
- B. The model says it's 92 percent confident in the claim.
- C. The sentence reads as specific and detailed.
- D. A person opens the cited document and the exact claim is actually written there.
Show hint
Three of these are about how the claim looks. Only one is about whether it's actually true.
Show answer
D. A citation that resolves is necessary but not enough, and a confidence score is the model grading its own homework. The only real pass is a person checking the document against the sentence.
True or false
3. True or false: this problem gets fixed by adding a line to the prompt telling the model to double check its citations before answering.
Show hint
Ask who would ever know if the model just didn't.
Show answer
False. A prompt instruction can't be tested, sampled, or audited. It's the model telling you it checked its own homework. You still need a person pulling real claims every week and checking them against the real document, or there's no way to know the instruction is doing anything at all.
Short answer
4. Name a place in this same product where a low citation-support score genuinely would NOT be worth pulling the feature over.
Show hint
Look at the pile marked "98 percent is already plenty," not the one marked "needs the hard floor."
Show answer
Model answer: "The list of related articles you might also search for. It's a suggestion, not a factual claim about the world, so a citation that's slightly off doesn't put anything false into a story. Only claims that could end up as a sentence in print need the hard floor."
Short answer, apply it yourself
5. Pick a product you use yourself. Name one thing it tells you as a fact that you've never actually checked against the real source. What would the weekly-sample version of this check look like for it?
Show hint
Think of anything that gives you a number or a fact with no link back to where it came from.
Show answer
Model answer: "A fitness app that tells me 'you burned 640 calories' after a run, with no way to see the math. The weekly check: pull thirty logged runs, recompute the estimate by hand from pace, weight, and distance using the published formula, and see what percent land within a reasonable range of the app's number. If it drifts, I'd catch it before I'd spent months trusting a number nobody had checked."
Multiple choice
6. Which of these would make the weekly score look good without actually making the assistant safer?
- A. Raising the sample from 50 claims to 200.
- B. Only sampling claims about dates and filing numbers, never the sentences that combine two documents into a conclusion.
- C. Turning down the model's temperature setting.
- D. Having two different people check each sampled claim instead of one.
Show hint
The dangerous move hides the hardest sentences inside an average full of easy ones.
Show answer
B. An average score can sit at 98 percent while the riskiest sentences, the ones stitching two documents into a new claim, never get checked at all. That's why the answer scores those separately, on a tighter bar.
