Why is AI a poor fit for tasks where users cannot verify the output?

Kestrelwyn Health builds PlainChart, a feature that turns a doctor's clinical note into a plain-language summary a patient can actually read after a visit. Berengaria Draxler is the product manager who owns it. Dr. Mordecai Fitzalan is the family physician whose notes PlainChart translates. Valdemar Skjold, a retired dock foreman on blood thinners for an irregular heartbeat, read one of those summaries after a routine knee procedure, and very nearly acted on it.

How to answer this, stage by stage

Nobody is grading whether you'd feel bad for Valdemar. They're grading whether you can turn "users can't verify it" into the actual mechanism, and a fix specific enough to defend.

Let's learn

PlainChart is a feature inside Kestrelwyn Health's patient app. After a doctor's visit, it reads the clinician's note and rewrites it in plain language, so a patient can understand their own diagnosis before they've even left the parking lot.

Four parts. One of them, the caution shorthand, is the part a patient can never check for themselves.

Before PlainChart, a patient at one of Kestrelwyn's partner clinics got a photocopy of the doctor's own note, dense with abbreviations only another clinician could read, or waited an average of two days for someone at the front desk to call and explain it in plain words.

PlainChart turns forty-two thousand of these notes into plain language every month, most of them inside a minute of the note being signed. Not every note gets the same treatment, though. PlainChart scores each one by shape: how long it is, how closely it follows the clinic's standard template. Seventy-two percent of notes, about thirty thousand two hundred forty a month, come back short and templated enough to publish the moment they're drafted, no clinician looks at them again. The other twenty-eight percent, the longer or more unusual ones, already went to a clinician for a quick check before PlainChart ever existed. That part of the workflow never changed.

Here's the turn. Being short and templated has nothing to do with being safe. Of the notes PlainChart auto-published untouched, about four percent, roughly twelve hundred and ten a month, contained a real caution about a drug interaction, an allergy, or a dosage change that the plain-language rewrite dropped or softened on its way out.

At its worst, this is a patient reading a confident, friendly sentence with no idea it's wrong, and taking it at face value, because nothing on the screen ever told him to check.

The lesson: a summary that reads clearly and a summary that reads correctly are two different promises, and only one of them can be checked by someone who's never been to medical school. The job isn't writing clearer sentences. It's knowing exactly which sentences the reader can never check for themselves, and treating only those differently.

Now here is the same thing as a story

Stage five above compresses this into four sentences. Here's the six weeks underneath, the part a stand-up answer skips.

Valdemar Skjold ran a dock crew for thirty-one years before he retired, the kind of job where you learn to read a situation fast and act on it. Four years ago a doctor found an irregular heartbeat, atrial fibrillation, and put him on warfarin to keep his blood from clotting. Valdemar took that seriously. Twice-weekly blood draws, a diet he actually stuck to, and a habit of calling the clinic before he took so much as a new vitamin.

When PlainChart arrived, it made his life a little easier. After every visit, instead of squinting at a photocopy full of abbreviations, he got a clean paragraph on his phone before he'd even started the car. For months he read every line carefully, and sometimes called the clinic to double check something that sounded odd. It was always fine. So, slowly, he stopped calling. The summaries kept being right, and a habit that costs you ten minutes stops feeling worth it once it's cost you ten minutes forty times running for nothing.

Then, in April, Valdemar had a minor procedure on his right knee, a steroid injection for arthritis. Dr. Mordecai Fitzalan's note was short, the kind PlainChart handles without a second look: pain managed with acetaminophen, avoid anti-inflammatories given his anticoagulation. Structured, templated, four sentences long. It sailed straight through the shape-based gate and published itself in under a minute.

Four of these five steps happened exactly as designed. The one that mattered, a check on what the note actually said, was never built into the path.

What Valdemar read on his phone that evening was one confident sentence: "For pain, you can take ibuprofen as needed." Nothing hedged, nothing flagged, nothing to make him think twice. The note's actual instruction, avoid NSAIDs because of the anticoagulation, use acetaminophen instead, a caution wrapped around two abbreviations, hadn't survived the rewrite. PlainChart kept the idea of something for pain. It lost which drug that was, and which drug it wasn't.

Kestrelwyn's team decided what needed a second look before it shipped. Valdemar got whatever that decision produced, and nothing else.

He had no way to know that. Valdemar isn't a nurse. He's never had a reason to learn which painkillers fight with a blood thinner and which don't. The sentence read exactly as sure of itself as every other sentence PlainChart had ever gotten right.

Three days later, picking up his regular warfarin refill, he mentioned it to the pharmacist, almost in passing, that he'd grab some ibuprofen for the knee since the app said it was fine. The pharmacist stopped mid-sentence. Ibuprofen and warfarin together can tear up a stomach lining before either of you notices anything's wrong. She pulled up his chart, saw the anticoagulation flag that had been sitting there the whole time, and told him plainly: not this one, ever, not while you're on warfarin.

Nothing in between these two moments belonged to the product. It belonged to luck, and to a pharmacist paying attention that afternoon.

Nobody at Kestrelwyn found out from a complaint. Ulyana Ravinia, the analyst who runs PlainChart's quarterly safety audit, pulled her usual sample three weeks later, comparing a slice of auto-published summaries against the source notes underneath them. Valdemar's was one of forty-one in that sample where a real caution had gone missing on the way to plain language. Scaled up to the whole month, that put the number at roughly twelve hundred and ten summaries carrying a dropped or softened warning, every single month, since the day the shape-based gate shipped.

Six weeks from a near miss nobody at Kestrelwyn saw, to a routine audit that finally did.

I keep coming back to the meeting where Kestrelwyn decided how PlainChart would route notes for review. Full clinician review of every summary was on the table, and it got ruled out fast: Dr. Fitzalan's own panel alone generates hundreds of notes a week, and reviewing all of them before publish would have meant summaries landing days late instead of before the patient left the building. So the team picked a proxy instead: if a note is short and follows the clinic's standard template, it's probably routine, publish it instantly. That was a reasonable bet. Most short, templated notes really were routine. Nobody had ever checked whether "short and templated" and "contains a real caution" could be true at the same time.

Here's the redo, run properly. Same note, same four sentences. But this time a second, independent check reads it for content, not shape: it sees "anticoagulation" and "NSAID" sitting in the same note and holds it, regardless of how short or templated the note looks. The flagged summary lands in Dr. Fitzalan's queue that afternoon. He confirms the caution in under a minute. PlainChart's plain-language line now reads: "For pain, use acetaminophen. Don't take ibuprofen or other anti-inflammatories, they don't mix with your warfarin." Valdemar gets that version by early evening instead of instantly, four hours later than before, and it's the version that's actually true.

One design let the shape of a sentence decide whether a person could trust it. The other one checked what the sentence actually said.

What I'd tell myself, sitting in that first meeting: we asked how to keep clinicians from reading every note. We never asked which notes a patient could never check for himself, no matter how clearly we wrote them.

GUARD, checked against a sentence Valdemar had no way to verify

This was never really about whether PlainChart's writing was clear. It was clear, that was the whole problem. GUARD is for naming who pays when a fluent sentence and a correct one are impossible for the reader to tell apart.

Shape and risk are two different axes. PlainChart's old gate only ever looked at one of them.

The trade-off, said out loud: routing about nine percent of summaries, roughly thirty-eight hundred a month, to same-day clinician review instead of instant delivery adds real turnaround, typically a few hours, and costs about four minutes of a doctor's time per flagged note, real time pulled from an already full clinic day. Kestrelwyn took that trade on purpose, for the slice of summaries where a wrong line can't be walked back, instead of either reviewing every single summary and losing the speed patients actually valued, or leaving the gate exactly as it was and trusting a shape-based proxy to keep catching a content problem it was never built to see.

And if you want to be sure it really works, try it somewhere else

Same five letters, an insurance claim-denial letter instead of a diagnosis, and this time the unverifiable line is a legal deadline instead of a drug.

Egmont Mutual, an insurer, built Claerity, a feature that rewrites a claim-denial letter's dense policy language into plain English: what got denied, why, and what the policyholder can do next. Sylwester Lundqvist owns Claerity's roadmap the way Berengaria owns PlainChart's.

Same fork Kestrelwyn faced, a completely different kind of business. The line the reader can't check is still the one that decides.

Claerity used the same kind of shortcut PlainChart did. Denial letters that cited a standard, boilerplate exclusion clause, the same paragraph reused across thousands of policies, auto-published instantly. Letters that got more specific, a dollar threshold, a coverage-limit calculation, held for an adjuster to check first. It felt like a safe split, boilerplate language rarely hides a real problem.

Same rank, mapped onto Claerity: size it by what the reader can never check for themselves. A routine internal review found that six percent of the auto-published letters restated the policyholder's appeal deadline wrong, not because the exclusion clause was misquoted, but because a state-specific rider shortened the real appeal window from the standard sixty days to thirty, and the boilerplate paragraph never mentioned the rider at all. A policyholder reading "you have sixty days to appeal" has no way to know a rider changed that number. By the time most of them called to ask a question, the real thirty-day window had already closed. The fix is worded for a different unverifiable line, but it's the same fix: any denial letter whose policy carries a rider affecting deadlines or dollar amounts now gets held for an adjuster, no matter how standard the rest of the letter reads, and the letter shows the policyholder's actual appeal deadline pulled straight from the policy file, not from the boilerplate paragraph's own guess.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: never let a confident AI sentence reach someone who can't check it, without either a citable source or a qualified human sign-off standing behind it.
Cost: no budget this quarter for a full clinician or adjuster review pipeline. Do the cheap version: show the original source wording, the doctor's note, the actual policy clause, right next to the plain-language line for anything that changes what the reader does next. It's checkable by whoever they call next, even if nobody reviewed it first.
The model got better, for real: say PlainChart's summarizer gets retrained and its accuracy on negation-heavy notes climbs to 99.7 percent next year. That's a reason to re-test the content check's threshold and maybe loosen it a little, on purpose, with fresh eval data behind the change, not a reason it should have skipped the check from day one.

Where people run it wrong.
They treat a fluent, confident sentence as proof it's also correct, when fluency and accuracy are two completely different things a model can get separately.
They let a strong overall accuracy number, ninety-six percent, stand in for "verified," when nobody ever asked whether the reader could catch the other four percent themselves.
They wait for a complaint, a lawsuit, or a pharmacist's lucky catch to reveal the gap, instead of asking before launch whether the target reader has any real way to check the output at all.

How to use it live. Before answering, ask out loud: "if this exact line were wrong, would the person reading it ever find out?" If the honest answer is no, that question alone is most of the real answer.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0