How do sector regulations change the AI PM's job in healthcare or finance?

What happens when "just describe it in your own words" turns out to have a catch nobody built on purpose? Meridian Health's triage assistant reads a patient's intake note and scores how urgently they need to be seen. Zanele Dube is the nurse practitioner who leads the clinical product team behind it.

How to answer this, stage by stage

Seven stages. The question sounds like a policy essay. The job is to make it one real number, fast.

Let's learn

Say a clinic gives its front desk a tool that reads a patient's intake note and scores how urgently they need care, so the sickest patients get seen first instead of by arrival order.

At launch, the clinic told every staff member the same reassuring line: describe the patient's symptoms in your own words, no special format needed, the tool understands normal language. For months, that promise held up fine. Notes stayed messy, real, and full of the odd detail a patient actually says.

Then, without anyone deciding it, front-desk staff started noticing that short, tidy notes seemed to get faster, more confident scores. Nobody was told to write that way. It was simply the cheapest adjustment available, and people take the cheap adjustment.

At its worst: a patient's intermittent chest pain and arm tingling gets typed up as "chest pain" to fit the fastest form field, scores as a moderate case, and she waits six hours before an on-call physician happens to flag it as worth an urgent look.

What I would leave alone: a short note for a real prescription refill or a sprained ankle is exactly as good as it looks. The fix isn't to make every note longer. It's to stop treating brevity itself as a sign of a clear case.

The lesson: a friendly onboarding promise with no follow-up leaves people to discover, on their own, exactly what the model rewards, and they will learn to feed it that, whether or not it's the honest version of the patient in front of them.

Now here is the same thing as a story

The short version above is what you'd say defending this metric to a clinical director. Read this one for how Zanele actually found the drift.

The triage desk at Meridian's clinic gets a new intake note every few minutes during the morning rush, and for a long while, those notes stayed exactly as messy as real patients actually are.

There was no single bad morning that started the change. It built up slowly, the way a habit does when nobody's watching for it. A staff member would type a full, rambling symptom description, notice the score come back a little slower and a little less confident, and quietly learn that a shorter, cleaner version scored faster next time. Nobody taught this. Everybody discovered it separately, at their own pace, over months.

Five steps, and the second one quietly stopped reflecting the first one honestly.

Zanele found it during her own quarterly model-health review, the kind of routine check that usually turns up nothing. This time, clean-note rate had nearly doubled over five months, with no matching change in how complicated patients' actual visits were.

The third branch is the one nobody designed. It just turned out to be the cheapest path available.

She pulled the note behind that six-hour wait and compared it to the actual recorded encounter. The patient had described pain that came and went, and a strange feeling down her arm. The note on file said two words: chest pain.

One of these four parts had been quietly getting smaller for months, and nobody had a number on it until now.

She thought about the two clocks running underneath the whole system.

One clock rings after the damage is done. The other one had been ringing for months.

With the redesign, the model no longer lets confidence rise from note length or tidiness alone. It calibrates against real complexity signals, multiple symptoms, hedging words like "comes and goes," and flags an ambiguous presentation for a higher tier regardless of how short the note reads. And staff now see, plainly, that a tidy note doesn't score faster just for being tidy.

The old design rewarded a clean note. The new one asks whether the case behind it was actually clean.

I told every new hire the tool understood plain language because it was true and it made onboarding easy. It took my own quiet, routine number-check, not a single dramatic failure, to see that a true promise with no upkeep quietly turns into a false one as people learn what a system actually rewards.

LEAD, in one screenNot a dashboard metric. LEAD is what forces you to find the number that moves before the one you actually care about.

The form can look perfectly filled in while the actual person behind it walks right past the thing that mattered.

The recap, one line per letter: link is a patient who needs urgent care actually getting flagged for it, early signal is clean-note rate drifting five months ahead of the outcome, abuse is staff templating notes to hit the number without truly listening, and decision is auditing a sample the moment the drift shows up, not after an incident forces the question.

And if you want to be sure it really works, try it somewhere elseSame four letters, a trading desk instead of a clinic. This time the input isn't a symptom, it's a trade, and the leading indicator lives in a review queue instead of a note.

Ridgeline Trading runs an AI copilot that flags potentially manipulative trades for a compliance team to review. Henrik Solberg manages that surveillance model.

Mapped onto LEAD: link is a manipulative trade actually getting caught and escalated to a regulator, not the model's raw flag-accuracy on whatever gets reviewed. Early signal is same-day review completion rate on flagged trades, which fell from 95 percent to 30 percent over three months as flag volume grew faster than headcount, long before any regulatory inquiry ever landed. Abuse is analysts clearing a backlog by batch-approving clusters of similar-looking flags, hitting a healthy-looking queue-length number while barely reading half of them. Decision is that once same-day completion drops below a set floor, new flags above a risk threshold get a second reviewer automatically, rather than waiting for a compliance backlog to become a genuine inquiry.

A different desk, a different input, and the same shape of gap: a queue that looked handled without actually being read.

Swap the trigger and it still runs.
Speed: an interviewer caps you at a minute. Say "the leading indicator is clean-note rate against real complexity, not raw accuracy," and stop there.
Cost: there's no budget this quarter for a full complexity-calibration model. Start by simply flagging any note under a certain length for a light human glance, a cheap floor while the real fix gets built.
The model gets better, for real: if the underlying triage model's overall accuracy improves, the pre-editing problem doesn't go away. A more accurate model reading a quietly sanitized note is still reading the wrong input.

Where people run it wrong.
They track the lagging outcome alone and call it "watching quality," when it only tells them after the fact.
They assume a rising "good-looking" metric like clean-note rate is progress, instead of checking what's actually driving it.
They blame the model for getting worse, when the real change was in what the model was ever shown.

How to use it live. When this question comes up, ask yourself what the honest input to the model actually looks like under real time pressure, not in a demo. That's usually where the regulated-sector answer actually lives.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0