Who should be in the room for an AI incident, and who usually is not but should be?

Everything else in the incident waits on the first box. The monitor's read of the four dosed charts is the one nothing else can happen before.

How to answer this, stage by stage

The trap in this question is answering it as a list of job titles, engineering, support, product, comms, when it is actually asking which specific person's absence lets the room make a call it cannot walk back. Eight moves.

Let's learn

Matchlight is a tool that reads a cancer patient's electronic chart, the biomarker test results, the diagnosis notes, the treatment history, and works out which open clinical trials that patient might qualify for.

Before Matchlight, a trial coordinator at Aldermoor Biosciences read each chart by hand, checking it line by line against a trial's written eligibility rules. That took about 40 minutes a chart. A coordinator could screen around 12 patients a day, across a network of six cancer centers.

With Matchlight, that same chart takes about 5 minutes to screen. Coordinators went from 12 patients a day to over 40. Referrals to open trials nearly quadrupled in the first year.

The extra referrals were never the problem. Here is the turn. One trial needed patients confirmed negative for a gene mutation called EGFR, because a patient who actually carried it needed a different, targeted drug instead, not this combination. A model update meant to catch more eligible patients started misreading hedge language in doctors' notes, phrases like "cannot rule out EGFR positivity" or "status pending, treat as positive given family history." On a slice of charts written that way, Matchlight quietly flipped them to eligible.

What that costs at its worst has nothing to do with accuracy scores. A patient dosed on a mismatched trial can face a real, avoidable safety risk, and separately, the trial itself is required to report that kind of error to its safety office and its ethics board inside a set window once it is confirmed. Miss that window and there is no version of "we'll fix it in the next release" that gets it back.

What I would leave alone. Most of what Matchlight gets wrong is a coordinator's problem, not a safety one, a trial ranked second instead of first, a note the model summarized a little oddly. Those don't need the medical monitor at 9pm. Reserve that page for the case where the wrong call changes what actually happens to a patient's body.

The lesson. A model that is wrong about biology does not announce itself as a safety incident. It announces itself as a match, with a confidence score, sitting in a queue that looks exactly like every correct one next to it. The room has to already know which kind of wrong that is, or it will spend its first hour asking the wrong question entirely.

Now here is the same thing as a story

The short version is above. Keep reading if you want to feel why nobody thought to call Dr. Kirtley until the fourth hour.

Kavita Shenoy has run clinical trial operations at Aldermoor for six years, and what she is good at, the thing people actually page her for, is getting six cancer centers to agree on anything inside a day. When Matchlight's automated audit flagged an anomaly at 6am on a Tuesday, a spike in eligible matches for one trial arm that didn't match the historical rate, she was in the incident channel within eleven minutes.

By 6:20, Priam Castellane, the VP of engineering, had a working theory: a model update three weeks earlier had changed how the parser handled negation in clinical notes. By 6:45, his team had a candidate fix ready to test. The room moved fast, and it moved the way incident rooms are trained to move. Confirm the bug. Scope the blast radius. Ship the patch. Nobody in the room was doing anything wrong.

At 7:30, the data lead pulled the list: fourteen patients across six sites had been flagged eligible for the EGFR-negative combination trial since the parser started misreading hedge language. Ten were still in screening, nobody dosed. Four had already started the combination.

That number sat in the channel for eleven minutes before anyone said out loud what it actually meant. Kavita read it three times. Then she paged Dr. Absalom Kirtley, the trial's medical monitor, for the first time that morning.

Pausing enrollment for a few days is a decision you can reverse the moment the monitor clears it. Missing the disclosure window is not.

Dr. Kirtley asked one question nobody in the room had asked yet: what does EGFR-positive mean for someone on this specific combination. The answer was not "a slightly worse outcome." It was a real interaction risk, the exact one the negative-only eligibility rule existed to prevent. He pulled the four charts himself, live, on the call. Two of the four patients turned out to have EGFR status that had never actually been confirmed either way, only hedged in a note the model had flattened into a false negative.

He also said the second thing nobody in the room knew: the moment those four charts were confirmed, a clock had already started. The sponsor's safety office needed notice inside a day. The site ethics boards needed it inside the week. That clock does not wait for a patch to ship, and it does not restart if you find a cleaner explanation later.

The old decision, told as a memory of a meeting. When Matchlight's incident process was written, a year before this morning, the on-call rotation was built entirely from engineering and trial ops, the two teams who owned the tool day to day. Someone asked, in that planning meeting, whether a clinical role should be on the page list. The answer at the time was that clinical staff would obviously be looped in "if it's serious." Nobody wrote down who decides serious, or how fast.

The replay, run forward with the fix already in place: the on-call list pages the medical monitor automatically the moment any incident touches a trial's eligibility criteria, not after the room decides on its own that it's serious enough. Same Tuesday, same 6am spike. Dr. Kirtley is on the call by 6:20 instead of 8:11. The four dosed patients get their safety review inside two hours instead of most of a day, and the sponsor notice goes out with nine hours to spare instead of getting written after the fact and dated honestly late.

What I'd tell myself, back in that planning meeting a year earlier: "if it's serious" is not a paging rule. It is a bet that the room will correctly notice its own emergency, and the whole point of this incident is that it did not, for almost two hours.

ORDER, for who belongs in the room

GUARD would fit if the question were about the patients themselves, the people with no lever over the match they're given. This question is about staffing a room under time pressure, which decision costs the most if you get the order wrong. That's ORDER's job.

Same rank, a gas utility instead of a trial desk

Caldergate Gas & Electric runs PipeSense, a tool that reads sensor and inspection data to flag which pipeline segments are at highest risk of a leak, so field crews know where to inspect first.

Swap the trigger and it still runs.
Speed: an interviewer caps the answer at ninety seconds. Skip straight to naming the medical monitor and the one call that can't be undone.
Cost: the on-call rotation gets cut to save budget. Don't cut the monitor's page to save money. Cut the size of the room around them instead, fewer routine attendees, same one non-negotiable page.
The model got better, for real: suppose the parser update genuinely improved matching everywhere except this one hedge-language case. That still doesn't change who gets paged first. A rare, high-consequence miss needs the same clinical read as a common one, because the cost isn't set by how often it happens.

Where people run it wrong.
They staff the room by who owns the tool, engineering and product, and treat clinical or safety expertise as an escalation path instead of a standing seat.
They wait for the room to decide something is "serious enough" to page the missing expert, instead of writing a rule that pages them automatically the moment the incident touches their domain.
They let the technical fix set the pace of the whole response, so the safety review waits on the patch instead of running in parallel with it.

How to use it live. Say the ranking rule before naming a single role: "I'd rank the room by what's hardest to undo if a decision gets made without that person there, not by who built the tool." That line buys you the room to name the medical monitor specifically, instead of reciting "cross-functional stakeholders" and moving on.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0