Describe the difference between an AI PM and an ML PM at a company that has both.

Almanac is Hollowbrook's chat widget: SaaS companies embed it in their signup flow so a brand-new customer can type a plain question and get unstuck without waiting for a person. Wystan Zabala owns the model that reads each question. Jocasta Ravenscroft owns everything the customer actually lives through: what happens when the model isn't sure, and whether a new sign-up finishes setup without a human at all. Six months after Wystan shipped his best retrain yet, Jocasta's own number had barely moved, and neither of them could say, right away, whose job it had been to notice.

How to answer this, stage by stage

Nobody is grading whether you can describe two job titles politely. They're grading whether you can name the one piece of shared surface between them that nobody owns by default, and say what it costs when it slips.

Let's learn

Almanac is a chat box. A SaaS company drops it into the first screen a brand-new customer sees, and when that customer types a plain question, like why won't my bank feed connect, Almanac either answers it right there or hands the whole conversation to a real person.

Before Almanac, a brand-new customer had three slow ways to get unstuck, and no fourth option that answered back.

Before Ledgerkeep, one of Hollowbrook's customers, had Almanac, only 34 out of every 100 new customers finished setup completely on their own. The rest needed at least one live conversation, usually two, and the ones going it alone took about five and a half days start to finish.

Five steps. The fourth one, checking the score against a cutoff, is the step this whole answer turns on.

Almanac reads a question, scores how sure it is, and checks that score against one cutoff. Score high enough, it answers right there. Score too low, it hands the whole conversation to a person, no guessing involved. In its first version, Wystan's model called the right intent 81 times out of 100 on his own test set. Solo completion jumped from 34 percent to 61. Customers going it alone finished in about a day and a half instead of five and a half days. For six months, that held.

Then Ledgerkeep shipped a new feature, recurring invoice templates, and two months later Wystan retrained the model on six more months of real conversations. His score climbed to 89 out of 100, a real jump, measured against three thousand held-out conversations nobody had trained on. He shipped it, reported the win at the next model review, and moved on to the next intent.

Here's the turn. Jocasta's own number, solo completion, the one that says whether a new customer finishes setup without ever needing a person, went from 61 percent to 63. Eight points on his scoreboard turned into two points on hers.

The reason was sitting one layer down. New customers asking about recurring invoice templates, now about 18 out of every 100 new sign-ups and growing, were finishing setup on their own only somewhere between 26 and 29 percent of the time, month after month, while the average above them quietly inched up. The model wasn't even wrong about that intent very often, it called it correctly about 84 times out of 100. But its confidence on that specific intent averaged around 58, well under the 70 cutoff, so almost every one of those conversations got handed to a person automatically, right or wrong.

At Ledgerkeep's volume, about 108 new customers a month asked something about recurring invoices. Almanac had already gotten roughly 91 of those right. But because its confidence sat under the cutoff, about 80 of those correct answers got handed to Tullia Vaas and her team anyway, every month, on top of the ones that genuinely needed her. Over the five months before anyone connected it to the cutoff, that's roughly 200 extra specialist hours spent re-answering questions the model had already answered correctly the first time.

What I would leave alone: for Almanac's dozen original intents, the ones with thousands of examples each, one shared cutoff is still fine exactly as it is. Splitting every intent's cutoff apart on day one would be solving a problem that, for that group, doesn't exist yet.

The lesson: a model getting better and a product getting better are two separate claims. At a company with both roles, somebody has to own the translation between them, or a real improvement can climb eight points and a customer will never feel a single one of them.

Now here is the same thing as a story

Read the short version above when the interviewer is watching the clock. Read this one when you want to feel what five quiet months actually cost.

Jocasta Ravenscroft has run product for Almanac for three years, since before Hollowbrook had a name worth putting on a slide. In its first year, every time Wystan's team shipped a retrain, she made a habit of sitting in on the review herself, asking one question before she'd sign off on anything: which intents actually moved, and by how much.

Through six retrains, that habit paid for itself. Every time, Wystan's score moved and her own number followed within a point or two, close enough that after a while she stopped needing to prove the connection to herself. It just kept being true.

Nobody decided to stop checking. It just kept being fine, the way a habit fades when nothing ever punishes it for fading.

By the fourth retrain, she'd stopped pulling the confidence charts herself and started reading Wystan's own summary in Slack instead. By the seventh, she wasn't in the room at all, she'd send a thumbs-up on his write-up and move to her next fire. Nobody decided this on purpose.

Then, on an ordinary Tuesday call, Tullia Vaas, who's fielded Almanac's handoffs at Ledgerkeep for three years and can tell a real problem from a busy week before she's finished her coffee, mentioned something in passing. "Not a big deal," she said, "but I keep getting handed recurring-invoice questions where Almanac already wrote the right answer in the transcript before it handed off to me. Is that supposed to happen?"

Jocasta didn't have an answer. That was the whole thing. She hung up and pulled the confidence charts herself for the first time in five months.

The whole gap in one picture. A real improvement ships, gets reviewed by one person, and nothing catches whether it can actually reach anyone.

The decision Jocasta would take back reaches back to Almanac's first launch week, a fifteen-minute meeting, a dozen intents on the whiteboard, all roughly the same size. Someone asked whether the cutoff should differ by intent from day one. The answer, reasonable at the time, was to ship one number and split it later if it ever mattered. Nobody wrote down whose job later would be.

Run the same retrain again, with the one thing they added afterward: every retrain now ships with a shared table, confidence against accuracy, one row per intent, both their names on the review. Recurring invoice templates would have flagged itself in its first week, thirty points of gap sitting right there in a shared document instead of buried in two separate dashboards. The cutoff for that one intent alone would have dropped to something realistic, and solo completion for that slice would likely have climbed toward 70 percent within the same quarter, not sat near 28 for five months.

Two true dashboards, read by two different people, that quietly stopped agreeing with each other for five months.

What I'd tell myself, back in that fifteen-minute meeting: a shared dashboard between two roles was never the same thing as a shared job. Nobody had actually agreed whose job it was to notice when the two numbers quietly stopped agreeing with each other.

PICK: whose job the gap between two dashboards actually was

Not a diplomatic way to describe two job titles. PICK forces you to say, in one sentence, where the model's job ends and the product's job starts, and which side of that line is expensive to get wrong.

One kind of wrong gets a support ticket inside a week. The other one just runs, unnoticed, until someone happens to ask sideways.

Three things worth stating directly, since the real judgment sits here. Two alternatives got rejected before landing on a shared, triggered check. Giving Wystan sole ownership of the cutoff was rejected, because it's a product lever, not a modeling one, and he has no visibility into which segments the business actually needs to win. Giving Jocasta sole ownership was rejected too, because she can't safely pick a number without knowing what the model's confidence distribution structurally means per intent, she'd be guessing that 0.70 sounds safe without knowing new intents run quieter for reasons that have nothing to do with whether they're right. The AI-specific failure worth naming by name is silent threshold miscalibration after a retrain: a routing cutoff tuned against one model's confidence distribution quietly stops meaning the same thing once that distribution shifts, especially for any intent with a thinner training set, and a rising headline score gives no warning that it's happened. The guardrail is the shared table, confidence against accuracy, per intent, attached to every retrain, not an annual audit. And the trade-off is real: dropping the recurring-invoice cutoff from 0.70 to about 0.55 would likely take that segment's solo completion from 28 percent toward something near 70, in exchange for a few more wrong answers reaching customers directly, maybe one in five instead of one in six on that specific slice. Ledgerkeep accepted that trade on purpose, because a wrong answer on a recurring-invoice question is a two-minute correction, not the kind of mistake that can't be undone.

And if you want to be sure it really works, try it somewhere else

Same four letters, a hospital instead of a signup flow, and the fragile edge isn't a chat cutoff. It's a mandatory second read.

Fallowridge Diagnostics builds Lucerna, a tool that reads chest and limb X-rays for fractures a radiologist might be moving too fast to catch, and flags anything worth a second look. Placida Rutherford owns Lucerna's detection model. Lysander Padgett owns what a radiologist actually sees on their worklist, including the rule that decides when a case needs a mandatory second read before it's signed off. Dr. Sibeal Cottrell reads a large share of Amberleigh General's pediatric caseload, and for two years, Lucerna made her mornings noticeably shorter.

Different building, same shape of question. A chat cutoff became a mandatory-second-read rule, and the gap transferred without changing.

Placida's team retrained Lucerna last spring. Sensitivity, the share of real fractures it actually catches, climbed from 91 to 95 out of 100 on their held-out set, a real, defended win. For adult fracture reads, that improvement reached radiologists directly: the share of adult cases needing a mandatory second read dropped from 40 percent to 22, real time back in real mornings. For pediatric wrist fractures, a newer, smaller category in Lucerna's training data, the story never moved. The retrained model was already getting those right about 88 percent of the time, close to the adult numbers, but its confidence on that category still sat under the fixed second-read threshold, exactly like before the retrain. A hundred percent of pediatric wrist cases still required a second read, whether the first read needed one or not, and nobody had rechecked whether that threshold still made sense once the model actually improved.

Same rank, different lever, mapped straight onto PICK: the position is the same shape, Placida owns the model as a built thing, Lysander owns what a radiologist actually lives through, and the second-read threshold belongs to neither by default. The impact splits the same way: Placida can ship a real sensitivity win that never reaches a pediatric radiologist, and a vendor rollout that oversold a capability across every fracture type equally would have been the loud, fast-caught version of the same mistake. The cost asymmetry lands the same place: a vendor overclaim gets caught inside a week by an unhappy contract review; a stale, category-blind threshold just quietly keeps every pediatric case in the second-read queue, indefinitely, with no complaint ever filed, because nothing about that queue looks broken from the outside. And the kill criteria transfer directly: any fracture category sitting fifteen points or more off the average second-read rate, for two straight review cycles, forces the same joint check, whatever the headline sensitivity number says.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: the model's his, the product's hers, and the line between them needs a name on it or a real win vanishes before anyone feels it.
Cost: no budget this quarter for a shared calibration dashboard. Ship the manual version first, a spreadsheet review at every retrain, and automate it once it's proven it catches something real.
The model got better, for real: say the score climbs from 89 to 95 next quarter. Still run the shared check. A higher score has never once told you whether the improvement reached anyone.

Where people run it wrong.
They treat a rising model score as proof the product got better, and stop checking there.
They let whichever side notices a gap first quietly own the whole fix, instead of naming who's actually supposed to catch it next time.
They wait for a loud complaint to trigger a review, when the expensive failures are exactly the ones that never complain at all.

How to use it live. Ask the split question before naming a fix: "when this model gets better, whose job is it to check that the product actually got better too?" That question alone buys real thinking time, and it's usually exactly what the interviewer wants to hear you ask.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0