How does the AI PM role differ at a company that trains its own models versus one that calls an API?

Coracle sells a customer-support chat box that answers online shoppers in their own language, and ships it two ways: a model it fine-tunes itself, and a thin layer over someone else's API. Ferelith Hawksmoor owns both tiers. This is the six weeks she learned those are not two versions of the same job.

How to answer this, stage by stage

Nobody is grading whether you can define "trains its own model." They're grading whether you can say, out loud, what that actually changes about a Tuesday.

Let's learn

For ten months, Ferelith watched one number on both of Coracle's tiers: how often the chat box gave a wrong answer about a refund. It stayed under 2 in 100. Good enough that Coracle promised its biggest sellers a 97% accuracy floor, and kept it without much drama.

Coracle Core fine-tunes and hosts its own model. Coracle Flash wraps a third-party API. Same chat box, from the outside, to a shopper.

Then, in the same six weeks, both tiers picked up the same kind of mistake at once, for two completely different reasons. Company-wide, the number barely moved: 1.8% wrong to 2.6% wrong. Nobody on the leadership team blinked at that. But Coracle serves eleven languages and dialects, and one of them, Aduna, spoken by a small slice of southern sellers, told a different story. Wrong-refund replies in Aduna climbed from 2% to 14% on Core, and jumped from 2% to 11% almost overnight on Flash.

On Core, she spent nineteen days finding four hundred real Aduna refund chats, checking them by hand, folding them back into the training set, retraining, and re-running the eval before she trusted the fix enough to ship it. On Flash, she spent one day: pin the model back to the version from two weeks earlier, and patch the prompt to spell out the refund rule in plain language.

What it costs at its worst: Coracle's biggest Aduna-speaking seller, a phone-repair marketplace, nearly lost its storefront rating over those nineteen days, because eleven wrongly denied refunds turned into public complaints before anyone thought to escalate past the chat box. On Flash, the quick patch held for three months. Then the vendor announced it was retiring the pinned model version, on sixty days' notice Ferelith had no say in, and she lost eleven more days to an unplanned second migration.

What I would leave alone: the chat widget itself, the box a shopper actually types into, and the handoff to a real person when the bot gives up. Both are identical no matter which tier answers, on purpose, because a shopper should never be able to tell which one they got.

The lesson: training your own model and calling someone else's aren't two difficulty settings on the same job. They're two different jobs that happen to share a title. Staff, page, and interview for the one you're actually running, not the one that sounds more technical on a slide.

Now here is the same thing as a story

Say the short version out loud in an interview. Read this one when you want to feel exactly how two rooms can hold the same bug and demand two completely different afternoons.

Ferelith Hawksmoor has run product at Coracle for three years, and before that she spent four years doing localization QA by hand, reading machine-translated support tickets in eleven languages and marking which ones would embarrass the company if a shopper ever saw them. She can tell a genuinely wrong answer from an oddly-phrased right one in about the time it takes to read one sentence.

She built Coracle's two-tier structure herself, eighteen months ago. Core for the sellers big enough to want a model trained on their own return policies and product names. Flash for the sellers who just needed something live by Friday. For most of that year and a half, the two tiers behaved so much alike that Ferelith ran one weekly review for both: pull the eval scores, glance at the wrong-refund rate, ship a fix if either one dipped below bar. It worked well enough that she stopped splitting her attention between them. One dashboard, one Tuesday-morning half hour, done.

Then, over six weeks in the spring, the wrong-refund rate on both tiers started climbing, and Ferelith's first instinct was the one anyone would have: something in her own logging must have changed.

None of the easy explanations held up. That's exactly what made the real one worth digging for.

It hadn't. She checked the language tags, checked the ticket volume, flat, checked whether some new hire had touched the eval pipeline. Nothing. The company-wide number was barely worth a Tuesday meeting, up from 1.8% to 2.6%. It was a support lead, forwarding three angry emails from one Aduna-speaking seller, that made Ferelith slice the number by language instead of trusting the average. Aduna alone: 14% wrong on Core, 11% wrong on Flash. Hidden entirely inside a number that looked almost fine.

Two things shipped in the same week, and both were described, honestly, as improvements.

She opened Core's training pipeline first, the way she always had. What she found took her by surprise: three months earlier, to cut a rising GPU bill, an engineer had trimmed the fine-tuning set, dropping older, rarely-used examples to shrink the retrain time. Buried in the cut: most of Coracle's Aduna refund examples, a small, unevenly-labeled corner of the data nobody had flagged as load-bearing. The model hadn't broken. It had quietly forgotten something it used to know, one merchant onboarding wave at a time.

Flash told a completely different story, and it took a day to find, not three months. The vendor's release notes for that week said, in exactly these words, "improved reasoning and multilingual handling." It was a real update, and by every metric Coracle didn't track, probably a genuine one. It also, for reasons the vendor never fully explained, made Flash worse at exactly one thing: telling a valid Aduna refund complaint from a scam attempt, a distinction that hinges on two words that sound almost identical in that dialect.

Here's what I mean by two different toolkits. To fix Core, she needed a data lead, four hundred real chats, and nineteen days she had to defend in a planning meeting. To fix Flash, she needed a vendor support contact and one afternoon, and the fix she shipped that afternoon was the rougher of the two: an older model version, not a corrected one, held together by a prompt patch until the vendor could explain what changed.

On Core, Ferelith could turn the dial herself. On Flash, she had to ask, and wait for someone else's answer.

I want to say the real story is that the Flash fix was cheap and the Core fix was expensive. It wasn't really about cost. Ferelith never had a single number that told her these were different kinds of problems. She had a feeling: on Core, the fix lived inside Coracle's own walls. On Flash, the fix lived on someone else's calendar. Two settings. No dial between them.

The decision she'd take back sat in a planning meeting eighteen months earlier, the one where she built one dashboard and one on-call playbook for both tiers, because at launch they behaved identically and splitting the review felt like busywork. It made sense then. A tier is a tier, she remembers thinking, until it very much isn't one.

Run the same six weeks again, with two playbooks instead of one. Core's still slow, because a training-data problem is always going to be slow: nineteen days, no faster, because a rushed retrain is how you trade one bad regression for a different one. But Flash gets caught in three days instead of six weeks, because a second, smaller dashboard tracks provider release notes against Coracle's own eval set every time the vendor ships anything, and "improved reasoning" stops being a claim anyone takes on faith.

What I'd tell myself, back in that first planning meeting: I built two products that would only ever look the same until the day they didn't, and I picked a review process that assumed they always would.

TRACE, run twice on the same bug

Not a way to prove owning the model is harder. TRACE is what forces you to name what actually splits two jobs wearing the same title, instead of guessing from the label on the company's stack.

Two of the four reasons were dead ends. The other two needed completely different fixes.

One question, two very different answers. That's the whole evidence test in one picture.

Three things worth saying plainly, since interviewers push here. Ferelith considered a second option before splitting the playbook: hire a dedicated vendor-relationship PM to own Flash full time, separate from whoever owns Core. She rejected it, because the two jobs still need the same eval set and the same definition of "wrong," and splitting them fully just moves the coordination problem instead of solving it. The AI-specific failure worth naming by name is silent model-version drift: a provider ships an update it calls an improvement, and the failure only shows up on a language or segment small enough to hide inside the aggregate. The guardrail is unglamorous: diff any new model version against your own eval set, by segment, before you let it serve real traffic, never trust a vendor's changelog as the eval. And the trade-off is real and accepted on purpose: Flash's same-day fix used the rougher tool, an older model version patched with a prompt, not a properly corrected one, because a fix that ships that afternoon beats a perfect one that ships in three weeks while shoppers keep getting denied refunds they're owed.

And if you want to be sure it really works, try it somewhere else

Same five letters, a livestock triage line instead of a refund chat box, and this time the thing nobody separates is a vet co-op's own model from a telehealth vendor's quiet cost cut.

Cloverhoof triages livestock health calls for smallholder farmers, in the farmer's own dialect, and decides whether a case waits for a morning vet visit or gets flagged urgent tonight. Ptolemy Braunfels runs product there, and hit Ferelith's exact split eleven months into the job: Cloverhoof Ground, a triage model the co-op trains and hosts itself, for the regions with the compute to support it, and Cloverhoof Relay, a thin layer over a telehealth API, for the regions that came online too recently to justify their own model.

Different reading room, same shape of split. Only one of the two answers to Cloverhoof's own roadmap.

A bloat symptom, described in a regional dialect Cloverhoof serves, started getting under-triaged on both tiers within the same fortnight, sent to "wait for morning" instead of "urgent tonight." Ground's version came from six weeks of new dialect audio slowly diluting an already-thin fine-tuning set. Relay's version came from the telehealth API's provider swapping in a cheaper, faster default model to cut its own costs, a change it announced as a "speed improvement" in a release note nobody at Cloverhoof read closely.

Mapped onto TRACE: the timeline shows Ground drifting for six weeks and Relay breaking in one release. The recut shows the same split, data curation and training-run decisions against prompt and vendor-relationship decisions. The assumption Ptolemy had walked in with, that the co-op's own trained model was obviously the safer of the two because Cloverhoof controlled it, held up on drift speed but not on stakes, since Relay's sudden change hit more farmers per day simply because it served more regions. The cause candidates were the same three axes, reworded for livestock instead of refunds. The evidence test gave the same answer: on Ground, the vet-data lead and Ptolemy retrain; on Relay, Ptolemy and the vendor argue about a changelog.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: the difference isn't how technical the job is, it's who controls the fix and whose clock it runs on, test that directly.
Cost: no budget to interview four backgrounds this quarter. Ask whoever's already in the room one question: last time something broke, did you open a training pipeline or a support ticket?
The model got better, for real: say Flash's vendor genuinely improves Aduna handling next quarter. The deprecation risk doesn't shrink, because Coracle still doesn't control when that model gets replaced again.

Where people run it wrong.
They treat "calls an API" as shorthand for "less technical," and stop checking what that PM actually decides.
They let a flat company-wide average retire the question of whether one segment is quietly cratering underneath it.
They fix the coordination gap by hiring a second, fully separate role, and end up with two people who can each blame the other's changelog.

How to use it live. When an interviewer asks how the AI PM job differs by company type, ask one thing back before answering: "when quality breaks, who's in the room, and what do they reach for, a retrain button or a support ticket?" That question is usually exactly what a TRACE-shaped question like this one is listening for.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0