Tell me which AI PM variant you want and defend the choice against the other three.

Undercroft is Quillhaven Systems' enterprise search tool: type a plain question, get an answer pulled from every wiki, contract, ticket, and old Slack thread the company owns. Four real jobs sit around it: applied, platform, infra, and research. Folasade Osunkoya is in the final round for the infra seat. Faramond Winterhalter, running the panel, has one line on Folasade's resume circled in pen. She's about to ask the question that line is there to answer.

How to answer this, stage by stage

Nobody is grading whether you can name four job titles. They're grading whether you can hold one of them under three people, one at a time, trying to talk you out of it.

Let's learn

Undercroft is Quillhaven Systems' enterprise search tool. An employee types a plain question. Undercroft pulls the answer from every wiki, contract, support ticket, and old Slack thread the company owns. It answers about 900,000 employee questions a week, across every customer that runs it.

Four real jobs sit around one product. Each one is a genuine seat, not a made up title to fill out an org chart.

An applied PM owns the answer box: the words a searcher reads, the confidence ranking, the eval set that scores whether an answer is actually right. A platform PM owns the retrieval API three other Quillhaven products already call: a ticketing tool, a Slack bot named Ask Undercroft, and a CRM plugin. An infra PM owns the embedding index underneath all of it, the thing that turns a document into a list of numbers a computer can compare. A research PM chases a better way to find the right document at all, instead of the way Undercroft uses today.

Folasade Osunkoya is in the final round for the infra seat. Before this, she spent two years at Vaultkeep Discovery, a legal document search company, running exactly this kind of index for 1.2 million case filings. The panel is asking her this question because of one line on her resume: eleven weeks, once, the index looked completely fine and wasn't.

The third box is the one nobody was watching. That's the whole story in one picture.

Vaultkeep Discovery swapped its embedding model to cut inference cost, from about $38,000 a month to $19,000. Every document that arrived through the live intake path, new filings, about 1,900 a day, got re-embedded under the new model automatically. But 410,000 older filings, about a third of the whole corpus, had come in years earlier through a separate bulk import pipeline. Nothing in that pipeline knew to check which model had embedded a document. It just assumed "already embedded" meant "done."

The gap didn't announce itself. Comparing a new model's numbers to an old model's numbers is like comparing two different rulers and hoping they happen to agree. Nothing in the output said so. The search just kept returning a confident top result every time, and for a third of the corpus, it was quietly the wrong one.

What it costs at its worst: across those eleven weeks, queries a week quietly landing in the stale third of the corpus climbed from 340 to 825, and none of it showed up on the dashboard the team actually watched. Vaultkeep issued about $210,000 in service credits to twelve affected customers, and lost one account worth $86,000 a year. Roughly $296,000, and nobody saw a single number move on the screen they trusted, because that screen was built to sample the part of the corpus that was never broken.

What I would leave alone: a one-off sandbox instance a team spins up for a hackathon, throwaway data nobody depends on for real work. It doesn't need this kind of versioning rigor. Let that one stay simple; nothing real is riding on it.

The lesson: a gap in coverage isn't a missing feature. It's a debt that sits quietly until the one week a real question happens to land exactly where nobody was looking.

Now here is the same thing as a story

The short version above is what you actually say in the room. Read this one for the two years, then the eleven weeks, that actually built the answer.

The resume sitting in front of three panelists at Quillhaven Systems had one line circled in pen: "led embedding index recovery, Vaultkeep Discovery, eleven weeks." Faramond Winterhalter had circled it herself before Folasade Osunkoya ever sat down.

Folasade had spent two years at Vaultkeep Discovery keeping the machinery under its legal search product running: the pipeline that turned scanned court filings into an index a lawyer could actually search. She was good at it in the boring way that matters, the kind of good where nothing breaks long enough that nobody thinks about the job at all. New filings landed, got embedded, got indexed, and showed up in search within the hour. By her second year, she mostly watched dashboards that stayed green.

Then Vaultkeep swapped its embedding model to cut inference cost. Folasade's team retested the live intake path, the 1,900 new filings a day, and it looked perfect. Nobody retested the bulk import pipeline, the one that had carried 410,000 older filings in from an old acquisition, because nobody had ever needed to touch it since it was built. It had no owner, no test, and no reason, on paper, to be any different from the rest of the index.

It was different. The old filings kept their old model's embeddings. New queries got the new model's embeddings. Nothing about the mismatch looked broken from the outside. The search kept returning a confident top result every single time. It was just, for a third of the corpus, quietly the wrong one.

Eleven weeks between the migration and the first real question that happened to land in the wrong third of the corpus.

It surfaced on a Thursday, in a business review nobody expected to be memorable. A compliance associate had spent forty minutes searching for a specific precedent ahead of a regulatory audit and gotten nothing back. She found the case herself, through a competitor's tool, and asked, in front of her whole team, why Vaultkeep's own search had missed something that had been sitting in Vaultkeep's own corpus the entire time.

Folasade got the ticket that afternoon. It took two days to find the actual cause, because the first instinct was to blame the new model's quality, not its coverage. The break wasn't in what the model knew. It was in what had never been re-embedded at all.

The decision Folasade would take back sits in a fifteen minute standup, eighteen months before any of this. An engineer building the bulk import pipeline mentioned, almost as an aside, that it didn't tag which model had embedded each vector. Someone suggested adding one. It got shelved, because at the time there was exactly one embedding model in production, and tagging a single, unchanging value felt like busywork nobody had a sprint for.

Run the same eleven weeks again, with that tag in place and a shadow eval sampled across every ingestion path, not just the live one. The new model ships on the same Tuesday. The shadow eval flags the bulk import path within a day: recall on that segment craters to 54 percent before a single real customer query ever sees it. The fix ships before the compliance associate's search ever comes up empty. Two days, not seventy seven.

That's the whole story behind one circled line on a resume. In the room, across the table from Osazee, Kentigern, and Uzoamaka, it took her about four minutes to say. It took Vaultkeep Discovery eleven weeks and roughly $296,000 to teach her.

PICK, said about your own seat instead of somebody else's feature

Not a way to rank four job titles by prestige. PICK is what forces you to say which seat you want, then defend it against the best case for the other three, one at a time, without hedging and without flinching.

Four real reasons, four real seats. Only one of them is backed by a scar she can point to.

One mistake shows up on a calendar the same week. The other one hides inside a search that looked, on the surface, like it was working.

Infra sits in the one corner nobody applauds and everybody pays for, if it's wrong. That's exactly why she wants it.

Three things worth stating directly, since the real judgment sits here. The alternative worth naming and rejecting isn't only the other three seats, it's also "let each product team own its own slice of embedding versioning independently." That option loses because embedding spaces have to stay globally consistent: you can't have applied's eval set say the index is fine while platform's three downstream consumers are quietly pulling mismatched results from a corpus slice that eval set never touched. The AI-specific failure worth naming by name is an embedding-space mismatch hiding behind confidence: a model migration leaves part of a corpus embedded in an old vector space while new queries arrive in a new one, and nearest-neighbor search returns a result anyway, with nothing in the output flagging that the comparison never meant anything. The guardrail is a version tag on every vector, model id plus a checksum of the embedding config, and a shadow eval sampled across every ingestion path, not just the live one, run before any embedding model ships. And the trade-off is real and accepted on purpose: that shadow eval adds roughly a week to every model migration, on purpose, because a fast migration that silently breaks a third of the corpus costs far more than the week anyone saved.

And if you want to be sure it really works, try it somewhere else

Same four letters, a vet clinic instead of a law firm, and this time the honest answer isn't infra at all.

Clarivue, built by Vetraine Diagnostics, reads X-rays and ultrasounds for 340 vet clinics and flags likely fractures, tumors, and foreign objects for a vet to confirm. Ozioma Amberidge is interviewing for AI PM, Platform, there. Corentine Wrackham, Vetraine's Head of Applied Diagnostics, pushes the same question from the other side of the table: why not applied, where you'd catch a missed fracture the same day a vet does?

Different office, different scan, a genuinely different honest answer this time. That's the point of the method, not a flaw in it.

Ozioma's answer runs the same four letters and lands somewhere different, on purpose. Position: the platform seat, owning the shared scoring contract that Vetraine's imaging app, its referral portal, and its insurance-claims tool all call. Impact: not choosing applied gives up seeing a missed fracture the same day it happens; not choosing infra gives up ownership of the raw scan-to-vector pipeline; not choosing research gives up the newer, multi-view architecture Vetraine's research team is testing. Cost asymmetry: Ozioma is openly not optimizing for the fastest individual bug fix. A contract change ships slower, on purpose, because it has to be checked against three consumers instead of one. Kill criteria: if any single consumer's needs diverge so far from the shared contract that platform decisions stop being able to serve all three at once, that's the sign to split the API, not to keep forcing one contract to fit three jobs.

Same method, a different, equally defensible answer. That's the whole point: PICK doesn't hand out one universal right seat. It makes you defend the one you actually want, on the evidence you actually have.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Skip straight to it: name the seat, name the one scar or reason that earns it, name one thing you're giving up, done.
Cost: no time to build the full case live. Lead with the number: "I broke this once, it cost $296,000 and eleven weeks, and I don't want to do that again" earns you the follow-up question that lets you say the rest.
The model got better, for real: say the new embedding model genuinely outperforms the old one on every benchmark. Keep the shadow eval anyway. A better model can still land its new documents in a different corner of vector space than the old one, and "better" was never the same claim as "compatible."

Where people run it wrong.
They pick the seat that sounds most impressive instead of the one they can actually defend with evidence.
They wave off the other three seats as a group instead of naming what's genuinely lost by not choosing each one.
They give a position with no kill criteria, which reads as rigid instead of confident the moment a good interviewer pushes back twice.

How to use it live. Before answering, ask yourself one question: what's the one scar or fact that makes this pick actually mine, not just a job description I liked the sound of? If you can't name it, you haven't picked yet, you've just recited an org chart.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0