Describe the handoff artifact between a PM and an ML engineer starting a new feature.

Vestibule is where people rehearse interview answers out loud, against an AI interviewer, the night before it counts. Cinzia Vachon is the product manager who writes the one page document that kicks off every new model feature there. Wangechi Kiptoo is the engineer who builds from it. Fourteen months and thirty feature launches in, the process was fast and clean, right up until one document left out a single case, and then, trying to fix that, put in far too many.

How to answer this, stage by stage

Nobody is grading whether you can define "handoff." They are grading whether you can point at one real document and say, out loud, what belongs in it and what doesn't.

Let's learn

Vestibule is the app people open the night before a job interview, to practice answering out loud against an AI interviewer that talks back.

Before Vestibule had any standard way to hand off a new model feature, Cinzia Vachon and Wangechi Kiptoo started each one with a call and a Slack thread. It worked, mostly, but scope kept slipping mid build. Across the first ten model features, the team lost an average of two and a half weeks per feature to rework, because half the assumptions were only ever said out loud, once, in a meeting nobody wrote down.

Fourteen months ago they started writing one page before any model feature began: the Callback Brief. The real problem, three example inputs, one line for good enough, one line for budget. Thirty feature launches since, and rework dropped to under two days on average.

Five parts, one page. Everything the brief owes the engineer, nothing that belongs to her instead.

Here's what that brief actually looked like, the first time it went out, for a feature called the Readiness Score.

Read it again and look for the gap. Nowhere on that page is a transcript with words missing because a call dropped or a mic cut out. Nothing tells the model what to show when it isn't even sure it heard the whole answer. Three examples, and the fourth one, the one that would actually matter, was never written down.

The v1 brief got this part right, on accident, by never mentioning the fallback at all. That's the same gap as leaving out the example. Nobody had actually decided.

What that costs at its worst: for five weeks, the missing case sat there quietly. About 1.4 percent of Readiness Score sessions had a transcript the system's own speech to text step wasn't sure about, and the model scored every one of them anyway, full number, full confidence. Then a candidate named Ferdous Rahman lost six seconds of audio to a dropped connection mid answer, twenty minutes before a real interview, and got back "Readiness: 24%. Weak. Unclear structure." He hadn't rambled. He'd been cut off.

What I would leave alone: the one page length. Doubling the brief to twelve pages to be safe would only slow down thirty launches a year to guard against one kind of gap. Keep it short. Just make sure the short version has the right five things on it, and nothing more.

The lesson: a brief fails the same way whether it says too little or too much. Both versions take a decision that belonged to someone else, the engineer's how, or the missing case's shape, and quietly decide it without anyone noticing they'd decided anything.

Now here is the same thing as a story

The short version above is what you actually say out loud. Read this one for the meeting where the case got left off, and the five months it took to feel that mistake's twin.

The brief is one page, printed for a design review that starts in ten minutes. Cinzia Vachon has read it through twice already. She built the Callback Brief template herself, fourteen months back, after watching two features slip their timeline over an assumption nobody had written down. Since then she has run twenty nine of them through it, and every single one shipped close to on time.

She is good at this. Ask her what a brief needs and she'll tell you before you finish the question: the real problem, three or four honest examples, a number for good enough, a budget, and a line for what's out of scope. She built the habit of listing the ugly examples first, the ones that go wrong, because a clean example teaches an engineer nothing they didn't already assume.

For the first year, that habit held. Then, slowly, across a run of four features that all shipped clean, the example list started getting written the night before instead of the week before. Three good examples became two good examples and one she typed from memory in the review itself. Nobody noticed, because nothing had gone wrong yet, and a habit that hasn't cost you anything doesn't feel like a habit slipping. It just feels like efficiency.

The whole brief only works if each side stays on their own half. Cinzia had never once, on purpose, written Wangechi's half for her.

The Readiness Score brief was one of the fast ones. Three examples, all clean enough to write in an afternoon. Wangechi Kiptoo built it in nine days, shipped it, and it worked exactly as written. Score, one line of feedback, back in under four seconds, agreeing with a human coach's band on the labeled set 87 percent of the time. Better than the bar. Everyone moved on.

Nobody had told Wangechi what to do with a transcript that was missing words. Nobody had told her not to score it either. So when the system's own speech to text step came back unsure, low confidence on a stretch of the audio, she made the call herself, alone, the way an engineer has to when a brief goes silent on a real case: score it anyway. It was the reasonable choice. Refusing to answer felt worse than an ordinary product, and nobody in that first review had raised the case for her to weigh against.

Nobody decided, on any single day, to score a broken transcript. The brief just never told anyone not to.

Ferdous Rahman had a real interview in twenty minutes. He was running through one more answer, "Tell me about a time you had to deliver bad news to a client," when his connection dropped for about six seconds, mid sentence. Vestibule's own transcript of that stretch was three words and a gap. The Readiness Score didn't know that. It only knew a transcript, and it did what it had always done: it scored it. Twenty four percent. Weak. Unclear structure.

He read it twice. He closed the laptop. He didn't practice again before the real thing.

We did not lose Ferdous's confidence because a model got a number wrong. We lost it because nobody had ever told the model what to do when it didn't have enough to go on.

Wangechi pulled the logs that afternoon, at Cinzia's ask. She found the gap in about twenty minutes, the same afternoon it was reported, and traced it back to a case the original brief had simply never shown her. About 1.4 percent of sessions over the prior five weeks had the same shape: a shaky transcript, scored anyway, in plain sight in a table nobody had reason to query.

Here is the decision Cinzia would take back, and it happened in a ten minute meeting, hers, not Wangechi's. Writing three examples felt complete because three examples covered every case Cinzia had personally imagined. The fourth case, a transcript she couldn't trust, wasn't hard to imagine. It was just the kind of case that feels too obvious to write down, right up until it's the one that reaches a real person.

What she did next is the part worth naming honestly, because it looked like the responsible fix. The next brief, for a feature called the Structure Flag, meant to catch answers that ramble without a clear shape, came back fourteen pages long, more than twice the length of the problem it was solving. It didn't just name the case. It named the fix: if a transcript runs under forty words in its first thirty seconds, flag it, using a fixed word count rule, written into the brief itself. Do not train a model for this, it said. We already know how.

Wangechi built exactly that. It worked, technically. It also caught barely a fifth of the answers a human coach would have flagged as genuinely unstructured, because word count alone misses a fluent ramble and over flags a short, sharp, correct answer.

Nobody found that out for five months. During a slow week, Wangechi tried a small classifier on the side, trained on the same labeled set, just to see. It caught 68 percent of the coach confirmed cases against the rule's 22. The better approach had been available the entire time. The brief had simply never let anyone reach for it, because the brief had already decided, on Cinzia's page, not the engineer's.

One alternative got seriously discussed in that postmortem, and it's worth naming because it also looked tempting: throw out the template entirely and go back to a fully open brief, no examples at all, trust the engineer to find every case herself. It lost, for the same reason the original gap happened in the first place. An open ended brief with no examples doesn't stop a thin brief, it guarantees one. The fix was never more freedom or more control. It was drawing the line in the same place every time: show her the case, and never write her answer for her.

Run the same week again, with the fixed brief. Ferdous's dropped connection produces the same gap in the transcript. This time the brief already named the case and the exact line to show. Vestibule returns "Not enough signal. Try that one again." within four seconds, no number, no verdict. He tries it once more, seven minutes later, gets a real score, and walks into his interview having practiced one extra time instead of zero.

What Cinzia would tell herself, back in that first ten minute meeting: the brief doesn't need to be longer to be right. It needs the one case you're most sure doesn't need explaining, because that's exactly the one you never got around to writing down.

PICK: what the brief owes, and what it doesn't

Not a ranking of thin versus thick. PICK only earns its place here if you can point at the exact line a brief should never cross, in either direction.

One mistake shows up in a review, in daylight. The other hides inside a working feature that looks, from the outside, like it shipped fine.

Missing pieces cluster cheap and fast, bottom left. One kind of mistake sits alone in the expensive, slow corner: telling the engineer how.

Three branches, decided once, on the page, before the engineer ever has to make the call alone at nine at night.

The trade off worth saying out loud: checking a transcript's own confidence before scoring it adds about 200 milliseconds and a fraction of a cent per answer, comfortably inside the four second, half cent budget the original brief already allowed. The fix cost nothing structural. It was a case that had never been named, not a resource anyone had to fight for.

And if you want to be sure it really works, try it somewhere else

Same four letters, an insurance claim instead of a mock interview, and this time the missing case is a blurry photo instead of a dropped call.

Claimhurst triages incoming auto claims, deciding same day which ones need a human adjuster and which can fast track. Zohreh Boztepe runs product there, and the handoff brief for its fraud likelihood score had the same shape of gap. Three example inputs: a clean photo of real damage, a clean photo staged too neatly, a photo that didn't match the claim's own description. None of them was a blurry, rotated, half cropped photo, the kind an actual customer takes standing in the rain.

Same failure as Vestibule's, one industry over. A case never shown, an engineer guessing alone, a number that looks exact and isn't.

An engineer facing the same silent gap made the same reasonable call: score every photo, regardless of quality. A genuine customer's blurry, sideways photo of a fender bender came back "82 percent likely staged," and her claim sat in review for three weeks before anyone looked at the actual image. About 900 claims run through Claimhurst a day, and for roughly six weeks, close to 2 percent of them had photo quality poor enough that the score shouldn't have been trusted, and was anyway.

Mapped onto PICK: the position holds its shape, name the case, don't dictate the fix. The impact splits the same way, a thin brief lets an engineer guess wrong on a real customer's claim, a prescriptive one would have handed her a fixed blur detection threshold that might miss the next kind of bad photo entirely. The cost asymmetry lands the same too, the missing example got caught and fixed inside a day once someone finally asked about photo quality, while a dictated rule would have taken months to reveal a better approach nobody was allowed to try. The kill criteria transfers without changing a word: can the engineer read this and know what done looks like, without being told how to get there.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: show the messy inputs, write the eval bar, name the budget, and never write the fix. That's the whole brief.
Cost: no time for a design review this sprint. Fine, but at minimum ask one question of every brief before it ships: what's the ugliest input this could ever see, and is it on the page.
The model got better, for real: say the underlying model's overall accuracy improved on a fresh benchmark. Doesn't change the argument. A better model still needs to be told what a low confidence input should look like on the page, because that's a product decision, not something a benchmark score answers for you.

Where people run it wrong.
They treat a short brief as automatically risky, and pad it with implementation detail that was never theirs to write.
They assume a case is too obvious to need an example, which is exactly the case that turns out to matter later.
They fix a thin brief by overcorrecting into a fully dictated one, instead of asking what specific thing was actually missing.

How to use it live. Before answering, picture the actual document. One page, five sections, the ugliest example you can think of on the page, and the fix nowhere on it. If you can describe that page out loud, you've already answered the question.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0