An AI platform PM's users are internal engineers. How does that change discovery?

Loomtext is Thistlecroft Systems' internal tool. Point it at a service's code and it reads the files, then drafts the README, the endpoint list, and the architecture notes an engineer would otherwise write by hand. Fennela Wexleigh owns Loomtext's product calls. She spent five years running discovery for a consumer shopping app before this job, and she ran Loomtext's discovery the same way. Threnody Marchetka, senior backend engineer on the billing team, sat through one of those sessions, said all the right things, and then quietly built a way to never use Loomtext again.

How to answer this, stage by stage

Nobody's grading whether you can spell out FLIPS from memory. They're grading whether you'll notice the moment a discovery script built for a stranger meets someone who could already write the spec, and change the script.

Let's learn

The hook Threnody wrote is twelve lines long. It does one thing: strip anything Loomtext generated out of a pull request before that pull request can merge.

Loomtext is Thistlecroft Systems' internal doc writer. Point it at a service's code and it reads the files, then drafts the README, the endpoint list, and the architecture notes an engineer would otherwise write by hand.

Before Loomtext, writing that kind of documentation from scratch cost an engineer about two hours a service, on top of actually building the thing. Most people put it off. The average page on Thistlecroft's internal wiki sat fourteen months out of date.

With Loomtext, a draft is ready in under ninety seconds. Teams stopped writing docs by hand almost overnight. Inside two months, thirty-one of Thistlecroft's forty internal services had a Loomtext draft as their main reference.

Four steps. The last one, what an engineer actually does with a wrong answer, is the step this whole answer turns on.

Here's the turn. The mistakes Loomtext makes on a big, unfamiliar file are not the real problem. The real problem is what an engineer does the first time it gets confidently, specifically wrong about her own code. She doesn't file a bug. She builds around it, quietly, and nobody on the product side finds out unless they happen to go looking.

Threnody's billing service has one file, the settlement module, that runs about forty thousand lines, somewhere near three hundred and ten thousand tokens. Loomtext reads two hundred thousand tokens in a single pass. Past that line, it can still write a confident paragraph about a function it never actually saw.

At its worst, an engineer who's been burned once stops trusting Loomtext for anything in her repo, writes her own tooling to keep it out for good, and the product team finds out months later, if ever, from a Slack thread they weren't part of.

Same forty five minutes on the calendar. Two very different scripts, built for two very different customers.

What I would leave alone: services whose whole codebase sits under two hundred thousand tokens never had this problem. Their wrong-answer rate held at two percent the entire time. I wouldn't touch their discovery script or their docs pipeline chasing a failure that lives somewhere else entirely.

The lesson: a discovery script that works fine on a stranger can go quietly wrong on someone who actually knows the answer, if it never gives them a place to put the number they're holding.

Now here is the same thing as a story

The short version above is what you actually say in the room. Read this one when you want to feel exactly what a twelve-line hook cost, and why nobody noticed for six weeks.

Fennela Wexleigh ran product discovery for five years before Thistlecroft, on a shopping app used by people who'd never read a line of code in their life. She was good at it. Sit her across from a stranger for forty five minutes and she'd walk out with three real insights and a person who felt heard. That was the whole craft, and she was genuinely excellent at it.

She brought the same forty five minute script to Loomtext. Open questions. How do you feel about documentation today. Walk me through your morning. What would make this easier. It had worked on every team she'd ever talked to, so there was no reason to think it wouldn't work here too.

For the first five months, it looked like it was working. Engineers were polite, thoughtful, generous with their time. They said documentation would help a lot. They said the current wiki was rough. Fennela wrote it all down, synthesized it into a roadmap, and shipped features against it. Adoption climbed. Nobody complained.

Nobody decided any single day to stop trusting Loomtext. It thinned out in three quiet beats, until the fourth one had nothing left to check.

Threnody Marchetka sat through one of those sessions in March, same script, same forty five minutes. She said documentation would help, especially for the settlement module, which was a nightmare for new hires. She meant it. She also never once said a number, because Fennela never once asked her for one, and by then Threnody had learned that answering with a number just got the conversation redirected back to how something felt.

Loomtext shipped comment generation for the billing repo in April. For the first two weeks, Threnody read every comment closely. It was good, mostly, on the smaller files. By week three she was skimming. By week five she'd stopped opening the generated comments at all, because checking them properly took nearly as long as writing her own, and the ones she did check kept being fine.

Then, on one Tuesday in June, a comment described a function in the settlement module that did the opposite of what the code actually did. Confidently. In the same clean voice as everything else Loomtext had ever written. A junior engineer read it, trusted it, and shipped a fix based on the wrong description. It got caught in review, an hour before it would have touched real settlement data.

Threnody didn't file a ticket. She didn't email Fennela. She opened her own terminal that same afternoon and wrote twelve lines: a pre-commit hook that strips any Loomtext-authored comment out of a pull request on the billing repo before it can merge. She told her own team about it in standup. She never told product.

Fennela found out by accident, six weeks later, scrolling through an old Slack thread while researching something else entirely. Fourteen pull requests on the billing repo had been silently stripped of Loomtext's comments in that time. Nobody had complained. Nobody had filed anything. Threnody had simply, quietly, stopped.

I want to say the problem was the settlement module being too big. It is too big, for one pass. But that's not really the story. Threnody never had a dial to turn down slowly. She had a switch, and the day it flipped, it stayed flipped, because nobody had ever given her a way to say what she actually knew.

Here's the decision I'd take back. Six months before launch, in the kickoff for Loomtext's discovery process, someone suggested reusing the interview script UX research had already built for the shopping app, since it had a good track record and standing up a new one from scratch felt like overkill for one internal tool. Nobody in that room thought to ask whether an engineer would actually answer differently than a shopper would. It was a reasonable thing not to ask, for a room full of people used to interviewing shoppers.

Run the same Tuesday again, with a technical pre-read in place from day one. Before that March session, Threnody gets a one page form: how big is your biggest file, in tokens, and how wrong can a comment be before it's worse than useless to you. She writes back the same day: the settlement module is around three hundred and ten thousand tokens, and anything over two percent wrong on code Loomtext hasn't actually seen isn't worth shipping to her team. Fennela has the real spec in March instead of a warm anecdote. Loomtext ships with a chunked pass for anything over the context window, and a plain flag on any comment written from a partial read. Threnody never writes the hook. She spends that Tuesday afternoon reviewing a real pull request instead of writing one to keep Loomtext out.

One version of that spring ends in a hook nobody found for six weeks. The other ends in a flag on one paragraph, and an engineer who trusts the other ninety nine.

What I'd tell my past self, the one who reused a script built for shoppers: a discovery interview is a container for whatever the person across the table is capable of giving you. Ask a shopper for a number and you get silence. Ask an engineer for a feeling and you get a warm, useless answer, and you'll never notice it's useless until she stops answering you at all.

FLIPS, or what a discovery script owes someone who could already write the spec

Not a trick to sound technical. FLIPS is what forces you to notice which of your interview questions only work on someone who can't hand you a real number back.

Four setup and payoff letters, and one hard question sitting in the middle of all of them.

Fennela's discovery habit was never a dial she nudged. It was a switch with two positions, and Threnody flipped it herself, quietly.

The whole answer, in one picture. Nobody designed a dial. Everybody got a switch.

Three things worth being direct about, since this is where the real judgment sits. We considered the cheaper fix first: just tell Loomtext to say nothing at all below some confidence score, instead of building a chunked pass. Rejected, because that leaves the settlement module with zero documentation instead of partial, flagged documentation, and an engineer with nothing is worse off than an engineer with something clearly marked unverified. The AI-specific failure worth naming is a context-window truncation hallucination: past two hundred thousand tokens, Loomtext doesn't refuse to answer, it writes a plausible, confident paragraph about a function it structurally could not have read. The guardrail is the chunked pass itself, paired with a flag on any comment built from a partial read, checked against the file actually existing in the repo. And there's a real trade-off, accepted on purpose: a single-pass draft takes under five seconds, a chunked pass on an oversized file takes closer to forty, because a slower, honest comment beats a fast, confident, wrong one on code that touches real settlement data.

And if you want to be sure it really works, try it somewhere else

Same five letters, a different kind of technical customer, and this time the flip runs the other way, not toward more checking, but away from it entirely.

Hearthstone Mutual built Tallyframe, a tool that reads a claim file and a policy's history and drafts the narrative section of the risk memo an actuary would otherwise write by hand. Amaranthine Corrigold owns its product calls. Evadne Bramwick is one of the actuaries who use it every week, and she is exactly as sophisticated a customer as Threnody, just in the opposite direction.

Tallyframe's citation accuracy, whether it cited the right policy clause, climbed three releases straight: seventy four percent, eighty six, ninety six. A real, honest win. In month one, Evadne's team fact-checked every single memo before it left their desk. By month six, spot checks had fallen to about six percent of memos, because the citations kept being right, and checking a memo that was almost always fine started to feel like a waste of a busy actuary's afternoon.

Same five questions, asked of two different sophisticated customers. Only the I step changes shape.

F · Amaranthine Corrigold, product lead at Hearthstone Mutual, who watched Tallyframe's citation accuracy climb for six straight months.
L · She stopped requiring a fact check on every memo once accuracy crossed ninety percent, and let teams set their own spot-check rate instead.
I · The over-trust flip, running the opposite direction from Fennela's. Old setting: checks a real sample of memos against the underlying claims data. New setting: stops checking anything at all, because the model kept being right about the one thing it was being measured on.
P · Nobody ever separated "cited the correct clause" from "the causal story in the memo is actually supported by the claims data." The team let quarterly reconciliation reviews lapse once citation accuracy crossed ninety percent, and never built a check tied to the memo's causal claims, only its citations.
S · With every causal sentence checked against the claims-linkage table before it renders, unsupported ones get flagged "unsupported inference, verify" instead of shipping clean. The near miss, a hail claim linked to an unrelated water claim on the same policy, gets caught in Evadne's own ninety-second read the same day, instead of three weeks later in outside counsel review.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: ask an internal technical customer for the exact number they're working against, not how something feels, and treat "I'm not sure" as an answer that still needs a spec built around it.
Cost: no budget this quarter for a new pre-read process. Send the three technical questions in a Slack message before the next discovery call, free, and start there.
The model got better, for real: say Tallyframe's citation accuracy reaches ninety nine percent next year. Still not proof the causal story in the memo is right, because citing the correct clause and drawing the correct conclusion from it are two different claims, and only one of them was ever measured.

Where people run it wrong.
They assume a sophisticated internal customer needs less onboarding, when what they actually need is a harder question, not a lighter touch.
They let one metric, citations, precision, whatever shipped first, stand in for the whole document's trustworthiness, because it's the one number everyone's already watching.
They wait for a ticket from someone who has every reason not to file one, because filing it means admitting the tool that's supposed to save them time cost them an afternoon instead.

How to use it live. When an interviewer asks how discovery changes for a technical audience, ask one thing back before answering: what's the one number this person could hand me right now that a typical end user never could? That question is usually the exact distinction the interviewer is listening for.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0