Describe how synthetic data changes the calculus of a cold-start problem.

Ferrotype Labs sells an AI tool that reads a codebase's history of real bugs and drafts new regression tests before a release goes out. Talia Bergstrom is the PM who owns what happens the day a brand-new client connects a repo with no bug history at all.

How to answer this, stage by stage

Nobody is testing whether you know synthetic data exists. They're testing whether you know which kind of wrong you can afford while you wait for the real thing.

Let's learn

Here is what happens when an AI product has to launch for a customer it knows nothing about yet.

Before Ferrotype had any synthetic-data option, a brand-new client waited about 47 days before the test-generation feature produced its first useful suggestion, the time it took for enough real failures to build up in their repo. With synthetic data, seeded from the client's own code rather than a generic bug library, Ferrotype can produce a first suggestion on day one instead.

The gap in step two is the whole problem. Nothing real has happened yet for the model to learn from.

Here's the turn: the extra day-one suggestions are not the risk. The risk is what the team does with the model's confidence once those suggestions start looking good. A confidence score built entirely from invented bugs measures how well the model catches the bugs it invented, and that is not the same thing as how well it catches this client's real ones.

At its worst, the model's synthetic-earned confidence gets treated exactly like a real one, an unproven suggestion clears the same 92 percent bar used for established clients, and it auto-merges into a release branch nobody double-checked, because the number looked fine.

What I would leave alone: for an established client with months of real failure history, I wouldn't touch the existing autonomous-merge bar. It's proven against real bugs already, and adding a synthetic-data gate there would just slow down something that already works.

The lesson: a confidence number only means what it was measured against. If the measuring stick changes, quietly, from invented bugs to a client's real ones, the same number stops meaning the same thing, and nobody tells you when that happens.

Now here is the same thing as a story

The short version above is what you'd say defending the launch plan in a design review. Read this one for how a reused threshold nearly let a real bug through.

The wall by Ferrotype's engineering bay has a monitor bolted to it, showing every pull request across every client repo, color coded green and amber and, rarely, red.

Four branches off the same root. Ferrotype picked the third one, on paper. The story below is about the branch it almost slid into instead.

Driftwood Games connected their multiplayer repo to Ferrotype in early spring, a brand-new codebase with nothing to learn from yet. Talia's team seeded synthetic bugs drawn from Driftwood's own code, and by day one the model was drafting its first test suggestions. For three weeks, everything Driftwood's engineers saw in the review queue looked plausible, careful, even a little impressive for a tool that had never watched their code actually break.

Three of these four were true for Driftwood by week three. The fourth, showing the confidence number honestly, was the one nobody had actually wired up yet.

By week five, real failures started trickling in from Driftwood's own CI runs, the ordinary kind: a null pointer here, a timeout there. The model kept drafting tests, and its own confidence scores kept climbing, because nothing it had seen yet, real or synthetic, looked hard.

Then came week seven. A synthetic-trained test suggestion for a matchmaking function cleared the 92 percent confidence bar Ferrotype had used for years on established, real-data repos. Under the standing rule, anything over that bar auto-merged without a human in the loop. It very nearly went through.

An engineer named Owen, doing a routine skim of the merge queue out of habit rather than obligation, caught it. The synthetic-trained test passed cleanly on a benign case and never touched the actual race condition that showed up under real matchmaking load, the exact kind of bug synthetic data almost never invents, because nobody thinks to write "simulate two players joining within four milliseconds of each other" into a bug library.

Nobody decided, on any single day, to treat synthetic confidence like real confidence. The 92 percent bar was just already there, from a different kind of repo entirely.

The real question was never whether synthetic data belonged in a cold-start launch. It was whether a number earned on invented bugs should ever be allowed to open the same door a number earned on real ones does.

Rerun the same seven weeks with a separate gate for cold-start clients: the near-miss suggestion still gets drafted, still scores 92 percent, but it never reaches the auto-merge lane, because Driftwood hadn't logged its thirtieth real failure yet. It sits in Owen's ordinary review queue instead, gets caught in the same twelve minutes he spends there every morning, and never comes near a release branch at all.

What I'd tell myself, watching that suggestion clear the same bar as a proven client's work: the 92 percent number was never wrong. It just stopped meaning the same thing the day it started scoring something other than what it was built for, and nobody had agreed to notice when that happened.

PICK, the case for committing earlyNot a rule against synthetic data. PICK is what tells you exactly which kind of wrong you're choosing to live with.

The recap, one line per letter: position is launching day one on synthetic data but advisory only, impact is Driftwood absorbing a slow launch versus a release absorbing a missed bug, cost asymmetry says optimize against the hidden, silent one, and kill criteria is thirty real failures at matched precision before autonomy is earned.

And if you want to be sure it really works, try it somewhere elseSame four letters, a food distributor instead of a dev-tools company. Very different kind of bug to miss.

Passamore Foods runs demand forecasting for the small restaurants it supplies. A brand-new restaurant client has no order history at all, so Passamore's model launches on synthetic demand curves built from similar restaurants in the same cuisine category. Mapped onto PICK: position is recommending order quantities from day one but keeping a human buyer as the final approver. Impact says waiting for real orders costs a new restaurant a week or two of manual guesswork on their own, a visible, bounded cost they already know how to manage. Trusting the synthetic category average at full autonomy costs something quieter: a restaurant whose real spice-mix demand doesn't match the generic cuisine average ends up silently over-ordered, and nobody notices until the walk-in cooler is full of stock that will spoil before it sells. Cost asymmetry favors the hidden one, the spoilage nobody's watching for. Kill criteria: eight real weeks of orders, or until the model's forecast error on this specific restaurant matches its historic bar, before the buyer steps back.

The same two boxes work for a repo or a walk-in cooler. One side is loud and small. The other is quiet and expensive.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "synthetic data buys day one, not trust, keep it advisory until real data earns autonomy," and stop.
Cost: no budget to build a dedicated cold-start gate before the next client launches. Say so honestly, and default to advisory only until that gate exists, rather than skipping the gate to save time.
The model got better, for real: if synthetic-trained confidence starts matching real-data confidence on a held-out sample, that's a legitimate reason to shorten the thirty-failure bar, not a shortcut being taken to launch faster.

Where people run it wrong.
They treat "we have synthetic data now" as if it answers the trust question instead of just the launch-day question.
They reuse a threshold built for one kind of data on a different kind of data, without ever revisiting whether the number still means the same thing.
They let a promising three weeks of synthetic-only results substitute for the real evaluation that was supposed to happen before autonomy was granted.

How to use it live. The moment an interviewer asks about synthetic data and cold start, ask yourself: what did this confidence score actually get tested against, and would I bet a customer's production release on that being the same thing as the real world? Answer that, and the rest of the call follows.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0