Explain how data strategy differs when you are prompting rather than training.

Gatework sells ticketing software for concerts and festivals, and behind every purchase sits a fraud score deciding whether to let it through. Selin Karahan runs trust and safety product there, and watched what happened the quarter her team stopped fine-tuning that score and started prompting for it instead.

How to answer this, stage by stage

Nobody is testing whether you know prompting exists. They're testing whether you noticed that switching to it quietly deleted a safety step nobody meant to remove.

Let's learn

Here is what happens when a real safety step quietly disappears because nobody rebuilt it somewhere new.

Before the switch, Gatework fine-tuned its fraud model on a quarter's worth of labeled transactions, about 40,000 of them, and a data steward spot-checked a sample before each new version shipped. That review took about two days, every quarter, and it caught obviously bad labels before they ever reached a live model.

The second step is the one that quietly had no equivalent once the model stopped being trained at all.

With prompting, the fraud score comes from a live call at the moment of purchase: the buyer's account age, their recent order velocity, a device fingerprint, assembled fresh and handed to a general model with instructions on how to weigh them. There is no quarter, no curated batch, no natural pause where a version gets built and checked before it goes live.

Here's the turn: the missing review checkpoint is not visible as a bug. Nothing crashes. The fraud score keeps returning numbers that look perfectly reasonable, right up until a small, ordinary-looking config edit quietly changes what those numbers mean.

At its worst, three weeks pass before anyone connects a quiet dip in the fraud numbers to a specific config edit, because there was never a log that could point to the moment it happened, only a feeling that things had gotten slightly worse lately.

What I would leave alone: the model itself, the general-purpose one Gatework prompts, doesn't need Gatework's own review process. It's someone else's job to test that model broadly. Gatework's job is reviewing what it feeds that model and what it asks it to do, which is exactly the part that went unreviewed.

The lesson: a review step you got for free from one architecture doesn't carry over to a different one automatically. If you don't notice it disappeared, you find out the hard way, in an audit, months later.

Now here is the same thing as a story

The short version above is what you'd say defending a new review process to a skeptical engineering lead. Read this one for how a private spreadsheet ended up being the only real record Gatework had.

Selin could read a fraud-loss report before the second page loaded. Six years watching that number will do that to a person.

The memory aid for the whole method, held up as one page before the story starts.

For two years, the training era ran like clockwork. Every quarter, a steward pulled a sample of that quarter's labeled transactions, checked them by hand, and signed off before the new fraud model shipped. It took two days. Nobody loved it, but nobody questioned it either.

Then Gatework moved the fraud score to a prompted call: live account signals, assembled fresh, handed to a general model at the moment of purchase. No more quarterly training run. No more two-day review, because there was nothing left that resembled the thing being reviewed before.

The training era's review was slow and visible. The prompting era's changes are instant and, unless someone builds a gate, invisible.

Marek Sadowski, one of the engineers who maintained the fraud prompt, noticed early on that nothing tracked what had changed between one week's fraud score and the next. So he started his own spreadsheet: a row for every edit to the prompt or the retrieval config, a note on what changed, and a quick before-and-after check on catch rate whenever he had ten minutes. Nobody asked him to. Nobody else knew it existed.

Three real signals feeding the prompt, and one glaring absence sitting right alongside them.

Three months in, a colleague tweaked which fields fed the prompt, trimming a field that seemed redundant, to make the request a little faster. The change looked harmless. The catch rate slid from 91 percent to 78 percent within days, and nothing on the fraud dashboard called it out as a change. It just looked like a slightly worse stretch, the kind every team has now and then.

Gatework designed as if changes to the fraud prompt were a dial you could nudge gently. They were a switch with no memory of its own settings.

Three weeks later, finance flagged a spike in ticket-fraud chargebacks during a routine quarterly review, the same kind of review that used to catch bad training data, now pointed at a system with nothing equivalent to check. Nobody could say exactly when the catch rate had dropped, or why, until someone remembered Marek's spreadsheet.

Nobody decided, on any single day, to stop logging changes. The training era's checkpoint just never got rebuilt somewhere new.

Rerun the same quarter with a version log and a lightweight review gate in place: the field-trimming edit still gets proposed, but it can't ship without a before-and-after replay against a fixed held-out set. That replay shows the thirteen-point drop in an afternoon, before a single real transaction is affected. Marek's private spreadsheet becomes the official change log instead of a shadow record only he happened to keep.

What I'd tell myself, hearing that one engineer had quietly built the only real audit trail the system had: the training era's review wasn't slow because anyone loved process. It was slow because it was the only pause point that existed. Losing the pause, not losing the paperwork, was the actual danger.

FLIPS, held up as one pageNot a case against prompting. FLIPS is what shows exactly which safety step needs to be rebuilt, not assumed.

The recap, one line per letter: find the person is Marek, keeping the log the system wouldn't. Locate the habit is the quarterly review that had nowhere to land once training stopped. Identify the flip is quietly building a private workaround instead of trusting an official record that doesn't exist. Pinpoint the old decision is never rebuilding a version log for prompt changes. Show the replay is catching the same edit in an afternoon instead of three weeks.

And if you want to be sure it really works, try it somewhere elseSame five letters, a library consortium instead of a ticketing platform. A different flip family entirely.

Rooksbridge Library Consortium used to train a cataloguing model on years of past acquisition decisions, deciding which subject headings and shelf categories fit a new book. It later switched to prompting a general model with each new title's live metadata: publisher description, table of contents, a sample chapter. Mapped onto FLIPS: find the person is Rhona Callaghan, a senior cataloguer across four branch libraries. Locate the habit says she used to spot-check a sample of the model's category assignments each month, back when a new model version came with a change note. Identify the flip here is not a workaround, it's pre-editing: once catalogers noticed the live-metadata prompt guessed wildly whenever a publisher's description was thin or missing, they started writing a fuller, hand-composed description themselves before ever submitting a book to the system, quietly grooming the input to avoid a bad guess rather than reporting the gap. Pinpoint the old decision is that the prompt was built to always return a confident category, never an "I'm not sure," so a thin real description produced a confident wrong guess indistinguishable from a confident right one. Show the replay: with the model allowed to say "insufficient metadata, needs a human," Rhona's team gets flagged the actual thin cases instead of quietly rewriting descriptions for every book, and the ones that truly needed rewriting shrink from most of the intake to about one in twelve.

A different building, a different kind of record, the same missing piece: nothing telling anyone when the input wasn't enough to trust the answer.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "training gives you a free review checkpoint, prompting removes it, so you have to build the checkpoint back on purpose," and stop.
Cost: no budget to build a full versioning system before the next prompt change ships. Say so honestly, and start with a simple changelog and a one-person sign-off, rather than skipping review entirely to save time.
The model got better, for real: if a newer base model starts flagging its own low-confidence guesses reliably, that's a legitimate reason to lean on it more, not a shortcut being taken to avoid building the review gate.

Where people run it wrong.
They treat "we're prompting now, not training" as if it means less data discipline is needed, instead of a differently shaped kind.
They let one engineer's private tracking substitute for an official log, without noticing the system's whole memory now depends on that one person staying.
They review the prompt's wording carefully and never audit the live data being fed into it, which is usually where the real damage happens.

How to use it live. The moment an interviewer asks about prompting versus training, ask yourself: what free safety step did the old approach give us, and did anyone rebuild it, or did it just quietly stop existing? Answer that, and the rest of the comparison falls into place.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0