Describe the decision criteria you would set before the spike begins.

Cobalt Ledger sells expense software to mid-size companies. SnapAudit is its feature that reads a photographed receipt and files it into the right expense category. Marisol Deschamps is the AI PM who ran the spike that decided whether SnapAudit shipped. Renske Bakhuizen runs the finance desk that has to live with what it got wrong.

How to answer this, stage by stage

Nobody is scoring whether you can name a good accuracy number. They're scoring whether you'd write the bar down before you're standing in front of a result you already like.

Let's learn

Every month, about 600 employee expense receipts landed on Cobalt Ledger's own finance desk. Someone in accounts payable read each one and typed in a category by hand: travel, meals, supplies, or one of a dozen others. Three minutes a receipt, thirty hours a month, done the same way for years.

With SnapAudit, a photographed receipt gets a category back in under two seconds. Marisol's team ran a first spike on sixty receipts, hand-picked from the tidiest folder in the archive, and it hit 97 percent. The team was thrilled, and nobody had written down what number, on what data, would have counted as a real pass.

The four letters, held up as one page. Early signal is the step this question is really testing.

Here's the turn: the spike's 97 percent was never wrong, exactly. It was just a fact about sixty receipts nobody had disagreed about the category of. The real 600-a-month distribution held handwritten tips, foreign currency, split bills, and vendor names that could mean two different things depending on the trip. None of that showed up in the curated sixty.

At its worst, an employee's legitimate travel dinner gets mis-coded as a policy violation, a manager gets an automated flag about it before anyone checks, and the tool that was supposed to save thirty hours a month has quietly cost someone their afternoon and their patience.

What I would leave alone: I wouldn't demand a full production-data eval for an internal-only draft feature that a person already reads and edits before anything reaches a customer, since a curated spike is plenty to greenlight more building there.

The lesson: a spike's number is a fact about the data it ran on, not a fact about the feature. Decide the data and the bar before you see the number, or the number will happily decide them for you.

Now here is the same thing as a story

The short version above is what you'd say defending a spike plan in a review meeting. Read this one for what it felt like the eight weeks nobody wrote the bar down at all.

Renske Bakhuizen can smell a mis-coded travel expense before she's finished reading the vendor line. Nine years on Cobalt Ledger's own finance desk will do that.

The third step is where a curated spike's promise either holds up on a real receipt or quietly doesn't.

When SnapAudit launched, Petra loved it. A photo came in, a category came back, and for the first few weeks she checked a handful at random just to see it behave. It always did. She let it run.

Same word, test set, and only one of the two versions ever touched a hard case.

Then came a run of vendor names that could be either a client dinner or a personal one, and a foreign hotel bill split three ways. SnapAudit filed them fast and filed some of them wrong. Petra didn't complain. She just quietly started re-checking that category by hand before it posted.

Four things a written bar needs, and the spike that shipped SnapAudit had none of them on paper.

By week eight, Petra was quietly re-doing about a quarter of SnapAudit's work by hand, and nobody upstream knew it, because the dashboard everyone watched still said the model was 97 percent accurate. Then a legitimate client dinner got coded as a personal expense on an employee already under a performance review, and it nearly reached that employee's manager as a flag before Petra caught it, purely because she happened to be looking that morning.

The number everyone watched sat in the obvious corner. The number that actually mattered sat in the hidden one.

The real question was never whether SnapAudit's model was good. It was whether anyone had ever written down, before the spike ran, what data that goodness had to survive.

When the spike was first proposed, someone said, "let's just run it on whatever receipts we've got handy and see how it does," and it sounded reasonable, since nobody expected a small sample to lie.

Rerun the same spike with the bar written down first: test on a full real month, require 90 percent there, and name the cost of an HR-sensitive miss out loud. The 71 percent shows up before launch, not eight weeks after it, and the fix, a stronger rule for ambiguous vendor names, ships before Petra ever needs to quietly cover for it.

What I'd tell myself, hearing how close that flag came to reaching a manager: sixty clean receipts were never a test. They were a compliment the model paid itself.

LEAD, the bar that has to exist before the number doesNot a script for distrusting every good spike result. LEAD is what tells you exactly which data that result is actually describing.

The recap, one line per letter: link is the thirty hours of finance-desk time actually being saved, early signal is Petra's override rate climbing while the accuracy score stayed flat, abuse is a curated sample quietly inflating the launch number, and decision is a written 90 percent bar on real data, watched by the override rate afterward, not the launch-day score.

And if you want to be sure it really works, try it somewhere elseSame four letters, an event-ticketing marketplace instead of an expense desk. Different flip family entirely, the same missing bar.

Conrad Aldous runs trust and safety product at Foxglove Ticketing, where a model flags likely fraudulent resale listings for a review team to confirm. The spike ran on a curated batch of obvious fakes and hit a strong score. Mapped onto LEAD: link is real chargebacks avoided, not the flag rate itself. Early signal here is a concealment flip, not an override rate: once a review analyst's flagged case got auto-closed the moment a form was filed, analysts stopped bothering to double check a flag that looked routine, and the form-filed count looked healthy while a slower, subtler class of fake listing kept slipping through unflagged. Abuse is the same shape as Cobalt Ledger's: the curated batch of fakes was all obvious ones, so the score never touched the subtle cases. Decision is requiring the eval set to include listings a human analyst originally disagreed about, not just ones everyone already agreed were fake.

A different flip entirely: not a person doing more manual work quietly, but a person doing less checking, quietly, because a form made it look done.

The gap between the launch number and the real number was never a surprise. It was just never measured until the near miss forced it.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "write the pass bar and the real data source down before the spike runs, not after," and stop.
Cost: no time to build a full production eval set before a deadline. Say so honestly, and pull even 150 real, unfiltered receipts instead of relying on sixty curated ones.
The spike result is a genuine improvement, for real: if a later model scores even higher on the real set, that's worth trusting more, since the bar was always about the data, not about being suspicious of good news.

Where people run it wrong.
They agree on a pass bar informally, after the number is already sitting in front of them.
They build the eval set from whatever data was easiest to gather, not the data the feature will actually face.
They watch the launch-day score forever and never build a second signal for what happens after.

How to use it live. The moment an interviewer asks what criteria you'd set before a spike, ask yourself: what real data will this be graded on, what number does it have to clear there, and what does a miss cost in the worst case. Say those three out loud, in that order, and the rest of the answer follows on its own.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0