How does the discovery phase differ when feasibility is genuinely unknown until you build?

Holdfast is Falconbridge's AI tool for litigation teams: it reads a lawyer's draft argument and finds real case law to back it up, citation and all. Vianne Underhill owns its rollout. Fairhurst and Merriwether, the mid-size litigation firm running the pilot, will not put Holdfast anywhere near a real filing until someone can tell managing partner Zdenka Woolgar exactly how it fails, and how anyone would know.

How to answer this, stage by stage

Nobody is grading whether you can say "we'd run a spike" and sound careful. They're grading whether you can turn a genuine unknown into a real number, and a decision that survives someone asking where that number came from.

Let's learn

Holdfast reads a lawyer's draft argument and finds real court cases to back it up. It writes the citation, quotes the part that matters, and hands it back ready to drop into a brief.

A citation looks like one small fact. It's really four separate things that all have to hold at once, and only two of them are a simple lookup.

Before Holdfast, an associate at Fairhurst and Merriwether spent four to six hours on a single motion just finding and checking case law: searching a database, reading each case's actual holding, and confirming nothing had been overruled since. With Holdfast's first draft in hand, that search time drops to under twenty minutes. The associate reviews what Holdfast found instead of starting from a blank search box.

Five steps. The fourth one, verifying what the model just wrote, is the step the whole feasibility question actually turns on.

Falconbridge ran a three week spike before promising anyone anything. 150 real citations, pulled from five practice areas at Fairhurst and Merriwether, scored one at a time by Jerric Stanmore, a contract attorney hired just for this. 127 came back clean. 6 cases didn't exist at all. 3 were real but had been overruled. And 14 were real cases that simply didn't say what the brief claimed.

Three separate tripwires. A citation only clears on its own when none of the three fire, and the middle one is the only tripwire that isn't a certainty.

A fabricated or badly mischaracterized citation that reaches a filed brief doesn't just embarrass someone. It can get a lawyer sanctioned, fined, and named in a public order any client can read. Fairhurst and Merriwether weren't going to risk that to save an associate four hours.

What I would leave alone: formatting. Whether a citation follows the firm's house style, commas in the right place, doesn't need a feasibility spike. That's ordinary string matching, already reliable, and testing it here would have burned time on the one thing that was never actually in question.

The lesson: a feasibility spike isn't a smaller version of the product. It's the one question that decides whether to build the product at all, answered with real numbers instead of a guess that happens to sound confident.

Now here is the same thing as a story

The short version above is what you actually say in the room. Read this one when you want to feel why a single citation, almost logged clean in four seconds, nearly decided the whole answer.

Jerric Stanmore has checked citations for eleven years, first as a summer associate, now as the person firms call when they need someone fast and exact. He can read one paragraph of a court's holding and tell you in ten seconds whether it says what a brief claims it says. He has a habit nobody taught him: he always reads the actual paragraph a pin cite points to, never just the case name and the page number, because the two can drift apart in ways a fast scan won't catch.

Falconbridge brought him in for three weeks, at half time, to score Holdfast's citations one at a time. Clean, fabricated, mischaracterized, overruled. Real cases, pulled from real work Fairhurst and Merriwether had already filed.

The first week went well. Contract and employment citations mostly checked out, and the ones that didn't were easy calls: a case that plainly didn't exist, a holding that had obviously been reversed. Jerric started to relax into the rhythm of it, the way anyone does with a task that keeps agreeing with them.

Then insurance coverage.

Citation 94 was a coverage dispute, cited for the idea that a specific policy exclusion didn't apply to a particular loss. It was formatted perfectly. Holdfast's own confidence score on it read 91 out of 100. Jerric almost logged it clean on the strength of the case name and the pin cite alone, the way he might have by citation 200 if the rhythm had kept holding.

He read the paragraph anyway. Old habit.

The case didn't just fail to support the point. It said the opposite. The court had specifically held that the exclusion did apply, on nearly identical facts. Cited the way Holdfast had cited it, the case would have told a judge the reverse of what the actual court decided.

The whole spike, in one picture. Existing is a fact you can check in seconds. Being right is a judgment call, and citation 94 passed the first test and failed the second.

Jerric flagged it. It took him four minutes to catch, on a citation Holdfast had scored 91 out of 100 sure of itself.

Here's the decision that traces back to a kickoff call, ten days before the spike started. Someone had asked whether the scoring should track flaw type separately from day one, or just mark each citation right or wrong to move faster inside three weeks. The team picked one column. Reasonable, given the calendar. Splitting into flaw types meant more setup, more categories for Jerric to keep straight while moving fast.

I would take that back. Not because one column is wrong exactly, it is faster to build. But by day 9, with one column, all we would have had was "84.7 percent clean," a number that sounds like a passing grade. We would have had no way to tell Zdenka Woolgar the thing buried inside that 15.3 percent: that only 6 of the wrong ones were fabricated, easy to catch, and only 3 were stale, also easy to catch. The other 14, the mischaracterized ones, are the only kind a person like Jerric can catch and a lookup can't. That's the entire question this spike exists to answer, and one column would have hidden it inside an average.

Run the spike again with flaw types split from day one, the way it actually ran. By day 12, nine days early, Vianne already had the number that mattered: a verification layer catching mischaracterization somewhere between 64 and 86 percent of the time, tested against Jerric's own scoring. She didn't need all three weeks to know what to bring Zdenka. She needed the right four columns.

One design tells you whether Holdfast is roughly right. The other tells you exactly which kind of wrong could end a career, and whether anything catches it before a judge does.

What I would tell myself, back on that kickoff call: a single column isn't really faster. It just moves the cost from this week to the week someone asks why the number everyone trusted didn't say what it actually meant.

Three weeks after that call, Vianne told Zdenka Woolgar the truth in one number, not a pitch. Build it, but only as verify-first, until the catch rate proves out past 80. Fairhurst and Merriwether signed on for that version. Nobody ever found out about citation 94 from a judge.

BOUND, or turning "nobody knows yet" into a number Zdenka could act on

Not a way to sound more careful about an unknown. BOUND is what turns "we can't know until we build it" into a real number, and a decision that survives someone asking exactly where that number came from.

All four numbers on the same line. The measured range straddles the ship-it line instead of clearing it, which is the whole reason this became a pivot decision and not a launch.

Two of the three flaw types sit exactly where a good estimator stops worrying: dangerous, but solved by a lookup. The third sits where the real work still is.

Three things worth naming directly, since this is where the real judgment sits. The AI-specific failure mode here is a mischaracterized holding: a real case, cited for a point it never actually decided, which is a specific and severe way for a legal tool to be wrong, not a generic bug. The guardrail is layered, a deterministic lookup for existence and staleness, plus a separate verification model for holding match, with every flagged citation and every high-value argument routed to a lawyer regardless of score. The alternative Falconbridge rejected was interviewing litigation partners for a target accuracy number instead of running the spike. It lost because nobody in that room had seen a real Holdfast mistake yet, so any number they gave would have been a guess wearing a requirement's clothes. And there's a real trade-off accepted on purpose: routing every flagged and every high-stakes citation through mandatory review means citations that could return in seconds now wait on a lawyer's actual time, slower and costing real billable hours, a cost Fairhurst and Merriwether accept because the alternative is a fabricated citation reaching a judge's desk.

And if you want to be sure it really works, try it somewhere else

Same five letters, a paper mill instead of a law firm. This time the unknown isn't a mischaracterized holding. It's whether a rare, quiet signal even exists inside years of sensor noise.

Different building, same shape of question. A mischaracterized holding became a bearing about to fail, and the method transferred without changing.

Faultline, built by Wrycroft Industrial, reads years of vibration, temperature, and pressure data off the slurry pumps at Winslowe Paper Co. and tries to flag a bearing failure before it happens. Nairne Oleander, the plant's reliability engineering lead, doesn't know if the failure signature is even detectable inside all that noise until someone actually builds and backtests a model. Nobody can spec that in a meeting.

Nairne runs the same five letters. Break it down: catching a failure only matters if the crew gets at least 48 hours of warning to schedule a repair without an emergency shutdown, so the spike needs to test against real, confirmed failures, not just "did the model notice something." Own the numbers: 40 confirmed pump failures over two years, full sensor history for each, backtested over four weeks by one engineer against the plant's own maintenance logs. Use a range: the model catches somewhere between 26 and 33 of the 40 failures with at least 48 hours warning, 65 to 82.5 percent, while throwing somewhere between 3 and 9 false alarms a month across the fleet of 600 pumps, a range because tightening the threshold to catch more failures always means more false alarms too. Nail the sanity check: an unplanned failure costs about $40,000 in downtime and damage, a false alarm costs about $800 in an unnecessary inspection, so even at the low end of detection and the high end of false alarms, the math still lands strongly in Faultline's favor. Direction: the swing assumption isn't the failure count, it's the false-alarm rate at whatever threshold catches enough real failures, because a crew that gets paged too often stops trusting every alert, the same way Jerric might have stopped reading a paragraph if the rhythm had kept holding.

Same rank, different lever, mapped onto BOUND: the breakdown is the same shape, four numbers hiding behind one word, unknown. The owned numbers are different but reasoned the same way, a real sample, a stated budget, a stated definition of "caught." The range is the same kind of honesty, an estimate from a backtest, not a census. And the swing number plays the same role in both stories: not the flashiest figure, but the one number that decides whether the team trusts the flagging enough to actually use it.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: how much of the outcome the model needs to get right, what it actually does on a real labeled sample, what a verification step catches, and what's left over compared to what the real thing can survive.
Cost: no budget for a full labeled sample this quarter. Run a smaller spike, 40 to 50 citations instead of 150, say so plainly, and treat the range as wider until a bigger one is affordable.
The model got better, for real: say the next version's raw mischaracterization rate drops to 3 percent instead of 9.3. The verification layer doesn't get removed. A lower raw rate isn't the same as zero, and the cost of one that slips through unflagged hasn't gotten any smaller.

Where people run it wrong.
They report the overall clean rate as the finding and stop, without ever isolating the one flaw type that actually needs a person.
They treat a small spike's range as a flaw in the method instead of the honest limit of what a three-week sample can tell you.
They test on the easy, obvious errors and call the feature validated, without checking whether the hard, judgment-only failures were even in the sample.

How to use it live. Ask yourself out loud, before naming any number: "which part of this is a lookup, and which part actually needs a person to decide?" That question alone buys real thinking time, and it's usually exactly what the interviewer is listening for.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0