How do you set expectations about AI capability without sounding like you are blocking?

Kolrath AI builds Scrivane, an AI tool that drafts a first-pass legal brief from the facts and the relevant law before an associate edits it. Vesperine Ottsworth is the AI PM who owns Scrivane's roadmap. Odovacar Brinsleigh is a senior litigation partner at Ashenmoor and Draycombe, the firm that has run Scrivane against real filed briefs for fourteen months.

1How to answer this, stage by stage

Nobody is grading whether you can sound diplomatic. They are grading whether you can turn "I am not sure" into something with evidence behind it and a plan to close it, instead of either a guess dressed as confidence or a no dressed as caution.

2Let's learn

Before Kolrath built anything, a litigation associate at a firm like Ashenmoor and Draycombe drafted a brief like this by hand: pulling case law, checking every citation against the reporter, checking the citation format, building the argument. On a typical motion that ran close to fourteen hours.

Scrivane reads the facts and the relevant law, then drafts a first-pass brief an associate edits before it goes out the door. Since Kolrath launched it fourteen months ago, Ashenmoor and Draycombe has run it against more than 640 real filed briefs. Every one of those briefs sat inside a jurisdiction Scrivane's eval set had actually been checked against: federal courts, New York, Texas, California, Illinois. Citation accuracy across that eval set, checked case by case by a licensed reviewing attorney, sits at 98.4 percent.

This is the gap the anchor sentence exists to close. Not a lack of caring on anyone's part. A lack of any designed way to say "not checked yet" out loud.

Then Odovacar's Bexdale matter got removed to Louisiana state court, on a venue ruling nobody at the firm expected. For the first time in Scrivane's life, someone needed it to cite Louisiana's Civil Code, articles numbered in a code, not case law pulled from a reporter, a structure Scrivane's eval set had never once tested.

The problem isn't that Scrivane might get a citation wrong here or there. Every model does, sometimes. The problem is what happens if nobody says out loud that this exact combination, Civil Code citations, has never been checked, and Odovacar files what Scrivane wrote with the same trust fourteen months of tested jurisdictions had earned.

Nothing about the trigger was dramatic. A partner asking for the same thing he always asks for, on a matter that had quietly moved somewhere new.

What that costs at its worst: left unchecked, Scrivane invents a Civil Code article that does not exist, and that citation reaches a Louisiana court in a real filed brief. Opposing counsel checks it in fifteen seconds and it isn't there. That isn't a rough edge, that's the kind of finding that gets a motion to strike filed, a reviewing judge's attention, and Ashenmoor and Draycombe's name attached to a story about an AI hallucinated citation. Worse than if the firm had never used Scrivane on this matter at all.

Same model, same untested jurisdiction. What changes the outcome is only whether anyone named the open question out loud, early enough to check it.

What I would leave alone: Scrivane's handling of the argument and the fact pattern in the Bexdale brief was never in question, that part runs on case law and reasoning it has done a thousand times. I wouldn't slow that part down just because the citations needed a look. Only the part genuinely never tested gets the extra check.

The lesson: "I don't know yet" said with a number and a plan sounds like calibration. Said with nothing behind it, it sounds like an excuse, or it sounds like fear. The words are almost the same. What's standing behind them is the whole difference.

3Now here is the same thing as a story

The short version above is what you'd actually say out loud. Read this one for why the fix had to be a sentence pattern, not a promise to communicate better.

Vesperine Ottsworth has PM'd Scrivane for two years now, long enough to tell within a sentence whether a request from a firm is routine or about to test something the eval set has never seen. Odovacar Brinsleigh has practiced litigation for nineteen years, and he stopped double checking Scrivane's citations somewhere around month six, once he'd watched it get a hundred in a row right on cases he already knew well.

For fourteen months that trust was earned honestly. Ashenmoor and Draycombe ran Scrivane against federal filings, New York contract disputes, Texas commercial litigation. Vesperine watched the weekly sign-off rate from the firm's reviewing attorneys: ninety seven, ninety eight, ninety eight point four, steady. She never had reason to ask which jurisdictions actually built that number, because every jurisdiction the firm used it in was one Scrivane had been checked against since day one.

Then, on a Monday, Odovacar's Bexdale matter got removed to Louisiana state court on a venue ruling the whole team had bet against. He's argued contract disputes in a dozen states. He'd never once needed to cite Louisiana's Civil Code, articles by number, not case law by reporter, and he didn't think twice about it.

The trigger was small. He messaged Vesperine's account manager Monday afternoon, almost as an aside: "Same as always, right? Just point it at the Civil Code articles instead of case law."

Here's what Vesperine didn't do. She didn't say yes, because she had no evidence Scrivane's civil code citations were any good, none of its 2,300 eval cases came from a code jurisdiction. She didn't say no either, because she had no evidence they were bad, and a flat no over an assumption would have cost the firm's trust in Scrivane for nothing.

What she said instead: "Every case in our eval set is common law. Louisiana's different, articles instead of case law, and we've never checked Scrivane against that structure. Give me one day. Twenty real citations from the Bexdale brief, checked by a Louisiana licensed attorney, by tomorrow evening. If it holds, full speed ahead. If it doesn't, you'll know Tuesday, not Thursday night."

Odovacar agreed, mostly because the ask cost him nothing he wasn't already planning to do, a reviewing attorney checks every brief anyway. What he didn't expect was the number that came back.

Fourteen of twenty citations were right. Four had the wrong article number, close enough to look plausible, wrong enough to matter. Two pointed at Civil Code articles that simply do not exist, invented whole, with the same confident formatting as every real one around them.

Vesperine brought the six wrong citations to Odovacar Wednesday morning, expecting frustration. She got something closer to relief. Nineteen years of practicing law had taught him that a source you haven't checked isn't a source, no matter how confidently it's written. What he hadn't expected was that a tool with a 98.4 percent track record could sit at thirty percent wrong, silently, the moment it crossed into ground nobody had walked yet.

She'd considered two other paths first. The first, say nothing and let the citations go out as drafted, the same trust that had worked for fourteen months. She rejected that the moment she saw two fabricated articles sitting in the sample, a court filing is not a place to find out the hard way. The second, tell Odovacar flatly that Scrivane couldn't be trusted for civil code jurisdictions and shouldn't be used on the Bexdale brief at all. She rejected that too. Thirty percent wrong on a twenty citation sample is real signal, but it isn't proof the tool is useless there, and refusing outright over one small sample would have cost the firm's confidence in Scrivane for every jurisdiction, not just this one.

What they built instead, over Wednesday and Thursday: every Civil Code citation in the Bexdale brief got checked, one by one, by a paralegal, against the actual code text, a step that added about ninety minutes to Thursday's edit pass. Scrivane still drafted the argument, the structure, the case law parts of the brief, the parts fourteen months had already proven. Only the untested part got the extra set of eyes. The opposition brief went out Friday morning, on time, with every citation verified, including the two that would have gone out wrong.

What I'd tell myself, watching that thirty percent come back on Tuesday: the mistake was never that nobody had tested civil code jurisdictions yet. The mistake was that nothing in Scrivane noticed, on its own, that this was the first time. We fixed that after, not before. I wish it had been before.

4SPARK, five lines that keep a maybe from sounding like a no

Not a script for sounding careful. SPARK is what forces you to name the exact open question, and prove the answer survives both the day you stayed quiet and the day you said no too fast.

The whole answer in one picture. Four parts, said in one breath, every time an untested combination shows up.

Same finding, same firm, same untested jurisdiction. What changes the outcome is only whether the anchor sentence gets said out loud, early.

This is the line Vesperine holds on purpose. Not caution for its own sake, and not confidence with nothing behind it.

The recap, one line per letter: situation is a genuinely untested combination nobody had ever flagged, payoff is treating "not checked yet" as useful information with a plan attached, anchor is the named gap plus the same day test, risk is a fabricated citation in a filing on one side and a partner who stops asking on the other, and keep out draws the line at claims with no evidence and at caving to the deadline.

5And if you want to be sure it really works, try it somewhere else

Same five letters, a farm cooperative instead of a law firm, and this time the untested ground isn't a citation format. It's a crop the model has never been shown.

Ottermere Aerial builds Loamsight, an AI tool that flies over farmland and flags crop disease from the photos its drone takes. Ilberic Sennwick owns Loamsight's roadmap. Loamsight's eval set, like Scrivane's, was built entirely from the crops Ottermere's first customers grew: corn and soy, in flat Midwest fields, at low altitude. Disease flagging accuracy across that eval set sits at 96 percent.

Quennifer Marsvane manages a farming cooperative that grows quinoa on terraced fields above 3,000 metres, nothing like the corn and soy Loamsight was built and checked against. She asks Ilberic for the same thing Odovacar asked Vesperine for: just point it at our fields, same as everyone else.

Same shape, different ground. The anchor doesn't ask Loamsight to already know quinoa. It asks someone to say, out loud, that it doesn't, yet.

Mapped onto SPARK: the situation is a disease model that has never seen quinoa canopy or the thinner, harsher light above 3,000 metres. The payoff is Quennifer treating "we haven't checked that yet" as a real answer with a plan attached, not a brush off. The anchor is the same sentence pattern: name the exact gap, quinoa, high altitude, never in the eval set, say what already works, the drone, the flight planning, the report format, all proven on other crops, and give a same day way to close it, a sample flight over twelve hectares, checked field by field by Quennifer's own agronomist against what Loamsight flagged. The risk runs the same both ways: stay silent and Loamsight quietly misses a disease pattern it was never trained to recognise on a crop it's never seen, and a cooperative loses a season's yield before anyone notices; sound too blunt, tell Quennifer flatly that Loamsight can't handle quinoa with no evidence, and she takes her cooperative's business to a competitor over an assumption nobody tested. Keep out draws the same line: Quennifer's agronomist helps flag what the sample flight got wrong, never how the disease model itself gets retrained.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to the anchor, name the gap, say what already works, give a same day test.
Cost: no budget for a same day sample flight or a same day citation check every time. Run the sample smaller, five citations instead of twenty, five hectares instead of twelve, coverage still climbs off zero, just with less certainty per check.
The model got better, for real: say Scrivane's overall accuracy climbs to 99.5 percent. The anchor still matters, because that number was never measuring civil code citations. A better common law score doesn't teach the model anything about an article format it's never seen.

Where people run it wrong.
They let "let's check it" become an open ended maybe, no deadline, no sample size, and the check never actually happens before the real deadline arrives anyway.
They let the attorney or the farmer start dictating the technical fix, turning a same day sample check into a week long argument about model architecture nobody scheduled.
They treat one good sample as permanent proof, and stop checking future jurisdictions or crops the same way, instead of treating each new untested combination as its own open question.

How to use it live. Before answering a "how do you set expectations" question cold, ask yourself one thing: what specific claim about this AI's capability do I actually have evidence for, right now, and what's the fastest real check for the part I don't. Naming that gap, not a soft word like "cautiously optimistic," is what the question is actually listening for.

6Flashcards (tap any card to flip it)

7Check yourself Score: 0 / 0