How do you evaluate a vendor's quality claims without running your own eval?

Lissett Group is a mid-size law firm handling vendor contracts for a string of manufacturing clients. Priya Sundaram runs legal operations there. Clarisage sells an AI tool that reads contracts and flags risky clauses, and publishes a 94 percent clause-extraction score on its own benchmark.

How to answer this, stage by stage

Nobody is scoring whether you can recite what a good eval set looks like. They're scoring whether you know what to check when you don't have time to build one.

Let's learn

Here is what happens when a team accepts a vendor's published number as the whole story.

Before Clarisage, Priya and two paralegals split contract review by hand, about forty-five minutes each, roughly ninety hours a month across a caseload of a hundred and twenty vendor contracts. Clarisage read every contract in seconds and flagged clauses to check, which cut the team's hands-on time to a fraction of that, on paper.

Lissett Group picked the left side for the first five weeks, without deciding to.

Here's the turn: the extra speed was never the risk. The risk was that Lissett's leadership accepted Clarisage's published 94 percent score as the bar for going live, and never asked what documents that score was actually tested on. It felt reasonable: the firm had never bought an AI tool before, and a benchmark number felt like the same kind of proof a case citation would be.

At its worst, trusting a published score without checking it doesn't just waste review time. It risks a client receiving a contract with a real hole in it, stamped "no issues" by a tool everyone assumed had been checked.

What I would leave alone: Clarisage's basic clause-flagging on plain, boilerplate sections doesn't need this scrutiny. Standard governing-law and notice clauses are the same in almost every contract, and the tool catches those reliably regardless of which benchmark it was trained on.

The lesson: a number a vendor built to sell you the tool is not the number that protects you from it. The one that matters is smaller, cheaper to get, and it's the one nobody thinks to ask for until something almost goes out the door wrong.

Now here is the same thing as a story

The short version above is what you'd say defending the pilot to the partners. Read this one for how close the missed clause actually came.

Priya Sundaram had run legal ops at Lissett Group for six years, long enough to read a vendor contract and know which clause a client would ask about first. When Clarisage went live, its "high confidence, no issues" tag agreed with her own read often enough that within a month she'd stopped opening every flagged contract herself, trusting the green tag to mean what it said.

Nobody at Lissett Group had actually measured the right side until five weeks in.

Five weeks in, over coffee with a legal ops peer at another firm, Priya mentioned the tool's accuracy score in passing. The peer set down her cup: "Wait, you just believed their number? Did you test it on your own stuff first?" Priya hadn't. That night, she pulled twenty recent contracts and ran them back through Clarisage herself.

Three NDAs from the same vendor family had been marked "high confidence, no issues" and were sitting in a folder ready to go out to a manufacturing client. All three were missing a standard indemnification cap the firm always included, a clause Clarisage's training documents had apparently never needed to flag, because the benchmark it was built on didn't use the firm's specific templates.

Priya pulled the three NDAs the same night, fixed the missing clause by hand, and built a standing rule the next morning: every "high confidence" tag on a new document type gets a human spot-check for the first month, whether it feels necessary or not.

This is what Priya started asking Clarisage for, in writing, after that night.

The colleague's remark didn't create the gap. It just made someone finally look for it.

LEAD, in one screenNot a replacement for a real eval. LEAD is what tells you the fastest, cheapest way to catch a bad claim before it costs you.

Four moves, none of them requiring you to build a full evaluation pipeline first.

The recap, one line per letter: link is tying trust to Lissett's own contracts, not the vendor's benchmark, early signal is the weekly disagreement rate that rose for a month before the near miss, abuse is a benchmark quietly built on easy documents, and decision is a stated threshold for when confidence stops being enough.

The published score sits in the worst corner. An afternoon of your own testing moves you to the best one.

And if you want to be sure it really works, try it somewhere elseSame four letters, a veterinary chain's diagnostic-imaging vendor instead of a law firm. A calibration setting breaks the second story, not a benchmark.

Cutlass Veterinary Group piloted Vetraya, an AI tool reading X-rays for fractures, published at 91 percent sensitivity on a public veterinary imaging set. Mapped onto LEAD: link is trusting Vetraya's read on Cutlass's own patients and their own aging X-ray machine, not the vendor's benchmark equipment, early signal is the rate at which a vet overturns a "clear" read on a small sample of recent films, abuse is that public imaging benchmarks skew toward common breeds and obvious fractures, and decision is Owen Kessler's team agreeing in advance: above one overturned "clear" read in fifteen, pause and recalibrate before reading another film.

The old decision here isn't an accepted benchmark, it's a calibration one: the clinic used Vetraya's default confidence threshold straight out of the box, tuned on the vendor's own newer imaging equipment, without adjusting it for the contrast levels their older X-ray unit actually produced. That made sense in the vendor's demo, which used its own machine. It stopped making sense once Cutlass's equipment quietly shifted what "confident" should even mean on their films.

Swap the trigger and it still runs.
Speed: an interviewer caps you at a minute. Say "twenty of your own documents, spot-check the confident ones, watch the gap," and stop.
Cost: there's no time or budget for even a small sample before go-live next week. Say so honestly, and add a mandatory human check on every confident result for the first month instead of skipping the check entirely.
The vendor gets better, for real: if a later version genuinely closes the gap between claimed and measured accuracy, that's the early signal doing its job, telling you it's safe to loosen the spot-check rate.

Where people run it wrong.
They treat a published benchmark score as proof instead of a claim still waiting to be tested.
They spot-check only the outputs the tool already flagged as uncertain, missing the confident wrong answers entirely.
They wait for a real mistake to reach a client instead of watching the disagreement rate that was rising for weeks first.

How to use it live. If an interviewer asks how you'd evaluate a claim with zero time to build anything, say "twenty real documents and an honest spot-check of the confident ones" and give the actual disagreement threshold you'd set. A number beats a philosophy every time in this kind of question.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0