What should the eval spec say about who breaks a tie when scores are ambiguous?

How to answer this, stage by stage

Seven moves. Name the decision before the story, or the answer sounds like a process diagram instead of something you'd actually put in a spec.

Let's learn

What happens when two graders look at the same AI summary and can't agree whether it's good enough? Say a law firm uses a tool called Casefold. It reads a legal brief, thirty pages of filings and exhibits, and writes the first draft of a one page summary an associate needs before a hearing.

Before Casefold, an associate read the whole brief by hand and wrote that summary themselves. A thirty page brief took about ninety minutes to read and summarize well.

Now Casefold drafts the same summary in under two minutes. To make sure the drafts hold up, two senior associates score a sample of forty summaries a week against a five point rubric. A score of four or five passes. Anything under that fails and goes back to be rewritten by hand.

Here's the turn. When the two graders' scores land on opposite sides of that line, one says four, one says three, that's not a pass or a fail yet. It's a tie. And the eval spec never said who breaks it. So whoever happens to be free breaks it, case by case, with nobody keeping count.

At its worst, the pass rate held around 90 percent for ten straight weeks, so the weekly QA report looked healthy the whole time. Underneath it, the share of summaries landing as a tie crept from 6 percent to 24 percent, almost one in four. Nearly all of those ties were landing with one reviewer, Owen Radcliffe, who happened to answer fastest on the QA Slack channel and who passed nine out of ten of the ties he broke. One of those passes was a summary that left out a filing deadline. It reached a partner's binder marked clean, and the firm came within two days of missing the response window.

Same tie, same shrug. The deadline still slips off the page.

What I'd leave alone. Ties over style, is the summary too long, does a sentence read stiffly, don't need a named tiebreaker. Nothing about the case depends on those, so two graders can just pick one and move on.

The lesson. A tie rate nobody counts isn't proof the rubric is fine. It can be proof one reviewer has quietly become the whole rubric. The only way to know a pass rate still means something is to watch who's deciding the close calls, not just how many pass.

Now here is the same thing as a story

Use this version when you've got the time. The short version is above. This is for when you want to feel why it mattered.

Elias Thorne has run product for Casefold for two years, and he wrote the first version of its eval spec himself, back when the tool was still being tested on one partner's caseload.

For the first several months, the rubric earned its trust the honest way. Every Friday, two senior associates scored a sample of the week's summaries, and on the rare case where they disagreed, whoever was in the office finished the argument out loud, in about two minutes, and moved on.

Then the checking faded, in three small steps, none of them looking wrong at the time. First, because ties were rare enough early on, Elias never wrote down a real rule for who settled them, just a line in the spec saying the graders would sort it out between themselves. Second, as the firm grew, the QA Slack channel filled up with side conversations, and the fastest replier on any given tie started being whoever happened to be at their desk. Third, nobody was logging which reviewer answered, only whether the week's sample passed or failed.

Then Naomi Kessler, a first year associate shadowing QA for the month, asked an offhand question in a Friday review. Why does Owen end up deciding almost every close call?

Elias didn't have an answer. He pulled the raw grading log instead of the weekly pass rate, every score, every tie, and who resolved it, plotted by week. Owen Radcliffe had gone from breaking about a third of the ties in week one to almost nine in ten of them by week ten. And Owen passed 90 percent of what he broke. Everyone else passed about 55 percent.

It would be easy to say Owen did something wrong. He didn't. He was fast, he was available, and every time a close call landed on his desk he made a real call. Nobody decided the tiebreaker should be whoever answers fastest. It just quietly became that, one Slack reply at a time.

So here's the decision Elias would take back.

Two years earlier, in the meeting where the eval spec first got written down, someone asked whether they should name an actual tiebreak owner. It felt like overbuilding a rule for something that would barely ever happen. Elias remembers agreeing they'd deal with it if it ever became a real problem.

I would name the tiebreak owner in the spec from day one instead. Same five point rubric, same weekly sample, but a fixed rotation for who breaks a tie, and a real number on the tie rate itself that someone has to answer to.

Here's the replay. Same ten weeks, tie rate tracked from day one. It crosses the 12 percent floor in week six, and that crossing is the trigger, not Naomi's question in week ten. Elias pulls that week's tied cases, rewrites the two rubric lines graders keep reading differently, and starts a weekly tiebreak rotation from there. The tie rate falls back under 10 percent by week eight. The quarter closes with 1 confirmed missed fact reaching a partner's binder, instead of 6.

One eval spec watches whether this week's summary passed. The other watches whether passing still means the same thing it did in week one.

And the thing Elias would tell himself, back in that first meeting: the pass rate was never the risk. Leaving nobody in charge of the close calls was.

LEAD, run on who gets to say yes when the rubric doesn't

This reads like a process question, who signs off on a form. Underneath it, it's still asking whether the pass bar stays the same bar for every case, or starts depending on who happened to answer. That's LEAD, run on a tiebreak instead of a single metric.

And if you want to be sure it really works, try it somewhere else

A used car marketplace called Fairmile Auto Exchange has two inspectors score every vehicle condition report before it goes live: does it list every real issue with the car, yes or no. Same shape of question. A pass rate that holds steady can hide the same thing it hid at Renwick & Hale: one inspector quietly deciding every borderline report, because they're the one who happens to be online when the disagreement comes in.

Same idea, a car lot at midnight.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0