Write a scoring rubric for the quality of a generated customer support reply.

How to answer this, stage by stage

Six moves. Ground it in one reply type that's already auto-sending, before touching the rubric itself.

Let's learn

Here is what happens when a support reply gets judged on one number, and nobody ever writes down what that number is actually supposed to check.

Say we build Wingman, a tool that drafts the first reply inside Silvertail Airlines' live chat, so a support agent can read it, send it as is, fix it, or throw it out and type their own.

What scoring a reply looked like, before this rubric existed

Before anyone scored anything on paper, every Friday four reviewers split a sample of two hundred replies from that week and gave each one a number, one to five, for quality. Nothing more specific than that. Two straight weeks, baggage-delay replies averaged 4.3. That crossed the line the team had already agreed on: hold 4.2 or higher for two weeks running, and that kind of reply stops going through an agent at all. Wingman's draft goes straight to the customer.

On a normal week that saved real time. Baggage-delay chats ran about ninety a day, and skipping the agent glance freed up minutes for the calls only a person could take, the kind that spike hardest during a storm delay.

Then one reply inside that well-scoring batch offered a customer two hundred dollars for a bag delayed six hours. Silvertail only owes that two hundred dollars once a bag has been missing twenty-four hours. Under six hours, an agent is supposed to say the bag is being traced and offer to cover receipts if the delay passes a day. The reviewer who read it gave it a five. It sounded warm. It sounded confident. It sounded exactly like the tone Silvertail wanted.

Here's the part that's easy to miss. A dollar figure that wasn't owed was never really the problem, because that was one reply. The real problem was that "quality" had never meant anything sharper than does this sound like something we'd send. A reply that broke policy in its second sentence could still be the best-reading reply in the whole sample.

At its worst, this costs real money and a real fight. The wrong reply kept going out for eleven days after auto-send switched on, thirty-four times out of the nine hundred and ninety baggage-delay replies sent in that window, every one to a customer whose bag hadn't actually been gone a full day. Silvertail chose to honor every promise it had already made rather than call thirty-four customers back to take money away, an unplanned $6,800, on top of a specialist's whole afternoon spent walking one customer's bank through why a payment she was owed had never landed.

The choice I would take back. When this scoring was first built, reviewers pushed back on anything more than one number, said a five-part scorecard would triple how long the Friday sample took, and back then Wingman only touched a slice of chats, so a slower review felt like the safer trade. That held up fine while a flat five just meant "read this one again if you're not sure." It stopped holding up the day a flat five became the only thing standing between a bad promise and a live customer.

What I would leave alone. A reply to something like "what's my seat number" doesn't need this. There's one right answer, it's already sitting in the reservation system, and the reply either matches it or it doesn't. No policy judgment call lives in there for a rubric to catch. Only score the kind of reply where a person had to decide what to promise.

The lesson. Splitting a score into five parts doesn't fix reviewers disagreeing with each other. It fixes what any one reviewer's number was ever measuring. If a wrong promise and a right one can land on the same score, we never wrote a rubric. We ran a taste test.

Now here is the same thing as a story

Read this one when you've got a few minutes, because eleven quiet days say more than any rubric ever could.

Saanvi Rajguru owns Wingman's quality bar at Silvertail Airlines, inherited from the contractor team that had built the scoring process the year before she took the role.

She trained all four reviewers herself, in one shared room, reading transcripts together and arguing out loud about what a five actually meant next to a three. For months she sat in on every Friday session.

Then storm season hit and her calendar filled with two other launches. She stopped sitting in on Fridays. Then she stopped reading the transcripts behind any score above a three, trusting the number on its own. Reviewers, scoring faster as volume tripled through October, stopped writing why behind a score and just typed the digit.

On the Tuesday that mattered, baggage-delay hit its second straight week at 4.3. Auto-send flipped on for that intent automatically, the way the team had agreed it would, months earlier, back when 4.2 sounded like a number worth trusting.

Eleven days later, a complaints-line email landed with a screenshot attached: a customer's own bank dispute form. Her bag had been six hours late. Wingman had told her two hundred dollars was coming. It never did, because the offer had never been real to begin with. She hadn't asked for a refund. She'd asked her bank why a payment Silvertail promised her, in writing, hadn't shown up.

Saanvi pulled the Friday scores for that whole window. Every one of them read 4.1 or higher. Nothing on the dashboard had ever looked wrong.

She found the reviewer who'd scored the original reply a five. It hadn't been carelessness. Read on its own, it was a good reply: warm, clear, fast. Nothing in the one number he'd been asked to give had ever asked him to check the twenty-four-hour rule against the timestamp sitting right above it in the same chat.

Saanvi pulled all eleven days of transcripts. Thirty-four of the nine hundred and ninety baggage-delay replies sent automatically in that window had made the same offer to a customer whose bag hadn't been gone a full day. Nine hundred fifty-six had been fine.

The anchor: one rubric card, five named rows, one of them a gate

Here's the rubric she wrote after. Five things, not one: policy accuracy, resolution progress, completeness, tone, and escalation safety, each with a written line for what a low score looks like and what a high one does. Policy accuracy stands apart from the other four. Score it a one or a two, on its own, and the whole reply fails, no matter how the rest reads.

The day it was wrong, with and without the gate in place

Run the same eleven days forward with the rubric in place. The reply reads exactly the same, warm, clear, fast. But policy accuracy checks the twenty-four-hour rule against the chat's own timestamp before anything sends, and this reply fails that one check by itself. It doesn't auto-send. It queues for the next agent, who glances at it, fixes one sentence, and sends the real answer in under a minute. The other nine hundred fifty-six replies in that window never touch a human at all, because they never trip the gate.

Thirty-four wrong promises become zero. Nine hundred fifty-six replies still move at the same speed they always did.

And the thing I'd tell myself, back when the reviewers first asked for one simple number: I answered how fast can we score them. I never asked what happens the day the fast number is wrong in a way that reads as right.

SPARK, sized for a rubric instead of a screen

This question asks for a scoring rubric, not an interface, but it's still one concrete design decision about the exact moment a reply either earns trust or breaks it, so SPARK still fits. A question asking how to measure Wingman's overall quality across the whole product would reach for LEAD instead.

And if you want to be sure it really works, try it somewhere else

A home-insurance claims chat is a different business entirely, wearing a different kind of promise, but the same gap between "it read well" and "it was true" shows up there too.

Same framework, a different desk, a different kind of promise

S. Tamsin Halloran leads claims intake at Cresthaven Mutual, where a chat tool drafts the first reply to anyone filing a water-damage or fire claim. Today, without a written rubric, a reply gets judged on whether it sounds reassuring, with nobody checking whether the settlement figure it quotes matches the actual policy.
P. The habit worth building: whoever reviews a claims-chat log can tell in one read whether the number it quoted was ever real, not just polite.
A. Same shape, different gate. Coverage accuracy, checked against the real policy's sublimit table for that kind of damage, fails the reply outright if the quoted figure doesn't match, the same way policy accuracy did for Wingman.
R. Loosen the gate to a vibe check, does this number feel about right, and it never actually fires, the same failure as before, wearing new clothes. Tighten it to require a full manual policy lookup on every single reply, even a plain "we've received your claim" with no figure in it at all, and response time collapses during the one week claims volume actually spikes, a real storm.
K. No attempt yet to score empathetic phrasing differently for a fire claim versus a flood claim. Just whether the number quoted was ever true.

It took a homeowner calling to ask where her fifteen-thousand-dollar repair estimate had gone, when her policy's water-damage sublimit was five thousand, before anyone checked whether the reviewers' warm scores had ever looked at a number at all.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0