How do you communicate a quality bar to people who have never seen an eval report?

Kessley Mail sells hosted business email with a built in phishing and spam screen called Talonguard. Ottavine Thorbecke is the AI PM who owns Talonguard's quality bar. Brumley Grimwade runs Customer Success at Kessley Mail and has to answer for that quality bar on the phone. Wickstead Hardware is a fourteen store customer that nearly lost real money to a phishing email Talonguard let through.

01How to answer this, stage by stage

Nobody is grading whether you can define precision and recall. They are grading whether you can turn a probability into something a person with no model background can actually act on, and defend the exact words you'd use.

02Let's learn

What do you say to someone who has never once looked at an eval report? That's really the whole question, and the honest answer is: not the report, and not a vibe either.

This is the gap the card exists to close. Not a lack of care from Brumley's team. A lack of anything real to hand a worried customer.

Talonguard reads every email that reaches a Kessley Mail inbox and decides, in a fraction of a second, whether it's safe, spam, or a phishing attempt. Before Kessley Mail built it, a business like Wickstead Hardware relied on its own spam folder and a wary eye, and still lost the occasional afternoon to a fake shipping notice or a bad link. Talonguard screens roughly 40,000 emails a day for Kessley Mail's customers, and about 1 in 900 of those is a genuine phishing attempt.

Over the last quarter, checked case by case against a set of real, confirmed emails, Talonguard caught 95 percent of confirmed phishing attempts, about 19 of every 20. It wrongly held about 1 in 500 real customer emails for review. Those two numbers on their own sound close to finished. They aren't, because they hide exactly where the misses live.

A single badge like this one looks calm because it's built to look calm. The real failures underneath it were never spread evenly.

What it costs at its worst: hand a worried customer a single reassuring number, and the very first bad email that reaches them lands as proof the number was a lie, not proof that a rare, hard category slipped through. Hand them the raw eval sheet instead, and most people can't read it well enough to know whether they should be relieved or terrified, so they assume the worst by default.

What I would leave alone: Talonguard's engineers already read the full eval report every release, threshold sweep and all, and they should keep doing exactly that. The card isn't for them. It's only for the people who will never open that report, on purpose or otherwise.

The lesson: a number on its own isn't understanding, it's a placeholder for understanding. Brumley needed a real example and a real second number before "95 percent" meant anything he could act on.

03Now here is the same thing as a story

The short version above is what you'd actually say out loud. Read this one for why the fix had to be a one page card, not a promise to "communicate more clearly."

Every Thursday afternoon, Brumley Grimwade clears his calendar for renewal calls. He's run Customer Success at Kessley Mail for five years, long enough to know that a calm voice and a real answer save an account faster than any discount ever has.

Talonguard had been quiet in his calls for months. Nobody at Wickstead Hardware, a fourteen store chain that had used Kessley Mail's inboxes since 2019, had ever asked him a hard question about it. He didn't bring it up either. Some Thursdays he didn't think about Talonguard at all.

Then, on a Tuesday, an email reached Salvestro Corcoran, Wickstead's accounts payable clerk. It looked exactly like an invoice from Wickstead's regular lumber supplier: same logo, same layout, one line changed, a new routing number tucked into the wire details. Salvestro did what anyone would have. The logo matched. The format matched. Nobody double checks a routing number by habit. Talonguard let it through, and he wired $22,650 before the mismatch turned up on the next month's statement.

Two honest attempts, two failed calls. Neither one gave Wickstead's ops director anything she could actually use.

Wickstead's ops director called Brumley that same afternoon. Her question was simple: "How good is this thing, actually?" Brumley did the thing that felt safest. He forwarded her Ottavine's eval sheet: precision 0.951, recall 0.949, F1 0.950, AUC 0.978, a threshold sweep chart attached underneath. She called back an hour later angrier, not calmer. She hadn't understood a single row of it, and now assumed nobody at Kessley Mail knew whether Talonguard worked either.

On the next call, three days later, Brumley tried the opposite. "It's working really well," he said. "This was a rare thing." No number behind it, nothing she could check against anything else. She didn't believe that one either. By Friday she'd drafted a note to her own team recommending they start evaluating a different vendor.

I want to say the problem was that Talonguard missed one email. It did miss one, the kind it's genuinely worst at: a first contact sender pretending to be a known vendor, no history to compare it to. But that's not really the story. Wickstead's ops director never had a real number in her head about Talonguard, not once in five years. She had a mood, built from a smooth onboarding call and two quiet years. One bad Tuesday, and there was nothing underneath the mood to catch her.

So here's the decision I would take back. A year earlier, Ottavine's team built a single line for Customer Success to quote: "Trust score: 97 out of 100." A single number was the easiest thing to hand a salesperson, and nobody built it to mislead anyone. It just flattened every kind of mistake Talonguard makes into something that looked evenly spread, when the real failures were nothing like even.

I would put a real quality card back in its place instead. Not a bigger number, a different shape of information: catches about 19 of every 20 phishing emails, wrongly holds about 1 in 500 real ones, and here's the exact kind it still misses, the kind that reached Salvestro. Replay that second call with the card in Brumley's hand: he opens with the real numbers, names the one real miss, and explains the fix already in motion. That call runs about ten minutes, not forty. Wickstead renews for two more years instead of drafting a cancellation that Friday.

And the part I'd want to tell myself, if I could go back: we thought we were protecting customers from a confusing spreadsheet. We were really just leaving them with nothing real to hold onto instead.

04SPARK, the five beats behind the one page card

Not a script for sounding reassuring. SPARK is what forces you to name the one concrete card, and prove it survives both ways a quality bar actually breaks.

This is the whole anchor in one picture. Not a claim about Talonguard in general. Two real numbers and one real miss, on a single page.

Same fraud email, same Brumley. What changes the outcome is only whether the card exists before the call happens.

Ottavine drew this line early and held it on purpose. Not because customers can't handle detail. Because detail wasn't the problem.

The recap, one line per letter: situation is a customer success team with no designed way to translate a number, payoff is one shared, calibrated question carried into every hard call, anchor is the one page card built from two real numbers and one real miss, risk is either extreme costing a real account, and keep out draws the line at a threshold walkthrough and a sales pitch, never at the numbers themselves.

05And if you want to be sure it really works, try it somewhere else

Same five letters, a municipal water utility instead of an email screen, and this time the blind spot isn't a vendor's invoice. It's a leak nobody can see from the street.

Cinnabar Penrhyn owns the quality bar for Leaklight, an AI tool Yewcroft Regional Water uses to flag meter readings that look like a hidden leak or a tampered meter, so a crew truck gets sent out before a street floods or a bill goes wrong. Yewcroft's council formed its picture of Leaklight the same ungrounded way Kessley Mail's customers did: a neighboring utility's press release claimed it "catches nearly every leak," while a resident's angry call after a truck showed up at a house with nothing wrong left the opposite impression. Hallward Osbaldeston, the council member who owns the water budget, had never seen one real Leaklight flag before either story reached him.

Mapped onto SPARK: situation is a council forming its picture of Leaklight from a rival utility's marketing and one angry resident, never from a real flag. Payoff is one shared question the council carries into every future budget vote: is this the kind of anomaly Leaklight is actually good at, or the kind it only sounds confident about. Anchor is the same shaped card: Leaklight catches about 9 of every 10 real leaks worth a crew visit, and sends a crew to about 1 in 200 homes with nothing wrong, paired with the one thing it still misses most, a slow leak inside a house wall, where the meter barely moves for weeks before a resident notices water damage on their own. Risk runs the same both directions: call it "90 percent, basically solved" and the council assumes every miss is minor, when the in wall leaks are exactly the expensive kind. Keep out draws the same line: no pitch for a bigger Leaklight contract, no walkthrough of the anomaly model's feature weights.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to the anchor, the two numbers and the one named miss.
Cost: no budget for a standing quarterly rebuild. Rebuild the card once, tied to the next real renewal or vote on the calendar, rather than a fixed schedule, the calibration still lands, it just isn't refreshed automatically.
The model got better, for real: say Talonguard's first contact catch rate climbs to 80 percent. The card still matters, because Brumley still needs the words to say "which kind of miss is this," even when the honest answer is "a rarer one now." A better model doesn't teach anyone how to ask a sharper question on its own.

Where people run it wrong.
They let an engineer take over the call and start explaining how the classifier's threshold actually works, and the card quietly becomes the technical walkthrough it was built to avoid.
They quote the card's numbers from launch day and never touch them again, so a customer hears "97 out of 100" eighteen months after the real number moved.
They tack a renewal pitch onto the end of the card, and every honest number in it gets read backward as a sales tactic.

How to use it live. Before answering a "how would you explain the quality bar" question cold, ask yourself one thing: what's the one real, current example, a genuine catch and a genuine miss, you'd actually put in front of the room. Naming that example, not a value word like "transparency" or "trust," is usually exactly what the question is listening for.

06Flashcards (tap any card to flip it)

07Check yourself Score: 0 / 0