Describe the difference between an eval spec and a test plan.

How to answer this, stage by stage

Eight moves. Most of the weight sits in stage four: this question is really asking whether you understand that a checklist can pass every single time and still never once ask if the thing is good. Every stage has the actual words to say.

Let's learn

Every year, before any of this existed, two writers on Rahel Tesfaye's team wrote every line of copy shown during Tidemark's signup by hand. A new screen took about two weeks: a draft, a compliance read, a legal read, a round of changes. In a good year they touched maybe twenty screens total.

Then Tidemark built a tool that drafts that copy instead: the welcome screen, the identity-check prompts, the error messages, the little nudges that keep someone finishing signup. Last quarter alone it produced 140 new variants, across screens, customer groups, and the three languages Tidemark supports.

Before any of that copy ships, it has to pass an eval spec Rahel and a QA engineer built eighteen months ago, when the tool first launched. It checks the character limit. It checks that the customer's name and the dollar amount fill in correctly. It checks that the required legal disclosure text is there, word for word. It checks for leaked personal data. It checks that a toxicity filter comes back clean. Six checks, every one pass or fail.

For the first two months, Rahel read every new variant herself before it shipped, even the ones that already passed. Then she started reading only the first of each new type. Then, once the eval spec had come back green for twenty straight weeks, she stopped reading personally at all. A pass was the ship signal, on its own.

One of those variants was built for a specific moment: a customer's first transfer over $2,000, which triggers Tidemark's routine identity reverification. It read: "We've noticed unusual activity on your account. Verify your identity now or your account may be frozen." It passed every check. Right length. Right amount filled in. Disclosure text present. No leaked data. Clean toxicity filter.

It ran for eleven days. About 3,400 people saw it. Completion on that screen fell from its usual 71% down to 38%, and the ticket count above kept climbing, a little more each day, until a support lead noticed the tag count on a Friday and messaged Rahel almost as a joke: was Tidemark trying to scare people during a normal ID check?

The extra tickets were not really the problem. The real problem is that nothing in the eval spec had ever asked whether that line sounds like an accusation. A hundred percent pass rate never once meant the copy was good. It only ever meant the copy ran.

At its worst, this is what a regulator reads when they ask why a routine ID check sounds like a fraud accusation to the person who got it. Nobody at Tidemark meant to accuse anyone of anything. That's not what shows up in a support chat log, though, and it's not what a customer who just wanted to move their own money feels reading it.

What I would leave alone. A tooltip explaining that a field auto-fills from a linked bank account doesn't need any of this. Nobody's anxious reading that line, and there's no accusation it could make by accident. That one can stay a pass/fail check for good.

The lesson. A checklist built to test whether something runs will always miss whether it's good, because that was never the question it was built to ask. You have to build a second, different question on purpose, or nobody's asking it at all.

Now here is the same thing as a story

Pull this one out when there's room to sit with it, not just tick it off a list.

Every Monday morning, Rahel opened the eval-spec dashboard before she opened anything else, and for a long time, all it ever showed her was green.

She'd written Tidemark's identity-check copy by hand for three years before the generator existed, and she had a good ear for the one thing that mattered most there: whether a line sounded like a routine ask, or like an accusation. Ask her to read a paragraph of KYC copy and she could tell you in five seconds which one it was.

The generator launched eighteen months ago. Rahel and a QA engineer built its eval spec in one afternoon meeting, working from the QA team's existing test-plan template for a different feature, because building something new from scratch felt like a week nobody had. Six checks. Right length, right fields, disclosure text present, no leaked data, clean toxicity filter, reading level under a target grade. All pass or fail.

For the first two months, Rahel read every new variant herself before it shipped. Not because she distrusted the checklist. Because the checklist was new, and she wasn't yet. Every Monday she'd open the week's new copy and read it top to bottom, and every week it read fine.

Then she started reading only the first of each new type, the rest she trusted to look the same. Then, once the eval spec had run clean for twenty straight weeks, not one flagged issue, she stopped opening the dashboard's copy tab at all. She just watched the pass count. Green meant done.

People are switches, not dials

The trigger was small. A support team lead, doing a normal end-of-week look at ticket tags, saw a cluster mentioning "frozen" and "flagged" that hadn't been there a month ago. He messaged Rahel almost as a joke: were they trying to scare people during a routine ID check, or was that on purpose?

Rahel pulled up the eval-spec dashboard first. Fully green, same as every week. Then, for the first time in months, she opened the actual copy and read it herself. "We've noticed unusual activity on your account. Verify your identity now or your account may be frozen." Written for a customer's first transfer over $2,000, a completely routine check. It had passed every one of the six checks. It had been live for eleven days. About 3,400 people had read it.

She didn't just fix that one line and move on. She and a compliance lead spent two days reading every new variant touching identity or account limits from the last twenty weeks, about sixty four of them, and found eleven more with the same pattern: technically clean, functionally fine, and written in a way that would make a stressed reader feel accused.

I want to say the problem is that someone wrote a careless line. They didn't. "Verify now or your account may be frozen" is a completely reasonable thing to write if the only questions in front of you are about length and fields and disclosure text, and nothing tells you tone is also your job. That's not really the story either. Rahel never had a number in her head for how carefully to read a new variant. She had a feeling, and it only had two settings: this checklist is proof, or this checklist is a start. Twenty clean weeks flipped it for good, and one bad line flipped it back.

Here's the call I'd take back. In that first afternoon meeting, eighteen months ago, when we opened the QA team's test-plan template and started filling it in, nobody in the room was doing anything careless. There wasn't a real quality template yet, and everyone had used a test plan before, so it was the fastest thing on hand. Nobody ever came back and asked whether a checklist built to test "does it run" could also carry the weight of "is it kind."

I'd add a real quality rubric. Not instead of the six checks, alongside them. Two people score anything touching identity or account limits on three questions: does this imply wrongdoing with no cause, is the tone right for the actual stakes, would a stressed reader feel accused. That runs before ship, same as the six checks, just asking a different kind of question.

Run the same trigger again, three months after the fix. A new market needs its own KYC copy, at a lower transfer threshold. The first draft reads a little urgent, some pressure language that isn't quite an accusation but is close. It passes all six of the old checks, clean as anything. The quality rubric catches the tone inside a day. It ships rewritten. No real customer ever sees the first draft.

If I'm honest, nobody made a bad call in that first afternoon meeting. The bad call was mine: three years writing that copy by hand, and I still reached for the QA team's template instead of asking what a checklist for "good" would even need to look like.

The five moves behind a checklist that couldn't see tone

The letters matter less than which one breaks first. Here's the same five steps, mapped onto Rahel's eval spec.

FLIPS, five rows

A small move in the streak. A hard snap in what she did about it.

"She got less careful about reading copy" is a diagnosis anyone can offer after the fact. The harder part is naming the exact habit that had to break first, opening the copy tab on a Monday, and showing there was no smaller version of it left once it did.

And if you want to be sure it really works, try it somewhere else

Saltmarsh Family Pharmacy runs a tool that drafts the counseling scripts pharmacists read aloud, or print, at pickup, across its 40 stores. Same question, a pharmacy counter instead of a signup screen, and a flip that fires on a joke, not a bad number.

F. Jocelyn Panganiban, pharmacy operations lead at Saltmarsh, who set the standard for what a safe counseling script sounds like across every store.
L. Whenever a pharmacist felt a draft script sounded off, too clinical, too rushed for an older patient, they'd tag it in the shared queue: "AI draft, needs a look." That tagging habit is what fed Jocelyn's team the pattern of what needed fixing.
I. A different flip. Pharmacists don't stop using the tool, and they don't check less. They stop saying they used it. Flags every off script for review, or quietly tweaks a word and lets it go with no flag at all. Nothing in between once a regional supervisor joked, in front of the team, that a pharmacist had "just let the robot write it" after a patient complained.
P. The flag was visible by design, named next to whoever raised it, so Jocelyn's team could ask follow-up questions. That was free when using the draft was normal. It became a cost the moment it was a mark against you.
S. Make the flag anonymous: it records that a script was AI-drafted and needs a look, never who flagged it. A new script for a complex, high-dose medication gets flagged again within days, not silently patched by an embarrassed pharmacist and left in rotation.

Flashcards (click a card to flip it)

Eight fixed slots, pulled straight from the answer above.

Check yourself Score: 0 / 0