Explain how to write acceptance criteria that a QA team can actually execute.

How to answer this, stage by stage

Seven moves, from pinning the question to one real rubric to the line you'd close on.

Let's learn

Here's what happens when a written rule feels clear to the person who wrote it, and turns into a coin flip for the person who has to use it.

Say a clinic uses an AI tool that reads a scanned insurance card and a patient's handwritten history sheet, and fills out the intake form on its own: name, date of birth, insurer, current medications, allergies. Before any new version of it goes live, a small QA team checks a sample of its outputs against a written rubric, ten short criteria, one sentence each.

When the rubric only covered simple fields, name, date of birth, insurer ID, it worked fine. Two testers reviewing the same 25 outputs a week landed on the same call almost every time, agreement close to 98 percent, and a release cleared QA in under a day.

Then the tool grew a current-medications field, and one more sentence got added to the rubric: "the current medications list must be accurate and complete, matching the source record." The blended agreement rate across the whole rubric barely moved, still around 94 percent. But on that one criterion by itself, agreement dropped to 61 percent. Two careful testers, looking at the exact same output, split on it almost as often as they agreed.

Here's the part that matters. The extra disagreement isn't the real cost. Testers splitting on one field is annoying but survivable. The real cost is that whichever tester's queue a case lands in now decides what ships, and nobody chose that on purpose.

At its worst, a discontinued medication gets listed as current, passes because it happened to land with the tester reading the rule one way, and ships. It sits quietly on a chart until someone, a nurse, a doctor, a patient correcting her own file, happens to notice it by hand. The rubric didn't fail loudly. It failed the way a coin flip fails: fine most of the time, and silent about the rest.

What I would leave alone. A field like preferred pharmacy doesn't need this. If the autofill gets it wrong, the front desk asks the patient and fixes it in five seconds, no real cost either way. Writing worked examples for a field like that would spend effort protecting against a mistake nobody's actually hurt by.

The lesson. A criterion that reads clearly to the person who wrote it can still be a coin flip for the person applying it. The only way to find out is to write down the hard case and its answer together, before two testers find it for you.

Now here is the same thing as a story

The short version sits above. Read this one for how one added sentence turned into a three-month gap nobody was watching.

Beatrix Coleman could read a scanned intake sheet faster than most people could read their own handwriting. Three years testing Two Rivers Pediatrics' intake assistant had taught her to spot a mistake in a completed form almost on sight, a transposed digit in a birth date, an insurer ID one character short.

For the first year, that skill was mostly enough. The rubric she worked from had ten criteria, one sentence each: name matches the source, date of birth matches, insurer ID matches. Straightforward stuff. She and two other testers, rotating through each week's release candidates, agreed on almost every call. A release cleared QA by Thursday afternoon most weeks.

New fields arrived slowly. Allergies in April. Current medications in June. Each time, the product manager, Simeon Farrow, added one line to the rubric and posted it in the team channel: new criterion, go ahead and start grading against it.

At first, every odd case went straight to Simeon. A patient's sheet listed a medication with "no longer taking" scrawled next to it in different ink, clearly added later. Beatrix messaged him before marking anything. He wrote back within the hour: that one's a fail, it's listed as current on the source, mark it wrong. Fine. She had her rule, at least for that one case.

By August, the odd cases weren't so odd anymore. A discontinued medication showed up in roughly one release candidate out of four. Beatrix stopped pinging Simeon for each one. She'd decided, mostly on her own, that any medication with a stop note next to it counted as inaccurate if the model listed it plainly as current, and she graded it that way from then on. She never announced the decision. It just became how she did the job.

Aaron Buckley, one of the other two testers rotating through the same queue, had quietly landed somewhere else. He read the same rubric line, "the current medications list must be accurate and complete, matching the source record," and to him, accurate meant matching the paper exactly, stop note or not. If the source said the drug, the output said the drug, that was accurate, full stop.

Neither of them was being careless. Neither had ever seen the other's reasoning written down anywhere, because there was nowhere to write it. The rubric had one sentence for the field, and each of them had built a private rule underneath it.

Simeon held the pen. Beatrix and Aaron each got the same sentence, and had to decide what it meant on their own.

In September, a discontinued blood pressure medication landed in Aaron's queue instead of Beatrix's. He marked the output a pass; it matched the source. The release shipped that Friday.

Three weeks later, a nurse at the Fairhaven office pulled up a patient's intake form before a well visit and read off her medication list to confirm it. The mother stopped her partway through. "I haven't taken that one in years," she said. "I told the front desk that when I filled out the paper." Nobody was hurt. The nurse caught it in the fifteen seconds it took to ask. But the form had been wrong since the day it was filled in, and it had cleared QA clean.

Simeon ran a routine consistency check that quarter, mostly out of habit, before he'd even heard about the near miss. He had two testers blind-grade the same sample of 40 release candidates and compared their calls. Overall, they agreed on 94 percent of it, which looked fine on a dashboard. Broken out by field, current medications sat at 61 percent. Eleven borderline stop-note cases had come through that month. The two testers had split on six of them.

The step that should sit third, and doesn't

Then the near miss reached him, and the number stopped being a dashboard curiosity. Beatrix's queue and Aaron's queue had each been applying a private rule the rubric never wrote down, and neither of them had any way to flag that the rubric, not their own judgment, was the actual problem. Beatrix's manager had, a month earlier, mentioned in a one-on-one that her fail rate on medication fields ran higher than Aaron's, and asked if she was being too strict. She hadn't had an answer for that. She wasn't sure herself anymore.

Simeon had been in the room in June when the current-medications criterion got written. It was one line, added quickly, alongside a real release deadline. Writing worked examples for every possible edge case felt like overkill for a field that, on the surface, just meant "copy the medication list correctly." Nobody thought a single sentence could mean two different things to two people who both cared about getting it right.

Run the same quarter again, with the fix in place. Every criterion carries a real pass example and a real fail example, current medications included: pass is a drug transcribed with a discontinued flag, fail is the same drug listed plain with none. Beatrix and Aaron now work from the same rule instead of two private ones. The weekly blind check still runs, and this time it catches something fast: a new criterion, added for insurance secondary-payer notes, comes in at 68 percent agreement in its first week. It goes back to Simeon for examples before a single output ships, not three months and a doctor's visit later.

What I'd tell myself, back in June: we asked whether the sentence was clear. We never asked whether it was clear to the person who didn't write it.

GUARD, when one sentence reads two ways

This is a risk question wearing a process question's clothes. "Write criteria a QA team can execute" sounds like a documentation problem. The real test is whether the person applying the rule ever gets a say in whether the rule itself is the broken part.

And if you want to be sure it really works, try it somewhere else

A fabric mill runs an AI tool that grades incoming rolls from photos for shade variation, tears, and weave defects, before a batch ships to a garment maker. Different mill, same five letters, same trap.

G, groups. The quality manager who writes the defect-grading rubric, and the floor inspectors who grade each roll's sample photos against it.
U, unequal. The shade-variation defect has no attached example of how much variation counts as a fail. On a borderline roll, two inspectors reading the same photo and the same one-line rule land on opposite grades.
A, ability to contest. An inspector who passes a roll that later gets rejected by the garment maker gets told she read the light wrong. She has no way to say the rubric's shade-variation line was never actually defined.
R, reduce. Attach a labeled pass photo and a labeled fail photo to the shade-variation criterion, not a sentence describing acceptable variation.
D, detect. Re-grade a shared sample of rolls blind across inspectors every week, and track the agreement rate by defect type, flagging any type that drops under the bar.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0