Write the acceptance criteria for a feature I describe, live.

How to answer this, stage by stage

Nobody is grading whether you land on exactly 135. They're grading whether you scope the feature before naming a number, whether the riskiest category gets more than an equal share, and whether you close on an actual criterion someone could check. Eight moves get you there.

Let's learn

The product is a tool that reads a dish's name and its short ingredient list, then writes the two lines a customer sees under it on the menu and on the kitchen ticket.

Before the writer existed, whoever managed the menu wrote every description by hand, hunting for the right words for a new dish. About eight minutes each, and across a fifty-item menu that's over six hours, every time the menu changed. The writer drafts all fifty in under a minute. A manager just skims them and hits publish.

Before shipping any change to the writer, the product manager pulled 25 generated descriptions at random from that week's runs, read them over coffee, and shipped if most of them sounded right. About fifty minutes. When the menu only had four categories, apps, mains, desserts, drinks, that was roughly six checked lines a category. Good enough to catch a clumsy sentence.

Here's the turn. Over the next two years the menu grew to eight categories: kids, vegetarian and vegan, allergen-declared items, and seasonal specials all got added. The extra categories were never the real problem. The real problem is what the review kept doing. It stayed at 25, spread thinner every time a category got added, until some releases the allergen tab got checked by nobody at all, purely by the luck of a random pull.

At its worst, that gap cost a real customer a trip to urgent care, because the only signal an unchecked category ever produces is the complaint that comes in after it already went wrong.

The choice I would take back. The acceptance criterion for the writer said "a reviewer reads a sample of generated descriptions and confirms they sound natural and don't contradict the listed ingredients," with no category named and no bigger bar for allergen claims. That was fine when the menu had four plain categories and nothing on it carried a legal claim. It stopped being fine the day allergen-declared items became its own tab with its own claims to get right.

What I would leave alone. Drink descriptions don't need this same treatment. The writer already pulls the literal ingredient list straight from the same field the barista's ticket prints from, and an overwritten line about a latte is never actually wrong about what's in the cup, just a little flowery. Not every category is worth a doubled floor, only the ones carrying a claim someone could rely on.

The lesson. A sample size that can't say which categories it actually covered isn't really an acceptance criterion. It's a habit that felt like one.

Now here is the same thing as a story

Skip this if you already believe a flat sample and a category-shaped one aren't the same thing. Read on if you want to feel why they aren't.

Darius Kovac can tell in one read whether a menu description sounds like it came from the restaurant or from a robot pretending to be one. Three years running the description-writer feature at Countertop, a point-of-sale and online-menu platform a lot of independent restaurants use, will do that.

Darius inherited the writer the year it shipped with four categories: apps, mains, desserts, drinks. Before every release, he pulled 25 generated descriptions at random from that week's test runs and read them over coffee. If most of them sounded right, he shipped. It felt like the honest choice. Nobody was cherry-picking the good ones.

For most of a year, that felt like plenty. Early on, Darius would also open the raw ingredient data next to each description and check it line by line, just to see how the writer was actually reasoning. It usually lined up fine. So the line-by-line check got smaller. Then it stopped, replaced by one habit: read 25, ship if it sounds right.

Over the two years after that, Countertop kept adding what a menu could hold. A kids' tab. A vegetarian and vegan tab. An allergen-declared tab, for restaurants that wanted to flag gluten-free and nut-free items. A seasonal specials tab. Eight categories shared the menu now. The sample stayed at 25.

The same sign-off check, sized two different ways. One is a number that sounds thorough. The other can tell you exactly which category it actually covered.

Then came a Tuesday that wasn't an audit. A location using Countertop had listed a flourless torte as gluten-free online, copying wording the writer had generated for a similar dessert at another location months earlier. The torte at this location used a regular crust. A customer with celiac disease ordered it off the online menu, trusting the label, and ended up in urgent care that night.

Countertop's trust and safety team pulled every allergen-tagged description generated platform-wide that quarter, about 300 of them, and checked each one against the real ingredient feed by hand.

Twelve of the 300 had the claim wrong, a miss rate of about 4 percent, on the one category where a miss isn't a style problem. Darius pulled the acceptance criterion that release had shipped under. Twenty-five examples, read at random, with no line saying which category they came from. Across eight categories, there was no promise the allergen tab had ever been touched, and on the release that shipped the torte's description, it hadn't been.

Darius rewrote the criterion that sprint: eight named categories, a floor of 15 written cases each, doubled to 30 for allergen-declared items, for 135 test cases before any release could ship. Run against the writer as it stood, the allergen category alone scored 71 percent clean against the 98 percent bar Darius set for it. A fix that made the writer state an allergen claim only when it matched the structured ingredient tag, never inferred from a similar dish, took about a week to ship. Rerun, the category cleared 99 percent. Over the next two releases, zero generated allergen claims mismatched the source data, and the sign-off review, even with the bigger floor, took under five hours.

The thing I'd tell myself, the week I set that first flat number at 25: it felt thorough because it was a fixed effort, not because it had ever touched every part of the menu that could hurt someone.

BOUND, said out loud before you write a single test case

This is a sizing question about how many test cases to write and for which menu categories, not a person's trust flipping between two settings, so BOUND fits and FLIPS doesn't.

B, break it down. The sign-off test count equals the number of live menu categories, times a minimum floor of written cases per category, with a bigger floor for any category where a wrong claim can hurt someone. Countertop's default menu carries eight named categories.
O, own the numbers. A floor of 15 cases per category, since under that a reviewer can't tell a fixed failure mode from a few cases that happened to come out fine. Eight times 15 is 120. Allergen-declared items get that floor doubled to 30, adding 15 more. Total: 135.
U, use a range. A small independent place, three ordinary categories plus the required allergen tab, comes to 75. A franchise operator running fourteen categories across regional menus and catering pushes it past 220. Start at 135, the plain eight-category split most restaurants on the platform actually run, and only add a category once the menu structure really has one.
N, nail the sanity check. 135 test cases, at about two minutes each to check the line against the source ingredients and the ticket character limit, is around four and a half hours of one reviewer's time. A real afternoon, not a rounding error, and it fits inside the day before a launch sign-off meeting.
D, direction. A new category showing up moves the total more than the floor does. Going from eight categories to a franchise operator's fourteen adds 90 cases on its own. Raising the floor from 15 to 20 across all eight only adds 40. If the total needs to shrink, check whether two categories actually overlap before thinning the floor, since a thin floor is exactly what let the allergen claim slip through the first time.

And if you want to be sure it really works, try it somewhere else

A listing-description writer drafts the paragraph under a property's photos for a real estate platform, pulling from the agent's own notes on beds, baths, and the block.

B, break it down. The sign-off test count equals the number of property types the platform lists, times a floor of written cases per type. A typical set runs to five: single-family, condo or co-op, rental, new construction, and land or lots.
O, own the numbers. A floor of 12 cases per type, since a listing carries fewer distinct facts than a menu line. Five times 12 is 60. The category touching fair-housing-sensitive language, any wording near schools, safety, or "who lives here" framing, gets that floor doubled to 24, adding 12 more. Total: 72.
U, use a range. A small single-market brokerage running three property types, still required to carry the fair-housing category, comes to 48. A platform spanning several states and nine property types pushes it to 120. Start at 72, the plain five-type split most single-market platforms actually run.
N, nail the sanity check. 72 test cases, at about a minute each since a listing blurb is shorter than a menu line, is about an hour and fifteen minutes. That fits inside one afternoon before the writer ships to a new market.
D, direction. Here the lever isn't a new property type so much as a new way protected-class language sneaks in, a listing praising a "quiet, established block" right next to a school rating. Each new phrasing pattern like that needs its own written cases inside the fair-housing category, whether or not the platform ever adds a sixth property type.

Swap the trigger and it still runs.
Speed: an interviewer caps the answer at 90 seconds. Skip straight to the split: 15 per category, doubled to 30 for the one that can hurt someone, 135 total, checked against a five-hour review. The build-up backs it up if asked.
Cost: instead of asking for a case count, a manager caps the review budget at three hours. Work backward: at two minutes a case, that's 90 cases. Cover the allergen category's full 30 first and treat the rest of the floor as next sprint's work.
The model got better: a new version of the writer almost never invents an ingredient anymore in general use. The floor doesn't drop on its own. Rerun the allergen slice specifically, since a clean overall number says nothing about the one category that used to fail.

Where people run it wrong.
They size the check by what sounds thorough, "read a few dozen and see how they look," instead of a floor times a named category count.
They write the criterion once at launch and never add a floor when a new category gets added to the menu.
They let the allergen slice grow until it's most of the review, so the check stops reflecting what the writer does for the ordinary majority of the menu that carries no claim at all.

How to use it live. Say the equation before any number: "the test count is category count times a per-category floor, doubled wherever a miss actually costs someone something." That buys the time to name real categories instead of guessing a round number that sounds thorough.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0