Write acceptance criteria for an AI feature that extracts fields from an invoice.

How to answer this, stage by stage

Seven moves. The trap is naming one confident percentage that hides seven different fields, each with its own way of going wrong.

Let's learn

Here's what happens when one confident percentage in a spec meets an invoice that needs seven different things to go right at once.

Say Runwell, a bookkeeping tool for small businesses, ships a feature called Capture. Point a phone at an invoice, or forward it by email, and Capture reads it: vendor name, invoice date, invoice number, subtotal, tax, total, and PO number. Seven fields, every time, before the invoice can post to a client's books.

Before Capture, a bookkeeper typed all seven fields into the ledger herself, for every invoice, from every client. About five minutes each. A bookkeeper carrying five small-business clients might key sixty invoices a week. That's five hours gone before she's touched anything else on her list.

The acceptance bar the team wrote before launch was one line: field extraction must be at least 99% accurate. Pooled across every field on every invoice, Capture landed at 98%, close enough to the line that nobody questioned it.

Here's what that cost at its worst. Capture was built to auto-post any invoice above a confidence threshold, and the model was often confident and wrong on stamped PO numbers, not unsure and cautious about them. One invoice from Trellwood Supply, a Runwell client, posted against a PO number that had already been closed on a different order. Nobody caught it until the monthly reconciliation, and a real vendor payment sat a week behind schedule while the accountant untangled it.

The choice I would take back. Writing the bar as one blended number, 99% accurate, instead of a target per field plus a computed whole-invoice pass rate. That was a reasonable shortcut before launch, when the only test set was fifty clean demo invoices and every field behaved about the same. It stopped being reasonable the day a client with stamped, handwritten PO numbers started sending sixty invoices a week.

What I would leave alone. The memo field, the free-text line some vendors add about a return policy or a thank-you note, doesn't need any of this. Nobody reads it before approving a payment, so it can be wrong constantly and nothing downstream breaks. Save the rigor for the fields that actually gate whether an invoice posts.

The lesson. A percentage that sounds strict, ninety-nine, can still mean seven separate ways to fail on the same document. If the bar doesn't say what a whole passing invoice looks like, it isn't really a bar yet.

Now here is the same thing as a story

The short version is above. Read on if you want to feel why a closed purchase order almost cost Runwell a client.

Rosalind Fenn can read a spec and find the one sentence doing too much work before anyone else in the room notices it's there. She's been the product lead on Capture, Runwell's invoice and receipt extraction feature, since before it had a name, back when it only worked on the handful of demo invoices the team had scanned themselves.

She wrote the acceptance bar herself, the week before launch: field extraction must be at least 99% accurate, checked against fifty vendor invoices the team had gathered for the demo. Clean scans. Typed fields, mostly. A PO number here and there, usually printed, never stamped.

Seven fields hiding inside one number. On the demo set, none of them behaved differently from the rest.

For a year, that bar did its job. Runwell's early customers were small operations with a handful of vendors each. Capture hit 99% and stayed there. Rosalind checked the dashboard once a month, saw the number holding, and moved on to the next thing on her roadmap.

Then Runwell signed Trellwood Supply, a regional building-supply chain with sixty invoices a week and a purchasing department that still stamped its PO number in the corner of every vendor invoice by hand, the way it always had.

Noemi Salgado is the bookkeeper who runs Trellwood's books inside Runwell, along with four smaller clients. For the first two weeks, Capture looked fine. The dashboard still said 98%. She barely opened a flagged invoice.

Then a payment got stuck. A vendor called Trellwood's office asking why a $4,200 order hadn't been paid. Noemi pulled it up. Capture had posted it weeks earlier, against a PO number that had already been closed out on a different order. The model had read the stamp with total confidence. It was wrong.

She dug through the last month's invoices. Eleven more, same pattern. All against Trellwood, all PO number misreads, all posted with nothing flagged for review, because Capture's confidence threshold had never been tuned to catch a field it was consistently, wrongly sure about.

Here's the turn. Capture never stopped hitting its number. The dashboard was telling the truth the whole time. What the dashboard couldn't say was that the small share it was missing wasn't spread evenly across seven fields. It was landing, again and again, on the one field that decided whether an invoice matched the right purchase order.

Monday morning, Rosalind pulled the original bar she'd written herself, a year earlier. One line: 99% accurate. No mention of which fields, no mention of what a passing invoice actually required.

She rewrote it as seven fields, each with its own target and range, and a whole-invoice pass rate underneath all of them: 86% at the point estimate, 73 to 89% depending on which end of each field's range showed up on a given week. And one line flagged in bold: PO number is the field to watch, because its accuracy swings that whole-invoice number more than any other field, more than the total itself.

Trellwood's purchasing department switched to a printed PO number on new orders within a month, at Rosalind's request, and Capture's fallback started sending any stamped PO number below its new, tighter confidence bar straight to Noemi's queue instead of guessing. Six weeks later, Trellwood's flagged queue settled at about six invoices a week, caught and fixed, not swallowed silently into a wrong posting.

The part I'd go back and tell myself: I wrote down how accurate it needed to be. I never wrote down what a whole invoice needed to look like to actually pass.

BOUND, sized to Capture's acceptance bar

This is a sizing question wearing a critique's clothes, so BOUND fits and FLIPS doesn't. Nobody's habit snapped here. A number that was true on average quietly stopped describing the thing it was supposed to guarantee.

The range against a felt cost. Both bounds sit close enough to what one bookkeeper can absorb, until the low end.

B, break it down. Whole-invoice pass rate equals every required field's accuracy multiplied together: vendor name times invoice date times invoice number times subtotal times tax times total times PO number. Not their average.
O, own the numbers. Total, tax, and subtotal, the fields tied to money, sit at 99 to 99.5 percent. Invoice date and invoice number sit at 98. Vendor name at 97. PO number at 95, the field that's usually stamped or handwritten and the one the model struggles with most.
U, use a range. Multiplied together, that's 86 percent at the point estimate, 73 to 89 percent depending on which end of each field's own range shows up in a given week.
N, nail the sanity check. Trellwood sends about 60 invoices a week. At 86 percent, that's roughly 8 a week landing in Noemi's queue, about 4 minutes each. Real, but survivable. At 73 percent, the low end, it's 17 a week, over an hour just for one client, and that's the point where I'd delay a launch, not ship it.
D, direction. PO number's own range swings the whole-invoice pass rate by about 6.4 points, more than double vendor name's 3.1, the next closest, and more than ten times total amount's 0.6, even though total amount is the field everyone assumes matters most because it's the money. The field to watch first is rarely the one that sounds highest-stakes.

And if you want to be sure it really works, try it somewhere else

Basalt Diagnostics builds the software labs use to read a paper requisition form, patient ID, date of birth, ordering physician, test code, priority flag, specimen type, before a blood sample ever reaches a machine. The same one-number trap shows up wearing a different coat.

B, break it down. Whole-requisition pass rate is the product of six required fields, not their average.
O, own the numbers. Patient ID sits at 99.8 percent, because it's barcode-assisted, not typed. Date of birth at 99.5. Test code at 98. Priority flag at 99. Specimen type at 96. Ordering physician's NPI number, a ten-digit code usually handwritten on the form, at 97.
U, use a range. Multiplied together, about 90 percent at the point estimate.
N, nail the sanity check. Perrin Vasquez, Basalt's eval lead, checked that number against how many requisitions a lab tech already re-keys by hand on a busy morning, and 90 percent landed close to what techs already tolerated without complaint.
D, direction. Here the swing field isn't the safety-critical one. Patient ID is barcode-scanned and barely moves, about 0.4 points across its whole range. The physician's NPI number, handwritten and free-text, swings the total by about 6.5 points, nearly the entire gap between a comfortable pass rate and an uncomfortable one.

The old decision Basalt would take back is a cousin of Rosalind's, not a copy of it. Runwell had merged seven fields into one blended number. Basalt had picked one flat target, 97 percent everywhere, and applied it to a barcode-scanned field and a handwritten one alike. Same mistake, wearing a different shirt: treating fields that behave nothing alike as though a single number could describe both.

Swap the trigger and it still runs.
Speed: an interviewer wants a usable bar before the meeting ends. Same seven fields, same equation, PO number's target stated as a range instead of researched to the decimal, tightened once real client data exists.
Cost: engineering caps how much review capacity goes to flagged PO numbers. Same equation, solved backwards: at that fixed budget, the achievable pass rate falls from 86 toward 82 percent, so the flagged-queue estimate has to grow to match, not get quietly ignored.
The model got better: a new OCR model cuts vendor-name errors in half, from 97 to about 98.5 percent. The whole-invoice number only moves from 86 toward about 88, a small gain, well short of what fixing PO number alone would buy. Fixing a field that isn't the sensitive one buys less than it sounds like it should.

Where people run it wrong.
They write one blended number because it reads clean in a slide, and have no way to say which field to check when a real invoice fails.
They average field accuracy instead of multiplying it, so the bar always sounds better than the invoice-level reality behind it.
They chase the field that looks highest-stakes, usually the dollar amount, instead of the field that actually swings the total most, which is often the one nobody's watching.

How to use it live. Say the equation out loud before naming a single number: list the required fields, say you're computing a pass rate as a product, not an average, and that a real target needs a range once you know which field is shakiest. That buys thinking time and tells the interviewer a real number is coming, not a guess dressed up as one.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0