What is the right batch size for human review, and why does it matter?

Northfield Comfort Systems runs an AI model that reads incoming HVAC service tickets and drafts a diagnosis and urgency level for each one. Bettina Kirsch is the dispatcher who reviews the AI's calls before technicians get sent out, working through however many tickets happen to be sitting in her queue.

How to answer this, stage by stage

Nobody is grading whether you can name a number. They're grading whether you know which kind of mistake that number is protecting against.

Let's learn

Picture ninety service tickets sitting in one pile, all waiting for a single look at five o'clock.

Northfield Comfort Systems' AI reads each incoming HVAC ticket, drafts a likely diagnosis, no heat, a failing compressor, a refrigerant leak, and flags an urgency level. Bettina Kirsch reviews the AI's calls before a technician gets dispatched, and can bump, confirm, or reclassify any ticket in seconds.

Before the AI, every ticket got a phone call and a few minutes of Bettina's own judgment, one at a time, all day. Slower, but nothing sat unseen.

Same ninety tickets. Only the shape of the pile changes.

Now Northfield reviews tickets in one batch, at the end of the day. On a slow day, twenty tickets, that's fine. On a busy day, ninety or more, that same habit means whatever came in at 9am waits for a 5pm look, whether or not anyone's tired by then.

Here's the turn: the real cost of a bad batch size was never the extra minutes at the end of a long batch. It's that everything near the bottom of a big pile gets the least careful look, right when it might be the one call that can't wait.

What I would leave alone: routine maintenance tickets, filter checks, seasonal tune-ups, don't need this at all. A slightly stale review on one of those costs a scheduling inconvenience, nothing more.

The lesson: a batch size decided once, at one volume, doesn't stay right as volume grows. Nobody thinks of "how many tickets pile up before I look" as a safety setting, until it is one.

Now here is the same thing as a story

The short version above is what you'd say defending this to Northfield's operations lead. Read this one for the Friday it nearly went wrong.

Bettina Kirsch has dispatched HVAC technicians for Northfield for five years. She can tell a nuisance call from a real emergency before the ticket's even fully loaded, from the words a customer used and the time of year.

Both boxes are the same size on a screen. They are not the same size in real life.

For most of the year, the end-of-day batch worked fine. Twenty, thirty tickets, reviewed once at close. She'd never once missed something that mattered.

The ticket didn't get lost. It got buried under everything that arrived after it.

Then came a Friday in July. A commercial customer's walk-in freezer ticket landed at 9:40 in the morning, flagged urgent by the AI. Nothing looked wrong yet. Bettina still reviewed once a day, at five, the way she always had.

By five o'clock, ninety four tickets sat in that day's batch. She worked through them in order, the way she always did, and somewhere past ticket eighty, tired and moving fast to clear the queue before heading home, she confirmed the freezer ticket's diagnosis but didn't register how long it had already been sitting. The technician wasn't dispatched until the next morning.

The owner called back angry before any inventory actually spoiled. Nothing broke. It was close enough that it stuck with her.

With batches capped at ten and reviewed every forty five minutes, that same Friday plays out differently. The freezer ticket sits in a batch of eight others at 10am, gets Bettina's full attention, the same attention every ticket in a small batch gets, and a technician is on site by early afternoon.

I built the review process around one look a day because that was the simplest thing to ship, and it was true at fifteen tickets a day. I never asked what would happen to the same design once volume tripled.

PICK, in one screenNot a lecture on batch processing. PICK is what tells you which error you're actually protecting against.

Batch size looks like one number. It is really four smaller decisions wearing one number.

The recap, one line per letter: position is small, frequent batches over one big daily pile, impact is re-orientation cost against buried-ticket cost, cost asymmetry is optimizing against the hidden emergency rather than the visible delay, and kill criteria is loosening batch size only as real emergency-detection accuracy actually improves.

And if you want to be sure it really works, try it somewhere elseSame four letters, a veterinary telehealth line instead of an HVAC dispatch desk.

Cragmoor Veterinary Telehealth runs an AI model that triages pet owner symptom submissions, flagging likely urgency, from a mild skin rash to suspected toxin ingestion. A vet triage nurse reviews the AI's calls in batches before deciding which cases get a callback within the hour versus a routine morning slot.

The same four checks apply whether the queue is service tickets or vet symptom reports.

Mapped onto PICK: position is the same, small frequent batches over one large end-of-shift pile; impact is a routine rash case waiting a bit longer versus a suspected toxin case sitting unreviewed for hours; cost asymmetry is that a delayed rash costs an annoyed pet owner, while a delayed toxin case costs an animal's life, so the batch design has to optimize against the second one even though it's rarer; kill criteria is the same, loosen batch size only once the AI's own toxin-flagging accuracy is trusted enough that a missed case becomes genuinely unlikely rather than merely rare.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "small, frequent batches, because batch size decides how long an emergency gets to hide, not how fast the queue clears," and stop.
Cost: there's no staffing budget to review every forty five minutes around the clock. Say so honestly, and shrink batch frequency overnight only, when true emergencies are rarer, while keeping it tight during peak daytime hours.
The model gets better, for real: if the AI's urgency flagging becomes highly reliable, the safe batch size can grow, since the buried-emergency risk that set the original ceiling has genuinely gone down.

Where people run it wrong.
They set batch size once, at a lower volume, and never revisit it as volume grows past what made that size safe.
They treat batch size purely as a throughput or staffing question, with no thought to what's hiding inside a large pile.
They shrink to reviewing one ticket at a time to "be safe," and lose more time to constant interruption than the risk was ever worth.

How to use it live. When someone asks about batch size, ask one question back to yourself first: what's the worst thing that could be sitting at the bottom of the biggest batch I'd allow? Size the batch to that answer, not to whatever feels efficient.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0