How would you prototype an agent workflow before building the agent?

How to answer this, stage by stage

Nobody is grading whether you land on exactly 32. They're grading whether you define the sizing logic before touching a number, whether the costliest categories get more than an equal share, and whether the total survives a check against what a person can actually sit down and do. Seven moves get you there.

Let's learn

What does it actually prove when a prototype gets every test case right?

Say we build an agent that reads every ticket landing in a helpdesk queue and decides where it should go: billing, login, a bug report, a cancellation, a how-to question, or a plan upgrade. Before any of that gets built for real, tools wired up, a model calling them, someone has to prove the decision logic underneath it actually holds.

The first way to prove it: pull one easy ticket from each category, walk the logic through by hand, get six for six right. Fast. Clean. Convincing in a room.

The second way: pull a real week of tickets, nothing picked out on purpose, and run the same logic against all of it.

At its worst, that gap costs a whole team's quarter: engineers spend weeks wiring up real tool calls around logic that quietly falls over the first time it meets a ticket that mentions two things at once.

The choice I would take back. The first test set was built from the single newest, cleanest ticket in each category, instead of a real, unpicked batch. The fix: six categories, a floor of four scenario types each (clean, missing a detail, touching two categories at once, angry or garbled), twenty-four scenarios. Double the floor for cancellation and billing, since a wrong route there either loses a customer who wanted to be saved or moves money that shouldn't move, adding eight more. Thirty-two scenarios, not six, before a single tool call gets built.

What I would leave alone. How-to and feature-request tickets don't need this treatment. Route one of those wrong and a human just moves it, thirty seconds, no customer ever notices. Not every category is worth doubling the floor for, only the ones where a miss costs something a customer can actually feel.

The lesson. A number like "six for six" sounds like proof. It's only proof of whatever you were careful enough to leave out of the test.

Now here is the same thing as a story

Skip this if you already believe a hand-picked walkthrough and a real, unpicked batch aren't the same test. Read on if you want to feel why they aren't.

The whiteboard behind Sabina Okereke's desk still has six columns drawn on it in dry-erase marker: Billing, Login, Bug, Cancel, How-to, Upgrade. She drew them the week she pitched the idea, and nobody's erased it since.

Sabina has run product for ticket routing at Corbel for three years. Corbel is the helpdesk platform sitting under a few hundred other companies' support teams, the software their agents use to receive, sort, and answer their own customers' tickets. The router that ships with it today just runs down a list of keyword rules: sees "refund" or "invoice," calls it billing; sees "password" or "locked out," calls it login. It's been good enough for years. Sabina wanted to replace it with an agent that actually reads the ticket.

Before asking anyone to build it, she wanted proof the decision logic behind it actually worked. So one Tuesday, before her first meeting, she pulled the newest ticket sitting at the top of each of the six queues, one billing question, one login problem, one bug report, and so on, and walked each one through the routing logic she'd sketched out, playing the agent herself, deciding the way she wanted the real thing to decide. Six tickets. Six right.

She put it in front of the exec team that Thursday. Six for six, clean logic, ready to build. She asked for three engineers for the quarter. She got them.

Two weeks into the build, a new hire in support ops was sitting in on Sabina's walkthrough deck during onboarding and asked one small question: what happens on a ticket where someone says they want a refund and mentions canceling in the same message? Is that billing, or is that cancel?

Sabina didn't have an answer. None of her six tickets had asked that question. None of them had asked any hard question at all, because she'd pulled the newest ticket in each queue that morning, and the newest ticket in a queue is whatever just came in, not whatever was actually going to be difficult to route.

The same six columns on the whiteboard, filled two different ways. One only holds the ticket that was never going to be hard. The other holds whatever actually showed up that week.

She went back to the real queue that afternoon, and this time pulled a whole week instead of a morning, forty tickets, nothing picked out on purpose. Eleven of them didn't sit cleanly in one category. Three were angry enough that the actual request was buried past the second sentence. Run against her six-for-six logic, seven of the eleven came out wrong.

Here's the part that actually cost something. It wasn't the seven wrong tickets. It was that three engineers had already spent two weeks wiring up real tool calls around logic nobody had actually stress-tested, and finding that out in week two instead of week zero meant redoing work, not just rewriting a test.

So here's what I'd take back. Building that first test set from the single newest, easiest ticket in each category, instead of a real, unpicked batch. It made sense the week Sabina just wanted a fast yes or no on whether the idea was worth exploring before asking anyone for real engineering time. It stopped making sense the moment a whole quarter's sprint got committed on the strength of it.

Sabina rebuilt the test set properly: six categories, a floor of four scenario types each, doubled to eight for cancellation and billing, thirty-two scenarios total, and this time nobody hand-picked which tickets went in. Every scenario type was written to match a real pattern from that forty-ticket week. Run against the logic as it stood, twenty-four of the thirty-two came out right on the first pass, and every miss sat in the same two places: multi-issue tickets, and the angry, garbled ones. Two fixes: check for a cancellation phrase before checking anything else, and ask one clarifying question the moment a ticket touched two categories at once. Rerun: thirty of thirty-two.

The real agent went live that quarter. In its first month, it routed 91 percent of tickets correctly on the first pass. The keyword router it replaced had managed 74. And the ticket that used to sit in the billing queue for two days while a customer's account quietly canceled underneath it, the one where someone asks for a refund and mentions canceling in the same breath, gets caught on the first read now.

The thing I'd tell myself, the Tuesday I pulled six clean tickets and called it proof: a test built from the easiest example in every category will always look finished. That's exactly what makes it useless. You built it specifically to avoid the part where the real work was hiding.

BOUND: the letters, run against a real week of tickets

This is a sizing question about how many scenarios to hand-run and for which named categories, not a person's trust flipping between two settings, so BOUND fits and FLIPS doesn't.

B, break it down. The number of scenarios to hand-run through the prototype equals the number of task categories the agent has to sort, times a floor of scenario types to test per category. Not one pass of the easiest ticket in each bucket.
O, own the numbers. Corbel already names six ticket categories: billing, login, bug reports, cancellations, how-to questions, and plan upgrades. A floor of four scenario types per category, clean, missing a detail, touching two categories at once, angry or garbled, gives six times four, twenty-four. Cancellation and billing tickets cost something real if the route is wrong, so their floor doubles to eight each, adding eight more. Total: thirty-two scenarios.
U, use a range. If the prototype only tests the classification decision itself, three scenario types per category is probably enough, and the total drops to about twenty-four. If the real agent chains three tool calls per ticket, looking up the account, checking the subscription, and drafting a reply, each one is a place the workflow can branch, and covering those branch combinations pushes the floor to six types per category, toward forty-eight. Start at thirty-two, the plain four-type floor, and only widen it once the tool chain is actually decided.
N, nail the sanity check. Thirty-two scenarios, hand-run at about eight minutes each, reading the ticket, working the logic by hand, writing down the route, comes to a little over four hours. Under half a working day, which is the whole point of a prototype: cheap enough to run before a single tool call gets built. If that number ever crept past what it'd take to actually code a rough version of the real tool calls, the prototype would have quietly turned into doing the job by hand instead of testing it.
D, direction. The scenario floor moves this more than the category count does. Raising the floor by one type across all six categories, with the doubled ones counting double, adds eight. A whole new ordinary category only adds four. If this number needs to shrink, the floor is the first place to look, not the category list, since cutting a category means an entire kind of ticket goes untested.

And if you want to be sure it really works, try it somewhere else

A veterinary chain runs the same idea on the phone instead of a ticket queue. An agent reads a transcript of an incoming call or message and decides where it goes: routine scheduling, a prescription refill, an urgent same-day symptom, or a true emergency that needs an immediate transfer to the ER.

B, break it down. The number of scenarios to hand-run equals the number of triage categories, times a floor of scenario types per category. Four named categories: routine scheduling, prescription refill, urgent same-day, and true emergency.
O, own the numbers. A floor of three scenario types per category, clean, missing a detail, and language that makes the urgency ambiguous. Four times three is twelve. True emergency gets that floor tripled, not doubled, to nine, since a missed emergency risks an animal's life rather than just an inconvenienced customer, adding six more. Total: eighteen.
U, use a range. Testing only the triage decision itself, two scenario types per category is close to enough, dropping the total to twelve. Chaining in the patient record lookup, the appointment calendar, and the refill history, the real tool calls a finished agent would need, pushes the floor to five types per category, toward thirty.
N, nail the sanity check. Eighteen scenarios, at about five minutes each for a short phone transcript, comes to ninety minutes. Well inside one afternoon, which is exactly what a phone-based prototype should cost before anyone builds the real call-handling integration.
D, direction. Same shape as Corbel's. Raising the floor by one type across four categories, with the tripled one counting three times, adds six. A new ordinary category only adds three. The floor moves it more.

Swap the trigger and it still runs.
Speed: an interviewer caps the answer at 90 seconds. Skip straight to the split: floor of 4, doubled to 8 for the two costliest categories, 32 total, about four hours to hand-run. The build-up backs it up if they ask.
Cost: instead of a scenario count, a manager caps the prototype at two hours of one person's time. Work backward: at 8 minutes a scenario, that's about 15 scenarios, cover the two doubled categories first and leave the rest for the next pass.
The model got better: a newer base model rarely misses an obvious billing ticket anymore. The floor doesn't drop on its own. Rerun the cancellation and multi-issue slices specifically, since a clean overall pass rate hides the one pattern that used to fail.

Where people run it wrong.
They size the prototype by how convincing the walkthrough looks, six clean tickets in a row, instead of a floor times a category count.
They pull the test tickets by hand instead of a real, unpicked batch, which quietly excludes exactly the tickets the logic most needs to meet.
They let the scenario count grow until hand-running it costs more hours than actually building the real tool calls would, so the prototype stops paying for itself.

How to use it live. Say the equation before any number: "the scenario count is category count times a per-category floor, doubled wherever a wrong call actually costs someone something." That buys the time to name real categories instead of guessing a round number that sounds thorough.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0