What criteria would you set for an agent's tool-calling accuracy?

How to answer this, stage by stage

Nobody is grading whether you land on exactly 99.7 percent. They're grading whether the criteria separate calls that can be re-run for free from calls that can't, and whether the end-to-end number survives a check against how often a person has to step in. Seven moves get you there.

Let's learn

What happens when a chain has five tool calls, and each one is only allowed to be wrong two times in a hundred?

Placesetter is an agent that books a restaurant table for you. You tell it what you want, and it calls the real booking system on your behalf, the same system a host at the front desk would use, and it gets you seated.

Before an agent like this, a person called the restaurant directly, or opened a booking app and did it themselves, checking the time, the party size, and the table by hand, about four minutes for a popular Friday night spot.

Now Placesetter does the whole thing, start to finish, in about twenty seconds.

At its worst, the agent calls the booking system, gets back a real confirmation number, and books a genuine table, just the wrong one: the wrong branch of a restaurant with two locations in the same city. Nothing in the chain failed. Every call returned success. The person just shows up somewhere nobody is expecting them.

The choice I would take back. Placesetter's launch bar graded every tool call, reads and writes together, against one 97 percent accuracy target. It never asked whether a miss on that call could be walked back.

What I would leave alone. Which two candidate restaurants the agent checks availability at first doesn't need the same rigor. If it checks a slightly worse second choice before the best one, the booking still happens fine, and nobody ever knows.

The lesson. An accuracy target that treats every tool call the same is measuring the agent, not the risk. The risk lives in exactly one call, the one that turns a guess into a fact.

Now here is the same thing as a story

Skip this part if you already believe a write call and a read call need different floors. Read on if you don't.

Devrim Aksoy can read a false-positive rate the way some people read a weather report, at a glance, knowing which cloud is actually going to rain. He'd shipped two agent features at Placesetter before this one, and both launches had cleared their eval with room to spare.

For the first ten weeks after the reservation agent went live, it earned that confidence back every night. Diners asked for a table, Placesetter called the booking system, and within twenty seconds a confirmation landed in their messages. Devrim spot-checked the call logs most mornings, reading through a sample of the previous night's bookings against what diners actually said they wanted. They always matched.

So he stopped checking every morning. Then every few mornings. The dashboard read 97 percent, week after week, and a number that steady stops asking to be watched.

The five calls behind one booking. Four can be silently retried. One can't.

Then came a Saturday. A support ticket landed with a screenshot: a couple standing outside Ferrante's on the Riverwalk, no table, no record of a booking, holding a confirmation text for a reservation that, as far as the host in front of them could tell, did not exist. It existed. Just eleven minutes away, at Ferrante's on 9th, the restaurant's other location, under a name the host there had never heard called.

Devrim pulled the call trace. Search had found both branches correctly. The availability check had come back clean for both. The booking call itself had returned a real confirmation number, a genuine, successful write to a genuine restaurant. It had simply used the ID from the wrong branch, carried over from an earlier lookup in the same chain, and nothing downstream had a way to catch that the two IDs no longer agreed with the address the diner had actually asked for.

Devrim went back to the 97 percent dashboard and asked it the one question it had never been built to answer: how does the booking call do, on its own, against duplicate-name restaurants specifically? He pulled every chain restaurant with more than one branch out of three weeks of logs and reran them. The booking call, isolated from the read calls padding the blended average, scored 91 percent on that slice, about one wrong in eleven. Every other call in the chain was still fine. It had only ever been the one call, and the one bar hiding it, that mattered.

He rebuilt the criteria that week: a 99 percent floor for the four calls that read and can be retried, a 99.7 percent floor for the one call that books, tested specifically against duplicate-name and near-identical restaurants, not just the clean single-location ones that made up most of the original eval. Run against the model as it stood, the booking call's duplicate-name score sat at 91 percent, well under its new floor. A fix requiring the agent to confirm a neighborhood or street name before booking, whenever two branches shared a name, took four days. Rerun, the duplicate-name score cleared 99.8 percent. Over the following six weeks, Placesetter completed just over 3,100 bookings with zero wrong-branch confirmations, the exact failure that had reached a real couple three weeks earlier.

The thing I'd tell my ten-weeks-in self: a steady 97 percent isn't proof nothing's wrong. It's proof the number was never built to find the one call where being wrong actually costs something.

BOUND, sized for a call that can't be undone

This is a criteria and sizing question about a chain of calls and a floor for each one, not a person's trust flipping between two settings, so BOUND fits and FLIPS doesn't.

B, break it down. End-to-end task success is per-call correctness raised across the number of calls in a typical chain, since one wrong call can derail the whole booking. A typical chain here is five calls: search, two availability checks, one booking call, one confirmation text.
O, own the numbers. A 99 percent floor on the four calls that read and can be silently retried. A 99.7 percent floor on the one call that books a real table. End-to-end target: at least 95 percent of full chains finish with no person needing to step in.
U, use a range. For a pilot where a person reviews every booking before it's sent, a 97 percent floor on the booking call is probably fine, the human is the real safety net. For an unsupervised release booking a stranger's evening, that floor needs to sit at 99.7 percent or higher, because a wrong write is a fact somebody has to discover in person.
N, nail the sanity check. At those floors, the chain clears about 95.8 percent. Placesetter books roughly 2,000 tables a week, so about a hundred chains a week still need a person to catch and fix something, close to four hours of one support person's week, not a research team's quarter.
D, direction. The booking call moves the number most. It's the one call in every chain, it's the least reversible, and it's the last call before the diner treats the task as done, so any earlier mix-up in the chain gets locked in there instead of caught.

And if you want to be sure it really works, try it somewhere else

A warehouse-restocking agent watches stock levels and reorders parts by calling a supplier's ordering API, without a person approving each order.

B, break it down. A typical reorder chain is four calls: check the current stock level, check price at two candidate suppliers, place the purchase order.
O, own the numbers. A 98 percent floor on the three read calls. A 99.8 percent floor on the purchase-order call, tighter than the restaurant booking call, because a wrong order can buy the wrong part, the wrong quantity, or from the wrong supplier, with real money and real dock space on the line. End-to-end target: at least 96 percent of reorder chains complete clean.
U, use a range. In a test warehouse where a buyer approves every order before it's sent, a 95 percent floor on the purchase-order call is workable. Once the agent places orders unsupervised, the floor needs to scale with what's being bought: 99.8 percent for a routine part, closer to 99.99 percent for anything over a few thousand dollars, since one bad order there isn't just wrong, it's expensive.
N, nail the sanity check. A human buyer placing manual orders sends the wrong part or quantity roughly once every 500 orders. At a flat 99.8 percent floor the agent already beats that. The refinement isn't the floor's average value, it's that a single flat floor still lets a $40,000 part get the same tolerance as a $9 one.
D, direction. Here it isn't just which call is irreversible, it's how much a miss on that call costs. The purchase-order call still moves the number most, but the real lever is letting its floor scale with dollar value instead of holding one number for every order size.

Swap the trigger and it still runs.
Speed: an interviewer caps the answer at ninety seconds. Skip straight to the split: 99 percent on the calls that can be retried, 99.7 on the one that can't, at least 95 percent of full chains finishing clean. The build-up backs it up if they ask.
Cost: instead of asking what the floor should be, a manager caps how many bookings a support person can review a day, say forty. Same rule, solved backward: raise the booking call's floor until the chain's failure rate produces no more than forty reviews a day, rather than picking a floor first and hoping it fits.
The model got better: a new version books correctly on duplicate-name restaurants almost every time. The floor doesn't drop on its own. Rerun the duplicate-name slice specifically, since acing the easy branches proves nothing about the one that used to fail.

Where people run it wrong.
They grade every tool call against one blended bar, because a single number is easier to put in a slide.
They set the floor once at launch and never revisit it when a new, riskier call type gets added to the chain, like modify or cancel.
They test the booking call against clean, unambiguous restaurants only, never the duplicate-name case that's the one actually likely to fail.

How to use it live. Say the split before any number: "tool-calling accuracy isn't one bar, it's a floor per call, tighter for the ones that write than the ones that read, multiplied across the chain." That buys the time to work out the real floors instead of guessing one number that sounds thorough.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0