What is the cost of running evals and how do you budget for it?

One automated-judge line looks like the whole cost of running evals. The real bill has three more parts sitting behind it, and the biggest one only shows up the week a rare, high-stakes call needs it.

How to answer this, stage by stage

Nobody's grading whether you can say the word "budget." They're grading whether you can show the arithmetic behind one number and say which part of it you trust least.

Let's learn

What does it actually cost to know whether an AI phone line is sending people to the right place?

Callpath is the voice assistant Corvasse built to answer exactly that. A caller speaks, Callpath transcribes it in real time using speech recognition, works out what they actually need, and routes the call to the right team, billing, technical support, emergency dispatch, before a human ever picks up.

Before Callpath, every call landed in one shared queue, and a live triage agent listened for the first few seconds before transferring it by hand. That took about 90 seconds on a typical call, and Corvasse kept roughly 40 triage agents staffed around the clock just to keep that queue moving across its retail, telecom, and utility clients.

Callpath cut that to about 12 seconds median, speech recognition and a language model deciding the right queue before most callers finish their sentence.

The turn: a handful of misroutes here and there were never going to sink Callpath. What could sink it was budgeting the eval program like a line item you set once and forget, then discovering the real bill nine months in, on the one call it mattered most to get right.

Here's the arithmetic behind that bill. Every month, four things get paid for, not one.

Automated sampling checks 1,000 calls a day, about 2 percent of Callpath's 50,000-call volume, at roughly a cent and a half a call. That's $450 a month, and it's the only number the original business case ever wrote down.

Human calibration is a weekly ritual: 300 calls double-scored by two reviewers, four minutes a call each, checking whether the judge's score matches what a real person would have said. That's 40 reviewer-hours a week, $5,160 a month, more than the other three parts combined.

Golden-set upkeep adds new labeled calls each quarter as Corvasse's clients add new call types, about $833 a month once you spread the cost out. And every time the routing prompt or the underlying model changes, a full regression run checks the new version against the golden set and flags anything it's unsure of for a human to look at, $2,180 a month at two deploys a month.

The honest answer was never one number. It was a range from about $7.5k to $11.1k, still under what it costs to run Callpath itself.

At its worst, an eval program that's underfunded is worse than no eval program at all, because everyone upstream believes the system is being checked, when really it's being sampled too thin to catch the one call that actually needed catching.

What I'd leave alone: low-stakes intents, store hours, balance lookups, appointment reminders, genuinely don't need this weekly double-human rigor. A miss there costs someone a re-ask, not a missed emergency. Spot-checking those quarterly instead frees reviewer hours for the intents where a miss actually costs something.

The lesson: a budget line can be completely honest about what it counts and still be the wrong number to plan around. $450 a month really was the cost of automated sampling. It never claimed to be the cost of trusting Callpath with a call that mattered.

Now here is the same thing as a story

Read the long version below when you want to feel why a tidy $450 line went so wrong, not just be told that it did.

Nomsa Dumisa could listen to eight seconds of a call transcript and tell you which queue it belonged in, back when Corvasse still routed every call by hand.

She'd run workforce planning for Corvasse's shared queue for three years before Callpath existed, building the shift schedules that kept 40 triage agents covering the phones around the clock. She knew exactly what that queue cost, and exactly how slow it could get on a bad morning.

Callpath launched in the spring, and the first few months were good, genuinely good. Median time to the right team fell from 90 seconds to 12. Uche Osazuwa, the finance partner who'd signed off on the build, ran the savings numbers in the first quarterly review and they were better than the pitch deck had promised.

The eval program that shipped alongside it looked modest on purpose: 300 calls a week, double-scored by two QA reviewers, checked against Callpath's judge. Nobody thought that number was sacred. It was just what the plan called for.

By month two, two of those reviewers had been pulled onto a new client's onboarding for "just this sprint." The weekly sample quietly slipped to about 220. Nobody changed the number in the plan; the plan just stopped matching what was actually happening. By month five it had slipped again, to around 140, and by month seven, someone in a status meeting asked why the calibration chart always looked the same shape, and nobody in the room could say when 300 had become less than half of that.

It came back on an ordinary Tuesday. A Brackwell Gas & Electric customer called in and said there was "a funny smell by the meter box." Not the word gas. Not the word leak. Callpath's classifier, trained mostly on the ordinary shape of a billing or service call, scored it as a service appointment request and routed it to the scheduling queue.

The customer called back eleven minutes later, more worried, and that second call landed correctly in emergency dispatch. Nobody was hurt. Nobody outside the team ever heard about it.

Nomsa found it two days later, not live, during her regular Monday spot listen through that week's flagged calls. She almost filed it as one bad transcript, a fluke of phrasing. Then she pulled the full year of calibration data and did the math she should have done the first week: out of roughly 350,000 calls a week, only about 1,400 are emergency-adjacent, gas smells, sparks, smoke. A 300-call random sample should have held about 1.2 of those most weeks. The sample Corvasse was actually running, 140 calls, held about half of one.

It was never really about whether 300 was the perfect number to start with. It was about nobody having the one job whose whole purpose was noticing the moment 300 quietly became 140.

The decision that opened the door went back to the very first budget meeting, before Callpath had a single production deploy. Eval got one line, $450 a month, automated sampling only, because that was the only cost anyone could see yet. Nobody was hiding the human calibration hours or the deploy regression cost on purpose. They just genuinely didn't exist yet, so nobody wrote them down, and a line item sized for a pilot quietly became the whole plan once the pilot stopped being a pilot.

Run that Tuesday again with one change: the weekly sample is stratified now, a guaranteed 25 emergency-adjacent calls reviewed every week no matter how rare they naturally are, on top of the general 300. And a hard-coded keyword check runs ahead of the classifier entirely, gas, smoke, spark, can't breathe, straight to dispatch, no confidence score involved. The same "funny smell" call still confuses the classifier's normal routing. The keyword check catches it anyway, in under four seconds, and there's no second call to make.

One design trusted a spend line written before the product had carried a single real emergency. The other design resizes the sample to the intent that actually carries the risk, not the intent that happened to be easiest to staff for.

What I'd tell myself, back in that very first budget meeting: a line item sized for a pilot is a placeholder, not a plan, and the day it quietly becomes the whole plan is a day nobody ever puts on the calendar.

BOUND, the five letters behind the $8,623

This isn't a story question wearing numbers. It's an estimation problem, and BOUND is what keeps a comforting single figure from hiding the four real ones underneath it.

Three things worth stating directly, since this is where the real judgment sits. The alternative Nomsa's team considered and dropped was skipping human calibration entirely and trusting the automated judge on its own, since automated scoring runs roughly twelve times cheaper than the human loop. It lost because a judge nobody checks against a real person can drift silently, with no way to know when it stopped being trustworthy, especially on the rare calls where being wrong costs the most. The AI-specific failure mode worth naming by name is a classifier that looks accurate in aggregate while staying unreliable on rare, high-stakes intents, because a flat random review sample almost never contains enough of them to catch the gap. The guardrail is two-part: a stratified minimum, at least 25 emergency-adjacent calls reviewed every week no matter how rare they naturally are, and a hard-coded keyword check that routes words like gas, smoke, spark, and can't breathe straight to dispatch, bypassing the classifier's confidence score entirely. That guardrail isn't free, the stratified sample alone adds about $250 a month nobody had planned for, small next to the $8,623 total, but the one addition that exists purely because of what a near miss revealed. And the bar Callpath's eval program holds itself to was never zero misroutes across 1.5 million calls a month, no probabilistic router can promise that. It's an intent-specific bar, at least 98 percent routing accuracy on emergency-adjacent intents specifically, checked every week against the stratified sample, not one company-wide accuracy figure standing in for every intent's actual stakes.

And if you want to be sure it really works, try it somewhere else

Same five letters, a city permits office instead of a phone carrier's support line, and this time the number that swings the budget most isn't the software team's own release calendar at all.

Permitline is an AI phone assistant Wexbury's permits office built to answer "what's the status of my permit" and to schedule inspections, without a caller waiting on hold for a clerk. Lena Devos runs its digital services program.

The build-up: Permitline handles about 8,000 calls a month, far smaller than Callpath. Automated sampling costs about $6 a month. Monthly human calibration, 150 calls double-scored, runs about $700. Deploy-triggered regression is small too, about $100 a month, since the office ships changes rarely. The number that dominates here is golden-set upkeep, about $891 a month, because Wexbury's zoning and fire code rules change on their own schedule, not the software team's.

That assumption broke the first time a new fire-code ordinance passed mid-year. Permitline kept confidently citing the old inspection requirement for weeks, because nothing had told its golden set the rule had changed. The fix looked the same shape as Callpath's: stop trusting a launch-day snapshot to still be true, and budget the refresh as a recurring cost, not a one-time build.

Same method, different lever: for Callpath, deploy cadence was the assumption worth stress testing first. For Permitline, it's how often the world underneath the product changes, not how often the team ships. If Wexbury's code updates sped up from every six weeks to every three, golden-set cost alone would nearly double, from about $891 to roughly $1,782 a month, the single biggest swing in that budget by far.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: budget eval as four line items, size it as a range anchored to the biggest lever, and sanity check the total against what the product costs to run.
Cost: there's no budget this quarter for the full stratified sample and the deploy regression run both. The stratified sample wins, twenty five extra reviewed calls a week on the riskiest intent beats a bigger general sample that just re-measures the same average more precisely.
The model got better, for real: say Callpath's classifier gets upgraded and headline accuracy climbs. That's not proof the emergency-adjacent intent improved with it. The easy, common intents could have gotten even easier for the model while the rare ones stayed exactly as under-sampled as before.

Where people run it wrong.
They budget eval as whatever the automated scoring API costs, and never ask what keeps that score honest.
They cut the human calibration sample quietly, a person here, a person there, without ever formally changing the plan or telling finance the real coverage dropped.
They size the review sample by overall call volume instead of by which intents actually carry risk, so the rare, expensive-to-miss ones stay chronically under-checked.

How to use it live. Say the real question out loud before quoting a figure: "before I give you one number, is this asking what the API call costs, or what it costs to actually trust the system?" That buys a beat to think instead of repeating the small, comfortable number that got written down on day one.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0