Explain how an eval spec should handle cost and latency alongside quality.

How to answer this, stage by stage

Nobody is grading whether you land on exactly nine cents. They are grading whether all three limits are real, checked against something outside the spec, with an honest answer for which one gives first. Seven moves get you there.

Let's learn

Callridge is a tool call centers plug into their phone system. It listens to every support call and scores how the customer felt, angry, neutral, happy, so a supervisor doesn't have to sit through the whole call to find out.

Before Callridge, a supervisor at a forty-agent floor could only sit and listen to a handful of calls by hand. About two per agent a month, roughly eighty calls out of nine thousand that came in, fifteen minutes each to do it properly.

Now every call gets scored the moment it ends. A live add-on goes further: while an agent is still on the phone, if the sentiment turns negative, a supervisor's screen lights up inside twenty seconds, while the call is still running.

At its worst, someone hits the cost ceiling by cutting the one part of the pipeline that tells the model which voice is the agent and which is the customer. The dollar number stays fine. The score is now scoring the wrong person, and nothing in the spec catches it, because nothing in the spec ever named that part as protected.

The choice I would take back. We wrote the three limits with no order and no protected parts. I'd name cost as the one that bends first, and name the one thing inside it that never gets cut: keeping the two speakers apart. A cheaper transcription vendor is fine. A transcript that can't tell who's talking is not.

What I would leave alone. The little coaching-tip line the tool suggests to a supervisor, something like "try acknowledging the wait time first," doesn't need the same floor. It's a suggestion, not a record. If it's ordinary two days out of ten, nobody's file is touched by it.

The lesson. Three numbers with no order are really three teams each assuming someone else is holding the line. Someone has to write down which one bends, and what it can't take down with it, before the quarter where compute costs money nobody budgeted for.

Now here is the same thing as a story

Skip this part if you already believe a cost ceiling can be met on paper while the product quietly breaks underneath it. Read on if you don't.

Samira Haddad wrote Callridge's first eval spec two years ago, back when the company was four people pitching one regional call center. In those early months she sat in on the customer calls herself and listened, by ear, for the moments the model's score and her own read of a call didn't match.

The same spec page, before and after Samira wrote down what cost couldn't touch

After the funding round, Callridge grew to sixty call-center customers in a year. For the first stretch, every model update still went through Samira's full review, including a manual listen-through of ten calls where she checked that agent and customer voices stayed cleanly split apart.

Then the review quietly thinned. An engineer took over the compute budget, and ten calls became five. Five became "spot check if something looks off." By the quarter live coaching's per-call cost ran thirty percent over its own ceiling, the listen-through hadn't happened in months, because cost fixes had stopped being something her team reviewed at all. They'd become finance's problem to solve, quietly, on their own.

The fix that quarter was a cheaper transcription vendor. It didn't include the add-on that separates the two voices on a call. Nobody flagged it, because the dollar number came in fine, and dollar numbers were the only thing anyone was still watching.

The near miss came on an ordinary Tuesday. A shift supervisor pulled up a live-coached call to write up an agent for sounding curt with a customer. The agent asked to hear the recording first. The "angry" voice tagged to them was the customer, raised near the end of a long hold. The agent hadn't said a harsh word the whole call.

We didn't lose the number. We lost which voice was which, and the number never knew.

Samira pulled the pipeline logs that afternoon. The diarization step had been off, quietly, on every live-coached call for three months. The cost line had never once crossed its ceiling. The transcript had crossed a line the spec never drew.

She remembered the meeting where the original spec got written, four people around a laptop, three numbers on a slide: quality, cost, latency. Someone said, only half joking, "if we ever have to choose, obviously it's cost." Nobody wrote that down. It felt too obvious to need a sentence.

She rewrote the spec that week. Cost stays the limit that bends first, in writing this time. But one line inside it is boxed off: whatever changes to hit the cost ceiling, the two speakers on a call stay separated, always. A monthly check now listens to ten calls by hand and compares the pipeline's own speaker labels against what a human hears. Twenty minutes, once a month.

In the next two quarters, that check caught one more vendor swap before it shipped, the same thirty percent in savings, this time with the speaker split still intact.

The thing I'd go back and tell myself, in that four-person meeting: writing three numbers down isn't the same as writing down what happens the day we can only hit two of them. I left that page blank and called it obvious.

B O U N D, mapped onto three ceilings at once

This is a sizing question with a priority problem sitting inside it, so BOUND fits, and FLIPS doesn't. Nobody's trust is flipping here. It's a budget with three lines, and an honest answer for which one moves first.

B, break it down. A call only passes the spec if it clears all three limits at once: the score at or above the quality floor, the cost at or under the ceiling, the answer back before the latency ceiling.
O, own the numbers. Quality: 90% agreement overnight, 82% live. Cost: about 5.5 cents overnight, about 9 cents live. Latency: six hours overnight, twenty seconds live. Six real numbers, not one.
U, use a range. The range comes from how the score gets used, not from how long the call ran. An overnight record and a live nudge get genuinely different numbers on all three limits, because different people are relying on them in different ways.
N, nail the sanity check. Ninety percent checked against ninety percent human-to-human agreement. Cost checked against the thirty-five cents the call center actually pays. Latency checked against the forty-five seconds a supervisor notices trouble on their own.
D, direction. Cost bends first, because it's the only one of the three whose failure doesn't trip anything on its own. That's the whole reversal: name what cost can never cut on its way down.

The range, with what the call center pays marked for scale, not as one of the bounds

And if you want to be sure it really works, try it somewhere else

A wholesale produce packer runs an AI camera over the belt that sorts avocados, grading ripeness and bruising as each one passes, before a mechanical arm diverts it to export-grade or local-grade.

B, break it down. An item passes the spec only if the grade clears the accuracy floor, the compute cost per item is under the ceiling, and the sort decision comes back before the diverter physically needs it.
O, own the numbers. Accuracy floor: 92% match to a certified human grader. Cost: about $0.004 per item on the full-speed line, ceiling $0.006. Latency: 400 milliseconds, a hard limit set by how fast the belt moves the fruit to the diverter.
U, use a range. The full-speed export line grades every item at that 400-millisecond ceiling. A slower compliance-audit line re-checks a sampled fraction with a bigger model, allowed 2 seconds and a $0.02 ceiling, since it only runs on a fraction of the fruit.
N, nail the sanity check. 92% checked against 94% agreement between two certified human graders, the same shape of check as Callridge, new numbers. Cost checked against about $1.80 of margin per case, both ceilings are close to free against that. Latency checked against the belt itself, not a business tolerance, a mechanical deadline that doesn't move.
D, direction. Quality bends first here, not cost. Latency can't move, it's set by the belt's physics. Cost is close to free per item. That leaves quality as the only lever standing, and it's the one that quietly slips when a model gets tuned to always beat the 400-millisecond deadline.

Swap the trigger and it still runs.
Speed: an interviewer caps the answer at ninety seconds. Skip straight to the rule and the six numbers, five and a half cents overnight, nine cents live, checked against ninety percent human agreement and thirty-five cents of revenue. The build-up backs it up if they ask.
Cost: Callridge's finance team caps the whole live-coaching compute budget at a fixed monthly figure instead of asking what one call should cost. Same rule, solved backward: divide the budget by expected call volume and the live-versus-overnight mix to find what ceiling the numbers can actually afford.
The model got better: a new scoring model ships that's both cheaper and faster. The spec doesn't need a rewrite, just the same three checks re-run with the new price and the new speed in place of the old ones.

Where people run it wrong.
They write one flat number that quietly averages two very different jobs, live and overnight, instead of splitting the range by how the score is used.
They set a cost ceiling with no note on what's protected inside it, so any cut that keeps the dollar figure under the line looks fine on paper.
They pick a sanity check that isn't actually external, comparing the quality floor to last quarter's own score instead of to something outside the system, like how often two human reviewers agree with each other.

How to use it live. Say the pass or fail rule before any number: "this only counts as passing if quality, cost, and latency all clear their bar on the same call, not on average." That sentence buys the time to work out where the three numbers should land instead of guessing one that sounds safe.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0