ConceptAdvancedEval-Driven Specification / Writing an eval spec / #13
How do you specify evals for a multi-step agent rather than a single response?
The direct answer
Grade every step in the agent that commits an irreversible action, a booking, a cancellation, a payment, on its own, right after it runs. Grade the steps that only draft or suggest something on the final output alone. For Farflung, an agent that books flights, hotels, and activities, that means checking each booking and each cancel the moment it happens, not just checking whether the finished itinerary looks right, because a run can pass every outcome check and still have cost real money getting there.
How to decide which steps earn their own check, in order
Grade every step that commits an irreversible action on its own, pass or fail, right after it runs.Why: booking, and cancelling a booking, is where real money moves, and an outcome-only score never looks there.
Leave draft-only steps, like the activities list, graded against the final itinerary alone.Why: grading a step nobody has ever complained about just spends eval-engineering hours somewhere it isn't needed.
Track the dollar cost hiding inside "correct" runs, not just the pass rate.Why: a 97 percent match score can sit right on top of real money leaking out the back.
Say who actually pays for each kind of miss.Why: naming finance, weeks later, on the corporate card, is what makes the case for step-level grading real instead of theoretical.
Set a kill line: drop step-level grading on a step once its error rate falls below where checking it costs more than it saves.Why: grading everything forever is its own kind of waste.
The four moves for grading a run that has more than one step
A question about evaluating an agent wants a pick about which steps earn their own check, not a rulebook for grading every step the same way, so PICK does the work here.
1
Ground it in one real agent, and say your structure up front
Say it like this
"Let's put this on Farflung, an agent that books a flight, books a hotel, and drafts a few activities from one trip request. I'm going to pick a position on which steps get graded on their own, then say who feels each kind of miss, then which kind actually costs more, then what would change my mind."
Why this works
Gives the interviewer one concrete agent to push on, and tells them the shape of the answer before you start it.
2
Say what the question is really testing
Say it like this
"This isn't really 'should you grade every step the same way.' It's 'do you know a run can look perfect at the end and still have spent real money getting there.' So I'm not going to grade every step the same way."
Why this works
Shows you see past the surface ask to the judgment it's actually testing.
3
Give the position, with the actual mechanism in it
Say it like this
"My position: any step that commits an irreversible action, booking a flight, booking a hotel, cancelling either one, gets its own pass or fail check, run right after that step happens. Any step that only drafts or suggests, like the activities list, gets graded once, against the final itinerary."
Why this works
PICK rewards a real mechanism, not a vague promise to grade things more carefully.
4
Name who feels each kind of miss, then prove the asymmetry with a real number
Say it like this
"Here's the split. If Farflung's flight search comes back with three bad options, nothing's booked yet, it just searches again in the same run, and it costs nobody anything. But last quarter Farflung finished 3,000 trips, and our one eval, the outcome score, said 97 percent matched what the traveler asked for. When we pulled the traces behind that number, 210 of those 'correct' trips had a flight or a hotel that got booked, found wrong a few steps later, and cancelled and rebooked. That cost about 86 dollars a trip in fees and pricier fares. 18,060 dollars in one quarter, sitting under a score that said everything was fine."
Why this works
Real numbers make the asymmetry checkable instead of asserted.
5
Say what you'd leave alone
Say it like this
"I wouldn't grade the activities step this way. Nobody's checking whether the third suggested walking tour is worded well against some rubric, they're checking whether they liked the trip overall. Splitting that into its own graded step spends hours we could put where real money actually moves."
Why this works
Shows judgment instead of grading every step just because you technically can.
6
Name the kill criteria, say what you'd keep watching, and close
Say it like this
"I'd drop the per-step grading on bookings the day the miss rate falls low enough that catching it costs more than it saves, or the day we add a hard confirm-before-booking gate, because then the gate is doing that job instead. Until then, I'll keep watching the dollar cost hiding inside 'correct' trips, not just the pass rate, because a 97 percent score that's quietly wrong about what a trip cost isn't really 97 percent."
Why this works
Shows confidence without stubbornness, and closes on the line worth remembering.
One more thing before this moves on to the long version: this pick covers the steps that spend money, not every step in the run. Most candidates hear "evaluate a multi-step agent" and answer by grading every step the same way. Say which steps earn their own pass or fail check and which ones stay graded at the end, and you've shown real judgment instead of reciting "add more evals."
Let's learn
Every trip used to take a person about 45 minutes: search a flight, call to hold a hotel, check both against the same dates, before anything got booked.
Farflung does that same trip in under 90 seconds. It reads one request, then books a flight, books a hotel, and drafts a short list of things to do.
Knowledge spark: what is a multi-step agent?
A tool that doesn't answer once. It takes a few actions in a row, search, pick, book, check, and each action can change what the next one sees. Grading only the last action misses everything that happened to get there.
For months, the team had one eval, and it looked at the finished itinerary: did it match what the traveler asked for. It scored 97 percent, most weeks.
Here is the turn. A 97 percent match score only looks at where the trip ended up. It says nothing about how the agent got there. So when a booking step went wrong partway through a run, say a flight matched the wrong week, Farflung just fixed it and kept going. It cancelled the bad booking, made a new one, and shipped an itinerary that looked completely right.
We never asked what a correct trip cost to produce. We only asked if it looked correct.
At its worst, that looks like this. Farflung booked a return flight for the wrong week for a family trip to Denver. It caught the mismatch two steps later, while booking the hotel, cancelled the flight, and booked the right one. The cancellation carried a 75 dollar fee. The good seat on the right flight was already gone, so the replacement cost 34 dollars more. None of that showed up anywhere. The itinerary that reached the family was completely correct.
The choice I would take back
We built one eval and pointed it at the finished itinerary, because that's the piece the traveler actually sees. I would take that back. I'd grade the steps that book or cancel something, on their own, right when they run, separate from whether the final trip looks right.
Cost, in dollars, of each kind of miss
The green bar is small on purpose: a bad search never reaches a booking, so it costs nothing beyond a wasted second inside the same run. The red bar is what one cancelled-and-rebooked trip actually costs: about 86 dollars between the cancellation fee and the pricier replacement fare. Across 210 such trips last quarter, that's 18,060 dollars sitting under a score that read 97 percent the whole time.
What I would leave alone. The activities step, since it never spends anything, and gets checked anyway when the whole itinerary is reviewed at the end.
The lesson. We built Farflung to answer "does this trip look right." It does. We never asked whether the run cost more than it should have to get there, and an outcome score can't tell you that, it never looks at the steps in between.
The eighty-six dollars nobody was watching for
You don't need this to answer the question. Read it if you want to feel why the per-step check has to sit on the booking steps before the next quarter, not after.
The laptop Beatriz Pham carries between meetings has a crack across the left hinge, from being packed one too many times for a launch trip she almost missed.
She's been the PM on Farflung for three years, since before it booked a single flight on its own. Before that she built booking flows by hand at a small agency, so she knows exactly how many places a trip like this can go sideways.
For the first several months after launch, Beatriz pulled five finished trips a week and read the full trace behind each one, every search, every booking call, every step, not just the itinerary that came out the other end. They all held up. Routine trips, nothing strange.
So she stopped reading five. She started reading whichever ones the dashboard flagged as unusual.
Then she stopped reading those too. If the weekly score said 97 percent, she moved to the next thing on her list.
There wasn't one Thursday where it broke. Nothing broke, exactly. Over about six weeks, someone in finance noticed the corporate travel card kept showing small disputes, a duplicate-looking charge here, a cancellation fee there, and flagged it as something to look into eventually, not urgently.
Same tool, two very different ways to cost money
Beatriz pulled the full quarter of trace logs. It was not a quick read.
210 of 3,000 trips had a booking that got made, found wrong a few steps later, and cancelled and rebooked. Every one of those 210 trips had passed the outcome eval. Every one of them had shipped a correct itinerary.
We didn't get any of those trips wrong. We just never asked what it cost to get them right.
It was never really about the 97 percent. Beatriz never had a number that told her what a trip cost to produce. She had a dashboard, and the dashboard only had two colors: green, or not green yet.
In the twenty-some hours it took to read a full quarter of traces by hand, Beatriz counted 18,060 dollars in fees and pricier fares, sitting quietly under a score that had read 97 percent every week for three months.
I want to say the problem was that Farflung made 210 mistakes. It didn't, not really. Every one of those 210 trips ended up correct. The problem is that nobody had ever decided whether a step that gets fixed mid-run should count against anything, because the only eval anyone had ever pointed at was the finished trip.
Two years earlier, when the team first scoped Farflung's eval, someone in the room asked whether the booking and cancel steps needed their own checks. It felt like a lot of extra plumbing for something that had never once gone wrong. They wrote one eval, pointed it at the finished itinerary, and moved on to shipping the next feature.
Beatriz put a check on each booking and cancel step the month after. Fail the step the moment a booking gets reversed, not just the trip. Run the same quarter again, and the pattern shows up in the first week's worth of step-level failures, not eight weeks late on a finance report.
One design waits for the corporate card to say something's wrong, weeks after the fact. The other watches the exact moment a booking gets undone, and says so immediately.
And the thing I'd tell myself, back in the room where we scoped this: we asked whether the finished trip was right. We never asked what it cost, in dollars, to get there. On a quiet month, that question never came up. Over one real quarter, it cost more than the plumbing would have.
PICK, applied to Farflung's booking steps
This is a tradeoff about which steps earn their own check, not a rule for grading every part of the run the same way, so PICK is the tool.
P, position. Grade every step that commits an irreversible action, booking a flight, booking a hotel, cancelling either one, on its own, the moment it runs. Grade steps that only draft or suggest, like the activities list, against the final itinerary alone.
I, impact. Nobody feels a bad intermediate booking if Farflung quietly fixes it before the itinerary ships, since an outcome-only score only asks whether the final trip matches the request. Finance feels it: cancellation fees and pricier replacement fares, showing up on the corporate card weeks after the trip shipped.
C, cost asymmetry. A bad search, caught before anything books, is loud and cheap: it shows up in the trace and gets retried in the same run for free. A booking that gets cancelled and rebooked is quiet and expensive: it passes the outcome eval every time, and the cost only turns up later, on a statement nobody was watching closely.
K, kill criteria. Drop the per-step check on a booking step the day its miss rate falls low enough that grading it costs more engineering time than the money it protects, or the day a hard confirm-before-booking gate exists, because then the gate is doing that job instead.
Knowledge spark: why not just have someone review every trip before it books?
Because trips have to book in under 90 seconds, and a person reading every itinerary before it goes out would erase the whole reason Farflung exists. The fix has to live inside the run, not after it.
Action-stage miss rate, month by month
Still far above the kill line
This month
Kill line: 1% miss rate
Even the best month on record, 6.8 percent, sits nowhere near the 1 percent line where step-level grading would stop paying for itself. Until Farflung's booking steps get an order of magnitude more reliable, the check earns its keep every month.
Run PICK again, on an expense report
Warrendale Manufacturing runs the same question on its internal expense agent. It reads a submitted receipt, categorizes the expense, checks it against the policy limit, routes it to a manager for approval, and issues the reimbursement transfer, five steps from one photo of a receipt.
P. Grade the step that issues the reimbursement transfer, and any step that auto-approves one, on its own. Leave the receipt-reading and categorizing steps graded against the final reimbursement record. I. A receipt mis-categorized and then corrected before approval costs nobody anything. A transfer that goes out twice, or for the wrong amount, and gets "fixed" by clawing money back from someone's next paycheck costs that employee trust, on top of the money. C. A bad category is loud and cheap, caught the moment someone reviews the record. A duplicate or wrong-amount transfer is quiet and expensive, invisible until an employee notices their paycheck is short. K. Drop the check on the transfer step the day the payment system itself catches every duplicate before money actually moves, the same evidence that would let Farflung fold its booking check back into one outcome score.
What I would leave alone, on the reimbursement agent
The receipt-categorizing step. Nobody's paycheck depends on whether a taxi ride got tagged "ground transport" or "other," it gets fixed for free when the record is reviewed.
Swap the trigger and it still runs
Speed: Farflung finishes a trip in 20 seconds instead of 90. Doesn't move the pick, because the position is about which steps spend real money, not how fast the whole run finishes.
Cost: grading each booking step turns out to cost about as much in compute as the booking step itself. Still doesn't flip it, a few extra model calls are cheaper than a fee and a pricier fare.
The model gets better: if Farflung's booking accuracy improves enough that action-stage misses nearly disappear, that's exactly the evidence that would fold the booking check back into one outcome score.
Where people run it wrong
Grading every step, including the activities draft, the same strict way as the booking steps, and spending eval-engineering hours on a step nobody has ever complained about.
Treating a high outcome score as proof nothing costly happened along the way, without ever checking whether a step reversed itself mid-run.
Adding the check to the flight-booking step and forgetting the hotel-booking step gets cancelled and rebooked exactly the same way.
If you are asked this cold
Say the reframe out loud before picking a side. "Give me a second, I want to separate the steps that spend real money from the ones that only draft something, before I say how each should get graded." That's true, it's already stage two of the walkthrough, and it buys you time to find the real asymmetry instead of guessing.
Flashcards (click a card to flip it)
1 · THE FRAMEWORK
Which framework fits this question, and what's the hardest step to nail?
Tap to flip
ANSWER
PICK, for a tradeoff. The hardest step is C, the cost asymmetry: naming why a booking that gets cancelled and rebooked is hidden until finance goes looking, while a bad search is loud and gets fixed for free in the same run.
2 · THE PERSON
Who owns Farflung's eval spec, and what's she done for three years?
Tap to flip
ANSWER
Beatriz Pham, the PM who has owned Farflung's eval spec for three years, since before it booked a single flight on its own.
3 · THE HABIT
What did Beatriz stop doing once the outcome score kept coming back green?
Tap to flip
ANSWER
Reading the full trace behind five finished trips a week. Then just the ones the dashboard flagged. Then just the weekly score itself.
4 · THE ASYMMETRY
Name the two kinds of miss here and what each one costs.
Tap to flip
ANSWER
A bad flight search, caught before anything books: costs nothing, fixed in the same run. A booking that gets cancelled and rebooked: about 86 dollars a trip in fees and pricier fares, 210 trips, 18,060 dollars in one quarter.
5 · THE POSITION
State the pick in one sentence, the way you'd say it out loud.
Tap to flip
ANSWER
Grade every step that books or cancels something on its own, right when it runs, and leave the draft-only steps, like the activities list, graded against the final itinerary.
6 · THE NUMBER
The end-to-end outcome score said Farflung's itineraries matched what travelers asked for ______ percent of the time, even in the quarter with the hidden cost.
Tap to flip
ANSWER
97 percent. The score never looked at how the trip got there, only where it ended up.
7 · THE KILL CRITERIA
What evidence would flip this position back the other way?
Tap to flip
ANSWER
Proof that the action-stage miss rate has fallen low enough that a per-step check costs more to maintain than the money it protects, or a hard confirm-before-booking gate that catches the same risk another way.
8 · THE TRANSFER
Section 4 runs PICK again on a different agent. Which one, and where does the position land there?
Tap to flip
ANSWER
Warrendale Manufacturing's expense-reimbursement agent. The check sits on the step that issues the transfer, not on the step that reads and categorizes the receipt.
Check yourself Score: 0 / 0
True or false
1. True or false: this position requires grading every step of Farflung's run, including the activities list, on its own.
True
False
Show hint
Think about which steps actually spend money.
Show answer
False. The position only grades the steps that book or cancel something. The activities step stays graded against the final itinerary, since it never spends anything on its own.
Multiple choice
2. Which of these is the actual mechanism behind this answer's pick?
A. Set every step of Farflung's run to its own strict pass or fail check, activities list included.
B. Grade the steps that book or cancel something on their own, right when they run, and grade the draft-only steps against the final itinerary.
C. Have a person review every itinerary by hand before it ships, every time.
D. Stop letting Farflung cancel and rebook anything, and fail the whole run instead.
Show hint
Three of these either grade a step that never caused a problem, or throw away the speed Farflung was built to deliver.
Show answer
B. A grades a step nobody's ever complained about, C erases the ninety-second promise, and D turns a fixable mid-run correction into a failed trip. Only B targets the miss that actually cost money.
Fill in the blank
3. Fill in the blank: last quarter, ______ of Farflung's 3,000 trips had a booking step that got cancelled and rebooked mid-run, even though every one of them still passed the outcome eval.
Show hint
Same number the walkthrough's fourth stage used to prove the asymmetry.
Show answer
210 (7 percent). Each one cost about 86 dollars in fees and pricier fares, for 18,060 dollars total, none of it visible on the 97 percent score.
Multiple choice
4. Why not just grade every step of Farflung's run the same strict way, to be safe?
A. It would take too much compute to check every step.
B. Because the draft-only steps, like the activities list, never spend money, so grading them the same way as the booking steps spends eval-engineering hours on a step nobody's ever complained about.
C. Farflung's contract with its flight suppliers forbids checking intermediate steps.
D. The model technically cannot be graded mid-run at all.
Show hint
Think about what step-level grading actually costs when the step never touches money.
Show answer
B. Step-level grading isn't free anywhere you put it, it trades engineering time for safety on a step that pays for it. The activities step doesn't pay for it.
Short answer
5. If the hidden cost had come out to 4 dollars a trip instead of 86, would the same position still hold? Walk through it.
Show hint
Think about whether the fix protects against the dollar figure, or against the fact that a step reversed itself without anyone noticing.
Show answer
Probably not, and that's the honest answer. At 86 dollars a trip and 210 trips, the per-step check clearly pays for the engineering time it costs. At 4 dollars a trip, the same 210 trips only add up to 840 dollars a quarter, likely less than the cost of building and maintaining the check. The asymmetry has to be real in dollars, not just real in principle, before it earns its own graded step.
Short answer, apply it yourself
6. Pick an AI agent you use yourself that takes more than one step. Name one step in it you'd grade on its own, and one step you'd only grade by the final result.
Show hint
Look for the step where something becomes hard to undo, a payment, a send, a delete, versus a step that just drafts something you'll see before it counts.
Show answer
Model answer: "A coding agent that reads a bug report, writes a fix, and opens a pull request. I'd grade the step where it actually pushes or merges code on its own, since that's hard to undo once it ships. I'd only grade the step where it drafts the commit message against the final pull request, since I'll read that before anything happens." Any answer works if you can name the step where something becomes hard to undo.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
