ConceptAdvancedResponsible AI & Advanced Practice / Agent product management specifics / #5
How do you evaluate an agent's trajectory rather than its final answer?
AUDIT the product is Cobblestitch, a coding agent that opens pull requests for Porthaven Systems
Porthaven Systems builds dev tools. Cobblestitch is the coding agent that reads a ticket, edits files, runs tests, and opens a pull request. Yuki Tanaka manages the engineering team that reviews its output before merge.
The direct answer
Don't judge a merged PR by its final diff. Pull the full, unedited trajectory, every file it opened, every edit it made and undid, every test run, and audit that log against a small, human-graded rubric of what a good process actually looks like. A correct diff reached by six blind edits and a lucky revert is a different, riskier thing than the same diff reached in three deliberate steps, even though they produce the same file.
Do this, in order
Get the full trajectory log, not just the final diff, before judging anything.Why: a summary that only shows the final state hides exactly the information process quality lives in.
Pin the exact agent version and date the trajectory ran on.Why: an agent that gets quietly updated can produce a very different process next week under the same name.
Score process against a small, human-graded rubric, not a single pass or fail.Why: "reverted a destructive edit before merging" and "never made one" look identical on the final diff.
Replay a real sample of trajectories yourself instead of trusting a self-reported summary.Why: the agent's own account of its steps can smooth over exactly the messy parts worth seeing.
Let simple, low-risk PRs skip the full audit once a category has a track record.Why: auditing every trajectory forever costs more reviewer time than the risk it's actually catching.
How to answer this, stage by stage
Six stages, since this is a focused concept question about one evaluation method, not a full case walkthrough.
Stage 1
Scope it to one real agent
Say it like this
"I'll answer this for Cobblestitch, a coding agent at a dev tools company that opens pull requests directly."
Why this works
Keeps "trajectory evaluation" from turning into an abstract lecture on evals.
Stage 2
Say why the final answer alone lies
Say it like this
"A merged PR with passing tests can still have gotten there by editing three unrelated files, reverting two, and stumbling onto the right change by trial and error. The diff looks identical to a clean, deliberate fix."
Why this works
Names the exact failure the question is testing before offering the fix.
Stage 3
Say your structure out loud
Say it like this
"I'd treat the trajectory log itself like a document to audit. Ask who ran it, what it's judged against, which version produced it, what it leaves out, and then test a sample myself."
Why this works
Signals this is a method for judging evidence, not a one-off gut check.
Stage 4
Name the actual rubric
Say it like this
"Did it check its own diff before claiming done. Did it run the real test suite, not just skim for errors. Did it touch only files the ticket actually named. Three yes-or-no checks, scored by a human on a sample."
Why this works
Turns "evaluate the trajectory" into something concrete enough to actually score.
Stage 5
Say what's missing from a self-report
Say it like this
"If the agent's own summary is all we look at, the messiest part of its own process is exactly what it has the least reason to highlight."
Why this works
Names the AI-specific failure mode: a self-reported trajectory can hide its own worst behavior.
Stage 6
Close on the cost you're accepting
Say it like this
"This costs reviewer time on a sample of PRs even when the diff already looks fine. I'd rather pay that cost regularly than find out about a lucky-guess pattern the week it finally guesses wrong on something that matters."
Why this works
States the trade-off plainly instead of pretending the audit is free.
Let's learn
Cobblestitch reads an assigned ticket, edits the code, runs the test suite, and opens a pull request, all without a person driving any of the individual steps.
Before Cobblestitch, an engineer did the same work by hand: read the ticket, make a focused edit, run the tests, open the PR. Reviewers judged that PR by reading the diff and the linked ticket, since a human's process was mostly visible in how small and deliberate their commits were.
Knowledge spark: what's a "trajectory"?
The full sequence of steps an agent actually takes: every file it opens, every edit it makes, every test it runs, in order, not just the final result it lands on. Two trajectories can end at the exact same final diff and still look completely different in the middle.
With Cobblestitch, PRs merge fast, most passing CI cleanly. That's the number in the sprint report. It's also not the number that tells you whether the agent is actually reliable.
The turn. A passing final diff isn't the real story. The real story is how it got there. A single-turn code-completion tool only ever shows you one suggestion at a time, so there's no "process" to separate from the "answer." An agent takes many actions to reach one answer, and that gap between action and answer is exactly where risk hides that a final-diff review can never see.
Correct final diffs, broken down by how they got there
All three bars produced a "correct, merged" PR. Only the diff review can tell them apart, and it can't, because they all end in the same clean file.
The decision I would take back
We reviewed Cobblestitch's PRs the same way we reviewed a human engineer's: read the diff, check it against the ticket, merge if it looked right. That made sense because it's exactly how code review has always worked. It stopped making sense once we realized a human's diff and their process are naturally correlated, since a person who edits six unrelated files usually produces a messy diff too, while an agent can wander through six blind edits and still land on a clean, minimal final diff that reads exactly like the deliberate version.
What I would leave alone: a trajectory that's short, touches only the files the ticket named, and runs the test suite exactly once before finishing doesn't need a deep audit. That process shape correlates strongly enough with reliability that a light spot-check is enough.
We weren't checking whether Cobblestitch could write correct code. We were checking whether it could write correct code the same way twice.
The lesson: reviewing an agent's output the same way you'd review a person's is a natural first instinct, and it's exactly the instinct that misses the one risk unique to agents: a good result reached by a bad process will look fine right up until the process finally produces a bad result too.
Now here is the same thing as a story
The short version above is what you'd say defending this audit process to Porthaven's engineering leadership. Read this one for how the gap actually got found.
Yuki Tanaka has managed the engineering team at Porthaven for four years. She reads the diff on every agent-authored PR herself before it ships to production code.
For months, the team only ever looked at the left panel.
Cobblestitch shipped in the spring, and for months its merged PRs looked exactly as clean as a senior engineer's. Then, in June, a routine trajectory sample, pulled for an unrelated audit, showed PR 482 had touched three files that had nothing to do with the ticket, reverted two of those edits, and only then landed on the one-line fix that actually mattered.
The final diff only ever showed the last box. The middle three never appeared in review at all.
The final diff was correct. Tests passed. Nobody would have known any of the middle three steps happened, because nothing in the PR review surface showed them. Yuki pulled a wider sample of trajectories and found the pattern wasn't rare: 22 percent of correct, merged PRs had reached that correctness through a genuinely messy path, and 7 percent had involved reverting an edit that would have broken something else entirely had it shipped.
PR 482 and PR 501 sit in the exact same row on this chart. A diff-only review can't tell which column they're actually in.
Yuki's team built a rubric off exactly what that sample revealed: did it check its own diff before finishing, did it run the real test suite rather than a partial one, did it touch only files the ticket named. A human grader scores a random slice of trajectories weekly against those three checks.
The version-pin check turned out to matter most: the messy-trajectory rate had climbed right after an unannounced upstream model update.
Replayed with the rubric live: PR 482's exact ticket comes back around a month later on a related file. This time the trajectory audit flags the pattern of touching unrelated files before the diff even reaches Yuki's desk, and a reviewer steps in mid-process instead of after merge, catching the wrong-file edits before either gets reverted live in production history.
Only the cleanest, most predictable trajectories skip a human look entirely, and even those get sampled later.
I reviewed Cobblestitch's PRs the way I'd always reviewed a person's, because that's the only review process the team had ever built. It took one wider trajectory sample to see that the thing making a human's diff trustworthy, a messy process producing a messy diff, simply isn't true for an agent.
AUDIT, applied to a trajectory logFive letters. The I step, isolate what's missing, is the one a final-diff review skips by design.
A
Ask who ran it.
The agent's own summary of its steps has no reason to highlight its messiest moments; a wider, independently pulled sample does.
Names whose account of the process you're actually trusting.
U
Uncover the eval set.
The rubric is graded against a human-labeled sample of real trajectories, not an abstract idea of "good process."
A rubric with no named sample behind it is just an opinion with extra steps.
D
Demand the version pin.
The messy-trajectory rate climbed right after an unannounced upstream model update, invisible if you only track "Cobblestitch" as one constant thing.
A trajectory pattern from last month may say nothing about this month's actual agent.
I
Isolate what's missing.
A final-diff review never shows the three unrelated files opened and reverted before the real fix landed.
This is the whole answer: the risk lives in exactly what a diff-only review can't see.
T
Test it yourself.
A human grader scores a random weekly sample of real trajectories against the three-item rubric, rather than trusting a self-reported pass rate.
Replaces "the agent says it went fine" with an independently checked number.
Destructive-edit revert rate, weekly (percent of trajectories)
The climb starts right where the version pin says the underlying model changed. Without pinning the version, that timing is invisible.
The recap, one line per letter: ask who ran it is trusting an independent sample over the agent's own summary, uncover the eval set is a human-graded rubric on real trajectories, demand the version pin is catching the climb tied to an unannounced model update, isolate what's missing is the unrelated file edits a diff review never shows, and test it yourself is the weekly human-graded sample replacing a self-reported pass rate.
And if you want to be sure it really works, try it somewhere elseSame five letters, a contract redline instead of a code diff. A completely different field, and the missing piece is legal precedent, not a test suite.
Briarcombe Legal runs an agent that redlines vendor contracts before a partner signs off. Nadia Okonkwo-Reyes manages the review team that checks its output.
Mapped onto AUDIT: ask who ran it means pulling an independent sample of redline trajectories rather than trusting the agent's own change-summary. Uncover the eval set means grading against a set of contracts a senior partner has already reviewed clause by clause, not an abstract sense of "sounds right." Demand the version pin means tracking exactly which model version produced a given redline, since a silent update once caused the agent to stop checking a specific indemnification clause against precedent entirely, invisible in the final redline because the clause simply looked untouched, not wrong. Isolate what's missing means noticing the final redline never shows which clauses the agent actually opened and compared against precedent versus which it left unexamined because they looked routine. Test it yourself means a human partner replaying a sample of trajectories against contracts with a known, deliberately buried risky clause, to see whether the agent's process would have caught it or walked right past it.
The final redline is only the fourth callout here. The other three never show up in a document a partner just skims.
Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "audit the full trajectory log against a human-graded rubric, not just the final diff, since a correct result can hide a risky process," and stop.
Cost: there's no budget this quarter for a dedicated audit tool. Say so honestly, and start with a manual weekly sample of ten trajectories, read by hand, since even ten beats zero.
The model gets better, for real: if Cobblestitch's overall pass rate climbs, that's still not a reason to stop sampling trajectories, since a version update that improves average quality can simultaneously introduce a new, narrower process failure the pass rate alone won't show.
Where people run it wrong.
They review an agent's output the same way they'd review a person's, assuming a clean result implies a clean process.
They trust the agent's own self-reported summary of its steps instead of pulling the raw, full trajectory log.
They build a rubric once and never re-check it against a newly pinned model version.
How to use it live. If you're stuck, ask one question: could this exact final output have come from a genuinely bad process that got lucky? If the honest answer is yes, you need the trajectory, not just the answer, and that's the whole case for auditing it.
Flashcards (tap any card to flip it)
1 · THE FRAMEWORK
What framework fits "how do you evaluate an agent's trajectory rather than its final answer"?
Tap to flip
ANSWER
AUDIT: ask who ran it, uncover the eval set, demand the version pin, isolate what's missing, test it yourself. It treats the trajectory log as a document to audit, not a claim to trust.
2 · THE PEOPLE
Who is this answer about?
Tap to flip
ANSWER
Yuki Tanaka, who manages engineering at Porthaven Systems and reads every agent-authored PR's diff herself before it ships.
3 · THE FAILURE MODE
What does a final-diff-only review always miss?
Tap to flip
ANSWER
Whether a correct diff was reached by a deliberate process or by blind edits and lucky reverts. Both can produce the exact same final file.
4 · THE ISOLATE STEP
What's the two-setting gap this whole answer turns on?
Tap to flip
ANSWER
A clean, deliberate trajectory versus a messy one that happened to land on the same correct diff. A diff-only review can't tell them apart.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Reviewing agent PRs the same way as a human's, assuming a messy process would show up as a messy diff, the way it usually does for a person.
6 · THE NUMBER
Fill in the blank: ___ percent of correct, merged PRs had reached that correctness through a genuinely messy trajectory.
Tap to flip
ANSWER
22 percent. Another 7 percent had involved reverting a destructive edit mid-trajectory before landing on the real fix.
7 · THE REPLAY
Same unrelated-file pattern, rubric now in place. What changes?
Tap to flip
ANSWER
The audit flags the pattern before the PR reaches Yuki, and a reviewer steps in mid-process instead of only ever seeing the final, already-clean diff.
8 · CROSS PRODUCT TRANSFER
Section 4 answers this again for a different product. Which product, and what's the equivalent of a test suite?
Tap to flip
ANSWER
Briarcombe Legal's contract-redlining agent. Its equivalent check is whether a clause was actually compared against legal precedent, not just left looking untouched.
Check yourself Score: 0 / 0
Multiple choice
1. Why did PR 482 pass review for months despite its messy trajectory?
A. The test suite was broken and never actually ran.
B. The review only looked at the final diff, which was clean, and never saw the three unrelated files opened and reverted.
C. Yuki approved it without reading it at all.
D. Cobblestitch's version was never updated during that period.
Show hint
Look at the comparison diagram of what most reviews check.
Show answer
B. The final diff hid the entire messy middle of the trajectory, which is exactly the isolate-what's-missing step.
True or false
2. True or false: this answer recommends a full human audit of every single PR Cobblestitch opens, forever.
True
False
Show hint
Look at "what I would leave alone."
Show answer
False. A short trajectory that touches only ticket-named files and runs tests once is reliable enough for a light spot-check, not a full audit every time.
Fill in the blank
3. Fill in the blank: the destructive-edit revert rate climbed from about 3 percent to ___ percent starting right around the week the underlying model quietly updated.
Show hint
Look at the line chart marking the model update.
Show answer
19 percent. Without pinning the model version, that timing connection would have been invisible.
Short answer, name the reversal
4. What old decision does this answer take back, and why did it make sense when it was made?
Show hint
Look at "the decision I would take back."
Show answer
Model answer: Reviewing Cobblestitch's PRs the same way as a human engineer's. It made sense because that's exactly how code review has always worked.
Short answer, where it wouldn't matter
5. Name a Cobblestitch trajectory shape where a full audit genuinely wouldn't add much.
Show hint
Look at "what I would leave alone."
Show answer
Model answer: A short trajectory touching only the ticket's named files, running the full test suite exactly once. That process shape already correlates strongly with reliability.
Short answer, apply it yourself
6. Pick an AI tool that shows you its steps, not just its final answer. What's one thing you'd want to see in its process before trusting the output, even if the output looks right?
Show hint
Think about what a good process would look like versus a lucky-guess process that reached the same final result.
Show answer
Model answer: Many people would want to see that it actually checked a source or re-read its own draft, not just that the final answer happens to sound plausible.
Before you close the answer
Why this works
Tests whether you understand that an agent's final answer and its process are two separable things, and whether you'd build a real, independently checked audit of the process instead of trusting a clean-looking result.
Follow-up traps
"Isn't grading trajectories by hand too slow to matter?" Response: a weekly sample, not every trajectory, catches the pattern in days instead of months, which is exactly how the version-update spike got found before it caused a production incident.
"What if the agent's process looks messy but the outcome is still always correct?" Response: "always" is doing a lot of work in that sentence; a messy process that's gotten lucky so far is still one bad edge case away from a wrong answer nobody catches, because nobody was watching the process.
If pressed
Porthaven's actual rubric also logs whether the agent re-read its own diff before opening the PR, since trajectories that skip that self-check correlate with a measurably higher rate of unrelated-file edits slipping through unnoticed.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
