ConceptAdvancedAI Opportunity & Model Strategy / Model selection from a PM lens / #18
How do you compare models when your task requires tool use rather than text generation?
LEADnobody graded whether the room actually held
Wrenfield Journeys sells trip planning to independent travel agents. TripPilot is the assistant they built on top of a large language model: a traveler describes a trip, and TripPilot searches flights, holds a hotel room, and sends a confirmation, by calling real booking APIs, not just by writing a nice reply. Noor Kessab is the AI PM who has to pick which model TripPilot runs on.
The direct answer
Score every model candidate on the one thing that happens after it decides to act: did the tool call go through, with the right parameters, matching what the model told the traveler. Build a golden set of real bookings and measure task completion on it, not chat quality. A model that writes a warmer confirmation message but sends the wrong check-in date to the hotel's API is not the better model for this job, no matter what it scores on a general leaderboard.
Do this, in order
Score candidates on tool-call task completion against your own golden set of real bookings.Why: a general chat leaderboard never once calls a real hotel API, so it can't tell you this.
Grade the parameters, not just the reply.Why: a wrong date and a wrong room count both read as a normal, friendly confirmation message.
Track a weekly completion rate on live traffic as your early warning number.Why: complaints show up weeks after the real drift starts, this number moves first.
Route anything under your bar to a human hold check instead of shipping it wider.Why: a calibrated cutoff beats "ship it and hope it holds."
Name how the eval could be gamed, like only testing easy single-city trips.Why: every metric has a cheap way to be hit without doing the real work.
Say plainly where chat fluency still wins, like small talk with no booking attached.Why: fluency matters somewhere, just not at the moment the model reaches for a tool.
How to answer this, stage by stage
Nobody is scoring whether you know that leaderboards exist. They're scoring whether you can name the exact number that would have caught this before a guest ever complained.
Stage 1
Scope it to one real system
Say it like this
"Let's ground this in TripPilot, an assistant that books real flights and hotel rooms by calling live APIs. That's the system I'd compare models against, not a general chat assistant."
Why this works
Keeps the answer from turning into a lecture on benchmarks in the abstract.
Stage 2
Say the structure out loud
Say it like this
"I'll run this as LEAD. Link, the business outcome that actually matters. Early signal, the number that moves first. Abuse, how the metric gets gamed. Decision, what I'd actually do at each threshold."
Why this works
Shows a repeatable way to pick an eval, not a one-off gut call about which model demo felt smart.
Stage 3
Reframe: it isn't "which model sounds smarter," it's "which metric would have caught this first"
Say it like this
"This isn't really a question of which model writes the better sentence. It's a question of which number would have told me, weeks early, that bookings were quietly failing behind a confident-sounding reply."
Why this works
This is where a strong answer separates from someone who just compares two leaderboard scores.
Stage 4
Give the one decision
Say it like this
"Here's the actual metric: weekly tool-call task completion on a golden set of 200 real itineraries, checked against the hotel and flight APIs' own confirmation, not the model's claim that it worked. Ship a model only once it clears 97 percent there, on real tool calls, not on a chat benchmark."
Why this works
This is the direct answer, stated as a real number and a real cutoff, not a vague preference for "the more capable model."
Stage 5
Prove it with the compressed evidence
Say it like this
"We shipped the model with the best chat benchmark score in the industry. Its own tool-call completion rate slid from 96 percent to 87 percent over six weeks, and nobody watched that number, because we were watching guest complaints instead, and complaints didn't move until week five."
Why this works
Compresses the whole case into the one number that was quietly telling the truth weeks before anyone noticed.
Stage 6
Name the AI-specific reasoning and the trade-off
Say it like this
"The honest reason this isn't a generic QA problem is that the model doesn't fail loudly. It tells the traveler 'you're all set,' in a perfectly warm sentence, while the actual API call behind it silently used the wrong date format. We accepted a model that reads slightly less polished in plain chat, in exchange for one whose tool calls are boring and correct almost every time."
Why this works
This is the load-bearing, AI-specific judgment. A normal software bug throws an error. This one writes you a nice paragraph instead.
Stage 7
Say what wouldn't change, then close
Say it like this
"I wouldn't touch which model answers a traveler's plain question, like 'what's the baggage policy,' since nothing there ever touches a booking API and a wrong answer there is a quick correction, not a lost room. For TripPilot, the rule holds: grade the tool call, not the sentence around it."
Why this works
Closes with real judgment about where the concern doesn't apply, and restates the direct answer in one breath.
Let's learn
Every evening before TripPilot, a Wrenfield agent typed out an itinerary by hand, checked flight times against three tabs, and called a hotel to hold a room, about 32 minutes for a single trip.
With TripPilot, an agent describes the trip in one sentence, and a booking that used to take 32 minutes comes back, done, in under a minute, for all 600 trips Wrenfield books on a busy day.
The four letters, held up as one page. Early signal is the one most model comparisons skip.
Here's the turn: the model Wrenfield picked had the best score on every public chat benchmark going. On real bookings, it silently sent the wrong date format to the hotel's hold API about 9 times in every 100 trips, and its own reply to the traveler never once mentioned that anything had gone wrong.
The model never dipped under its own launch bar of 97 percent by much in any single week. It just never stopped sliding.
At its worst, a guest arrives at a hotel with no room held, the trip's confirmation message says everything is fine, and Wrenfield finds out only when the guest calls, furious, from the front desk.
A confident sentence is not a confirmation. TripPilot said "you're all set" to 54 guests who were not.
The choice I would take back
Wrenfield's original eval process was simple: run the standard chat benchmark suite, ship whichever model scored highest. That made sense when TripPilot only answered questions in chat. It stopped making sense the day TripPilot started calling a hotel's real booking API, since that suite never once tests whether a tool call actually went through.
What I would leave alone: I wouldn't change which model handles a traveler's plain question with no booking attached, like asking about baggage rules, since a wrong answer there costs a quick correction, not a lost room.
The lesson: a model that talks a great game can still send the wrong date to a hotel's API. Grade the tool call. The sentence around it will always sound fine.
Now here is the same thing as a story
The short version above is what you'd say in a design review. Read this one for what it felt like the six weeks nobody was watching the right number.
Noor Kessab could read a candidate model's eval report and spot a padded benchmark before she finished the first page.
When Wrenfield first plugged TripPilot into the model with the best public chat score, the demo looked perfect: warm confirmations, natural back-and-forth, travelers loved how it sounded. For the first three weeks, the ops team's daily spot-check of 30 random bookings kept coming back clean.
Same word, two different questions. Only one of them was ever being asked.
By week four, the spot-checks started turning up something odd: a booking marked confirmed in the transcript, but no matching hold in the hotel's own system. Noor didn't panic. One weird case, she figured, could be anything.
Knowledge spark: why would a model sound confident and still be wrong?
A model that generates text is trained to produce a fluent, likely-sounding next sentence. It isn't automatically trained to check whether the tool call it just made actually succeeded. Unless you build that check in, the model will happily describe a failed booking as a success, because a success is what a confident sentence usually sounds like.
An audit two weeks later, pulling every booking from the past month and checking it against the hotel API's own logs, found the real number: 9 out of every 100 bookings had a malformed date sent to the hold API, and in every single one, TripPilot's reply to the traveler said the room was confirmed.
The step nobody's chat benchmark ever tests is the third one from the left.
The real question was never which model sounded better in a demo. It was which model could be trusted with the one part of the job that happens after the talking stops: filling in a form correctly and pressing send.
None of these four live inside a general chat benchmark. All four live inside a real booking.
When the model was first chosen, someone in the room said, "it's the best model on the market right now, look at its scores," and it sounded reasonable, since the scores were real, just measuring a different question than the one that mattered here.
Guest double-booking complaints per week
Complaints stayed calm for five weeks while the real number, task completion, was already sliding underneath them.
Rerun the same six weeks with a tool-use eval running from day one: the drift shows up in week two, at 95 percent, two full points under the ship bar. The model gets pulled back to shadow mode before a single guest ever calls a front desk confused.
What I'd tell myself, watching that audit land: the leaderboard graded what TripPilot said. Nobody graded whether the room actually held, and that gap is exactly where the real cost was hiding.
LEAD, the metric that would have caught this before a guest ever complainedNot a script for ignoring chat quality. LEAD is what tells you which number to watch first, and which one to watch second.
L
Link. The business outcome that actually matters.
A trip that actually happens the way TripPilot said it would: the flight is real, the room is held, the traveler never finds out otherwise at the front desk.
Naming the real outcome first stops the chat leaderboard from quietly standing in for it.
E
Early signal. The thing that moves weeks first.
Weekly tool-call task completion on a golden set of 200 real itineraries, checked against the hotel and flight APIs' own records, not the model's own claim of success.
This is the hardest step, and the answer to the question: it moved in week two, five weeks before guest complaints did.
A
Abuse. How the metric gets gamed.
A golden set built only from easy, single-city trips will always score high. Multi-city trips, date changes, and split payments are where tool calls actually break.
Every metric has a cheap way to be hit without doing the real work, name it before someone finds it by accident.
D
Decision. What you'd actually do at each threshold.
Above 97 percent, ship to all traffic. Between 94 and 97, shadow mode with a human confirming holds. Below 94, the model doesn't touch a booking API at all.
A metric nobody acts on is decoration. This one has a job at every level.
The recap, one line per letter: link is a trip that actually happens as promised, early signal is weekly tool-call completion on a checked golden set, abuse is a golden set padded with only the easy trips, and decision is the three-tier bar that decides shadow mode, full rollout, or no booking access at all.
And if you want to be sure it really works, try it somewhere elseSame four letters, an HVAC dispatch tool instead of a travel agent. The API changes, the gap between talking and doing doesn't.
Callan Osei runs product at Briarstoke Field Services, where DispatchAlly is an assistant that reads a technician's job description and calls a scheduling API and a parts-order API to line up the visit. Mapped onto LEAD: link is a technician showing up with the right part already on the truck. Early signal is weekly task completion on a golden set of real dispatch requests, checked against the parts-order system's own confirmation, not DispatchAlly's chat reply. Abuse is a golden set of only simple one-part jobs, when real jobs often need two or three parts ordered from different suppliers. Decision is the same three-tier bar: full rollout above 97 percent, shadow mode with a dispatcher confirming orders between 94 and 97, no parts-ordering access below that.
When technicians first started describing jobs by voice, DispatchAlly's tool calls broke on messy, run-on descriptions. Technicians didn't complain or stop using it. They started typing short, rigid phrases instead of describing the job naturally, a workaround nobody asked them to build and nobody could see from the dashboard.
The corner that matters for a booking tool is the bottom right of most leaderboards, not the top.
One full-page image to carry the whole answer: pick the mechanic, not the talker, for a job with real bolts in it.
Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "grade the tool call, not the sentence around it, on your own golden set," and stop.
Cost: the golden set of real bookings is expensive to build and check by hand. Say so honestly, and start with 50 real cases instead of 200, expanding as budget allows, rather than skipping the check entirely.
The model got better, for real: if a future version of the chat-benchmark model is retrained specifically on tool-call accuracy, that's exactly the kind of change worth re-testing against the golden set, not assuming from the vendor's announcement.
Where people run it wrong.
They compare models using only public leaderboards, never running a single real tool call themselves.
They watch user complaints as their only signal, missing the weeks of drift that happen before anyone calls in.
They build a golden set once and never touch it again, missing the new trip types that show up as the product grows.
How to use it live. The moment an interviewer asks how to compare models for a tool-use task, ask yourself: what's the one thing that has to be true after the model decides to act, and how would I check that, separately from what it says happened? Name that check, and the rest of the comparison follows on its own.
Flashcards (tap any card to flip it)
1 · THE FLIP FAMILY
What flip family is this?
Tap to flip
ANSWER
Verification flip: the ops team spot-checks a sample of bookings, then, after an audit finds a cluster of failures, moves to checking every single one by hand.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Noor Kessab, the AI PM at Wrenfield Journeys, who chose which model powers TripPilot's real bookings.
3 · THE HABIT
What did the ops team stop doing because it worked?
Tap to flip
ANSWER
They stopped worrying that the daily 30-booking spot-check might be missing anything, since it kept coming back clean for the first three weeks.
4 · THE FLIP, IN THIS STORY
What's the two setting switch here?
Tap to flip
ANSWER
Trusting the daily spot-check sample versus auditing every single booking against the hotel API's own logs. There was no in-between setting once the audit found a real cluster.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Using the standard chat benchmark suite as the default way to pick a model, a default that made sense before TripPilot ever called a real booking API.
6 · THE NUMBER
Fill in the blank: task completion on real bookings drifted from ___ percent to ___ percent over six weeks, while complaints stayed flat until week five.
Tap to flip
ANSWER
96 percent down to 87 percent.
7 · THE REPLAY
Same six weeks, tool-use eval running from day one. What changes?
Tap to flip
ANSWER
The drift shows up in week two at 95 percent, two points under the ship bar, and the model gets pulled to shadow mode before a single guest calls a front desk confused.
8 · CROSS PRODUCT TRANSFER
Section 4 answers this same question again for a different product, with a different flip family. Which product, and which family?
Tap to flip
ANSWER
Briarstoke Field Services' DispatchAlly. The flip is an input flip: technicians started typing rigid short phrases instead of describing jobs naturally by voice.
Check yourself Score: 0 / 0
Fill in the blank
1. Fill in the blank: the chat-benchmark model's tool-call task completion drifted from ___ percent in week one to ___ percent by week six.
Show hint
Look at the line chart in "Let's learn."
Show answer
96 percent, 87 percent. A steady slide, not a sudden cliff, which is exactly why nobody noticed without a dedicated metric watching it.
Multiple choice
2. Why did guest complaints stay flat for weeks while task completion was already dropping?
A. The hotel APIs were slow to report failures back to Wrenfield.
B. TripPilot's reply to the traveler said the room was confirmed even when the tool call had failed, so nobody had a reason to complain yet.
C. Guests generally don't notice booking problems until months later.
D. The audit team was slow to check the data.
Show hint
Look at the knowledge spark about a confident sentence.
Show answer
B. The model's own reply masked the failure, so the lagging outcome (complaints) had no way to move until a guest physically reached an empty front desk.
True or false
3. True or false: the chat benchmark used to originally pick TripPilot's model tested real hotel-API calls.
True
False
Show hint
Look at "the two different things called a benchmark."
Show answer
False. A chat benchmark grades the sentence a model writes. It never once calls a real booking API to check whether anything actually happened.
Short answer, where it wouldn't matter
4. Name a part of TripPilot where this same concern would NOT apply, and say why.
Show hint
Look at "what I would leave alone."
Show answer
Model answer: A traveler's plain question, like asking about baggage rules. No tool call happens there, so a wrong answer costs a quick correction, not a lost room.
Short answer, apply it yourself
5. Think of a tool you use that takes an action for you, not just answers a question. What would you actually check to know it worked, beyond what it tells you?
Show hint
Think about the difference between what a tool says happened and what you could independently verify.
Show answer
Model answer: A calendar assistant that schedules meetings. Instead of trusting its "invite sent" message, check the actual calendar of the other person, if you can, since a malformed invite can still generate a friendly confirmation message.
Short answer, work the number
6. If the tool-use eval had only sampled 20 bookings a week instead of 200, would it still have caught the drift by week two?
Show hint
Think about how a 9-in-100 failure rate shows up in a small sample versus a large one.
Show answer
Model answer: Less reliably. At a 9 percent failure rate, 20 bookings might show 1 or 2 failures some weeks and none in others, making the drift look like noise. A larger, consistent sample is what makes the early signal trustworthy instead of jumpy.
Before you close the answer
Why this works
Tests whether you'll reach for a generic leaderboard out of habit, or notice that a tool-using model needs its own eval built around the thing it actually does: act, not just talk.
Follow-up traps
"Isn't building a golden set of real bookings expensive and slow?" Response: yes, and it's worth starting small, 50 real cases instead of 200, rather than skipping it, since the cost of a bad model in production is a lost hotel room, not a slightly slower launch.
"What if the vendor's next release claims better tool-use scores?" Response: test the claim against your own golden set before trusting it. A vendor's tool-use benchmark is still not your booking flow.
If pressed
The golden set gets refreshed monthly with the newest 20 real bookings that included an edge case, like a split payment or a same-day date change, so the eval keeps pace with how travelers actually use TripPilot instead of testing the same 200 easy trips forever.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
