CaseIntermediateResponsible AI & Advanced Practice / AI product case study teardowns / #6

What does an AI customer support product get right that a generic chatbot does not?

PICK the product is Portside Assist, an AI support line for a shipping port's truck drivers and dispatchers

Portside Assist answers phone calls from truck drivers and dispatchers at a shipping port, handling berth assignments, container lookups, and gate schedules. Ben Okafor leads support operations there and has run both a generic chatbot and Portside Assist on the same call volume.

The direct answer
Portside Assist lets a caller fix one wrong detail without starting over. A generic chatbot treats any misunderstood detail as a reason to restart the whole request. That one design decision is the entire gap: it turns a ten-second correction into either a ten-second correction, or a full restart that costs minutes and, sometimes, a truck sent to the wrong berth.
Do this, in order
  1. Let the caller correct one detail without restarting the whole request.Why: this single decision explains nearly the entire time and error gap between the two products.
  2. Optimize against the hidden error, not the visible one.Why: a caller repeating themselves is annoying but cheap. A wrongly logged berth number is expensive and often invisible until a truck is already there.
  3. Confirm high-stakes details out loud before committing them.Why: a berth number or container ID is exactly the kind of detail where a silent mistake is costly.
  4. Leave low-stakes chit-chat, like a greeting or sign-off, unconfirmed.Why: nothing bad happens if that gets slightly misheard.
  5. Watch for rising call-abandonment if confirmations get too frequent.Why: that's the one signal that would flip this pick back toward fewer confirmations.
  6. Re-measure the cost gap as call volume grows.Why: the gap between the two designs gets wider, not narrower, under heavier load.

How to answer this, stage by stage

Six moves. The interviewer wants to see you commit to a position fast, then defend it with a real cost, not a feature checklist.

Stage 1
Scope it to one real support line
Say it like this
"I'll take Portside Assist, an AI phone line for a shipping port, and compare it against a generic chatbot handling the same kind of call."
Why this works
Turns a broad question into one concrete comparison you can actually defend.
Stage 2
Say your structure out loud
Say it like this
"I'll use PICK. Position first, then impact, cost asymmetry, and what would flip my pick."
Why this works
Signals you're committing to an answer, not hedging with "it depends."
Stage 3
Take the position
Say it like this
"The thing a good AI support product gets right is letting you fix one detail without restarting. That's it. Everything else follows from that one design call."
Why this works
One sentence, no hedging, exactly what a PICK answer needs up front.
Stage 4
Name who feels each kind of error
Say it like this
"A driver who has to repeat a berth number once loses ten seconds. A driver whose wrong berth number gets logged and confirmed anyway loses twenty minutes, and sometimes queues at the wrong gate entirely."
Why this works
Names both sides of the tradeoff in real units, not abstract accuracy percentages.
Stage 5
Name the cost asymmetry
Say it like this
"Repeating yourself is cheap and visible. A silently wrong berth number is hidden and expensive, since nobody notices until a truck is already at the wrong gate. Optimize against that one."
Why this works
This is the heart of PICK: naming which error kind actually matters more.
Stage 6
Give the kill criteria, and close
Say it like this
"If confirming every detail started making drivers hang up mid-call, I'd back off. Right now abandonment is low and misroutes are the real cost, so the pick holds."
Why this works
Shows the position isn't stubborn; it would change given different evidence.

Let's learn

Here is what happens the moment a phone support tool mishears one word: either it asks you to repeat just that word, or it throws away everything you've said so far and asks you to start over.

Portside Assist answers calls from truck drivers and dispatchers at a busy shipping port, handling berth assignments, container lookups, and gate schedule questions over the phone.

Knowledge spark: what's a "slot" in a voice support system? A slot is one specific piece of information the system needs to help you, like a berth number, a container ID, or a pickup time. A request usually has several slots. The question is what happens when only one of them comes through wrong.

Before Portside Assist, Ben's team ran a generic chatbot for phone support. It handled straightforward calls just fine: clear speech, common requests, no background noise.

Average handle time and misroute rate, one detail misheard
7 min / 5 per 100 0 6.2 min 4.1 / 100 Generic chatbot 2.1 min 0.6 / 100 Portside Assist handle time misroutes
Portside Assist isn't just faster. It also misroutes seven times less often, because it never has to guess its way through a restarted request.

At its worst: a full restart doesn't just cost minutes, it costs a second guess at every detail the caller already gave correctly the first time, and each guess is one more chance to get something wrong.

The cost asymmetry that matters here Repeating one word is cheap and visible: the caller notices, says it again, moves on. A silently wrong berth number is hidden and expensive: nobody notices until a truck is already queued at the wrong gate, sometimes racking up real demurrage costs. A good design optimizes against the second one, even if it means adding a small confirmation step to every call.

What I would leave alone: a misheard greeting, "good morning" instead of "good afternoon," costs nothing and needs no confirmation step at all. Confirming that kind of detail would just slow every single call down for zero real benefit.

A generic chatbot doesn't fail because it mishears things. It fails because it treats every mishearing as a reason to forget everything else the caller already said.

The lesson: the real skill in an AI support product isn't understanding speech perfectly. It's deciding what to do the moment it doesn't, and whether that moment costs the caller ten seconds or ten minutes.

Now here is the same thing as a story

The short version above is what you'd say defending this position live. Read this one for how the two designs actually compared on a real call.

Before Portside Assist, a call from a dispatcher named Marcus went like this: he stated his container ID, the berth he needed, and the pickup time, all in one breath, the way anyone in a hurry does. The chatbot caught the container ID and the pickup time, but misheard one digit in the berth number.

Hand sketched comparison diagram titled One wrong word two different fixes. Left panel a question mark box icon labeled Generic chatbot, caption restart the whole thing. Right panel a gauge icon labeled Portside Assist, caption fix just the wrong part.
Both systems mishear the same berth number. Only one of them makes Marcus repeat everything else he already said correctly.

It didn't ask him to just repeat the berth number. It said "let's start over," and asked for the container ID again, then the berth, then the pickup time, this time in a different order, which confused him further.

Hand sketched icon list titled What a generic chatbot never asks. Three rows: a question mark box icon labeled which container which berth, a box icon labeled one slot wrong all restart, a document icon labeled no memory of the fix.
Three small gaps, and all three point the same direction: nothing about the design assumes only one detail might be wrong.

Marcus hung up and called back a human dispatcher instead, the same shift he'd been assured the new AI line would handle. Somewhere in the call logs, that shows up as an "abandoned" AI call, no note about why.

Hand sketched flow diagram titled Portside Assist's clarify then resolve pause. Five boxes: Ben asks, clarify one slot highlighted, confirm, resolve, log the fix.
Five steps, and the second one is the entire design decision: ask about only the part that's actually unclear.

Portside Assist handles the identical call differently. It catches the container ID and pickup time cleanly, and when the berth number comes through unclear, it asks just one question: "Did you say berth fourteen or berth forty?" Marcus answers in two seconds, and the call moves on with everything else intact.

Hand sketched quadrant titled Sorting support requests by repair cost. Axes how often misheard from rare to common, and cost to fix if wrong from cheap to expensive. Container ID sits top right, common and expensive. Berth number sits nearby, also common and expensive. Shift time sits middle. Greeting tone sits lower left, rare and cheap.
Berth numbers and container IDs sit exactly where a confirmation step earns its cost. A greeting doesn't.

Cost per call as monthly volume grows
$2.50 $1.25 $0 Generic chatbot: $2.40 Portside Assist: $0.85 2,000 calls 10,000 calls
The gap doesn't just persist under load, it widens. More restarts at higher volume push more calls to costly human escalation.

Someone building the generic chatbot decided, early on, that any unclear detail should trigger a full restart, since it was simpler to build one clean reset path than to handle repair for every possible slot combination. That was a reasonable shortcut for a demo with clean audio and short requests.

It stopped being reasonable the moment real drivers, calling from a noisy truck cab with a rushed sentence covering three details at once, became the actual users. I would take that shortcut back and build the repair-in-place path instead, even though it takes more engineering work up front.

I picked the full-restart design because it was the fastest thing to ship, and every early test call had perfect audio and one detail at a time. It took hearing Marcus hang up and call a human instead, with no complaint logged anywhere in our system, to see that "simpler to build" and "cheaper to run" aren't the same thing.

PICK, in one screenFour letters, and the second one, cost asymmetry, is the one that actually decides the pick.

P
Position.
Repair one detail in place, don't restart the whole request. That's the single design decision that matters most.
Stated first, before any reasoning, exactly what a PICK answer needs.
I
Impact.
A driver repeating a berth number loses ten seconds. A driver whose wrong berth number goes uncorrected loses twenty minutes and sometimes a real gate misroute.
Names both sides in real units, not an abstract accuracy score.
C
Cost asymmetry.
Repeating yourself is cheap and visible. A silently wrong berth number is hidden and expensive. Optimize against the hidden one.
The hardest step, and the one the whole pick actually turns on.
K
Kill criteria.
If confirmation steps started causing real call abandonment, that would flip the pick back toward fewer confirmations.
Shows the position is evidence-based, not stubborn.

The recap, one line per letter: position is repair-in-place over full restart, impact is ten seconds versus twenty minutes, cost asymmetry is optimizing against the hidden, expensive error, and kill criteria is watching for rising call abandonment.

And if you want to be sure it really works, try it somewhere elseSame four letters, a community helpline instead of a shipping port. This time the caller has even less patience for a restart.

A regional nonprofit runs an AI-assisted phone line helping residents check eligibility and enroll in a local rent-assistance program. Callers are often stressed, sometimes on a shared or borrowed phone, and give their household size, income, and address in one anxious breath.

Mapped onto PICK: position is the same, repair one detail without a restart, this time applied to an address or a household size instead of a berth number. Impact: a caller asked to repeat one number loses a few seconds; a caller forced to restart the whole intake, after already working up the nerve to call, may simply hang up and never call back at all. Cost asymmetry: the hidden and expensive error here isn't a misrouted truck, it's a real family that needed help and didn't get it, because the design made "correcting a mistake" feel like being interrogated all over again. Kill criteria: if repair-in-place confirmations made the call feel slower without reducing actual restarts, that would flip the pick back toward a simpler, single confirmation at the very end instead of mid-call.

Hand sketched labeled parts diagram titled What a helpline agent needs to fix in place. Center person icon labeled Caller's request, with four callouts: which program, which household, correct just that, no full restart.
A different setting, a much higher emotional stake, and the exact same underlying design call: fix the one thing that's wrong, not everything the caller already got right.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "repair one detail, don't restart the whole request, because the hidden cost of a full restart is far worse than a caller repeating one word" and stop.
Cost: no engineering budget to build slot-level repair this quarter. Say so, and start with the cheapest version: a single "did I get that right?" confirmation at the very end of the call, catching most silent errors without full slot-level repair.
The model gets better, for real: even if the underlying speech model's overall accuracy improves, calls with background noise or fast speech will still mishear something occasionally. A better model narrows how often repair is needed. It never removes the need for a repair path entirely.

Where people run it wrong.
They chase overall accuracy percentage as the whole answer, missing that what happens after a mistake matters more than how rarely it happens.
They assume more confirmation steps are always safer, without checking whether those steps are actually causing callers to hang up.
They compare two support tools on tone and personality, when the real gap is almost always a structural one: restart versus repair.

How to use it live. When asked to compare two AI support products, ask yourself one question first: when either one misunderstands something, what does the caller have to do next? That single question usually reveals the real difference faster than comparing their scripts.

Flashcards (tap any card to flip it)

1 · THE FRAMEWORK
What framework fits "what does A get right that B doesn't"?
Tap to flip
ANSWER
PICK: position, impact, cost asymmetry, kill criteria. Built for tradeoff and A-or-B questions.
2 · THE PEOPLE
Who is this answer about?
Tap to flip
ANSWER
Ben Okafor, who leads support operations at a shipping port and has run both a generic chatbot and Portside Assist on real calls.
3 · THE POSITION
What's the one design decision this whole answer turns on?
Tap to flip
ANSWER
Repair one wrong detail in place, instead of restarting the whole request the moment anything is unclear.
4 · THE COST ASYMMETRY
Which error is cheap, and which is expensive, in this story?
Tap to flip
ANSWER
Repeating one word is cheap and visible. A silently wrong berth number is hidden and expensive, sometimes ending in a real gate misroute.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Building a full-restart design because it was simpler to ship, reasonable only when early test calls had clean audio and one detail at a time.
6 · THE NUMBER
Fill in the blank: the generic chatbot misroutes about ___ calls per 100, versus 0.6 for Portside Assist.
Tap to flip
ANSWER
4.1 per 100. Roughly seven times the misroute rate of the repair-in-place design.
7 · THE REPLAY
Same misheard berth number, repair-in-place design. What changes?
Tap to flip
ANSWER
Marcus answers one quick clarifying question and the call finishes in about two minutes, instead of restarting the entire request and eventually hanging up.
8 · CROSS PRODUCT TRANSFER
Section 4 answers this again for a different product. Which one, and what's the stake there?
Tap to flip
ANSWER
A nonprofit's rent-assistance helpline. The stake there isn't a misrouted truck, it's a family that needed help and hung up instead.

Check yourself Score: 0 / 0

Multiple choice
1. Why does this answer say the cost asymmetry favors optimizing against a silent wrong detail, rather than against a caller having to repeat something?
  • A. Because callers never notice when they're asked to repeat something.
  • B. Because a repeated word costs seconds and is visible, while a silently wrong detail is hidden and can cost a real misroute later.
  • C. Because repeating a word takes longer than a full restart.
  • D. Because generic chatbots never mishear anything.
Show hint
Look at the cost asymmetry key point.
Show answer
B. The hidden, expensive error is the one worth designing against, even at the cost of a small amount of visible friction.
True or false
2. True or false: this answer recommends confirming every single detail on every call, including greetings and small talk.
  • True
  • False
Show hint
Look at "what I would leave alone."
Show answer
False. Low-stakes details like a greeting don't need confirmation. Only details where a mistake is genuinely costly do.
Fill in the blank
3. Fill in the blank: Portside Assist's average handle time when a detail is misheard is about ___ minutes, versus 6.2 for the generic chatbot.
Show hint
Look at the grouped bar chart.
Show answer
2.1 minutes. Roughly a third of the generic chatbot's handle time for the same kind of call.
Short answer, name the reversal
4. What old design decision does this answer take back, and why did it make sense when it was made?
Show hint
Look at the story's paragraph about the "reasonable shortcut."
Show answer
Model answer: Treating any unclear detail as a reason to fully restart, reasonable when early test calls had clean audio and only one detail at a time.
Short answer, where it wouldn't matter
5. Name a detail in a Portside Assist call where a full restart-style design genuinely wouldn't cost much.
Show hint
Look at "what I would leave alone."
Show answer
Model answer: A misheard greeting or sign-off. Getting that wrong costs nothing and needs no confirmation or repair step.
Short answer, apply it yourself
6. Think of a voice assistant or phone system you've used. When it misunderstood you, did it ask you to repeat just the wrong part, or make you start over?
Show hint
Think about the last automated phone menu or voice assistant that misheard you.
Show answer
Model answer: Most people can recall a system that made them start over, and remember exactly how annoying that felt compared to a quick, targeted correction.
Before you close the answer
Why this works
Tests whether you can name a real, structural design difference between two AI products instead of comparing surface polish, and whether you can commit to a position with a defensible cost tradeoff behind it.
Follow-up traps
"Wouldn't confirming every detail just slow every call down?" Response: only low-value confirmations do that. Confirming a few genuinely high-stakes details, like a berth number, adds seconds while avoiding a much costlier silent error.

"Isn't this just a better speech-recognition model, not a design decision?" Response: no, even a perfect speech model still occasionally mishears in a noisy environment. The design decision is what happens next, and that's independent of how good the underlying model is.
If pressed
Portside Assist only asks a clarifying question when its own confidence on that specific slot falls below a set threshold, so clearly-heard details never get an unnecessary confirmation, keeping the added friction limited to the cases that actually need it.
From U2xAI Academy

From answering questions to owning outcomes.

A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.

  • A live AI agent you actually shipped
  • A launch decision you can defend under pressure
  • An interview-ready portfolio, not more flashcards
Know more