CaseAdvancedShipping & Model Lifecycle / Incident management for AI products / #16
Describe the incident where an agent took a wrong action, and your remediation.
Purchasing agent incident log
The direct answer
Put a hold on any order the agent generates that jumps past roughly double a vendor's normal size, and route that order to a person before it ever reaches the vendor, not after. Never let the agent auto-confirm anything a vendor cannot undo once they start building it. That one hold would have stopped the wrong order before a single part got made.
Do this, in order
Hold any order that jumps past double a vendor's trailing average, before it reaches the vendor.Why: that one hold is what would have stopped this order before Threadwell ever touched steel.
Cap the agent's own authority to confirm anything a vendor cannot undo once they start.Why: drafting a request and locking in real production are two different levels of risk, and only one of them needs a person watching every time.
Check every auto-approved order against that specific vendor's own order history, not just a dollar ceiling.Why: eighteen thousand dollars looked cheap enough to auto-approve; only Threadwell's own pattern showed it was five times too big.
Run same-day reconciliation on every order the agent sends, not a monthly one.Why: this sat unnoticed for eleven days because the only check left was an invoice that looked odd, and by then the parts were already made.
Give a vendor a way to reach someone who can actually see the system, not just someone who reads the number back to them.Why: Kojo called to check, and the person he reached had no more visibility than he did, so the one moment anyone tried to push back went nowhere.
Clear stale rows out of any table the agent reads for real decisions.Why: this is the smaller fix, it closes this exact cause but not the next one shaped like it.
GUARD starts here. Not what the agent got wrong, but which of these two people could have stopped it.
How to answer this, stage by stage
Nobody is grading whether you can describe a purchasing agent that misfired. They are grading whether you can name the two people a wrong action actually touches, and put the fix where the harm lands, not where the bug lives. Seven moves get you there.
1
Ground it in one order, one vendor, one number
Say it like this
"Let's ground this in one order. Cargast Industrial Supply runs a purchasing agent called Stockhand across about forty thousand parts. Gideon Baird owns it on the buying side. Threadwell Fasteners is one of nine hundred vendors it orders from, a fourteen person shop Kojo Antwi runs."
Why this works
Grounds the answer in one real transaction before any framework talk starts, so it never turns into a policy speech.
2
Say what the question is actually testing, before the story
Say it like this
"Here's how I'd frame it up front. This isn't really a story about an agent picking the wrong number. It's about who ends up holding a wrong number once nobody's left to check it before it leaves the building."
Why this works
Tells the interviewer you know this is a question about power, not accuracy, before you give a single detail.
3
Name both people the wrong action actually touched
Say it like this
"Two people sit inside this incident. Gideon, who built Stockhand and can pause an order from his desk any time he wants. And Kojo, who only ever sees a purchase order marked confirmed, with no way to see why it's the size it is."
Why this works
This is the G step. An interviewer listening for GUARD wants both names, not just the one whose system broke.
4
Say plainly where the harm landed, and how unevenly
Say it like this
"Cargast does something like forty million dollars a year. An eighteen thousand dollar order barely shows up on their books. Threadwell is fourteen people. Nine thousand four hundred dollars of steel and two weekend shifts is real money they can't get back, plus about fourteen thousand dollars of custom parts sitting in their shop that Cargast won't take."
Why this works
This is the U step, and it's the line that stops the answer from sounding like a generic bug report.
5
Show the exact moment Kojo tried to push back, and hit a wall
Say it like this
"Kojo actually called. The number looked five times too big, so he rang Cargast's order line before he touched a bar of steel. The person who picked up could see the order was marked confirmed. That's all they could see. They told him to go ahead."
Why this works
This is the A step, played as a scene instead of a claim. It proves nobody had the lever, not even the one person who went looking for it.
6
Name the design change, and the one you're turning down
Say it like this
"The fix isn't more training on Stockhand's model. I'd hold any order that jumps past roughly double a vendor's own trailing average before it reaches them, and stop letting the agent auto-confirm anything a vendor can't undo once they start building it. What I wouldn't do is add a monthly audit of Stockhand's biggest orders. A monthly audit finds this in week five. Threadwell's steel is already cut by day two."
Why this works
Names the reduce step and the rejected alternative in the same breath, and shows why the timing of the check is the real decision.
7
Say how you'd catch the next one before the vendor has to call
Say it like this
"Under the hold, every order gets checked same day against that vendor's own last eight cycles, not against a flat dollar ceiling. Run the replay and this exact order sits in Gideon's queue about twenty minutes after Stockhand drafts it. He follows the number back to a stale test row, deletes it, and Threadwell never touches a single bar of steel."
Why this works
Closes on the D step and a number someone could go check, not a promise to be more careful next time.
Let's learn
What happens the first time a purchasing agent gets to act on a number nobody taught it to doubt?
Say a distributor builds an agent that watches stock levels across thousands of parts, works out what to reorder, and sends that order straight to the vendor who makes it. No person has to type a purchase order by hand.
Before a tool like this, three buyers split the job across forty thousand parts and nine hundred vendors, about six hours a day each, reading each recommendation before it went anywhere. They caught a bad number the old way: a person who orders the same bolt every week glances at "twelve thousand" next to a part that always says "twenty-four hundred" and just knows something's wrong.
With the agent, most of that six hours disappeared. Orders inside a normal range go out in seconds. A buyer only opens the ones the system flags. Three months before this incident, Cargast raised how big an order could be before a person had to look at it, from five thousand dollars to fifty thousand, specifically so buyers would have even less to check.
Order size to Threadwell, last nine reorder cycles
Eight cycles sat close to 2,400 units. Cycle nine cleared a line double that width, and nothing was watching for a line, only a dollar ceiling Stockhand's number never got close to.
Here is the turn. The agent is right almost all the time, comfortably above ninety-nine cases in a hundred. That was never the problem. The problem is what happens to the one case in a hundred once nobody stands between it and the vendor who has to build it. And the case that hurts is never the one Stockhand flags as unsure. It's the one where a stray number sits next to real numbers and nothing in the agent doubts any single input on its own, so a five-month-old test row gets treated exactly like real demand. Checking that number against the vendor's own pattern is the only thing that catches a case like this; the agent's overall accuracy tells you nothing about it.
It wasn't Cargast's mistake to absorb. It was Threadwell's steel to eat, and nobody at Cargast had to feel it.
At its worst, this costs far more than the order itself. A vendor who can't return a custom run, and can't get back overtime pay or spent material, ends up carrying a loss a much bigger company created and never noticed.
Knowledge spark: what's an auto-approve ceiling?
A dollar line under which the agent can confirm an order to a vendor with nobody checking it first. Above the line, a person has to look. Raise the line, and more of the agent's decisions go out with no one watching.
The choice I would take back. Raising the auto-approve ceiling to fifty thousand dollars, on its own, without ever checking a new order against that specific vendor's own history first.
The decision that mattered
Cargast measured the ceiling in dollars, a number that made sense company-wide. It never measured it against what one small vendor's own pattern looked like. Eighteen thousand dollars is cheap for Cargast. It's five times too big for Threadwell.
What I would leave alone. Some parts really do swing wildly and it's normal, a seasonal item, a promotional part nobody could plan a year ahead for. A hold on every big swing everywhere would just slow down orders that were never wrong. The hold has to be sized to each vendor's own pattern, not applied as one flat rule across all nine hundred of them.
The lesson. An agent that can act, not just suggest, has to earn the right to skip the person standing on the other end of that action. Ninety-nine cases in a hundred doesn't earn it. The hundredth case is the whole job.
Now here is the same thing as a story
The shape above is the whole answer. What follows is what it actually felt like from Threadwell's side of the phone, on the Tuesday steel got cut for a part nobody needed twelve thousand of.
Gideon Baird has run procurement operations at Cargast Industrial Supply for four years. He knows which of his nine hundred vendors will call back if a spec looks off, and which ones will just build exactly what the order says, fast, no questions, because that's the service they're known for. Threadwell is the second kind. Kojo Antwi built that shop's name on doing precisely what a purchase order says, no follow-up call needed.
Stockhand went live for reorders under five thousand dollars that spring. For most of the year that was fine. Buyers stopped watching the small stuff and spent the saved hours somewhere that mattered.
Then, three months before the incident, Cargast raised the ceiling to fifty thousand dollars. Bigger orders started moving with nobody reading them first, and for a while, nothing went wrong. The agent really was good.
Somewhere in Cargast's forecasting tables sat a row nobody remembered: ten thousand units, logged five months earlier by a sales rep building a pitch deck, tagged TEST, never deleted. On a Wednesday morning, Stockhand's forecast for one hex bolt SKU pulled that row in as if it were real demand, stacked it on top of the SKU's normal signal, and recommended a reorder of twelve thousand units against Threadwell, five times the twenty-four hundred that SKU had ordered every three weeks for two years. The order value came out to eighteen thousand four hundred dollars. Comfortably under fifty thousand. Nobody at Cargast saw it before it reached Threadwell's supplier portal at 9:14 that morning.
Kojo saw it an hour later. Twelve thousand looked wrong to him immediately, the same way twenty-four hundred always looked right. He called Cargast's order line before he touched a single bar of steel.
The person who answered pulled up the order. It said confirmed. That was the only thing on their screen. They told him it was fine, go ahead.
So he did. Two weekend shifts, nine thousand four hundred dollars of raw steel rod, twelve thousand units cut, threaded, and shipped by the following Friday.
We didn't take a wrong number from Threadwell. We took their steel, their weekend, and the shelf space for parts nobody was ever going to buy.
Kojo's call landed in the middle of this path, not at either end of it. There was nothing built there to catch him.
Eleven days later, someone on Cargast's accounts payable team flagged an $18,400 invoice from a fourteen-person vendor as unusually large and pulled the order history to check it. Twenty-four hundred, every time, for two years, then this. That's when anyone at Cargast looked at the stale test row at all.
The old decision, told as a memory of a meeting. Three months earlier, the team that raised the ceiling talked about it as a trust question: Stockhand had been right often enough that a five-thousand-dollar cap felt like it was there out of habit, not need. Nobody in that meeting was wrong about what the data showed them. They just measured trust in dollars, a number that meant nothing to a shop the size of Threadwell.
The replay, run the same way forward with a hold in place. The same recommendation drafts at 9:14, but the hold checks it against Threadwell's own last eight cycles instead of Cargast's dollar ceiling, sees twelve thousand against a trailing average of twenty-four hundred, and it never reaches the portal. It sits in Gideon's queue instead. He opens it by 9:35, follows the number back to the stale test row, deletes it, and the real reorder, twenty-four hundred units, goes out that afternoon. Threadwell never touches the steel.
The thing I'd tell my past self, from that meeting about the ceiling: the number that mattered was never how much we trusted Stockhand. It was how much of what it gets wrong, someone else has to eat.
The five checks GUARD makes you run, against this exact order
This is a risk question with a real subject in it, someone outside Cargast's own walls who never signed up to be part of Stockhand's decision, so GUARD fits: name both people, find where the harm lands, and ask who never gets a lever.
G
Groups. Who does this actually touch?
Not "the vendor network." Two named people, one on each end of the same action.
Gideon Baird, who owns Stockhand and can pause any order from his desk. Kojo Antwi, who only sees a purchase order marked confirmed.
U
Unequal. Where does the harm land, and why there?
Put a real number on each side, not just "some vendors get hurt."
$18,400 barely registers against Cargast's roughly $40 million a year. It's $9,400 of real steel and two unpaid weekend shifts for a 14-person shop.
A
Ability to contest. Who never gets a lever, even when they try?
The hardest step, and the one most answers skip past.
Kojo called. The person who answered could only read "confirmed" off a screen, no more visibility than Kojo had himself. Trying to push back went nowhere.
R
Reduce. The actual design change, not a policy.
A specific product decision, sized and testable, not "add more oversight."
Hold any order past double a vendor's trailing average before it transmits, and cap the agent's authority to auto-confirm anything a vendor can't undo once they start.
D
Detect. How you'd know in production, before the person affected has to tell you.
Not a monthly report. Something that runs the same day the order goes out.
Reconcile every order Stockhand sends against that vendor's own trailing history, same day, not against an invoice that eventually looks odd eleven days later.
Time from wrong order to someone catching it
No confirm hold: caught by an odd-looking invoice11 days
Confirm hold, checked against vendor historyabout 20 minutes
A dollar ceiling was never built to catch this kind of order. Tuning it up or down changes when it fires, not whether it can see a number that only looks wrong next to one small vendor's own history.
And if you want to be sure it really works, try it somewhere else
Roundel Home Health runs Aidewell, a scheduling agent that reassigns home visits the moment an aide cancels, and texts the family the new time. No person reviews a reschedule before it sends.
G. Two people again. Priel Nakamura, the scheduling coordinator, can see every aide's calendar and can override any reschedule from her desk. The patient on the other end of the visit never sees Aidewell at all, only a text. U. When an aide cancelled a 2pm medication visit and Aidewell rebooked it for the next morning instead of finding same-day coverage, the harm landed entirely on the patient: one missed evening dose. Roundel's own schedule looked fine on paper, full, on time, nothing flagged. A. The patient had no way to push back. The text goes one way, to a phone she barely checks. Her daughter found out the next morning, when she called to ask how the visit went. R. Any reschedule that touches a medication-linked visit needs a two-way confirm from a family contact before it locks in, not a one-way text. A routine visit with nothing medical tied to it can keep moving the way it does now. D. Flag every medication-linked visit Aidewell reschedules in the last day and check it against a same-day-coverage rule before the day starts, not after a dose gets missed.
Same missing lever, a different shape. Aidewell's gap isn't a dollar ceiling, it's a text with nowhere to reply.
Same shape, different cost
At Cargast the missing check was a vendor's own order history. At Roundel it's a two-way line for exactly one kind of visit. Different fix, same real finding: GUARD always ends up pointing at the one lever the subject never got, not at the agent's overall accuracy.
Swap the trigger and it still runs.
Speed: an interviewer caps the answer at ninety seconds. Skip straight to naming the hold and who it protects.
Cost: engineering says a confirm hold slows down twenty percent of orders. Don't drop the hold to zero. Narrow it to smaller vendors or to actions a vendor can't undo, so the slowdown lands where the risk actually is, not everywhere.
The agent got better, for real: Stockhand's overall accuracy climbs to 99.9 percent. That still doesn't retire the hold, because the one case in a thousand it's wrong about is exactly the case a dollar ceiling was never built to catch.
Where people run it wrong.
They tune the model instead of asking who's standing on the other end of the action it takes.
They measure trust in a number that makes sense to the operator, dollars, overall accuracy, and never in a number that makes sense to the subject, steel, a missed dose.
They treat a person calling in to double check as the safety net, instead of building one that doesn't depend on someone happening to notice.
How to use it live. Say the reframe before any story: "the point of naming who can't push back is that the agent's accuracy was never the problem here. The problem is what happens to the fraction it gets wrong once nobody stands between the action and the person who has to live with it." That buys you room to name the actual hold, instead of promising to "improve monitoring."
Flashcards (click a card to flip it)
1 · THE FRAMEWORK
What framework fits a question about an agent's wrong action and its remediation, and what does it force you to name?
Tap to flip
ANSWER
GUARD. It forces you to name two people, the one who can pause the agent and the one who only sees the result, not just what went wrong technically.
2 · THE PEOPLE
Who are the two people this incident actually involves?
Tap to flip
ANSWER
Gideon Baird, who owns Stockhand at Cargast and can pause any order. Kojo Antwi, who runs Threadwell Fasteners and only ever sees a purchase order marked confirmed.
3 · THE DECISION THAT OPENED THE GAP
What earlier decision let a wrong order reach a vendor with nobody checking it?
Tap to flip
ANSWER
Raising Stockhand's auto-approve ceiling from $5,000 to $50,000, without ever checking a new order against that specific vendor's own history first.
4 · WHERE THE HARM LANDED
Why did this cost Threadwell so much more than it cost Cargast?
Tap to flip
ANSWER
$18,400 is small change against Cargast's roughly $40 million a year. It's $9,400 of real steel, two unpaid weekend shifts, and about $14,000 of unsellable custom parts for a 14-person shop.
5 · THE MOMENT HE COULDN'T CONTEST IT
What happened when Kojo actually tried to push back?
Tap to flip
ANSWER
He called Cargast's order line before cutting steel. The person who answered could only see "confirmed" on their screen and told him to go ahead.
6 · THE NUMBER
Fill in the blank: the stale test row said ___ units, sat unnoticed for ___ months, and turned a normal 2,400-unit reorder into a ___-unit one.
Tap to flip
ANSWER
10,000 units. Five months. 12,000 units.
7 · THE REPLAY
Same order, new design, what changes?
Tap to flip
ANSWER
A hold checks the order against Threadwell's own trailing average instead of Cargast's dollar ceiling, catches it in about twenty minutes, and Threadwell never touches the steel.
8 · CROSS PRODUCT TRANSFER
Section 4 answers this same question again for a different product. Which product, and what's the equivalent hold?
Tap to flip
ANSWER
Aidewell, a home health visit scheduler at Roundel Home Health. The equivalent hold is a two-way confirm on any reschedule touching a medication-linked visit, instead of a one-way text.
Check yourself Score: 0 / 0
True or false
1. True or false: this incident happened because Stockhand wasn't confident enough in its own recommendation.
True
False
Show hint
Ask whether the incident was ever the kind of case a confidence score would have flagged in the first place.
Show answer
False. Stockhand was confident and wrong. The number came from a stale test row it had no reason to doubt, not from a case it flagged as unsure.
Multiple choice
2. Why does Kojo's phone call matter to this answer, even though it didn't stop the order?
A. It proves Stockhand should message vendors in a friendlier tone.
B. It shows the harm was preventable if Kojo had simply trusted the system less.
C. It shows that even someone who noticed something was wrong had no way to actually check it, because the person he reached had no more visibility than he did.
D. It proves Threadwell should have a policy of never accepting large orders by phone.
Show hint
This is the GUARD "ability to contest" step, played out as a scene. Look for who had a lever, not who made a good or bad choice.
Show answer
C. Kojo did the sensible thing. The system had no lever anywhere for him to pull, not even through the one channel built for exactly this.
Fill in the blank
3. Cargast raised Stockhand's auto-approve ceiling from $___ to $___, and the wrong order came in at $___, comfortably under the new line.
Show hint
Check "Let's learn" and the story's third paragraph.
Show answer
$5,000; $50,000; $18,400. The gap between the old and new ceiling is exactly the room the wrong order had to travel through unseen.
Short answer
4. What decision does this answer take back, and why did it look reasonable when it was made?
Show hint
Look for the meeting memory, not a dial someone could just turn back down.
Show answer
Model answer: Raising the auto-approve ceiling to $50,000 without checking new orders against each vendor's own trailing average. It looked reasonable because Stockhand had been accurate for months, and the ceiling got framed as a company-wide trust question instead of a per-vendor one.
Short answer, apply it yourself
5. Think of an automated system at your own job that can take an action on someone else's behalf, not just suggest one. Who is the person on the receiving end who'd have no way to flag it if it went wrong?
Show hint
Look for something that acts and notifies, rather than something that only recommends and waits for a person to click go.
Show answer
Model answer: A scheduling tool that auto-confirms shift swaps for hourly staff. If it double-books someone by mistake, the coworker who was supposed to get the day off finds out from a text saying their shift moved back, with no one to call before the schedule locks for payroll.
Short answer, the number question
6. If Cargast had never raised the auto-approve ceiling past $5,000, would this exact order have gone straight to Threadwell the same way? Show the reasoning.
Show hint
Compare the order's dollar value against the old ceiling, not against the new one.
Show answer
No. At $18,400, the order would have landed above the old $5,000 ceiling and needed a person to look at it before it went out, the same person who could have caught the stale test row the way Gideon eventually did, just eleven days earlier.
Before you close the answer
Why this works
Tests whether you'll point a fix at the agent's overall accuracy, or at the one interaction where someone else absorbs its mistakes. Most candidates stop at "we'd retrain the model" and never ask who's standing on the other end of the action.
Follow-up traps
"Isn't a hold on big orders just going to slow down every legitimate large order too?" Response: only the ones that jump past a vendor's own trailing pattern, sized per vendor, not a flat rule, so a genuinely normal big order still moves the same day it always did.
"Why not just lower the auto-approve ceiling back to $5,000 instead of building a new check?" Response: because the next version of this mistake could just as easily show up as a $4,000 order. A dollar ceiling measures Cargast's comfort, not what a vendor can absorb, so it never actually closes the gap.
If pressed
The forecast table Stockhand reads from and the vendor order-history table live in two separate systems inside Cargast, one built for demand planning, one for procurement, and nothing had ever compared a single order against both before it went out the door.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.