ConceptAdvancedResponsible AI & Advanced Practice / Agent product management specifics / #18
How does the support model change when your product acts on the user's behalf?
ORDER the product is Halcyon Desk, Vireo Health Partners' scheduling agent at Brackenhurst Family Clinic
Interviewer's question: "How does the support model change when your product acts on the user's behalf?" Halcyon Desk texts patients to confirm and reschedule visits at Brackenhurst Family Clinic, and can move a patient to a new slot without a person clicking send. Cassia Threlkeld leads product. Averil Nightingale is a patient whose appointment moved on its own.
The direct answer
Build the plain "what did the agent just do" log first, since undo and escalation both depend on it existing. Then make every rescheduling action reversible without a phone tree, before scaling volume. Only after both of those exist should you invest in a dedicated escalate-to-human path, because a human can't actually help until they can see what happened.
Do this, in order
Build the plain action log before anything else.Why: undo and escalation both need it to exist first, so it unblocks everything that comes after.
Design every rescheduling action to be reversible without cascading into other patients' slots.Why: undo is the hardest thing to retrofit later, once one action has already bumped several other appointments.
Run a shadow month where the agent proposes but a person still sends, before committing a quarter to anything.Why: it's the cheapest way to learn where confusion actually happens before building a fix for the wrong problem.
Only build a dedicated escalate-to-human path after the log exists.Why: a person being escalated to can't actually help without first seeing what the agent did.
Track confused callback calls per week.Why: a rising number tells you the ranking was right, and it's time to build the next item on the list.
How to answer this, stage by stage
Nobody is grading whether you know support gets harder with agents. They're grading whether you can say which new support capability to build first, and why the others have to wait.
Stage 1
Scope it to one concrete action
Say it like this
"I'll answer this for Halcyon Desk, a scheduling agent at Brackenhurst Family Clinic that can move a patient's appointment without a person clicking send."
Why this works
A support-model question answered in the abstract turns into a wish list. One real action keeps the ranking honest.
Stage 2
Say your structure out loud
Say it like this
"I'll use ORDER. Outcome, reversibility, dependency, evidence, rank."
Why this works
Names a prioritization method up front, so the answer isn't three good ideas said in a random order.
Stage 3
Name what all the candidates compete to move
Say it like this
"Everything here is trying to protect the same thing: whether a patient trusts what happened to their appointment, and whether the front desk drowns in confused calls trying to explain it."
Why this works
The O step. Without a stated outcome, ranking candidates is just opinion in a list.
Stage 4
Find what's hardest to undo
Say it like this
"Undo is the hardest of the three to add later, because once a reschedule cascades into other patients' slots, reversing it isn't a database rollback, it's a round of phone calls."
Why this works
The R step. Says which decision has to be designed in from day one instead of bolted on after volume scales.
Stage 5
Say what depends on what
Say it like this
"You can't build a good escalate-to-human path, or a working undo, until you have a plain log of what the agent did. Both of the others actually depend on that existing first."
Why this works
The D step. Shows the order isn't a personal preference, it's forced by what unblocks what.
Stage 6
Say what's cheap to learn first
Say it like this
"I'd run a shadow month where the agent proposes the reschedule and a person still clicks send, so we learn where confusion actually happens before we build a fix for the wrong problem."
Why this works
The E step. Shows you'd rather learn cheaply than commit a whole quarter to a guess.
Stage 7
Give the rank and close
Say it like this
"So: the log first, then reversible actions, then a dedicated escalation path. Build them in the order things break, not the order they're fun to build."
Why this works
The R step, restated as the direct answer, in the exact order a reader could act on.
Let's learn
Halcyon Desk texts patients to confirm or reschedule a visit, and it can move a patient to a new slot without a person clicking send first.
Before Halcyon Desk, Brackenhurst's front desk staff spent about two hours a day on the phone rebooking missed and changed appointments.
Knowledge spark: why does an acting agent need a different support model at all?
When a tool only suggests something, a person is right there to catch a mistake before it happens. When a tool acts on its own, the mistake, if there is one, already happened by the time anyone notices. Support has to work backward from an action instead of forward from a suggestion.
Now the agent handles most rescheduling by text in seconds, and a person only steps in for the cases it flags as unusual.
Here is the turn: it isn't that Halcyon Desk sometimes reschedules to a bad time. It's that when it does, there is no fast way for the patient or the front desk to see what happened or why, so even a small mistake turns into a full phone call anyway, at a worse moment than the one it was supposed to prevent.
Confused callback calls per week, before and after the action log shipped
The agent's actual reschedule mistakes barely changed. What changed is that most patients could now see the reason without calling anyone.
At its worst: a patient shows up for an appointment the agent quietly moved, or believes it moved when it didn't, the doctor's slot sits empty, and nobody at the front desk can explain what happened without escalating to the vendor's own support line.
The right side isn't a design flaw so much as a design nobody built the left side to prevent.
The decision I would take back
Brackenhurst's team scoped the autonomous rescheduling action to ship before any visible log of what the agent had done, because the log felt like polish, something to add once the core feature worked. It shipped in the order that was fun to build, not the order that would actually break first.
What I would leave alone: routine appointment reminders, the ones with no rescheduling involved, don't need any of this. Nobody needs an undo button or an escalation path for a text that just says "see you Tuesday at 2."
The lesson: a support model for an acting agent has to be planned in the order things actually break, not the order that felt satisfying to build first.
Now here is the same thing as a story
The short version above is what you'd say defending the build order to the clinic's operations team. Read this one for how the gap actually got found.
Averil Nightingale had been a patient at Brackenhurst for six years, and she'd learned to trust the clinic's texts the way she trusted its waiting room chairs: reliably, without thinking about it.
Halcyon Desk had been live for two months when her Tuesday 2pm checkup got moved to Thursday 9am. She never saw the text, because it arrived while she was in a work meeting with her phone on silent, and by the time she checked it that evening, she'd already half-forgotten there'd been an appointment at all that week.
The first box is not the most exciting one to build. It's the one the other two can't exist without.
She showed up Tuesday at 2pm anyway, out of habit more than certainty, and found the front desk confused too. The system showed her moved to Thursday. Nobody at the desk could say why, because there was nothing to look at beyond that single line. Not what triggered the move. Not whether another patient had been slotted into her old time. Nothing.
A new front desk hire, two weeks into the job, asked the question that stuck: "Can't we just put her back in the two o'clock, since it's obviously still open?" It wasn't obviously open. Halcyon Desk had already offered that slot to someone else the moment it moved Averil, and that patient had already confirmed it.
The appointment wasn't lost. It was cascaded, and cascaded things don't undo with one click, they undo with a phone call to someone who has no idea they're about to get one.
Cassia Threlkeld's team had built the rescheduling action first, because it was the obvious value: fewer manual calls, faster fills on empty slots. The log showing why an action happened, and a clean way to undo one, sat lower on the roadmap, filed under "nice to have once the core thing works."
Four fields, and none of them existed the week Averil showed up on the wrong day.
The decision I would take back is exactly that ordering. Building the reschedule action before the log made sense in a demo, where nothing ever goes wrong on purpose. It stopped making sense the first Tuesday a real cascade meant nobody, not Averil, not the new hire, not Cassia's own team, could say what had actually happened without pulling raw database records.
Replayed with the log built first: the new hire pulls up Averil's entry in ten seconds. It shows the move, the reason, the other patient now in that slot, and a real undo option that offers both patients a fair alternative instead of just reversing blindly. The call takes ninety seconds instead of turning into an afternoon of confusion and an empty exam room.
I thought building the thing that acts was the hard part, and the log was bookkeeping to add once things settled down. It took one new hire's plain question to see that the log wasn't bookkeeping. It was the only thing that would ever let anyone answer her question at all.
ORDER, spelled out for one appointmentNot a roadmap debate. ORDER is what tells you which support capability breaks first if it isn't built yet.
O
Outcome. What every candidate competes to move.
Whether a patient trusts what happened to their own appointment, and whether the front desk drowns in confused calls explaining it.
Without a named outcome, ranking three good ideas is just opinion dressed as a plan.
R
Reversibility. What's hardest to undo.
Undo, because once a reschedule cascades into another patient's slot, reversing it isn't a database rollback, it's a round of phone calls to someone who didn't ask for one.
The hardest step. Flips don't flip back for free, so this is what forces the real ranking.
D
Dependency. What unblocks what.
Undo and escalate-to-human both need the plain action log to exist first, since neither a patient nor a person taking a call can act on something they can't see.
Shows the order is forced by reality, not by which feature sounded more exciting to build.
E
Evidence. What's cheap to learn first.
Run a shadow month where the agent proposes a reschedule and a person still sends it, to see where confusion actually happens before building a fix for a guessed problem.
Buys real information before committing a whole quarter to the wrong first build.
R
Rank. State the order and defend it.
The action log first, reversible actions second, a dedicated escalate-to-human path third, because each one depends on the one before it.
This is the direct answer, stated as an order a team could actually build against.
Month one looks like the least visible work on the roadmap. It's the one everything else depends on.
The fourth item is how you know if the first three are actually working.
Front-desk phone minutes per day, across the four-month build
The log alone barely moved the phone. Reversibility is what actually let staff stop calling people back to untangle a cascade by hand.
The recap, one line per letter: outcome is patient trust and front-desk time, reversibility is undo being hardest to retrofit, dependency is the log unblocking both undo and escalation, evidence is a shadow month before committing a quarter, and rank is log first, reversibility second, escalation third.
And if you want to be sure it really works, try it somewhere elseSame five letters, a personal finance agent instead of a clinic. A different action, the same shape of ranking.
Larkmoor Money runs an agent that moves money between a user's accounts to avoid overdraft fees, transferring funds automatically when a balance dips below a set line.
Mapped onto ORDER: outcome is whether a user trusts what happened to their own money, and whether support gets flooded with "why is my balance different" calls. Reversibility: undo is again the hardest, since a transfer that already cleared and triggered a second automatic transfer elsewhere can't be casually reversed the way a suggestion could have been ignored. Dependency: a clear transaction-level log explaining why each transfer happened has to exist before either undo or a real escalation path can work, since a support agent can't reverse or explain a transfer they can't see the reasoning behind. Evidence: run a shadow period where the agent recommends a transfer and requires one tap to confirm, to learn how often users would have said no, before committing to fully automatic transfers. Rank: the log first, then a same-day reversal window sized to how fast a second linked transfer can be caught, then a dedicated escalation path for cases the log alone can't resolve.
Swap the trigger and it still runs.
Speed: an interviewer caps you at thirty seconds. Say "log first, since undo and escalation both depend on it, then reversible actions, then a dedicated human path," and stop.
Cost: if building a full log this quarter is too expensive, ship a minimal version, just what changed and why, before the flashier reversible-action work, since even a thin log unblocks the ranking that follows.
The model gets better, for real: even if the agent's rescheduling accuracy improves to near-perfect, the log and undo still matter, because the rare mistake that does happen is exactly the one nobody could previously explain.
Where people run it wrong.
They build the acting feature first and treat visibility into what it did as a polish item for later.
They build a dedicated human escalation path before a log exists, so the human being escalated to still can't see what happened.
They measure success by how rarely the agent is wrong, instead of how fast someone can understand and reverse it the one time it is.
How to use it live. When someone asks how support changes for an acting product, ask yourself one thing out loud: if this action turns out to be wrong, can anyone, patient or staff, see what happened without calling a vendor. If not, that's the thing to build first, not last.
Flashcards (tap any card to flip it)
1 · THE FRAMEWORK
What framework fits a question about how support changes for an acting agent?
Tap to flip
ANSWER
ORDER: outcome, reversibility, dependency, evidence, rank. Built for prioritization questions with several real candidates competing for the same slot.
2 · THE PEOPLE
Who are the two people this answer centers?
Tap to flip
ANSWER
Cassia Threlkeld, who leads product for Halcyon Desk, and Averil Nightingale, a six-year Brackenhurst patient whose appointment moved without her seeing it happen.
3 · THE OUTCOME
What are all three candidate capabilities actually competing to protect?
Tap to flip
ANSWER
Whether a patient trusts what happened to their own appointment, and whether the front desk gets flooded with confused calls explaining it.
4 · THE DEPENDENCY
Why does the action log have to come before undo and escalation, not after?
Tap to flip
ANSWER
Neither a patient trying to undo something nor a staffer escalating a case can act on an action they can't see the reasoning behind.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Shipping the autonomous rescheduling action before any visible log of what it did, because the log felt like polish instead of the foundation.
6 · THE NUMBER
Fill in the blank: confused callback calls dropped from 41 a week to ___ a week after the action log shipped.
Tap to flip
ANSWER
12. The agent's actual mistakes barely changed. What changed is that most patients could see the reason without calling anyone.
7 · THE REPLAY
Same cascaded appointment, log built first. What changes?
Tap to flip
ANSWER
The new hire pulls up the entry in ten seconds, sees the move, the reason, and the other patient now in that slot, and resolves it in ninety seconds instead of a confused afternoon.
8 · CROSS PRODUCT TRANSFER
Section 4 answers this again for a different product. Which product, and what's the parallel ranking?
Tap to flip
ANSWER
Larkmoor Money's overdraft-avoidance transfer agent. Same order: transaction log first, then a same-day reversal window, then a dedicated escalation path.
Check yourself Score: 0 / 0
Multiple choice
1. Why should the action log be built before a dedicated escalate-to-human path, rather than after?
A. The log is technically simpler to build, so it should always go first regardless of the product.
B. A person being escalated to can't actually resolve a case without first seeing what the agent did and why.
C. Patients specifically requested the log before anything else.
D. Escalation paths are more expensive to maintain than logs.
Show hint
Look at the dependency step.
Show answer
B. Escalation and undo both need to see what happened before they can do anything about it. The log is what makes either of them possible at all.
True or false
2. True or false: Halcyon Desk's actual reschedule mistake rate changed significantly once the action log shipped.
True
False
Show hint
Look at the chart note under the callback-calls bar chart.
Show answer
False. The agent's mistakes barely changed. What changed is that patients and staff could now see the reason without escalating to a phone call.
Fill in the blank
3. Fill in the blank: before Halcyon Desk, front desk staff spent about ___ hours a day on the phone rebooking.
Show hint
Look at the opening of "Let's learn."
Show answer
Two. That's the time the agent was built to save, and the reason speed felt like the obvious thing to build first.
Short answer, where it wouldn't matter
4. Name a message type in this story where none of this support redesign is needed.
Show hint
Look at "what I would leave alone."
Show answer
Model answer: A routine appointment reminder with no rescheduling involved. There's nothing to undo or escalate about a text that just confirms an existing time.
Short answer, apply it yourself
5. Pick a product that's taken an action on your behalf without asking first. Could you have easily found out why it did that?
Show hint
Think of an app that auto-renewed something, or auto-sorted your calendar or inbox.
Show answer
Model answer: Many people can recall a calendar or email tool moving or filing something automatically with no visible reason shown, leaving them to guess rather than check.
Short answer, name the reversal
6. What old decision does this answer take back, and why did it make sense when it was made?
Show hint
Look at "the decision I would take back."
Show answer
Model answer: Shipping the rescheduling action before the action log. It made sense because the log felt like polish on top of a working feature, and stopped making sense the first time a real cascade left nobody able to explain what happened.
Before you close the answer
Why this works
Tests whether you can rank support capabilities by what unblocks what, rather than listing three good ideas in whatever order occurs to you. Most candidates name undo, a log, and escalation, but few say which has to come first and why.
Follow-up traps
"Isn't undo the most important thing, since it fixes the actual mistake?" Response: undo depends on the log existing first. Building undo before the log means building a button nobody can safely decide when to press.
"Why not just add more front desk staff to handle the confusion?" Response: that treats the symptom, not the cause. More staff answering "why did this move" calls doesn't reduce how often that question gets asked, it just answers it more expensively.
If pressed
The reversible-action design specifically limits automatic rescheduling to slots that haven't yet been offered to or confirmed by another patient, so undo never has to chase a cascade more than one step deep, which is what keeps a one-tap reversal actually possible instead of a multi-call untangling.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.