Describe a workflow where partial automation is worse than none.

Procurely builds OrderGate, a purchase order approval tool. It added a "Pre-Checked" badge that confirms a PO's format and vendor ID are filled in correctly before it reaches a human approver. Tavish Bellhurst is the procurement manager who signs off on POs. Ondine Marchetti is the PM who has to work out why a fraudulent-looking $42,000 PO made it all the way through.

How to answer this, stage by stage

Nobody is scoring whether you can define partial automation. They're scoring whether you'd have caught a green checkmark quietly meaning less than everyone assumed.

Let's learn

OrderGate is a tool that reviews purchase orders before a human approver sees them, catching obvious formatting problems so approvers spend their time on judgment calls instead of typos.

Nothing about the launch week predicted week eight. That gap is exactly what makes this failure mode dangerous.

Before the badge, Tavish reviewed every PO the same way, checking format, budget limits, and vendor legitimacy together, every time. It took about 14 minutes per PO on average, and manual review caught roughly 92 percent of problematic POs before approval.

After the badge shipped, the average review time fell to 4 minutes, a real, immediate, celebrated improvement. Nobody was watching the one number that actually mattered: how often a genuinely problematic PO still got caught.

Here's the turn: the pre-check's own accuracy was never the problem, it correctly validated format and vendor ID almost every time. The turn is that "Pre-Checked" read as a broader claim than it was ever built to make, and Tavish, busier that quarter than usual, took the shortcut the badge seemed to offer.

At its worst, this cost a real, near-miss loss: a $42,000 PO to a vendor with a barely-plausible name and a properly formatted request got the "Pre-Checked" badge, sailed through Tavish's queue in under a minute, and was only caught in a later audit, not by the workflow that was supposed to catch it.

What I would leave alone: the underlying format and vendor ID check stays exactly as it is, it's fast, reliable, and genuinely saves real minutes. The fix was never removing the automation, only naming what it does and doesn't cover, every time it appears.

The lesson: a badge that automates part of a job and stays silent about the rest doesn't split the work cleanly, it invites a human to quietly assume the rest got handled too. Name the edges of what got automated, out loud, every time.

Now here is the same thing as a story

The short version above is what you'd say out loud in the room. Read this one for what it actually felt like to watch a fast queue turn into a quiet blind spot.

Tavish Bellhurst had approved purchase orders at his company for five years. He was good at it in the unglamorous way: methodical, a little slow, and almost never wrong. Every PO got the same three checks, format, budget, vendor, whether it was PO number 4 or PO number 400 that week.

OrderGate's pre-check badge launched on a Tuesday. The first time Tavish saw a green "Pre-Checked ✓" sitting at the top of a PO, he still ran his usual three checks anyway, out of habit. It took him 13 minutes, barely faster than before.

Two of four things Tavish used to check, quietly narrowed to two, without anyone ever saying so out loud.

By week three, his queue had nearly tripled, a headcount gap nobody had backfilled yet. The badge started doing something nobody designed it to do: it became permission. A "Pre-Checked" PO felt safe enough to move through faster, and faster started meaning skipping the budget and vendor steps entirely, just this once, just for the ones with the green check.

There was no single bad Tuesday where Tavish decided to stop checking. It thinned out in small moves: skimming past the badge instead of reading it, trusting it a little more each week his queue grew a little longer.

Three real candidates for what caused the drift. Ondine's evidence test pointed hardest at the first one.

The PO that surfaced it was for $42,000, properly formatted, correct cost center code, vendor name close enough to a real supplier's that nobody glanced twice. It carried the green badge. It took Tavish under a minute to approve.

Ondine Marchetti, brought in to work out what happened, didn't start by blaming Tavish. She started by asking three suspects: was it the badge's wording, his climbing workload, or a training gap. She ran a quick test, seeding ten known-bad POs, budget overruns and shaky vendors, all correctly formatted, into a review batch, and watched what got caught.

Nine of the ten sailed through with the green badge attached. When she asked Tavish afterward what "Pre-Checked" meant to him, he said, without hesitating, "That it's good to go." Nobody had ever told him otherwise, in writing, once.

Full manual review never asked Tavish to remember which mode he was in. Partial automation asked him to remember it every single time, and that memory wore down under real workload.

Back when the badge shipped, skipping an explicit scope statement wasn't an unreasonable call. The team was focused on shipping something useful fast, and "obviously it just means format" felt self-evident in the room. It stopped being self-evident the moment a real, busy human needed a shortcut and the badge quietly offered him one.

Here's the replay: same badge, same launch week, but with its scope named plainly from day one.

The fork that should have run before the badge ever shipped, drawn out plainly.

With the badge relabeled "Format verified, budget and vendor still yours," Tavish's review time settles around 9 minutes, slower than the celebrated 4, but the seeded-error test now catches 9 of 10 known-bad POs instead of 1. The workflow gets a little slower and a lot safer, on purpose.

One version of this story spends eight quiet weeks looking like a win before a $42,000 near-miss proves it wasn't. The other spends those same eight weeks a little less impressive on a dashboard, and a lot more honest about what actually got checked.

What I'd tell myself, watching Ondine's seeded test come back nine wrong: a badge that doesn't say what it means will always get read as meaning more than it does. Say the edges out loud, or don't ship the badge.

TRACE, run on a checkmark that meant less than everyone assumedNot a script for blaming the human who trusted the badge. TRACE is what finds the real gap between what shipped and what actually broke, weeks apart.

The recap, one line per letter: timeline is the real gap between the launch and the eight-week-later drop, recut is why partial specifically, not automation broadly, causes this, assume nothing is the trap of reading a faster review time as a safer one, cause candidates is the honest shortlist of what might explain the drift, and evidence test is the seeded-PO check that confirmed which one actually did.

And if you want to be sure it really works, try it somewhere elseSame five letters, an insurance claims queue instead of purchase orders. The judgment call this time is whether a fraud score without a label reads as a full decision.

Delano Whitfield adjusts claims at Stonecrest Mutual. A new AI tool flags each claim with a fraud-risk score, "Low Risk," shown at the top of every file. It only ever scores documentation consistency, whether dates and numbers line up, never whether the claimed damage itself is medically or physically plausible. Mapped onto TRACE: timeline is that claim processing time dropped the week the score launched, and the real over-payment rate on implausible claims didn't visibly rise until a quarterly audit two months later. Recut is the same mechanism: adjusters started reading "Low Risk" as a broader signal than documentation consistency, the one thing it actually measured. Assume nothing means not trusting the faster processing time as proof nothing was slipping through. Cause candidates are the label's own vague wording, a real backlog that quarter, and no training on the score's narrow scope. Evidence test is seeding ten claims with medically implausible damage but perfectly consistent paperwork into the queue, seven got a "Low Risk" score and sailed through Delano's review in under two minutes each.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Skip straight to "name the scope of a partial automation step, every time it appears," and stop.
Cost: no time to run a seeded-error test before the meeting. Say so honestly, and propose it as the next concrete step instead of guessing at the cause.
The model got better, for real: say the pre-check's format accuracy climbs to 99.9 percent. The problem doesn't shrink, because the gap was never about the model's accuracy at its own job, it was about what a human silently assumed that job covered.

Where people run it wrong.
They measure the automated step's own accuracy and call the whole workflow safe, without checking the end-to-end catch rate.
They ship a badge or a score with no stated scope, assuming users will naturally infer the narrow thing it actually means.
They treat rising throughput as proof of a healthy workflow, when it can just as easily be proof that real checks quietly stopped happening.

How to use it live. The moment an interviewer describes a workflow where AI handles "part" of a decision, ask yourself first: does the human downstream know exactly which part, every time, or are they filling that gap with an assumption. That question buys real thinking time, and it's usually exactly where the risk is hiding.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0