CaseIntermediateDesigning for Uncertainty & Trust / Designing for failure and graceful degradation / #2

Design the fallback experience for an AI feature when the provider is down.

SPARK design the anchor for the minute the cloud connection dies

Brindle Metalworks runs a line of hydraulic presses that stamp brackets for truck chassis. ClearLine Vision is a cloud-based defect-detection system its inspectors run from a wall-mounted terminal beside the press. Deshawn Okafor has inspected parts on this line for eight years. Here is what happened the day ClearLine's cloud provider went down mid-shift, and what the terminal should have done instead.

The direct answer
The terminal has to say, within seconds, that the model has stopped answering, since when, and what manual step replaces it right now, a fixed statistical sampling rate the line already knows how to run. It never freezes on a blank or stale reading, and it never lets the press keep running on the assumption that "no answer" means "no defect."
Do this, in order
  1. Detect the outage within seconds and say so on screen, plainly, with a timestamp.Why: a frozen or stale reading looks identical to a healthy one unless the screen states otherwise.
  2. Switch the terminal to a fixed manual sampling rate the moment the outage is confirmed.Why: inspectors already know how to run a statistical sample by hand, so the fallback costs nothing to train.
  3. Never let the press treat "no reading" as "no defect."Why: parts keep stamping whether or not anyone is watching them, and an empty answer is not a pass.
  4. Log every part made during the outage window as unverified by the model, not as passed.Why: an unverified batch and a passed batch need different treatment the moment a defect turns up later.
  5. Resume automatic checking on its own once the provider recovers, without waiting on anyone to notice.Why: an outage that requires a person to remember to flip a switch back stays in manual mode far longer than it needs to.
  6. Leave the fast automatic path alone for the 99 percent of shifts where the connection never drops.Why: this whole design only has to earn its keep during the rare outage, not slow down every ordinary day.

How to answer this, stage by stage

Nobody is grading whether you know what an API outage is. They're grading whether steel keeps moving safely while it's down.

Stage 1
Scope it to one terminal, one press line
Say it like this
"I'll design this for ClearLine Vision, the defect-detection system Brindle Metalworks runs on its stamping line, for the exact minute its cloud provider stops responding."
Why this works
Turns "design a fallback experience" into one screen, in one factory, instead of a general essay on outages.
Stage 2
Say your structure out loud
Say it like this
"I'll use SPARK. Situation, how inspection worked before this tool. Payoff, the habit I want the fallback to build. Anchor, the actual screen. Risk, what breaks it. Keep out, what stays for later."
Why this works
Shows a repeatable method for a fallback design, not an ad hoc list of "what ifs."
Stage 3
Ground the anchor in what the line did before this tool
Say it like this
"Before ClearLine, an inspector sampled one part in ten by hand, with calipers and a printed checklist, catching most defect patterns but missing anything that only showed up on the parts nobody pulled."
Why this works
Proves the fallback isn't invented from nothing, it's a real process the line already knows.
Stage 4
Give the anchor, the one decision
Say it like this
"Detect the outage within seconds, say so on screen with a timestamp, and switch straight into the one-in-ten manual sampling protocol until the provider is confirmed back."
Why this works
This is the direct answer, stated as an actual design decision, not a promise to "handle errors gracefully."
Stage 5
Prove the anchor survives its own risk
Say it like this
"The first time an outage runs long, say forty minutes on an overnight shift with nobody senior on the floor, the terminal is already in manual mode and every part made during that window is tagged unverified, not passed."
Why this works
Answers the real question underneath: what happens if this outage is the bad kind, long and unattended.
Stage 6
Say what you'd measure afterward
Say it like this
"I'd track outage minutes per month, and separately, how many parts got shipped tagged unverified. Those two numbers together tell you whether the fallback is actually catching what it's supposed to."
Why this works
Shows you're thinking past the redesign itself, toward whether it's actually working in production.
Stage 7
Close on the one line
Say it like this
"A dead connection is not a passing grade. The terminal has to say what stopped, since when, and what to do by hand right now, every single time."
Why this works
Restates the direct answer in one breath, ready for a follow-up push.

Let's learn

Every part that came off Brindle Metalworks' line 3 press used to get checked one in ten, by hand, with calipers and a printed checklist, a slow but genuinely careful habit.

ClearLine Vision replaced that with a camera above the press feeding every single part into a cloud vision model, giving Deshawn a pass or fail reading in under two seconds, no sampling needed.

Hand sketched flow diagram titled Today, without ClearLine Vision. Five boxes: Press stamps part, Sample 1 in 10 highlighted, Check with calipers, Log pass or fail, Ship the rest.
ClearLine replaced the second and third boxes with an instant camera reading. It never replaced what the line should do if that reading just stops.

Here's the turn: the everyday speed was never the issue. The real problem showed up the one time ClearLine's cloud provider went down mid-shift, because the terminal had no idea what to do next. It froze on its last reading, gave no warning, and the press kept stamping the whole time, since nobody had ever told it that "no answer from the model" should mean anything at all.

Parts produced during a 40-minute outage, by inspection status
400 200 0 340 shipped unchecked Old design 34 sampled, 306 held for review New design
Same outage, same 340 parts. The old design shipped every one of them as if it had passed. The new design ships nothing without a status attached.

At its worst, a batch of parts with a hairline stamping crack ships to a customer, because the one reading the line had for those forty minutes was a stale "pass" from before the connection dropped.

Hand sketched comparison titled The day the API went dark. Left, a grey box icon labeled Old screen, caption frozen no explanation. Right, a blue gauge icon labeled New screen, caption names the gap manual mode.
Same outage, same forty minutes. Only what the terminal was willing to admit changed.
The decision I would take back When ClearLine replaced the manual sampling habit, the team folded camera capture and the pass or fail judgment into a single automated step, with no pause built in between them. That made sense while the cloud connection was reliable enough that a pause never seemed worth the cost. It stopped making sense the day the connection dropped and there was no seam left in the process for a person to step back into.

What I would leave alone: the automatic path itself doesn't need a rebuild. On the ordinary day, with the connection healthy, ClearLine's instant reading is exactly right, and nothing about this fallback design should slow that down.

The lesson: a fallback isn't something you bolt onto a finished system. It's a seam you have to leave in on purpose, or the system won't have anywhere to hand the job back to a person when it needs to.

Now here is the same thing as a story

The short version above is what you'd say defending this fallback design to Brindle's plant manager. Read this one for how quietly the risk built up.

The terminal is bolted to the wall two feet from the press, close enough that Deshawn can glance at it without stepping back from the line. For his first several years, that glance meant pulling a part, running calipers over it, checking a printed sheet.

ClearLine changed the glance into something else entirely: a green check or a red flag, updating every two seconds as parts rolled past the camera. Deshawn stopped pulling parts by hand within a couple of weeks. There was no reason to, the readings kept matching what his own eye would have caught anyway.

Knowledge spark: why would a cloud vision model just go dark? Most factory-floor AI tools like ClearLine don't run the model on the terminal itself. The camera sends each image out to a cloud provider, which runs the model and sends a reading back, usually in under a second. If that provider's servers go down, get overloaded, or the plant's own connection drops, every terminal relying on it goes quiet at the same time, with no warning built in unless someone designs one.

On an overnight shift, with the plant's newest network switch installed the week before, ClearLine's connection to its provider dropped at 2:12pm. The terminal's last reading, a green check from 2:11, stayed frozen on screen. Nothing told Deshawn it was stale. The press kept stamping.

Hand sketched timeline titled The forty minutes the line ran blind. Five milestones: API stops responding 2 12pm, Terminal freezes blank 2 13pm highlighted, Parts keep shipping 2 14 to 2 52, Defect batch flagged next morning, Recall traced back 2 days later.
Forty minutes between a dead connection and anyone noticing. The frozen green check looked exactly like a healthy one the entire time.
The screen did not say the model had failed. It just kept repeating the last true thing it had ever said.

By the time the connection came back at 2:52pm, 340 parts had shipped with no real inspection at all, everyone on shift assuming the frozen green check meant the line was clean. A hairline crack in one batch, the kind the vision model reliably caught but a rushed glance wouldn't, wasn't flagged until a customer's own incoming inspection caught it two days later.

Hand sketched labeled parts diagram titled The anchor, close up. A document icon at the center labeled Fallback screen, with four callouts around it: what stopped, since when, sample manually now, resume when back.
The old screen showed none of these four things during the outage. It just kept showing the last answer it ever got.

With the redesigned terminal, the moment ClearLine loses its connection, the screen switches within seconds to: "Model offline since 2:12pm. Sampling 1 in 10 by hand until connection returns. 34 parts checked, 306 held for review." Run the same afternoon forward: Deshawn is back on manual sampling by 2:13pm, and every uninspected part is tagged, not shipped blind.

The old terminal asked the line to trust a memory of a healthy reading. The new one tells the truth about the exact minute that memory stopped being current.

We built the fallback screen to be simple, on purpose, so it wouldn't distract from the automatic path doing most of the work. It took a two-day-late recall to see that simple and silent had quietly become the same thing.

SPARK, mapped to a dead connectionNot a lecture on redundant servers. SPARK is what forces the design to survive the minute the provider actually goes down.

S
Situation. How inspection worked before this tool.
An inspector sampled one part in ten by hand, with calipers and a printed checklist, a slower but genuinely careful process.
Grounds the fallback in a real, known process, not something invented under pressure.
P
Payoff. The habit this should build.
Treating a frozen or missing reading as a signal to act, not as a passing grade to trust.
Names the actual behavior change the fallback screen needs to produce.
A
Anchor. The one decision everything hangs on.
Detect the outage in seconds, name it with a timestamp, and switch straight into the known one-in-ten manual sampling protocol until the connection returns.
The hardest step and the direct answer: a concrete screen state, not a vague promise of resilience.
R
Risk. What breaks the first time it's wrong.
A long, unattended outage overnight. The anchor survives it because every part made during that window gets tagged unverified, whether or not anyone on shift is watching closely.
Proves the anchor was built against its worst case, not just its ordinary one.
K
Keep out. What we won't build, day one.
No offline model running on the terminal itself, no automatic pause of the physical press. Both are real ideas for later, neither is required to stop the silent failure.
Shows judgment about scope instead of trying to solve every related problem at once.
Hand sketched decision tree titled What the terminal decides. Root: is ClearLine responding. Four branches: yes normal speed leads to automatic check, slow still up leads to queue and retry, down under 2 minutes leads to hold last mode, down over 2 minutes leads to manual sampling.
Only the bottom branch is new. Everything above it was already how the terminal behaved.

The recap, one line per letter: situation is the one-in-ten manual sample, payoff is treating a missing reading as a signal instead of a pass, anchor is the timestamped fallback screen and its manual sampling switch, risk is the long unattended outage, and keep out is holding back an offline model and an automatic press pause.

Hand sketched icon list titled What we left for later. Three items: a box icon labeled No offline model on the terminal, a gauge icon labeled No auto pause of the press line, a document icon labeled No change to the normal fast path.
Each of these is a genuine next step. None of them is required to stop parts from shipping blind.

And if you want to be sure it really works, try it somewhere elseSame five letters, a customer support translation tool instead of a stamping line. A different anchor, aimed at a different silence.

Correlink Freight runs a support desk where agents use an AI translation feature to chat live with drivers and customers in a dozen languages. Marisol Petrov handles overnight escalations. Mapped onto SPARK: situation is an agent today without the tool, reaching for a printed phrasebook and a short list of bilingual colleagues to call; payoff is the habit to build, treating a translation tool that's gone quiet as a reason to reach for that same phrasebook immediately, not a reason to keep typing into a chat box that isn't replying.

The anchor here aims at a different silence: when the translation provider stops responding, the chat window doesn't just stop updating, it shows "Translation unavailable since 11:04pm. Message saved untranslated. Try phrasebook shortcuts, or escalate to a bilingual teammate." The risk the team designed against was an agent typing several messages into a chat that looked normal, unaware none of it had reached the other person in a language they could read, while a delivery delay dispute sat unresolved for hours.

Hand sketched labeled parts diagram titled The anchor, close up, reused here for the translation support tool. Center document icon labeled Fallback screen, with what stopped, since when, sample manually now, and resume when back around it, relabeled for a translation outage.
Swap "sample manually" for "use phrasebook shortcuts," and the same four parts still do the same job.
Minutes an unresolved chat sat silent, with and without the outage banner
60m 30 0 No banner, 52m With banner, 3m
Same outages, same agents. Naming the silence, instead of leaving the chat window looking normal, is what closed the gap.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "detect the outage fast, name it on screen, switch to the known manual fallback," and stop.
Cost: there's no budget this quarter for a backup inference provider. Say so honestly, and start with detection and the manual fallback banner, since those cost almost nothing and catch most of the risk on their own.
The model gets better, for real: if ClearLine's underlying accuracy improves and outages become rarer, that's still not a reason to drop the fallback screen, a rarer outage is exactly the kind everyone forgets how to handle by hand.

Where people run it wrong.
They design the fallback for the outage they can picture, a few seconds, and never test the one that runs for an entire shift.
They let a stale reading sit on screen looking healthy, instead of expiring it the moment it's no longer current.
They treat "the physical process kept running" as fine, when the whole point of the tool was that the process needed watching.

How to use it live. When someone asks you to design a fallback for when the provider is down, ask yourself one question first: what does the physical world keep doing while the software goes quiet? Design the fallback around that, not around the API call.

Flashcards (tap any card to flip it)

1 · THE FRAMEWORK
What framework fits a "design the fallback for when the provider is down" question?
Tap to flip
ANSWER
SPARK: situation, payoff, anchor, risk, keep out. Ground the anchor in a process that already exists, then prove it survives a long outage.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Deshawn Okafor, a quality inspector at Brindle Metalworks for eight years, who used to sample one part in ten by hand.
3 · THE SITUATION
How did inspection work before ClearLine Vision?
Tap to flip
ANSWER
An inspector pulled one part in ten, checked it with calipers against a printed sheet, and logged it by hand.
4 · THE ANCHOR
What's the one design decision this answer hangs on?
Tap to flip
ANSWER
Detect the outage within seconds, name it with a timestamp, and switch straight into the known one-in-ten manual sampling protocol until the connection returns.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Folding camera capture and the pass or fail judgment into a single automated step with no pause left for a person to step back into.
6 · THE NUMBER
Fill in the blank: during the 40-minute outage, ___ parts shipped with no real inspection at all under the old design.
Tap to flip
ANSWER
340 parts. All of them riding on a frozen green check from 2:11pm that nothing on screen ever flagged as stale.
7 · THE REPLAY
Same outage, redesigned terminal. What changes?
Tap to flip
ANSWER
Deshawn is back on manual sampling by 2:13pm, and the 306 parts made during the outage are tagged unverified and held for review instead of shipped as passed.
8 · CROSS PRODUCT TRANSFER
Section 4 answers this again for a different product. Which product, and what's the anchor there?
Tap to flip
ANSWER
Correlink Freight's support chat translation tool. The anchor shows an outage banner naming the silence and pointing agents to phrasebook shortcuts, instead of a chat window that looks normal but isn't reaching anyone.

Check yourself Score: 0 / 0

True or false
1. True or false: under the redesigned terminal, the press stops physically stamping the moment ClearLine's connection drops.
  • True
  • False
Show hint
Look at the Keep out step.
Show answer
False. Auto-pausing the physical press is explicitly held back for later. The fallback switches inspection to a manual sample, it doesn't stop production.
Multiple choice
2. Why did the old terminal's screen keep showing a green check during the outage instead of an error?
  • A. The camera itself had also stopped working.
  • B. The screen kept displaying its last real reading with nothing telling anyone that reading had gone stale.
  • C. ClearLine's provider sent a false "all clear" signal.
  • D. Deshawn manually cleared the error himself.
Show hint
Look at the highlight line in Section 2.
Show answer
B. There was no expiry on the last reading, so a stale answer looked exactly like a fresh, healthy one.
Fill in the blank
3. Fill in the blank: the outage started at 2:12pm, and under the redesigned terminal Deshawn would be back on manual sampling by about ___.
Show hint
Look at "the replay" flashcard or Section 2's redesigned paragraph.
Show answer
2:13pm. Detection and the switch to manual sampling both happen within about a minute of the outage starting.
Short answer, name the reversal
4. What old design decision does this answer take back, and why did it make sense when it was made?
Show hint
Look at "the decision I would take back."
Show answer
Model answer: Merging camera capture and the pass or fail judgment into one automated step with no pause. It made sense while the connection was reliable enough that a manual seam never seemed worth keeping.
Short answer, apply it yourself
5. Think of a tool you use that depends on an internet connection. What happens on your screen the moment it goes offline, and does it tell you clearly, or just go quiet?
Show hint
Think about whether the app shows a clear "offline" state or just stops updating with no explanation.
Show answer
Model answer: Many apps just freeze or spin silently, the same failure this answer's anchor is built to prevent, a stale state that looks healthy.
Short answer, where it wouldn't matter
6. Name a part of Brindle's process where this fallback design genuinely doesn't need to apply.
Show hint
Look at "what I would leave alone."
Show answer
Model answer: The ordinary automatic path on a healthy connection. It already works, and nothing about the outage fallback should slow it down.
Before you close the answer
Why this works
Tests whether you design for what the physical world keeps doing during an outage, not just for the software error itself.
Follow-up traps
"Why not just pause the press automatically the moment the connection drops?" Response: that's a bigger, riskier change held back on purpose. Stopping a hydraulic press mid-cycle has its own safety costs, and the fallback catches the actual harm, unverified parts shipping, without needing to touch the machinery.

"What if the outage is so brief nobody even notices the banner?" Response: fine, then almost nothing switches, since detection and reversion both happen automatically within seconds either way.
If pressed
ClearLine's redesigned terminal pings its provider every 3 seconds specifically to expire a stale reading, so a frozen "pass" can never sit on screen looking current for more than that window before the fallback banner takes over.
From U2xAI Academy

From answering questions to owning outcomes.

A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.

  • A live AI agent you actually shipped
  • A launch decision you can defend under pressure
  • An interview-ready portfolio, not more flashcards
Know more