ConceptIntermediateDesigning for Uncertainty & Trust / Feedback loops and data flywheels / #21
What is the minimum feedback instrumentation to ship with version one?
ORDER the product is LanePilot, an AI copilot at Portrail Logistics that suggests delivery routes and load assignments for the night dispatch desk
Portrail Logistics runs an overnight delivery fleet out of one regional warehouse. LanePilot suggests a route and a load plan for each truck before a dispatcher sends it out. Declan Osei has led the night dispatch desk for six years, radio in hand from the first truck out to the last one back.
The direct answer
Ship three things minimum, nothing fancier. First, log every suggestion as accepted or overridden, automatically, with no extra tap required. Second, add one optional tap for a reason when it's overridden. Third, watch the override rate by region or route type, with an alert if it climbs. Skip any of these and version one can't tell the difference between the tool working and the tool being quietly ignored, which is the exact gap that lets a real blind spot go unfixed for months.
Do this, in order
Log every suggestion as accepted or overridden, automatically, before anything else.Why: without this, silent abandonment and real usage look identical from the outside.
Add one optional, one-tap reason for an override, not a form.Why: a dispatcher on a live shift won't fill out a survey, but they will tap one reason if it costs three seconds.
Watch override rate by region, with an alert when it climbs past normal.Why: a real blind spot shows up as a rising number weeks before anyone would notice it by hand.
Hold off on a full feedback survey or a written explanation panel for v1.Why: neither one is load-bearing yet, and both cost more to build than the three signals above are worth trading away for.
Treat a rising override rate as a question to ask, not a score to defend.Why: the rate itself doesn't explain anything; it just tells you where to go look.
How to answer this, stage by stage
Nobody is grading how many instrumentation ideas you can list. They're grading whether you can say which one breaks everything else if it's missing.
Stage 1
Ground it in one desk, one tool
Say it like this
"I'll answer this for LanePilot, a route-suggestion copilot at Portrail Logistics, used by Declan Osei's night dispatch desk."
Why this works
Keeps "minimum instrumentation" from turning into a generic checklist with nobody using it.
Stage 2
Say your structure out loud
Say it like this
"I'll use ORDER. Outcome, what everything competes to move. Reversibility, the hardest gap to undo. Dependency, what unblocks what. Evidence, what's cheap to learn early. Rank, the actual order."
Why this works
Names a method, so the answer reads as a ranked argument instead of a feature wishlist.
Stage 3
Name what everything competes to move
Say it like this
"Every piece of instrumentation is competing to answer one question: can we tell the difference between LanePilot being used, and LanePilot being quietly worked around, within the first month?"
Why this works
Without naming this, ranking instrumentation candidates is just a matter of taste.
Stage 4
Find the least reversible gap
Say it like this
"Shipping with zero signal is the hardest gap to undo. If dispatchers quietly abandon a piece of it in week one, that habit sets in, and you won't find out until a usage report weeks later, if ever."
Why this works
ORDER's hardest step: the thing that can't be recovered later is what has to be built first.
Stage 5
Say what depends on what
Say it like this
"The one-tap reason only means something once the accept/override log already exists. Region-level alerting only means something once both of those are already flowing in."
Why this works
Some of the build order is forced by what the next piece needs, not by preference.
Stage 6
Name the cheapest real signal
Say it like this
"An accept/override log costs almost nothing to build, since the system already knows whether its own suggestion got used. That's the cheapest, earliest real signal available."
Why this works
Shows you'd learn something before spending a whole quarter building anything heavier.
Stage 7
Give the rank, and close on it
Say it like this
"So: accept/override log first, one-tap reason second, region-level alerting third. Everything heavier, a full survey, a written explanation panel, waits for a version after this one earns it."
Why this works
Restates the direct answer as a real sequence, ready to defend if someone pushes on the order.
Let's learn
The radio clipped to Declan Osei's belt has called every truck out of Portrail's yard for six years, long before LanePilot ever suggested a single route.
Before LanePilot, building the night's route plan by hand took Declan about ninety minutes at the start of each shift. LanePilot cut that to fifteen: it suggests a route and a load plan for every truck, and Declan approves or adjusts each one.
How the north-side routes were actually being handled, before anyone knew
Nobody built this number on purpose. Nobody was measuring it at all, until someone finally asked.
The turn: the ninety-two percent wasn't dispatchers rejecting LanePilot out of habit. It was LanePilot quietly missing something real, a road-network change nobody had told it about, and nobody on the product side ever finding out, because version one never asked the question.
One of these you can still walk back easily. The other one, by the time you notice, already isn't a decision anymore.
The decision I would take back
We shipped version one with no signal at all on why a suggestion got overridden, to hit a launch date. That made sense when the team believed early accuracy would be close to good enough everywhere. It stopped making sense the moment LanePilot had a real, systematic blind spot and nothing in the product could tell the difference between that and ordinary day-to-day override noise.
At its worst: an entire region of routes gets quietly rebuilt by hand, every single night, for a whole quarter, while the team's usage dashboard still shows healthy daily logins, because logging in and actually using a suggestion were never the same thing anyone measured.
What I would leave alone: a full, detailed feedback survey on every suggestion. It sounds thorough, but a dispatcher mid-shift won't fill one out, and building it first would have delayed the three signals that actually mattered.
The lesson: the minimum viable version of feedback isn't the smallest thing you can ship. It's the smallest thing that still lets you tell the difference between a tool working and a tool being quietly ignored.
Now here is the same thing as a story
The short version above is what you'd say defending the instrumentation plan to Portrail's engineering lead. Read this one for how the gap actually got found.
Declan Osei has run the night desk at Portrail for six years, the kind of dispatcher who can reroute a truck around a closed bridge from memory, mid-radio call, without looking anything up.
For LanePilot's first two months, the good nights were very good. Suggestions matched what Declan would have built by hand, across every region, and his ninety-minute planning block shrank to fifteen.
Four pieces, and the first one is the only one that doesn't depend on anything else existing yet.
Around week three, a real detour opened on the north side of the delivery zone, a construction closure LanePilot's map data hadn't caught up with yet. Its suggestions there quietly got worse. There was no button to say so, no field to explain why, just a route that was slightly wrong, night after night.
Knowledge spark: why is this a silent failure, not just an off day for the model?
A silent failure is a wrong answer with no signal attached to it, nothing telling anyone what went wrong or that anything went wrong at all. LanePilot wasn't loudly broken. It just quietly stopped being right about one part of the map, with nothing in the product built to notice.
Declan started building the north-side routes by hand again, the way he always had before LanePilot, without ever reporting it. There was nothing obvious to report to, and he had ninety trucks to get out the door before sunrise.
Nine weeks of quiet drift, and the new hire's question is the only reason anyone looked.
Three months in, a new dispatcher joined the night desk and asked Declan, mid-shift, why everyone just built the north-side routes by hand instead of using LanePilot like the rest of the map. Declan realized he couldn't actually explain it. He just did it. Everyone did. It had become the kind of thing you learned on your first week, never written down anywhere.
Nobody had decided LanePilot couldn't handle the north side. It had just quietly become true, the same way any unwritten rule becomes true: repeated often enough that asking why stops occurring to anyone.
That question was the only reason anyone went looking. Portrail added the three-part instrumentation the same week: an accept/override log, a one-tap override reason, and region-level alerting on the rate.
Four fields on every entry, and none of them existed anywhere before the new hire's question.
LanePilot acceptance rate, north-side routes versus the rest of the network
The gap was there from week two. Nobody was watching it until week ten, when a question with no easy answer finally made someone look.
Run the same construction closure forward under the new instrumentation: the north-side override rate crosses its alert threshold by day four, region-level, automatically, and an engineer is looking at the map data gap before a single new hire ever has to ask why.
I let version one ship with no feedback signal because the launch date mattered more than a hunch that most regions would be fine. It took a new hire's honest question to see that "most regions" isn't the same thing as "all of them," and a rule that only catches a real gap when someone happens to ask the right question out loud isn't a rule. It's luck wearing a launch date.
ORDER, in one screenNot a feature checklist. ORDER is what tells you which piece of instrumentation breaks everything else if it's skipped.
O
Outcome. What everything competes to move.
Whether you can tell LanePilot being used apart from LanePilot being quietly worked around, within the first month.
Without this named first, ranking instrumentation candidates is just taste.
R
Reversibility. The hardest gap to undo.
Shipping with zero signal. A quietly abandoned region sets in as an unwritten rule long before any usage report would ever catch it.
The hardest step and the direct answer's foundation: build first whatever prevents the least reversible gap.
D
Dependency. What unblocks what.
A reason for the override only matters once the accept/override log already exists. Region alerting only matters once both are already flowing.
Some of this order is forced by what each piece actually needs to mean anything.
E
Evidence. What's cheap to learn early.
The accept/override log costs almost nothing, since the system already knows whether its own suggestion got used.
The cheapest real signal available, before committing to anything heavier.
R
Rank. The actual sequence.
Accept/override log first, one-tap reason second, region-level alerting third. A full survey and a written explanation panel wait for a later version.
States the order and defends the top pick in one line.
The three that made the cut all sit in the same corner: cheap, and worth a lot.
The recap, one line per letter: outcome is telling real use apart from silent abandonment, reversibility is the unrecoverable cost of shipping with zero signal, dependency is the reason and the alerting each needing the log to exist first, evidence is the log's near-zero build cost, and rank is the three-item build order itself.
And if you want to be sure it really works, try it somewhere elseSame five letters, a radiology reading list instead of a dispatch desk. Nothing else about the two jobs is alike.
Ridgemoor Radiology Partners uses an AI tool that pre-marks likely areas of concern on chest scans before a radiographer reviews them. Owen Fitzgerald has read scans there for nine years.
Mapped onto ORDER: outcome is telling real trust in the pre-marks apart from radiographers quietly re-reading every scan from scratch regardless of what's marked. Reversibility: shipping with no signal on which marks get overridden is the hardest gap to undo, since a radiographer's private habit of ignoring the marks on a certain scan type could set in for months before anyone in engineering ever saw a number move. Dependency: a one-tap reason for dismissing a mark only means something once the accept/dismiss log already exists; alerting on dismissal rate by scan type only means something once both already flow. Evidence: the accept/dismiss log costs almost nothing, since the system already knows which marks a radiographer cleared without comment. Rank restates the identical three-item order, in a field with nothing else in common with overnight trucking.
The same three items, whether the suggestion is a delivery route or a mark on a chest scan.
Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "log accept or override automatically, add one optional tap for why, alert on the rate by segment," and stop.
Cost: engineering says even the one-tap reason field costs a sprint they don't have before launch. Say so, and ship the accept/override log alone first, since even that much beats zero signal entirely.
The model gets better, for real: if LanePilot's overall accuracy improves, this instrumentation still matters, because a rarer real gap is even easier to mistake for ordinary override noise without a number tracking it by region.
Where people run it wrong.
They build a detailed feedback survey for v1 and skip the cheap automatic log that would have caught the real problem sooner.
They treat login or usage counts as proof the tool is working, without checking whether suggestions are actually being accepted.
They wait for a support ticket that a silent failure, by definition, never generates.
How to use it live. When someone asks for the minimum v1 instrumentation, ask yourself first: could this ship with a real, systematic blind spot and nobody find out for months? If yes, that's the one signal you can't cut.
Flashcards (tap any card to flip it)
1 · THE FLIP FAMILY
What flip family is this?
Tap to flip
ANSWER
Abandonment flip. Declan quietly stopped using LanePilot on north-side routes, with no complaint or ticket, and nobody knew until a new hire asked why.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Declan Osei, who has led Portrail Logistics' night dispatch desk for six years, radio in hand every shift.
3 · THE HABIT
What did Declan stop doing because LanePilot worked at first?
Tap to flip
ANSWER
He stopped building routes by hand across the whole map, and started mostly just approving LanePilot's suggestions instead.
4 · THE FLIP, IN THIS STORY
What's the two-setting switch here?
Tap to flip
ANSWER
Using LanePilot's suggestions across the whole map, versus quietly building north-side routes by hand again with nothing reported anywhere.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Shipping version one with no feedback signal at all, to hit a launch date, when the plan assumed most regions would be fine without checking.
6 · THE NUMBER
Fill in the blank: on north-side routes, LanePilot's suggestions were accepted only ___% of the time.
Tap to flip
ANSWER
8%. Ninety-two percent were overridden with no comment at all, because version one gave nobody a way to leave one.
7 · THE REPLAY
Same construction closure, new instrumentation. What changes?
Tap to flip
ANSWER
The override rate crosses its alert threshold by day four, and an engineer is looking at the map data gap before any new hire ever has to ask why.
8 · CROSS PRODUCT TRANSFER
Section 4 answers this again for a different product. Which product, and what changed?
Tap to flip
ANSWER
Ridgemoor Radiology Partners' scan pre-marking tool. Same three-item order, applied to dismissed marks instead of overridden routes.
Check yourself Score: 0 / 0
Multiple choice
1. Which piece of instrumentation has to exist before a one-tap "reason for override" field means anything?
A. A full customer satisfaction survey.
B. The automatic accept/override log, so there's already a record of which suggestions were rejected.
C. A written explanation panel next to every suggestion.
D. Nothing; it can ship on its own with no dependency.
Show hint
Look at the D step, dependency.
Show answer
B. A reason field attached to nothing is just floating text. It only becomes a real signal once there's already a log of which suggestions were overridden in the first place.
True or false
2. True or false: a full, detailed feedback survey on every suggestion belongs in version one.
True
False
Show hint
Look at "what I would leave alone."
Show answer
False. A dispatcher mid-shift won't fill one out. Building it first would have delayed the three cheap signals that actually mattered.
Fill in the blank
3. Fill in the blank: north-side acceptance fell from 82% in week 2 to ___% by week 9, before anyone noticed.
Show hint
Look at the line chart in the story.
Show answer
8%. The gap was visible in the data from week two onward. Nobody was watching it until a new hire's question in week ten.
Short answer, name the reversal
4. What old decision does this answer take back, and why did it make sense when it was made?
Show hint
Look at "the decision I would take back."
Show answer
Model answer: Shipping version one with no feedback signal, to hit a launch date. It made sense assuming most regions would be fine; it broke the moment one region had a real, unmeasured gap.
Short answer, apply it yourself
5. Pick a tool you use yourself. What's the cheapest signal it could add that would tell its makers apart real use from quiet abandonment?
Show hint
Think about the difference between opening an app and actually acting on what it suggests.
Show answer
Model answer: Many tools already log opens or logins as their main usage metric, which hides the exact gap this answer is about: opening something and ignoring it look identical from the outside.
Short answer, where it wouldn't matter
6. Name a piece of feedback instrumentation that genuinely doesn't need to ship on day one.
Show hint
Look at "what I would leave alone."
Show answer
Model answer: A full, structured feedback survey, or a written explanation panel for every suggestion. Both are real ideas, just not load-bearing enough to justify delaying the three cheaper signals.
Before you close the answer
Why this works
Tests whether you can rank instrumentation by what actually breaks first if skipped, instead of listing every feedback idea that sounds reasonable.
Follow-up traps
"Isn't logging every accept and override a privacy or overhead concern?" Response: the system already knows what it suggested and what shipped; recording whether they matched adds no new personal data and almost no overhead.
"What if dispatchers just stop tapping the reason field too?" Response: that's fine, the automatic log still shows the override happening at all, which is the load-bearing signal; the reason field is a bonus, not the thing everything depends on.
If pressed
Portrail's actual alert threshold isn't a fixed percentage; it's a rolling comparison against that region's own baseline override rate, so a naturally harder region doesn't trigger false alarms while a real new gap still stands out fast.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
