ConceptAdvancedDesigning for Uncertainty & Trust / Feedback loops and data flywheels / #6
What is the latency between collecting feedback and improving the product, and how do you shorten it?
BOUND the product is Sparrowdock Helper, an AI chatbot that handles customer-support questions
Sparrowdock is a streaming service. Sparrowdock Helper is the AI chatbot that answers customer questions about billing, streaming quality, and account access. Otieno Kariuki manages the AI support team, and keeps a whiteboard by his desk he photographs at the end of most shifts.
The direct answer
Right now it takes about ten days from a bad chatbot answer to a fixed one, and the single biggest chunk of that, three and a half days on average, is just waiting for the next weekly triage meeting. Shrink that gap first: fast-track a same-day review for anything clearly severe or clearly repeating, instead of making every case wait for the same weekly clock.
Do this, in order
Break the total latency into its five real stages before trying to shrink anything.Why: you can't fix a number you haven't broken into parts.
Fix the biggest stage first: the wait for the next scheduled human review.Why: it's nearly as large as every other stage put together.
Build a fast-track path for clearly severe or clearly repeating issues.Why: not every issue needs the same slow, careful process as a genuinely unclear one.
Keep a staged rollout for any fix, even a fast-tracked one.Why: speed at the front of the pipeline shouldn't remove the safety check at the back.
Re-measure the real range regularly, not just the typical case.Why: a rare, complicated issue can still take three weeks, and that's worth knowing too.
Leave the full weekly triage meeting in place for genuinely ambiguous cases.Why: some issues really do need a careful human discussion, not a rushed guess.
How to answer this, stage by stage
Nobody is grading whether your number is exactly right. They're grading whether you can show your work and point at the one part worth fixing.
Stage 1
Scope it to one real system
Say it like this
"I'll answer this for Sparrowdock Helper, the AI chatbot that handles customer-support questions for our streaming service."
Why this works
Grounds an abstract question about "latency" in one concrete, measurable pipeline.
Stage 2
Say your structure out loud
Say it like this
"I'll use BOUND. Break it down, the equation. Own the numbers, my assumptions. Use a range, not one guess. Nail the sanity check. Direction, what swings it most."
Why this works
Signals this is an estimate with shown work, not a guess dressed up as a fact.
Stage 3
Break down the equation
Say it like this
"Total latency equals time to aggregate the signal, plus time to a human review, plus time to build a fix, plus time to pass eval, plus time to roll out to everyone."
Why this works
States the equation out loud before touching a single number.
Stage 4
Own the numbers
Say it like this
"I'd assume about half a day to aggregate the signal, three and a half days average wait for the weekly triage meeting, two days to build a fix, one day for eval, and three days for a staged rollout."
Why this works
Every number has a stated source, not a figure pulled from nowhere.
Stage 5
Give the range
Say it like this
"That's about ten days typical. Best case, a known issue an on-call engineer fast-tracks, maybe two days. Worst case, something ambiguous that needs two triage cycles, could run three weeks."
Why this works
A single number implies confidence nobody actually has. A range is the honest version.
Stage 6
Sanity-check it
Say it like this
"Ten days is longer than it takes a brand-new human support hire to notice and start improving on their own mistakes. If our AI system is slower than a rookie to visibly get better, that's a real problem, not a rounding error."
Why this works
Compares the estimate to something a listener already has a feel for.
Stage 7
Name the direction, and close
Say it like this
"The triage-meeting wait is the single biggest lever, nearly as large as every other stage combined. Fix that first, and the whole number moves the most."
Why this works
Names which single assumption is worth attacking, ready for whatever gets pushed on next.
Let's learn
Picture the ten days between a chatbot's bad answer and the moment that answer actually gets better, and ask yourself where all ten of them really went.
Sparrowdock runs Sparrowdock Helper, an AI chatbot that answers customer questions about billing, streaming quality, and account access.
Knowledge spark: what's a staged rollout?
Instead of shipping a fix to every customer at once, it goes out to a small slice first, then a bigger one, then everyone, with a short pause between each step to watch for anything going wrong. It's slower than shipping to everyone at once, but it catches a bad fix before it reaches the whole customer base.
For a long while, the team just knew "fixes take a while," with no real number attached to it. Nobody had ever added up the chain of small waits between a bad answer and a fixed one.
Where ten days go: typical latency by stage
The triage wait alone, 3.5 days, is nearly as large as fix-building, eval, and rollout put together.
At its worst: a genuinely urgent issue, one giving customers wrong billing information, sits in the same queue as a minor wording complaint, both waiting for the same weekly meeting, both taking roughly the same ten days to actually get fixed.
The decision I would take back
We built a single weekly triage meeting as the only review path for every flagged issue, since it was easy to schedule and gave the team a consistent, predictable cadence. That made sense when flagged-issue volume was low enough that a week's wait rarely mattered. It stopped making sense once daily volume grew and a severe billing error sat in the exact same queue as a minor wording complaint.
What I would leave alone: a genuinely ambiguous issue, one with no clear root cause yet, still deserves the full weekly discussion. Rushing an unclear case to a fast-tracked fix risks shipping the wrong fix with confidence.
The extra days were never really about engineering speed. They were about every issue, urgent or not, waiting in the exact same line for the exact same weekly clock.
The lesson: a latency number that nobody's ever added up end to end isn't a technical detail waiting to be measured someday. It's a decision, already made, about how long every customer's problem gets to sit before anyone even looks at it.
Now here is the same thing as a story
The short version above is what you'd say defending this estimate to Sparrowdock's support leadership. Read this one for how the number actually got built.
Otieno Kariuki ends most shifts the same way: he photographs the whiteboard by his desk before wiping it, a habit from years of tracking whatever the team was mid-fix on that week.
There was no single bad day that started this. It built up slowly, across months, as flagged-issue volume for Sparrowdock Helper kept climbing while the weekly triage meeting stayed exactly the same length, the same one hour, every Thursday.
Five stages, and the second one was the only one running on a fixed weekly clock instead of on how urgent the issue actually was.
Then it happened to a peer on another team, not Otieno's own. A billing-question issue sat in the queue for two full triage cycles, since the first meeting ran out of time before reaching it, and customers kept getting a wrong renewal date from the chatbot for another full week before anyone even started building the fix.
Otieno pulled up his own team's numbers that week and, for the first time, actually added the stages end to end instead of eyeballing each one separately.
Nobody had ever laid these four out side by side before. Once they were, the wait for triage was hard to miss.
Ten days was never the whole story. A genuinely unclear issue could sit for three weeks, and that number matters too.
He compared it against something everyone on the team already had a feel for.
A brand-new hire gets faster than this within their first week. That comparison did more work than any dashboard number could have.
With a fast-track path added for clearly severe or clearly repeating issues, the same billing-error case now skips the weekly queue entirely: an on-call engineer reviews it the same day it's flagged, and a compressed rollout gets the fix live in about four and a half days instead of ten, without skipping the staged safety check at the end.
The top right corner, a large wait that's actually easy to fix, is exactly where the triage bottleneck sits.
The old process asked every issue to wait for the same weekly clock, regardless of how urgent it actually was. The new one asks how urgent an issue is before deciding how long it gets to wait.
I built one weekly meeting because it was simple to run and easy to defend in a planning review. It took watching a peer's team leave customers with a wrong renewal date for an extra week, something that could just as easily have happened on my own team first, to see that a predictable cadence isn't the same thing as a fast one.
BOUND, the ten days broken openNot a single guess. BOUND is what forces every day in that number to say where it came from.
B
Break it down. The equation.
Total latency equals aggregation time, plus triage wait, plus fix-build time, plus eval time, plus rollout time.
Says the equation out loud before touching a single number.
O
Own numbers. Each assumption, sourced.
Half a day to aggregate, 3.5 days average triage wait, 2 days to build a fix, 1 day eval, 3 days staged rollout.
Every figure has a stated reason, not a number pulled from nowhere.
U
Use a range. Low and high, not one guess.
Ten days typical. Two days best case with a fast-track. Twenty-one days worst case across two triage cycles.
A single number implies a confidence nobody actually has.
N
Nail the sanity check.
Ten days is slower than a brand-new human hire takes to visibly improve on their own mistakes.
Compares the estimate to something the listener already has a feel for.
D
Direction. What swings the estimate most.
The triage-meeting wait, nearly as large as every other stage combined. Fixing it moves the total the most.
The hardest step, and the one that turns an estimate into a real plan.
How much each fix would cut total latency, by stage
Fast-tracking the triage wait alone saves more than the other three fixes combined. That's the direction step, in one picture.
The recap, one line per letter: break it down is the five-stage equation, own numbers is each stage's sourced assumption, use a range is the two-to-twenty-one-day spread around a ten-day typical case, nail the sanity check is the rookie-hire comparison, and direction is the triage wait as the single biggest lever.
And if you want to be sure it really works, try it somewhere elseSame five letters, a fisheries co-op instead of a chatbot. A completely different field, and this time the biggest lever isn't technical at all.
Wrackline Fisheries Co-op uses an AI tool to draft standardized catch reports for regulatory filing. Miriam Solheim manages that reporting tool, and fishers flag errors, a misidentified species, a rounded weight, through a dockside kiosk.
Mapped onto BOUND: break it down is total latency equals the wait until a flagged error reaches the co-op's shared log, plus the wait for the monthly co-op meeting that reviews flagged errors, plus the time to fix the report template, plus the wait for the next regulatory filing window to actually use it. Own numbers: about five days for the dockside sync, since boats dock at different times, fifteen days average wait for the monthly meeting, three days to fix the template, and up to thirty days for the next quarterly filing window to open. Use a range: about fifty-three days typical, with a worst case near ninety days, close to a full fishing season, since the filing window is fixed by regulation, not something the co-op can rush. Nail the sanity check: that's roughly the same delay the co-op's old paper-form process used to have, meaning the AI drafting tool hasn't actually sped up corrections at all, only the initial drafting step. Direction: the biggest lever here isn't technical, it's the monthly meeting cadence and the quarterly filing window, both process and regulatory bottlenecks a faster model would do nothing to fix.
Two of these four parts are set by regulation, not engineering. No faster model touches them.
Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "about ten days typical, and the triage wait is the biggest single lever," and stop.
Cost: there's no budget this quarter to build a full automated fast-track. Say so honestly, and start with a manual same-day escalation path for anything a support lead flags as severe, since even a manual shortcut beats none.
The model gets better, for real: if Sparrowdock Helper's overall accuracy improves, that's still not a reason to ignore this latency number, a better average model can still take ten days to fix the one narrow case it's still getting wrong.
Where people run it wrong.
They quote a single number with no stages behind it, which nobody can actually act on.
They assume the bottleneck must be technical, engineering speed, model retraining, when it's often a scheduling or process choice instead.
They speed up the whole pipeline evenly instead of finding the one stage that's actually the biggest lever.
How to use it live. When someone asks about the latency between feedback and improvement, ask yourself one question first: which single stage, if you fixed only that one, would move the total the most. Say that stage out loud, not just the total number.
Flashcards (tap any card to flip it)
1 · THE FRAMEWORK
What framework fits "what's the latency from feedback to improvement, and how do you shorten it"?
Tap to flip
ANSWER
BOUND: break it down, own numbers, use a range, nail the sanity check, direction. The direction step names which stage is worth fixing first.
2 · THE PEOPLE
Who is this answer about?
Tap to flip
ANSWER
Otieno Kariuki, who manages Sparrowdock's AI support team and photographs his whiteboard at the end of most shifts.
3 · THE EQUATION
What are the five stages in the total-latency equation?
Tap to flip
ANSWER
Signal aggregation, human triage, fix built, eval passed, rolled out. Each stage has its own real wait time.
4 · THE BIGGEST LEVER
Which single stage swings the total latency the most?
Tap to flip
ANSWER
The weekly triage-meeting wait, 3.5 days on average, nearly as large as every other stage in the pipeline combined.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Building a single weekly triage meeting as the only review path for every flagged issue, since it made sense only while volume was low enough that a week's wait rarely mattered.
6 · THE NUMBER
Fill in the blank: total typical latency runs about ___ days, of which the triage wait alone is 3.5 days.
Tap to flip
ANSWER
10 days. The real range runs from 2 days best case to 21 days worst case.
7 · THE REPLAY
Same severe billing issue, redesigned pipeline. What changes?
Tap to flip
ANSWER
It skips the weekly queue entirely, gets same-day review, and reaches customers in about 4.5 days instead of 10, without skipping the staged rollout safety check.
8 · CROSS PRODUCT TRANSFER
Section 4 answers this again for a different product. Which product, and what's different about its biggest lever?
Tap to flip
ANSWER
Wrackline Fisheries Co-op's catch reports. There, the biggest lever is a regulatory filing window and a meeting cadence, not anything an engineer could speed up.
Check yourself Score: 0 / 0
Multiple choice
1. Why does this answer name the triage-meeting wait as the stage worth fixing first, instead of, say, the eval stage?
A. Because eval can't be measured accurately.
B. Because the triage wait, at 3.5 days, is nearly as large as every other stage in the pipeline put together.
C. Because eval only matters for very large customer bases.
D. Because triage meetings are legally required to happen weekly.
Show hint
Look at the stacked bar chart of latency by stage.
Show answer
B. Fixing the single largest stage moves the total the most, which is exactly what the direction step in BOUND is built to find.
True or false
2. True or false: this answer recommends removing the weekly triage meeting entirely.
True
False
Show hint
Look at "what I would leave alone."
Show answer
False. The weekly meeting stays for genuinely ambiguous cases. Only clearly severe or clearly repeating issues get a fast-tracked, same-day path instead.
Fill in the blank
3. Fill in the blank: the worst-case latency, across two full triage cycles, runs as high as ___ days.
Show hint
Look at the timeline showing the real range.
Show answer
21 days. The typical case is 10 days, and the best case with a fast-track is about 2 days.
Short answer, name the reversal
4. What old decision does this answer take back, and why did it make sense when it was made?
Show hint
Look at "the decision I would take back."
Show answer
Model answer: Building a single weekly triage meeting as the only review path. It made sense while flagged-issue volume was low enough that a week's wait rarely mattered to anyone.
Short answer, where it wouldn't matter
5. Name a kind of flagged issue where the full weekly triage meeting still matters, even with a fast-track available.
Show hint
Look at "what I would leave alone."
Show answer
Model answer: A genuinely ambiguous issue with no clear root cause yet. Rushing it to a fast-tracked fix risks shipping the wrong fix with false confidence.
Short answer, apply it yourself
6. Pick a product you use yourself. If you flagged a problem with it today, roughly how long do you think it would actually take to get fixed, and what's the one stage you'd guess is slowest?
Show hint
Think about how long it usually takes an app to fix a bug you reported, and where the wait probably sits.
Show answer
Model answer: Most people guess weeks, not days, and usually point at "someone has to notice and prioritize it" as the slowest stage, the same shape as the triage wait in this story.
Before you close the answer
Why this works
Tests whether you can turn a vague question about "latency" into a real, broken-down estimate with shown work, and whether you can find the one lever worth pulling instead of proposing to speed up everything evenly.
Follow-up traps
"Couldn't you just make the whole pipeline faster across every stage?" Response: possible, but far more expensive, and the triage wait alone accounts for over a third of the total, so fixing it first gets most of the benefit for a fraction of the effort.
"Doesn't a fast-track path risk shipping bad fixes without proper review?" Response: no, the fast-track still keeps eval and a staged rollout, it only skips the wait for the next scheduled meeting, not the actual safety checks.
If pressed
Sparrowdock's real fast-track only applies once a rule-based severity score, based on issue volume and category, crosses a set threshold automatically, so no single person decides case by case which issues skip the queue.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
