Describe how streaming changes perceived latency without changing actual latency.

A recipe that takes eight seconds to write does not feel eight seconds long once real words start showing up in the first half second. It only feels broken when the screen stays blank the whole time, because a blank screen looks exactly the same whether the answer is nearly done or never coming at all.

How to answer this, stage by stage

Nobody is grading whether you can define streaming. They are grading whether you know wall clock time and felt time are two different numbers, and that you can fix the second one without touching the first at all.

Let's learn

What happens the first time you tap a button and nothing happens for eight seconds?

Broth is the AI recipe and meal planning assistant inside Skilletry. You photograph what's in your fridge, or just type it out, and Broth writes tonight's recipe plus a three day plan built around whatever's about to go off.

At first, Broth only wrote one recipe at a time. That took about two seconds. Nobody noticed two seconds. You tapped the button, you glanced up, the recipe was already there.

Then Skilletry shipped the three day plan. Same model, same button, but now it had to write about four times as much: tonight's dinner, plus breakfast, lunch and dinner for two more days. Typical generation time went from about two seconds to about eight.

Eight seconds doesn't look like a disaster on a spec sheet. It was still comfortably inside the ten second budget Skilletry had set at launch, and every dashboard the team watched said the product was healthy.

Here's the turn: those extra six seconds were never really the problem. The problem is what a person does with a screen that shows nothing at all for six of those eight seconds.

This is the whole answer to where perceived latency actually comes from. It isn't a slider that creeps up with every extra second. It's one flip, and streaming decides which side of it a person lands on.

At its worst, this trains people to treat a working feature like a broken one, forever, quietly, without a single complaint ever landing in a support inbox. Broth is built to rescue food that's about to be thrown out. Every session someone backs out of and never comes back to finish is a fridge that gets emptied into the bin anyway, which is the exact opposite of the whole product.

Before landing on real streaming, the team tried something cheaper first: a rotating carousel of cooking tips shown during the wait, meant to give people something to read while Broth worked. It barely moved the number, abandon fell from 18 percent to about 15 percent and stopped. The tips weren't real content. Once people learned the carousel always meant nothing was actually ready yet, they started backing out against the tips almost as fast as they had against a blank screen.

What I'd leave alone: the nightly batch job that rebuilds each person's "this week" suggestions on the home screen. It runs while nobody is looking at a spinner for it, so streaming that job would add engineering work for a screen no one is staring at.

The lesson: perceived speed is decided by the first visible word, not the last one. The actual time to finish can stay exactly the same, and the product can still flip from feeling broken to feeling instant, purely from when you choose to reveal the output.

Now here is the same thing as a story

Read the short version above when you're in the room. Read this one when you want to feel why an eighteen percent abandon rate can sit inside a healthy looking dashboard for three straight weeks.

Neriah Bircham has never needed a recipe to cook a real dinner out of whatever's left in the fridge. Long before Skilletry existed, friends would hand her three eggs, half an onion and a heel of cheese and she'd have something worth eating on the table in twenty minutes, no shopping trip, nothing wasted. That instinct is the whole reason she joined Skilletry to build Broth in the first place, and she still cooks on it herself most weeknights, partly because she likes it and partly because a product manager who never opens her own app stops noticing what's actually true about it.

For months, Broth was the easy part of her evening. She'd tap "get tonight's plan," start rinsing rice or washing a pan, and glance back two seconds later to a finished recipe. She never thought about the wait. There was nothing to think about. It was just there.

Then the three day plan shipped, and Neriah, like everyone else, kept tapping the same button without a second thought.

The habit did not wear down slowly over weeks. It held right up until one blank screen ran a few seconds too long.

On a Tuesday, she had two friends coming over, half a bag of spinach that needed using, and about forty minutes before they'd arrive. She tapped the button, propped her phone against the tile, and started chopping an onion. Four seconds in, the screen was still blank. Five. Six. She'd never had a reason to count before, so she wasn't really counting now either, she just felt the wrongness of it the way you feel a phone call that's rung one ring too many.

She backed out of the app, opened a delivery app instead, and ordered two pizzas. Her thumb was still on the confirm button when Broth's notification landed on the lock screen behind it: tonight's plan, ready. It had finished the whole thing in just over eight seconds, which was exactly what it was supposed to take. She just hadn't been there to see it.

What stayed with her wasn't the pizza. It was that she, of all people, the one who built this thing specifically so a half bag of spinach wouldn't end up in the bin, had just watched it end up in the bin anyway, and the app had done nothing wrong by any number on its own dashboard.

Eight months earlier, in the meeting where the three day plan's frontend first got built, someone asked whether the client should start rendering the recipe as soon as partial data arrived, or wait for the whole thing and show it at once. Streaming meant handling data that might arrive in pieces, in the wrong order, or not at all if something dropped. Waiting for the complete object meant one clean shape, easy to render, easy to test. Waiting won, reasonably, because back then the whole answer was ready in about two seconds and nobody had ever complained about a two second wait.

Neriah didn't find the eighteen percent number by watching a dashboard. She found it because she'd lived it, then went looking, and found that the exact same thing had been happening to strangers for three weeks, quietly, one closed app at a time, never once showing up as a support ticket.

Run the same Tuesday again, with one change: the reply streams the moment the model starts writing it. She taps the button, starts on the onion, and by the time she's set the knife down the first line is already on the screen, "Spinach and Chickpea Skillet, twenty five minutes." She keeps chopping. Words keep arriving. By the time the onion's done, the whole three day plan is sitting there, nothing wasted, no pizza ordered, no notification arriving behind a closed app.

One design waited for the whole plate to be ready before letting anyone see it. The other lets you watch it get cooked.

What I would tell myself, back in that first frontend meeting: the moment you decide to hide a real answer until it's completely finished, ask how long finished actually takes now, not how long it took when you made the decision. Nobody asked that question when the feature that would quadruple the wait was still eight months away. That's on the room, not on Neriah.

FLIPS, and the one letter that has no middle setting

Not five guesses about why a good product started feeling slow. FLIPS names the one thing that snapped, and asks which old choice made snapping the only option.

Five steps. Only the I step has no middle setting she could fall back on once the blank screen ran long.

Three things worth stating directly, since this is where the real judgment sits. The alternative Skilletry tried first was the cooking tip carousel, real, tested, and it lost: abandon only fell from 18 percent to about 15 percent, because a rotating tip is not the answer arriving, and people learned that fast. The AI specific failure worth naming by name is that streaming raw model output, word by word, risks showing a fragment that later turns out wrong or incomplete, a recipe step cut off mid sentence if a connection drops before the model finishes producing the third day of the plan. The guardrail is chunk safe streaming: the server only flushes a complete recipe step or a complete day of the plan to the client, never a half formed token, and if a stream drops partway, the client shows what's confirmed complete plus a small "still finishing" note while a background retry quietly completes the rest. That guardrail is not free. It costs real engineering time the old "wait for one object" approach never needed, a trade accepted on purpose, because an 18 percent weekly abandon rate on the feature meant to stop food waste costs Skilletry far more than the streaming infrastructure does. And the bar this enforces was never zero wait. Eight seconds to write a three day plan is allowed. The bar is about what's visible during that time, not about making the model faster.

And if you want to be sure it really works, try it somewhere else

Same five letters, an industry that has never planned a single dinner, and this time the private workaround isn't a pizza order. It's a second document nobody asked for.

Dovetail is an auto insurance claims company. Wickfield is the AI assistant its adjusters keep open on every call, drafting the claim summary and settlement note in real time from the adjuster's spoken notes and the photos coming in. Brennig Delacote is a claims adjuster who can look at a fender photo and land within a couple hundred dollars of the real repair estimate before a single form gets filled out.

The case for trusting it as built: on a simple claim, one car, one clear photo, Wickfield's draft lands in three or four seconds, and Brennig reads it straight back to the claimant while they're still on the line. Fast, accurate, and nobody ever had to think about the wait.

The case against it: a genuinely complicated claim, three cars, conflicting accounts, a dozen photos, makes Wickfield reason over far more input, and the draft can take twenty or thirty seconds to land. The screen shows nothing while it works. No progress bar, no partial line, nothing.

On the complicated calls, Brennig started keeping his own notes in a separate window the moment a call ran long, typing out a rough version of the summary himself so he'd have something to read to the claimant even if Wickfield never landed in time. Wickfield wasn't broken. It was working exactly as designed. But now, on the hardest calls, there were two documents: his and the machine's, and reconciling them after the fact was creating errors on about one call in twelve.

The fix that reached Brennig wasn't a smarter model. It was the same decision as Broth's: stream the draft the moment it starts, so the vehicle and damage lines fill in within about a second and Brennig reads along live with the claimant instead of typing a second version underneath the first. Average handling time on multi vehicle claims dropped from about nine minutes to about six, and the reconciliation errors disappeared, because there was only ever one document again.

Same rank as before, different family: a silent wait and a stalled process look identical from the outside, and a person who needs an answer regardless of which one it is will always build their own version rather than risk having nothing to say.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to the fix: stream the tokens, don't buffer the whole reply.
Cost: there's no budget this quarter for real token streaming. Flush the response in three or four finished chunks as the backend produces them, the title, the ingredients, then the steps, instead of one atomic blob. Most of the win for a fraction of the engineering.
The model got better, for real: say the model gets twice as fast, so total time drops from eight seconds to four. Still stream it. Four seconds of a blank screen still reads as "did my tap even land" to plenty of people. The threshold moves. The mechanism, nothing visible equals feels broken, does not disappear just because the model improved.

Where people run it wrong.
They watch total completion time against the budget and call the product healthy, without ever separately measuring time to first visible content.
They "fix" the blank screen with a generic loading animation or a tip carousel instead of real streamed content, which buys a few extra seconds of patience and nothing more.
They stream everything, including background jobs nobody is staring at, and spend engineering effort on the one place the fix was never needed.

How to use it live. Say the split out loud before answering: "is the problem that it's actually slow, or that it looks like nothing is happening." Naming that split buys you a beat, and tells the interviewer you won't reach for "make the model faster" as the only lever in the room.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0