How would you decide between a fast weak model and a slow strong one for autocomplete?

Loomstead picked Threadloom's model on quality alone and ran its best one everywhere. The real choice was between a suggestion that's late and invisible, or one that's fast enough to get tab-accepted without a second look, right or not.

How to answer this, stage by stage

Nobody's testing whether you know both models exist. They're testing whether you'll price the actual cost of arriving late against the actual cost of arriving wrong, or just say "speed matters" and stop there.

Let's learn

The suggestion is a few words of gray text that show up right after your cursor, before you've decided what to type next.

Threadloom is Loomstead's email tool. It watches a sentence as you write it and offers the next few words. Tap the tab key and they're yours. Keep typing and they vanish.

Before Threadloom, a sales rep writing a routine follow-up, about 90 words, typed the whole thing by hand. That took roughly three minutes when nothing needed looking up first.

Threadloom launched running Loomstead's biggest, best model, the same one the company trusted for everything else it built. When it worked, a rep could tab through most of a 90-word email instead of typing it, cutting that three minutes down to about 80 seconds.

The turn: "when it worked" was carrying a lot of weight. The big model averaged 640 milliseconds to produce a suggestion, and climbed past 1,300 milliseconds during the two hours every weekday morning when every sales team at once sat down to write follow-ups. A fast typist moves to a new character about every 170 milliseconds. By the time a 640-millisecond suggestion rendered, most reps had already typed three or four characters past where the model was aiming. The suggestion wasn't wrong. It was late. And late, in this feature, means invisible.

The late suggestion costs nothing and gets noticed instantly. The stale one costs nothing to notice, right up until a client reads it.

At its worst, this doesn't fail loud. Because the suggestion box didn't check whether the sentence underneath it had changed before letting a rep tab-accept whatever was already sitting there, a stale suggestion, built for an earlier, shorter version of the sentence, could still get inserted the moment it finally rendered. That's how this ships a wrong sentence into a real email, not because the model guessed wrong, but because the answer to a question had already changed by the time it printed.

The choice I would take back: in a fifteen-minute meeting the week before launch, Ingemar's team decided to run Loomstead's best model everywhere, live suggestions included, because "our best model should write every sentence" sounded right when only a hundred people used Threadloom and load was never a problem.

What I'd leave alone: the weekly digest email Threadloom drafts, summarizing a rep's week for their manager. Nobody's mid-keystroke waiting on that one, it has whole minutes to render, so it stays exactly where it is, on the big model.

The lesson: a suggestion that's a little wrong is easy to catch, you read it before you tab it in. A suggestion that's late doesn't get caught. It gets skipped, and skipped looks identical to working, right up until someone tab-accepts one that arrived after the sentence under it had already moved on.

Now here is the same thing as a story

The short version is above. Read the long version below for why one ordinary Tuesday morning changed how Threadloom decides which model gets to speak first.

Ingemar Riordan could read a week of Threadloom's usage logs and tell you which sales team was struggling before their manager noticed. He'd built two email tools before this one and picked Threadloom's original model himself, the week the company decided to invest in it seriously.

The first eight months went well. Threadloom launched to a hundred beta reps, and every morning a rep would open a draft, start typing, and watch gray text appear a beat later, usually right, sometimes worth ignoring, never much trouble either way. Loomstead's leadership pointed to it in every board update. Ingemar kept the big model running everywhere because nothing had ever given him a reason to look twice.

Then Loomstead signed three large customers in the same quarter, and Threadloom's user count jumped past three thousand. The habit thinned in three small beats nobody flagged at the time: reps stopped glancing at the ghost text during the busiest hour and typed straight through it, a few started keeping their eyes on the keyboard instead of the screen while they waited, and the tab key turned into something fingers reached for out of rhythm more than out of reading.

The trigger was small. It was a Tuesday, 9:40 in the morning, and a rep was typing "We can't ship before Friday" to a client who'd been pushing for an earlier date. The strong model had started scoring a suggestion off her sentence a beat earlier, before she'd finished typing the word "can't." By the time that suggestion rendered, 740 milliseconds later, she'd already finished the real sentence. Out of habit, mid-motion, her hand hit tab anyway. The stale suggestion dropped into the message and read back as "We can ship it by Thursday to be safe," the exact opposite of what she'd meant to send.

A teammate glancing over her shoulder, on the way to grab coffee, caught it before she clicked send. "Wait, didn't we just tell them we can't do that?" Four minutes of checking followed, then a slower rewrite of the whole paragraph by hand. Nobody outside that desk ever knew how close it came.

The real cost was never the extra half second the strong model needed. It was that for two hours every single morning, Threadloom quietly stopped being seen at all, and once in a while, on exactly the days it mattered most, an old answer sat there long enough to get mistaken for a new one.

The decision Ingemar would take back happened in that fifteen-minute meeting eight months earlier. "We only get one shot at people trusting this. Run the best model we've got, everywhere," someone said, and the room agreed. Nobody in that meeting asked what happens once three hundred reps are all typing at nine in the morning.

Run that Tuesday again with the fast model live and a freshness check underneath both models: the suggestion for "We can't ship before Friday" renders in about 45 milliseconds, already matching what she typed, or it renders nothing at all if the sentence moved on first. Either way, the wrong sentence never gets the chance to sit on screen waiting for a tab key. 9:40 becomes an ordinary minute instead of the one Ingemar's team spent the rest of the day re-reading the queue.

One design bet everything on the model being right. The other bet on the suggestion never getting old enough to be wrong by the time someone reads it.

What I'd tell myself in that fifteen-minute meeting: "best" isn't a property of a model sitting in a lab. It's a property of an answer that still matches the question by the time it arrives.

The four letters that made this a real pick, not a guess

This isn't a speed-versus-quality question wearing a technical costume. PICK is what keeps "the fast model felt right" from staying a hunch instead of a decision you could defend.

Two things worth stating directly, since this is where the real judgment sits. The alternative Ingemar's team considered and dropped was caching the strong model's most common completions to shave its latency, instead of switching the default suggestion to a smaller model. It lost because real work email is mostly unique sentences, not templates. In testing, fewer than 4 percent of live suggestions ever matched anything already sitting in the cache, so peak-hour latency barely moved and the two rush hours stayed exactly as invisible as before. The AI-specific failure worth naming by name is context staleness: a suggestion scored against a sentence that has since changed keeps sounding just as confident anyway, because the model has no way of knowing the world under it moved on before it finished answering. The guardrail is the freshness check itself, a plain check that a suggestion still matches the current sentence before it's ever allowed to render, run automatically on every single suggestion regardless of which model made it, plus a separate 500-example test set of risky spans, negations, dates, names, and numbers, that showed the fast model changing meaning about 1 in 40 times there against 1 in 250 for the slow model, which is why those specific spans get withheld rather than guessed on by either model.

And if you want to be sure it really works, try it somewhere else

Same four letters, a field technician's headset instead of a sales rep's keyboard, and this time the hidden cost isn't a sentence, it's a part that ships to the wrong address.

Coilnote is a diagnostic note assistant Wickfold Mechanical built for its HVAC field technicians. A tech talks while both hands are inside a unit, and Coilnote transcribes the visit and suggests the part number as she names the fault. Bronagh Amundsen is the technician whose crew tests every build before it reaches the whole company.

The build-up: Wickfold's techs log about 40 service visits a day across the company. The on-device weak model renders a part-number suggestion in about 150 milliseconds, fast enough to sit inside a normal pause in speech. The cloud-hosted strong model cross-checks the exact equipment model against the full manufacturer catalog, more accurate, but 2.5 to 6 seconds in a basement mechanical room with weak signal, an eternity when a tech is mid-sentence with a wrench in one hand.

Same rank as before, a different lever: for Threadloom, the hidden cost was a sentence that quietly said the wrong thing to a person. For Coilnote, it's a physical part that quietly ships to the wrong place. The fix isn't a freshness check this time, since nothing about a part number goes stale mid-sentence the way a written sentence does. It's a spoken read-back: Coilnote repeats any part number out loud before it's logged, and the tech has to say "yes" or correct it, turning a silent auto-fill into a checked one without slowing down the note itself.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: run the fast model for anything happening while someone's still mid-action, save the slow model for a moment they're already willing to wait on, and never let either one's guess through unchecked on the spans where being wrong actually costs something.
Cost: there's budget this quarter for either a bigger on-device model or a faster network link to the cloud model, not both. The bigger on-device model wins for Coilnote specifically, because a truck roll to fix a wrong part costs far more than the cloud model's extra accuracy is worth.
The model got better, for real: say the strong model's latency gets cut by two-thirds next year. That's real progress, and it narrows the gap. It doesn't remove the need for a freshness check or a read-back, since even a fast model can still be confidently wrong, just faster about it.

Where people run it wrong.
They treat "the fast model is less accurate" as the whole story, and never ask who actually pays for each kind of mistake.
They build a caching or infrastructure fix aimed at making the slow model faster, without first checking how often real usage would even hit that cache.
They design the guardrail once, at launch scale, and never revisit it once usage grows past the size it was tested at.

How to use it live. Say the real question out loud before naming a model: "before I pick one, is this asking what a suggestion should look like right now, mid-keystroke, or what it should look like once someone's already stopped to wait for it." That buys a beat to think instead of repeating whichever model got picked at launch.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0