Name three product decisions that change when a feature's output is probabilistic rather than deterministic.

Larderline is Milkwood Kitchen's meal-planning app. Tell it what's in your kitchen and it builds a week of dinners plus a matching grocery list. Aurelia Achterhoeve owns its quality bar. Fourteen weeks ago, Larderline shipped Freeform Fridge, a mode that reads one plain sentence about your week instead of a picklist, and composes the substitutions itself. A soy-free household almost got tofu on its cart. Nobody had written that exact sentence before, so nobody could have tested for it ahead of time.

How to answer this, stage by stage

Nobody is grading whether you can say "testing needs to be different for AI." They're grading whether you can name the three actual decisions that change, and say why a patched ticket doesn't mean the problem is gone.

Let's learn

Larderline is Milkwood Kitchen's meal-planning app. You tell it what's in your fridge and who you're feeding, and it builds a week of dinners plus a grocery list of exactly what's missing.

Before Larderline existed at all, a family spent about 45 minutes a week planning dinners and writing a list by hand. Larderline's first version cut that to about 6 minutes: pick meals from a grid of fixed recipes, and the app scales the quantities and swaps a disliked or allergenic ingredient for a safe one, using a fixed lookup table someone had hand-checked. That table has been live for two years. In that time, it has never once produced a wrong allergy swap that anyone found, because every entry in it was tested one at a time before it shipped.

Fourteen weeks ago, Larderline shipped Freeform Fridge. Instead of picking from a grid, you type a sentence: "cutting back on red meat this week, we have leftover rotisserie chicken and broccoli that's about to turn, feeding four, and Wren can't have soy." The model reads that, and writes the week, substitutions and all, in about 90 seconds. No picklist. That's the whole appeal.

A fixed recipe grid only ever had to get pantry math right. A freeform plan has to get all four of these right, every time, from one sentence.

By week 14, 96,000 households were using Freeform Fridge. About 9,600 of them, one in ten, had at least one allergy tag on file. Across those fourteen weeks, 19 tickets got filed for a wrong Freeform Fridge plan, of every kind: a missing ingredient, an odd quantity, one soy swap. Each one got traced, patched by an engineer named Baako Asante, and closed. Nineteen problems, nineteen fixes. It read like a healthy number for a brand-new feature.

Here's the turn. The 19 closed tickets were never the real number. On the Thursday of week 14, Freeform Fridge answered that sentence about Wren by adding pan-seared tofu to the Thursday dinner, for extra protein, and put a package of firm tofu on the cart. Wren's older sibling did the pickup, saw "organic firm tofu, 14 oz" on the screen before paying, and texted their mother: "wait, doesn't Wren do soy free? cart has tofu on it." She pulled it before checkout. Nobody got hurt. Baako found the exact generation, patched it, and closed it as ticket #19.

That question is what Aurelia couldn't let go of. There was nowhere in the product to look up how often this shape of mistake happened, only a queue of closed tickets that all said "fixed." So she pulled three weeks of raw request transcripts by hand, about 150 that combined an allergy tag with an "extra protein" style request, and read every one herself. Two more contained a soy-based swap for a soy-free household, neither ever reported. That's 2 in 150: about 1 in 75.

What it costs at its worst: Freeform Fridge could be quietly serving unsafe swaps to a real slice of its 9,600 allergy-tagged households, and nobody would know, because most people don't read their auto-generated cart line by line, especially from an app they've learned to trust.

A ticket queue is built to catch the misses in the bottom right corner. Nothing about it was ever built to catch the one in the top left.

What I would leave alone: Larderline's older fixed-recipe mode doesn't need any of this. It still scales the same hand-vetted recipes off the same fixed swap table it's always used, tested and provably safe. There's no rate to track there, only correct or a code bug, the same as it's been for two years.

The lesson: a ticket queue answers "did we fix the one we saw." It was never built to answer "how many did we not see," and once output is probabilistic, that second question is the only one that actually protects anybody.

Now here is the same thing as a story

Read the short version above when you're in the room. Read this one when you want to feel why a patched ticket felt like good news for exactly as long as nobody counted.

Aurelia Achterhoeve built Larderline's QA process the year the fixed-recipe engine shipped. One ticket per miss, one engineer, one root cause, one fix. For two years it worked so cleanly that she stopped thinking of it as a process at all. It was just what you did when something broke.

Freeform Fridge launched on a Monday in March, and the first weeks were good ones. Households described their kitchens in a sentence and got back a real week of dinners, no picklist. A handful of tickets trickled in, a repeated ingredient here, an odd quantity there, and each one closed the same way the fixed-recipe engine's bugs always had: Baako found the cause, fixed it, moved on. Aurelia closed every ticket without asking whether that exact shape of mistake could be happening again, right now, to someone who hadn't noticed. Why would she ask that. For two years, closing the ticket had always been the end of the story.

Nothing here looks dramatic until the very last milestone. That's what a quiet, constant risk looks like from the inside.

Then came a Thursday in week 14. Nothing about the day looked different. A household with a soy-free ten-year-old, Wren, typed a plain sentence about wanting less red meat and had leftover chicken to use up. Freeform Fridge read "more protein" and added pan-seared tofu to Thursday's dinner, then put firm tofu on the grocery cart. Wren's older sibling did the grocery pickup, saw the tofu on the screen before paying, and texted their mother one line: "wait, doesn't Wren do soy free?"

She pulled it off the cart. She also filed a support ticket, mostly annoyed. Baako found the exact generation within the hour, saw the model had weighed "more protein" more heavily than the soy-free tag on that one output, patched a rule to bump the tag's weight, and closed it. Ticket nineteen of fourteen weeks. Fixed.

Here's where the habit finally snapped. Aurelia went to write the same closing note she'd written eighteen times already, and stopped halfway through the sentence. This wasn't a lookup-table typo with one wrong row. There was no row. The model hadn't done anything you could point at in the code. It had simply, on this one generation, cared about "more protein" a little more than it cared about "soy free." Nothing said that couldn't happen again tomorrow, to a household nobody's teenager happened to be watching.

The ticket count never moved. The real number underneath it wasn't small, it was just never asked.

What Aurelia did next wasn't ask for more review, or tell Baako to be more careful. The product had no memory of how often this shape of mistake happened, so she built one herself. For three weeks she pulled every Freeform Fridge request that combined an allergy tag with a "more protein" style ask, about 150 of them, and read them by hand at her kitchen table most evenings. Two more had swapped in a soy ingredient for a soy-free household. Neither had ever been reported. Two in 150. About one in 75.

Scaled across the full 14 weeks, at roughly 50 of those requests a week, that's about 700 chances for this specific mistake to happen, and a 1-in-75 rate says it probably did, about nine times. Only one had ever come to light, and that one came down to a teenager glancing at a screen before paying, not to anything Larderline built to catch it.

The whole answer sits in this one picture. A dial has room to nudge. This never had a middle setting.

The decision Aurelia would take back traced to launch day. When Freeform Fridge shipped, nobody sat down and decided to keep the fixed-recipe engine's ticket-by-ticket QA process. It just kept running, because it had always run, and it had always worked. That made sense right up until the day a wrong output stopped having one findable cause.

Run the same Thursday again, with the fix in place. Freeform Fridge still occasionally weighs "more protein" over a safety tag, because that's what a probabilistic model does sometimes. But now a held-out eval set, stocked with exactly this shape of case, catches most of them before a release ships at all. Anything that slips through gets caught the instant it's generated, by a hard-coded allergen check that runs after the model, checking the plan against the tag no matter how confident the model looked. And the rate itself gets watched every week against a bar, fewer than 1 in 5,000, not discovered three weeks later at a kitchen table.

One design let "fixed" mean one ticket closed. The other lets "fixed" mean a number you can actually defend.

What Aurelia would tell herself, back on the Monday Freeform Fridge launched: a QA process built for a world where every wrong answer has one cause was never going to survive a world where most of them don't, and nobody had written down what should replace it when that day came.

FLIPS in five moves, not five slides

Not a story wearing a framework's clothes. This is what to actually run, in order, any time a question asks what changes once a feature stops being deterministic.

Five steps, one hard one. I is where a flat answer about "AI being different" becomes a real one.

Two things worth naming directly. The alternative Milkwood actually considered instead of building an eval set was cheaper and faster: keep a running blocklist of banned ingredient words per allergy tag, so "tofu" could never appear on a soy-free household's cart. That got rejected, because a blocklist only catches the literal word. It misses soy sauce, miso, edamame, tempeh, and textured soy protein, so it would have looked like a fix while missing most of the real risk category, and everyone would have relaxed, believing the problem was solved. The AI-specific failure worth naming is an instruction-weighting slip: nothing in the code was wrong, the model simply weighed "more protein" more heavily than the soy-free tag on one generation, which has no single line to patch. The guardrail is a deterministic check that runs after the model, not instead of it: every generated plan gets its ingredients checked against the household's allergy tags, and anything that matches gets blocked, no matter how confident the model looked when it wrote the plan. That check costs something real, a fraction of a second added to every single plan Larderline generates, in exchange for a floor under the allergy-substitution rate that the model's own reasoning can never fully promise on its own.

And if you want to be sure it really works, try it somewhere else

Same five letters, a poison control line instead of a kitchen, and this time the flip isn't a private tally. It's what the frontline staff start doing to the input before the model ever sees it.

Ringcall is the triage tool at Rooksmoor Poison Center: a caller describes what happened, and Ringcall reads the description and scores how urgent it is, replacing a rigid decision-tree script that used fixed thresholds on substance and amount. Ansel Bergqvist, the charge nurse, owns how Ringcall's severity scores get reviewed each shift.

Same framework, a genuinely different flip. Milkwood's problem was a missing memory. Rooksmoor's is what the input looks like by the time the model sees it.

F Ansel Bergqvist, charge nurse, reviews Ringcall's severity flags every shift. L He stopped re-listening to call recordings behind a "moderate" flag, since Ringcall's typed summary had matched reality for months. I A different flip entirely, the pre-editing flip: intake nurses started retyping what panicked callers actually said into a clean one-line summary before entering it into Ringcall, because a messy, rambling transcript sometimes tripped Ringcall into a low, wrong severity read. Cleaning it up strips out exactly the panic, the hesitation, the repeated phrase, that often signals how bad a case really is. P Ringcall was evaluated entirely on clean, well-formed transcripts, so the acceptance bar only ever asked "does it match severity on this tidy set," never "how does it do on a real, panicked, three-in-the-morning call." S The replay: evaluated against a held-out set of genuinely messy raw transcripts, with a "not sure, ask a nurse" fallback instead of a silent guess whenever confidence is low, Ringcall flags the ambiguous, panicked calls for a live read instead of confidently scoring them wrong, and nurses stop needing to clean up every transcript themselves before they dare trust it.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: triage becomes a shared eval set, the acceptance bar becomes a rate, and every release gets tested against the whole set, not one exact diff.
Cost: there's no budget this quarter to build a full eval-set pipeline. Ship the cheap version first: a weekly manual sample of fifty risky-combination requests, read by a person, tracked in a spreadsheet, revisited once real tooling exists.
The model got better, for real: say Freeform Fridge's next version cuts the real rate from 1 in 75 to 1 in 900. The three decisions don't change. A better model just moves where the needle sits on the same rate you're now able to see, it doesn't excuse going back to a ticket queue with no memory.

Where people run it wrong.
They treat a patched ticket as proof the underlying rate is fixed, instead of proof that one instance is fixed.
They build a keyword blocklist and call the risk category closed, when it only ever caught the literal word.
They write the acceptance bar as "no mistakes," which sounds safer and measures nothing.

How to use it live. Ask one question before trusting any "fixed" for a probabilistic feature: "Was that patch tested against a single case, or against everything that looks like it?" That question alone usually tells you whether you're looking at a real fix or a closed ticket.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0