Describe the maintenance burden of an AI feature that a simple heuristic would not carry.

Loworth Labs builds Quiet Inbox, an AI tool that files, sorts, and prioritizes a full inbox automatically. Xylia Ondrusek is the AI PM who owns its filtering roadmap. Emmerich Vallance, Loworth's CEO, wants every rule left in the product swapped for a model, to match how competitors market their inbox tools. Delvina Buranelli, an independent freight broker, depends on one particular filter to catch load offers from brokers she's never heard of before, the kind a plain rule can't recognize by sender alone.

How to answer this, stage by stage

Nobody is scoring whether you can recite four bullet points about drift and eval sets. They're scoring whether you'd have caught this before the ticket ever said done.

Let's learn

Quiet Inbox is a tool that reads a full inbox and automatically files, prioritizes, and archives it, so a person isn't sorting hundreds of emails by hand every morning. One of its filters, built for freight brokers, is meant to catch genuine load tender emails, the ones offering an actual shipment, and pull them out of the noise.

Before any model, this is a rule, plain and readable. Anyone could print it out and follow it by hand.

Delvina Buranelli works this filter every morning. She has about 35 brokers she deals with regularly, and the rule catches 92 percent of their tenders without her lifting a finger. But she also picks up spot market loads from brokers she's never worked with before, roughly five a week. Because their domains were never on the list, the rule only caught about 1 in 5 of those. The other four went straight into a folder she rarely opened.

So a small team built a classifier that reads the body of the email, not just who sent it, and routes anything that reads like a real tender to Delvina's priority list. It launched catching 89 percent of tenders from brokers she'd never heard from before.

Here's the part that isn't about accuracy at all. A rule doesn't need anyone watching it once it ships. A model does, quietly, forever. Someone now has to check whether it's still doing what it did on day one, whether the examples it was checked against are still current, whether the company that hosts it just changed something underneath.

At its worst, this cost showed up as a full workday every week that belonged to nobody's plan. An engineer, Jorick Groenveld, ended up spending 25 to 30 hours a month on this one filter a year in: chasing a slow accuracy slide nobody had flagged, relabeling a fresh batch of tender examples every quarter, and once, migrating the whole thing off a model version the provider was retiring with 60 days notice.

The actual decision this answer is about. Not a UI element. A ledger that has to exist before the model ships, not after it breaks.

What I would leave alone: the spam and newsletter filter never got touched. It's still a plain rule matching sender patterns and a few keywords. Marketing platforms don't quietly change their sending patterns the way freight tender language shifts season to season, so that rule just needs a new entry now and then. No drift to watch, no eval set to refresh.

The lesson: choosing a model over a rule isn't a one time build decision. It's opting into a bill that keeps arriving, monitoring, relabeling, migration. Pay that bill on purpose, only where the rule genuinely can't do the job, not because the model sounded like the more modern choice in a planning meeting.

Now here is the same thing as a story

The short version above is what you'd say out loud in the room. Read this one for what it actually felt like to watch a "done" ticket quietly turn into somebody's second job.

Jorick Groenveld had shipped features at Loworth Labs for four years, and he was good at it. Ship it, watch it for a bit, close the ticket, move to the next thing on the board. That was the rhythm, and it worked.

The tender classifier launched in a sprint like any other. For the first month, Jorick opened its accuracy dashboard every Monday morning, saw it holding around 89 percent, and moved on with his day. In the launch retro, someone floated the idea of a recurring drift check and a quarterly relabel cadence. The room was tired and the next quarter's roadmap was already full, so they voted to skip it. Freight language doesn't change that fast, somebody said, and nobody argued.

Weekly checks became monthly ones. By month three, Jorick only opened the dashboard when something else reminded him it existed.

There was no single bad Tuesday. That's the part that's easy to miss. A new online load board started sending auto generated tender emails in a format the classifier had barely seen in training. Around the same stretch, the accuracy on unfamiliar senders quietly slid from 89 percent down to 84. Nobody was looking closely enough to catch either one as it happened.

Nothing here looks like a failure while it's happening. That's exactly what makes it expensive.

What surfaced it wasn't a dashboard. It was three separate customers, in the same week, reporting missed tenders through support. Jorick had to reconstruct four months of drift by hand: pull a fresh batch of real examples, get them labeled correctly, retrain, and verify. Halfway through that, a deprecation notice landed for the underlying hosted model. Sixty days to migrate before the old version went dark, in the middle of everything else.

What had been "check the dashboard if someone complains" became "own this filter, indefinitely, on top of everything else." He started blocking something close to a full day a week for it. Every week. Forever, unless something changed.

He never had a number in his head for when this crossed a line. He had a feeling with exactly two settings: this is done, or this needs a person full time. The slow slide from 89 to 84 didn't flip it. Three customers complaining in the same week did.

Same ticket. The only thing missing at launch was a line that said who watches this, and how often.

Back in that launch retro, the decision that mattered wasn't obviously wrong at the time. Skipping a drift check schedule to protect the next quarter's roadmap is a normal, defensible call when a feature is brand new and nobody's load bearing on it yet. It stopped being defensible the exact week Delvina started trusting the classifier enough to stop checking her Other folder herself.

Here's the replay, same classifier, same eventual deprecation notice, but with the ledger written on day one instead of skipped.

Six to eight hours a month, blocked on a calendar from week one, instead of a surprise week twelve months later.

With the ledger in place, Jorick checks drift on a set schedule instead of by accident, catches the load board's new template in week three instead of month four, and gets the deprecation notice while there's still slack in his month to plan around it, not scramble through it. Same model, same real world shift underneath it. The only thing that changed is that somebody wrote the cost down before it arrived.

One design handed the team a bill nobody opened until it was already overdue. The other handed them a bill they signed for on day one, in hours they'd already agreed to spend.

What I'd tell myself, back in that retro: "done" is a fair word for a rule. For a model, the honest word is "owned," and if nobody's name is on that line, the bill goes to whoever's desk happens to be closest when it finally arrives.

SPARK: five moves for weighing what a model costs to keep, not just to buildNot a script for sounding thorough about AI maintenance in the abstract. SPARK is what forces the actual ledger out, and proves it survives the year nobody's watching closely.

Emmerich wanted the middle panel. The reasoning in this answer is exactly why the team didn't build it.

The recap, one line per letter: situation is a rule's real catch rate written down honestly, payoff is the habit of costing a model's upkeep before its build, anchor is a maintenance ledger on the ticket itself, risk is an unbudgeted bill landing on whoever's desk is closest, and keep out draws the line at the one judgment call that actually needs a model, leaving the rest as rules.

And if you want to be sure it really works, try it somewhere elseSame five letters, a veterinary booking form instead of a freight inbox. The judgment call this time isn't a load tender. It's whether a symptom description sounds like an emergency.

Bramblecoat Veterinary is a two person practice run by Kenrick Onuora. Its online booking form used a keyword rule for years: words like bleeding, can't stand, or won't eat flagged a booking as urgent for same day review. Fernpath Systems, which builds the booking software, pitched Kenrick on TriageLine, a model that reads the whole free text description instead of matching a fixed list, catching cases a first time pet owner might describe as "he's been really weird and floppy since this morning" with no keyword match at all. Mapped onto SPARK: situation is the keyword rule's real record, solid on obvious cases, blind to unusual phrasing. Payoff is the same habit again, cost the upkeep before picking the model, not after. Anchor is the same kind of ledger: for TriageLine, drift as pet owners describe symptoms in new ways season to season, a quarterly eval refresh using real vet reviewed cases, tracking Fernpath's own model version updates, and triaging occasional strange misfires, like a routine dental note flagged urgent because its language overlapped an unrelated emergency post in training. Risk is that a two person clinic doesn't own the model, Fernpath does, so if nobody at the clinic asks what gets watched and how often, that cost just becomes Fernpath's problem to under staff quietly across every clinic using it. Keep out is that the "which vet is free" scheduling logic stays a plain rule. Only the actual symptom urgency judgment, where free text genuinely varies, earns the model.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Skip straight to the anchor, say the four row ledger out loud, and stop.
Cost: no budget for a dedicated model owner at all. Keep the rule. A filter that only catches 70 percent, cheaply and predictably, can beat a model that catches 90 percent but nobody's paid to keep honest.
The model got better, for real: say a future version drifts far less and needs fewer eval refreshes. The ledger still matters, because "what does this cost to keep right, six months from now" is a question about the task and the team, not about how good any one model happens to be this quarter.

Where people run it wrong.
They budget the model like a one time feature instead of a subscription, so the ongoing hours never make it into anyone's plan.
They let a vendor's promise of "we handle maintenance" stand in for actually asking what gets watched, how often, and by whom.
They swap every rule for a model to sound more advanced, instead of asking which specific judgment call actually needs a model's flexibility.

How to use it live. The moment an interviewer asks you to compare a model to a simple rule, don't start with accuracy. Ask yourself first: what does each option cost to keep right, six months from now, with nobody thinking about it. That question buys you a real beat to think, and it's usually where the honest answer lives anyway.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0