Describe a framework for triaging inbound AI feature requests from across the company.

Coalfield Group's internal AI platform team owns Ledgerbird, a tool that reads every line on an expense report and sorts it into the right budget category for finance. It works well enough that requests for more of it pour in from every direction: sales, legal, and a newly acquired Antwerp freight firm all want a piece of the same small team's time. Product manager Evanthe Beaumaris used to answer whichever request had escalated most recently. One of those requests went out fast, on a promise made to the board, and it broke quietly for a month before anyone noticed. She needed a real system, not a longer list.

How to answer this, stage by stage

Nobody is grading whether Evanthe can describe a ticket queue for five minutes. They're grading whether the system she names would actually survive the next loud request landing in her inbox.

Let's learn

Ledgerbird is a tool that reads a line on an expense report, a taxi ride, a hotel bill, a client dinner, and puts it in the right budget category for finance, without a person doing it by hand.

Before Evanthe's team built any of it, a Coalfield Group finance clerk sorted about 4,000 expense lines a month by eye, close to three hours a day matching a receipt to a code. Ledgerbird cut that to a few minutes of spot-checking, and it worked well enough that four divisions were running it inside a year.

That's when the requests started. Sales wanted big client-entertainment spend flagged for account risk. Legal wanted policy violations caught automatically. A freight firm in Antwerp, Verhaeghe Forwarding, newly bought by Coalfield, wanted their own books on it entirely. By July, sixty-one requests sat in a shared doc with no order to them at all, and Evanthe answered whichever one had escalated most recently that week.

Here is the turn. The extra requests piling up were never the real problem. The real problem was what happened once Evanthe said yes to one of them too fast.

Verhaeghe's finance controller, Freydis Prost, had about nine weeks of migrated expense history when the request went out, roughly 340 labeled lines, and the old system's categories didn't line up cleanly with Coalfield's chart of accounts. Coalfield's core model runs on 14,000 labeled lines built up over four years and holds 97% accuracy against its own eval set. Verhaeghe's version shipped anyway, because a VP had already told the board it would be live by September. On a 120-receipt audit sample the following month, it was right 61% of the time, and it never once said it wasn't sure.

The check that never ran before the Antwerp request shipped.

At its worst, a model with too little data to learn from doesn't fail loudly. It answers every single line with the same flat confidence, right or wrong, and nothing on the screen tells anyone which is which. That's a more expensive kind of broken than a tool that simply stops working.

What I would leave alone: the currency-conversion step, plain arithmetic on a daily exchange rate, doesn't need any of this. It isn't a model guessing at intent, it's a lookup and a multiply, and it's been correct every single time anyone has checked it.

The lesson: a board update is not evidence. Treating a promise as if it were proof is how a real AI team ends up building the one request it was least ready to build, right when the whole company is watching.

Now here is the same thing as a story

The short version above is what you'd actually say in the room. Read this one for the eight weeks nobody at Coalfield was watching what the shortcut cost Freydis, receipt by receipt.

Freydis Prost has kept Verhaeghe Forwarding's books for eleven years, since before it was Coalfield's to buy. She knew every driver's usual fuel stop and every client's usual dinner spot well enough to sort a stack of receipts without looking twice.

Ledgerbird's Antwerp extension arrived in July, four months after the acquisition closed. For the first two weeks it looked, if anything, better than she expected. Fuel, tolls, hotels, the easy stuff, it sorted cleanly, and Freydis checked every single line anyway, because the tool was new and she didn't yet know what to trust.

By week four she was checking maybe one line in five. By week six, she skimmed the totals at the bottom of the report and moved on. Nobody told her to stop checking. The tool never once flagged a line as unsure, never showed a gap where a guess might be wrong, so there was nothing in front of her that looked like it needed a second look.

Nobody decided to stop checking. A tool that never looks unsure makes that decision for you.

In week eight, Coalfield's regional auditor pulled a routine sample of forty Antwerp receipts to close out the quarter. Nineteen were miscategorized, several badly enough that the wrong VAT treatment had already gone out on filed paperwork. A dinner with a supplier had been filed as internal training. A warehouse repair had been filed as travel. None of it looked wrong on a screen that never hedged.

Freydis wasn't careless. She did the sensible thing every single week: a tool that never once flagged a doubt earns less checking over time, not more. That's not a flaw in her. It's what any reasonable person does with a tool that never admits uncertainty.

The auditor's email reached Evanthe on a Tuesday. She didn't call it a model bug in the postmortem, because it wasn't one. Two years earlier, on a different request, she'd made the opposite mistake: she'd frozen every new AI request for a full quarter while the team "figured out a process," and the backlog that built up in that quarter took another two quarters to work through. She wasn't going to swing back to that either.

So instead, she pulled the full request list, all sixty-one of them, and ran each one through a real check for the first time: a stated problem, a stated data floor of 2,000 clean labeled lines before any model work started, and a tier based on what the request would actually cost to get wrong. Verhaeghe's own re-do went out nine weeks later, once enough clean data existed, alongside a plain crosswalk table that covered the gap in the meantime. On the same 120-receipt audit sample, the rebuilt version scored 94%.

One design let a confident wrong answer walk straight into a VAT filing. The other made Freydis wait nine weeks, and then actually trust what she got.

Here's what I'd tell myself, the day we fast-tracked that request: a board update is not evidence. I mistook a promise for proof, and Freydis paid for it in VAT corrections, not me.

Five letters Evanthe now checks before saying yes

PICK would fit if this were one tradeoff with a clean side to pick. Here there are sixty-one live requests competing for the same few engineers, and the job is sorting all of them by what's hardest to undo if the call is wrong. That's ORDER's job.

Five checks a request clears, in order, before it gets real engineering time.

Reversibility isn't a reason to avoid the harder call. It's a reason to check the data before making one.

Every box after the second one only means something once the second one is actually done.

One alternative is worth naming and rejecting directly: building Verhaeghe's categorizer as a full custom model from day one, which is what actually shipped. It lost, in hindsight, to a plain crosswalk table that could have covered the same gap in a week without any model at all, while real data accumulated behind it. The AI-specific failure worth naming is a model trained on too little data answering with the same flat confidence whether it's right or wrong, a kind of silent, confident wrongness that has no equivalent in a report or a button. The guardrail is the 2,000-line data floor plus a two-week shadow-mode check against a held-out audit sample before anything ships live. And the trade-off is accepted on purpose: a well-evidenced request can now wait up to two weeks for a formal decision instead of an instant yes, in exchange for never again burning a sprint on an AI feature that was never going to be reliable.

And if you want to be sure it really works, try it somewhere else

Same five letters, a veterinary call center instead of a finance team, and the honest answer this time is to say no.

Barrington Veterinary Partners runs TriagePaw, a tool that reads a pet owner's description of symptoms over the phone and sorts the call by urgency for front-desk staff at sixty-plus franchise clinics. Product manager Tuulikki Loach gets requests the same way Evanthe does: every clinic group has an idea. One regional director wants TriagePaw extended to exotic pets, reptiles, birds, rabbits, in time to announce it at the annual franchise conference.

The request in the top-left corner is the one worth saying no to, not the one worth speeding up.

Same steps, mapped onto Barrington. Outcome: protect against a real emergency getting sorted into the routine queue, not against looking slow to a regional director. Reversibility: a stage announcement at the conference is far harder to walk back than an internal pilot with no date attached. Dependency: the system holds only 85 logged exotic-pet cases network-wide, against the same kind of data floor that stopped Verhaeghe. Evidence: exotic-pet calls run about 40 a month across all sixty clinics, a small slice of total volume, and a manual routing rule already sends every one of them straight to a trained vet tech, a fix that already works. Rank: this one goes in the "not yet" tier, the conference announcement gets turned down, and the manual rule stays exactly as it is until real volume and real data justify building anything at all.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to Rank, name the tiers and the cadence, everything else is support.
Cost: there's no budget to check the labeled-data count properly before the meeting. Say so plainly, and use whatever's cheap and real, a quick count from the existing database, rather than assuming the data is probably fine.
The model got better, for real: a newer base model turns out to need far less labeled data to hit the same bar. Say that too, plainly, and move the request up a tier. The method never says never revisit, it says decide from real evidence either way.

Where people run it wrong.
They let a loud escalation skip the Dependency check entirely, the way Evanthe first did.
They build the ambitious version first and check for enough data only after something breaks.
They never revisit the queue on a real cadence, so it quietly turns back into an ever-growing, unranked list.

How to use it live. Before saying yes to anything, ask out loud: "do we actually have enough real, labeled examples for this, or are we hoping the model figures it out?" If the honest answer is "we don't know," that's the whole Dependency check, and it's reason enough to hold the request in queue until someone does.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0