Describe how you would build a model bake-off for your specific use case.

Here's what happens when a model gets chosen off a fifteen-minute demo instead of a real test. Halvergate County Permits Office built CodeReader, a tool that reads a submitted building permit application and flags which zoning and fire-code sections a reviewer should check first. Rosalind Achterberg runs the program, and inherited the job of picking CodeReader's model from a predecessor who left two weeks after the demo happened.

How to answer this, stage by stage

Nobody is scoring whether you know the word "bake-off." They're scoring whether your test set would actually catch something a fifteen-minute vendor demo wouldn't.

Let's learn

Say we build a tool that reads a permit application and tells a reviewer which sections to check first.

Before CodeReader, a Halvergate plans examiner read every submitted permit cover to cover, checking it against zoning maps and fire code by hand, about 35 minutes for a straightforward single-family permit and well over an hour for anything with a variance or an addition.

With CodeReader, a submitted permit gets flagged in under 20 seconds, telling the reviewer which sections likely need a closer look.

Four manual tasks, all riding on one examiner's memory of the whole code book.

Here's the turn: the real risk was never that CodeReader would be obviously wrong. It's that a model can look perfect on five clean demo cases and still have never actually been tested on the messy ones that make up a real counter's daily backlog.

The actual design decision behind the bake-off, drawn as five steps you could run this week.

At its worst, a permit office trusts a model that quietly clears a whole category of case, mixed-use conversions with hand-marked changes, without a single flag, and nobody notices until a building goes up with the wrong setback.

What I would leave alone: for straightforward single-family permits with no variances, which make up most of Halvergate's volume, CodeReader's flags have been reliable from day one, and I wouldn't add extra review steps there.

The lesson: a bake-off isn't a formality before you sign. It's the only thing standing between a model that looks perfect and a model that's only ever seen five perfect cases.

Now here is the same thing as a story

The short version above is what you'd say in a vendor review. Read this one for what it felt like the months nobody had drawn a clear line between watching a demo and trusting a model.

Rosalind Achterberg had reviewed permits herself for nine years before moving into the program manager seat, and she still knew a dangerous mistake from a cosmetic one on sight.

When CodeReader's vendor demo ran, it looked clean: five straightforward single-family permits, flagged correctly in under 20 seconds each. For the first three months after go-live, feedback from the front counter was good, examiners liked how much reading it saved them.

No single moment marked when it started going wrong. It built up slowly, permit by permit, and nobody could later say exactly which week it changed. Examiners kept trusting the flags for permit types nobody had actually tested the model against, because nothing about it looked different from the types that were working fine.

A demo is built to look effortless. Your real counter was never going to be that clean.

Three months in, a senior examiner reviewing a mixed-use conversion, an old house being split into two rental units, noticed CodeReader had cleared it with no flags, despite a hand-marked setback change on page four that clearly needed a zoning check.

Rosalind pulled every mixed-use permit CodeReader had processed since launch, 40 of them, and had a senior examiner manually re-check each one. Sixteen, 40 percent, had at least one real issue CodeReader missed entirely.

The real question was never whether CodeReader's model was good. It was whether anyone had ever tested it against a permit that looked like the ones actually piling up at Halvergate's counter, instead of the five the vendor chose to show.

Same mistake. One version costs three months. The other costs one afternoon of testing.

When the original rollout plan was drawn up, someone said, "the demo looked clean, let's move to contract," and it sounded reasonable, since nobody yet knew mixed-use conversions would be where the gap was hiding.

Rerun the same 40 mixed-use permits with a real bake-off run first: the 16 missed issues surface in the first afternoon of testing, on a 200-case set built from Halvergate's own backlog, three months before any of them would have shipped silently.

What I'd tell myself, hearing that senior examiner describe the missed setback: the demo was never lying. It was just never asked the question that actually mattered.

SPARK, applied to designing the bake-off itselfNot a script for distrusting every vendor. SPARK is what makes sure the test, not the demo, is what earns your trust.

And if you want to be sure it really works, try it somewhere elseSame five letters, a recycling facility instead of a permits counter. What "messy" means changes, the design still turns on it.

Anders Kettering runs operations for Kettering Materials Recovery, which uses SortCheck, a model that reads a conveyor camera feed and flags contaminated bales before they're baled and shipped. Mapped onto SPARK: situation is a sorter manually pulling suspect items off the line by eye, missing about a fifth of contamination in fast-moving mixed loads. Payoff is trusting an automatic flag enough to pull an item without double-checking it first. Anchor is the bake-off itself: 300 of the facility's own recorded conveyor clips, weighted toward its messiest mixed-material loads, scored blind by a veteran sorter's rubric before cost per camera or processing speed even enter the conversation. Risk is a contaminated bale clearing the line silently and getting rejected by a buyer weeks later, with no record of which clip caused it. Keep out is settling scores on hazardous-material contamination, since the facility only has four recorded incidents, far too few to trust any model's number on that category yet.

Plotted together, the cheapest option and the best deal turn out to be two different points.

Naming what the bake-off won't decide yet is as much a design choice as naming what it will.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "build the test from your own messiest real cases, score blind against a rubric written with a real reviewer, then compare cost," and stop.
Cost: no budget for 200 real cases before a decision has to be made. Say so honestly, and commit to even 40 of the messiest cases you can pull cheaply before trusting a live demo alone.
The model got better, for real: if a vendor claims an updated model fixes the exact gap you found, that's still worth re-running against your own messiest cases before trusting it, since "fixed" and "fixed on your specific gap" aren't the same claim.

Where people run it wrong.
They let the vendor pick which cases the demo shows, instead of bringing their own.
They build the rubric alone in a meeting room instead of with someone who actually does the job.
They compare cost and speed before any candidate has cleared a real quality bar.

How to use it live. The moment an interviewer asks you to design a bake-off, ask yourself: what's the messiest real example this team has, and is it actually in the test set? If the honest answer is "no, we'd probably use the vendor's demo cases," that's the gap worth naming out loud.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0