Design an eval spec live for a feature I describe.

How to answer this, stage by stage

Eight moves. Ground it in one real cart, since the interviewer is describing the feature live.

Let's learn

Here is what happens when a recipe gets every rule right, and still isn't something a person would ever cook.

Say we build Cook This, a feature inside Basketwise's grocery delivery app. It looks at whatever's already sitting in your cart and hands you three recipes you could make out of it, right as you're about to check out.

Checking a drafted recipe, before the rubric existed

Before the rubric, Cook This was checked against one bar: does the recipe use items from the cart, and does it break none of the shopper's flagged allergies. On the first ten test carts anyone ran by hand, every recipe that passed also looked fine to eat. So the team shipped the bar and moved on.

The regression suite grew. Forty test carts, three recipes each, a hundred and twenty cards a run. Nobody was reading all hundred and twenty by eye anymore. The bar was still doing the checking.

Here is the important part. The recipes that slipped through weren't factually wrong. They used the cart's items. They broke no rule. The real problem was what a person did when they actually read one: they stopped, stared, and said out loud that nobody would make this.

At its worst, that cost more than one odd recipe card. Reading all hundred and twenty cards by hand, to catch the ones like it, took a full day before every release. And once, it still wasn't enough. One slipped past the manual read and reached a beta tester, who screenshotted it and posted it, right as the team was heading into a launch review.

The choice I would take back. When Cook This first shipped, the team built one bar: uses the cart, breaks no allergy rule, ship it, tune the taste later. That held up fine when someone read ten test carts by eye every morning and nothing weird got past them. It stopped holding up once the regression suite grew to a hundred and twenty cards nobody was reading one by one anymore.

What I would leave alone. A cart with exactly spaghetti, marinara, and parmesan doesn't need the expensive plausibility check. Any recipe that names those three together is obviously right. Spending review time there takes it away from the carts where the model actually has to guess.

The lesson. Following the rules isn't the same as making sense. A recipe can pass every rule you wrote and still be something nobody would ever cook, so you have to check for "would a person do this" on purpose. It never shows up for free.

Now here is the same thing as a story

Read this one when you've got a few minutes. The short version is above. This is for when you want to feel why it mattered.

Meera Sundaram ran product for Basketwise's recommendations team. She could scan a recipe card and know in about two seconds whether it was a keeper, long before Cook This existed, back when the team still tested every idea by hand.

Cook This launched in the spring. For the first few weeks, it was the best thing on the roadmap. A shopper's cart filled up, three recipe cards appeared, and more people than anyone expected actually tapped one and cooked it that night. The team ran ten test carts by eye every morning before each release, and every one of them looked like food.

So they stopped reading ten. They read three. Then, most weeks, they read whatever the automated bar flagged, which was almost nothing, because the bar only checked two things: did the recipe use the cart, did it break no allergy rule.

Then came the audit.

Before a wider launch review, Basketwise's head of product pulled a random sample from the regression suite, the same forty test carts, run fresh, a hundred and twenty recipe cards. One of them, drawn from a cart holding chicken breast, a jar of marinara, a box of spaghetti, and a bag of shredded mozzarella, was titled Marinara Mozzarella Overnight Oats.

The anchor: three bars, one fail is enough

Nothing about it broke a rule. It used marinara. It used mozzarella. It technically touched the chicken. No allergy flag anywhere. It sat cleanly inside the sample slide, one recipe among many, ready for the leadership review.

I want to say the problem was that Cook This missed an ingredient, or broke a safety rule. It didn't. Every fact in that card was true. But nobody had ever given the model, or the eval, a specific way to check the other thing, the thing Meera had never had to write down because she'd always just known it by reading the card herself: would a real person actually make this.

Meera pulled the wider launch back a week. Someone spent a full day reading all hundred and twenty cards from that batch by hand, the way the team used to read ten, and found sixteen more like it. Marinara sauce dressing a salad meant for vinaigrette. A stir-fry recipe that skipped cooking the chicken through. Seventeen cards, out of a hundred and twenty, that the single bar had waved through clean.

So here is the decision she made differently.

Months earlier, in a stand-up, the team had agreed: check that it uses the cart, check that it doesn't break an allergy rule, ship it, we'll tune the vibe later. That made complete sense when someone read every test cart themselves each morning and caught anything strange before it ever reached a slide. Nobody had planned for a hundred and twenty unread cards sitting in front of a leadership review.

Meera built the rubric instead. Not one score. Three bars, scored apart: ingredient use, does each item play a real role, not just appear somewhere in the text. Plausibility, would a person actually make this. Dietary safety, right cook times, no unsafe combos, respects what the shopper flagged. A recipe had to clear all three, not average out to fine.

Checked against its own risk, both directions

Run the same batch forward with the rubric in place. A hundred and twenty cards, checked against three bars before anyone reads them by feel. Two get flagged, both genuinely borderline, both fixed inside the hour. The full day of manual reading is gone. The rubric found in an hour what a person used to need a day to catch.

One design trusts that "breaks no rule" is the same as "makes sense." The other checks for both, on purpose, separately.

And the thing I'd want to tell myself, back when we wrote that first stand-up decision: I built a check for what could go technically wrong. I never built one for what would look ridiculous on a slide, and ridiculous was always going to be the one that got there first.

SPARK, one cart at a time

This question asks for a live design of an eval spec, not the feature itself, but it's still one concrete decision about the exact moment a recipe passes or fails, so SPARK still fits. A question asking how to measure Cook This's overall recipe quality across every cart in the app would reach for LEAD instead.

And if you want to be sure it really works, try it somewhere else

A public library's reading recommendations run on a completely different shelf, but the same gap between technically matched and actually right shows up in a suggested next book.

S. Colm Radley runs digital services at Cranmoor Public Library, where a feature called Next Read looks at a patron's checkout history and suggests a book to try next. Today, without a rubric, whoever tests it checks one bar: does the suggestion share a subject tag with something the patron borrowed, and is a copy on the shelf.
P. The habit worth building: a patron picks up the suggested book and actually reads past chapter one, not a book that's technically related by tag but wrong for who's reading it.
A. Same shape, different bars. A seven-year-old checks out a gentle picture book about a rescue puppy. Next Read suggests a true-crime book about a police K9 unit, tag matched on "dogs," in stock, technically valid. The rubric adds age fit and tone match as their own bars, alongside tag relevance.
R. Tighten the reading-level bar to require an exact match every time, and it stops offering the "stretch" recommendations librarians actually want to make, a strong young reader ready for something a notch harder. Loosen it to tag match alone, and the K9 book passes again, related by subject, wrong for a seven-year-old.
K. No attempt yet to model tone across every genre crossover on day one. Age fit and tag relevance first, since that's where a wrong suggestion costs the most trust, refine tone matching later.

Same anchor, a different desk, tags and age instead of ingredients and dietary safety

It took a parent calling the branch to ask why a true-crime book had been suggested to her second grader, right after the picture book about the rescue puppy, before anyone noticed the tag match alone wasn't enough of a check.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0