How would you build an internal eval platform that other teams actually use?

Interviewer's question: "How would you build an internal eval platform that other teams actually use?" Gullford Grocers runs a regional chain of supermarkets. Ines Delacroix built ProofBench, the company's shared eval platform, starting with the loss-prevention team.

How to answer this, stage by stage

Nobody is grading how many platform features you can list. They're grading whether you can say which piece breaks everything else if you build it out of order.

Let's learn

ProofBench is a shared platform that runs a team's AI feature against a set of real cases with known correct answers, and reports whether it passes.

Before it, each team at Gullford tested their own model informally, spot-checking a handful of outputs by eye before shipping. Ines built ProofBench's first version around the loss-prevention team's shrink-detection model, using actual flagged incidents from the past year as the test set.

The turn: the climbing pass rate was never proof the support bot was improving. It was proof the support team had learned which test cases made the number go up, which is a completely different skill from making the product better.

The first step is the only one that doesn't depend on anything else existing yet.

At its worst: a team spends five months pruning its own test set instead of fixing its bot, reports a pass rate near ninety-six percent, and the actual production error rate never moves, because the two numbers stopped describing the same thing months earlier.

All three were true on Gullford's support team, for months, before anyone noticed.

What I would leave alone: the loss-prevention team's original golden set, pulled from real flagged incidents, never needed this level of concern. Nobody had an incentive to prune it, since it was built from real losses the company had already suffered, not a number anyone was being measured against.

The lesson: a shared eval platform's biggest risk isn't a wrong verdict on day one. It's a slow, plausible-looking drift where the number improves and the product doesn't, and nothing about the dashboard tells you which one is happening.

Now here is the same thing as a story

The short version above is what you'd say pitching ProofBench's build order to Gullford's engineering leadership. Read this one for how the drift actually got caught.

Ines Delacroix built ProofBench's first version around a real, painful problem: the loss-prevention team's shrink-detection model kept missing a specific kind of theft pattern, and nobody had a reliable way to measure whether a fix actually worked.

That first golden set, built from real flagged incidents, earned trust fast. Within three months, the loss-prevention team was gating every model change on ProofBench's verdict. Word got around, and the customer support team, building a bot to draft first-pass replies to routine complaints, asked to onboard next.

The support team's onboarding went smoothly, on paper. Their pass rate climbed steadily for months. Then a new hire, three weeks into the data team, pulled up both the ProofBench dashboard and the actual customer complaint reopen rate, side by side, out of simple curiosity.

One of these you can still walk back easily. The other, once it happens, doesn't undo.

"Why does the pass rate keep going up," she asked in a team meeting, "when the reopen rate on customer complaints hasn't moved at all?" Nobody had an answer. It turned out the support team, watching their own number sit on a company-wide leaderboard next to loss-prevention's, had quietly been removing the ugliest, hardest-to-handle complaint examples from their test set every time the bot failed one, rather than fixing the bot.

Here's the decision I'd take back: putting every team's pass rate on one shared, leadership-visible dashboard from the start. That felt like the transparent, healthy choice when only loss-prevention used the platform and nobody had reason to game a number built from real losses. It stopped making sense the moment a second team's incentive to look good outweighed their incentive to actually be good.

Replayed with pass rates kept private to each team's own leads, visible company-wide only as "onboarded, gated, and passing" or not, with no comparable number to game: the support team's test set stays intact, since there's no leaderboard rewarding a curated one. The reopen rate and the pass rate move together instead of apart, and the new hire's question never needs asking.

I built the shared leaderboard because I wanted every team to feel proud of using the platform, the same way loss-prevention did. It took one honest question from someone three weeks into the job to see that pride and gaming can look identical on the same chart.

ORDER, mapped onto one platform's rolloutNot a feature roadmap. ORDER is what tells you which capability breaks everything else if you build it first.

The fourth callout is the one Gullford's support team quietly broke, one deletion at a time.

The leaderboard looked cheap and safe to build early. It was the one that quietly broke the platform's honesty.

The recap, one line per letter: outcome is real gating versus quiet workaround, reversibility is a trust-destroying wrong verdict, dependency is a second team needing the first team's real cases to exist first, evidence is a cheap single-team pilot, and rank is the four-step build order itself.

And if you want to be sure it really works, try it somewhere elseSame five letters, a regional cinema chain instead of a grocery chain. Nothing about the two businesses is alike.

Palisade Cinema Group built a shared eval platform starting with its concession-demand forecasting model, before opening it to the staffing-prediction team. Dante Marchetti led that rollout.

Mapped onto ORDER: outcome is whether a second team genuinely gates decisions on the platform's verdict, same as Gullford. Reversibility: a wrong forecast verdict trusted blindly during a holiday weekend, before anyone had learned to double-check it, would have been the hardest thing to walk back, since a bad opening weekend of concession stockouts doesn't get a do-over. Dependency: the staffing team's onboarding only made sense once the concession team's golden set, built from three years of real holiday-weekend sales data, already existed and had proven itself. Evidence: piloting with just concessions, one team, one real problem, cost far less than building a platform for every team's hypothetical future use case. Rank restates the identical order, in a business with nothing else in common with a supermarket chain.

A different shape of picture than Section 2 used: a branching decision instead of two doors side by side. The same order still wins.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "one real team's failure cases first, earn trust, then generalize, dashboards last, because a wrong early verdict is the one mistake you can't undo," and stop.
Cost: if a full golden set is too expensive to build for every case type at once, start with just the highest-volume failure pattern and expand from there.
The model gets better, for real: even as underlying models improve, a shared leaderboard still creates the same incentive to curate a test set rather than fix a product, so the ranking risk doesn't go away just because accuracy does.

Where people run it wrong.
They build polished dashboards and self-serve onboarding first, because it's the most demoable part, before any team has a reason to trust the underlying verdict.
They put every team's score on one visible leaderboard, assuming transparency is automatically healthy.
They treat a climbing pass rate as proof of improvement, without ever checking it against a real-world number it's supposed to predict.

How to use it live. When someone asks how to build an eval platform other teams will use, ask yourself first: which mistake here can't be undone? Build against that one first, and let everything demoable wait.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0