What does responsible AI require in the eval spec specifically?

Aldergate Wealth runs the Aldergate Advisor, a robo-advisor that recommends how a customer's savings should be split between stocks, bonds, and cash. Renata Kowalska is the product manager who owns the eval spec every new model version has to clear before it ships.

How to answer this, stage by stage

Seven moves. Practice these until the words come without reaching for them.

Let's learn

Say a robo-advisor recommends how to split a customer's savings between stocks, bonds, and cash. Every quarter, a new model version ships, and every version has to clear an eval spec first.

For a long time, that spec was one number: run the new model against a golden set of 2,000 real customer conversations, check how often the recommended allocation matched what a licensed advisor would have suggested, and if the match rate cleared 95%, ship it. Version after version cleared comfortably, usually around 96%.

Nobody had asked how those 2,000 conversations were chosen. They were pulled at random from real traffic, and real traffic skewed young: most Aldergate customers are in their 30s and 40s, still adding to their savings every month. Only about 4% of the golden set represented someone within five years of retirement, starting to draw money out instead of putting it in, even though that group made up roughly 20% of actual customers.

Here is the turn. The 96% pass rate was never wrong, exactly. It was just answering a question nobody needed answered as urgently: how good is the model at the easy majority. It was silent on the one group where a bad recommendation does the most damage in the least recoverable way.

At its worst: a 61-year-old customer, five years from retiring, gets recommended an allocation built for someone decades younger, loses a chunk of it in a downturn with no time left to recover, and the eval dashboard says everything passed.

What I would leave alone: the blended score itself isn't the problem, and dropping it would be a mistake too. It's still useful for catching a broad regression across the whole product. The fix is adding segment bars next to it, not replacing it.

The lesson: "responsible AI" in an eval spec isn't a separate checklist item bolted onto the real one. It's the decision to measure the group most at risk on its own, instead of letting it disappear into an average.

Now here is the same thing as a story

The short version above is what you'd say in the room. Read this one for how Renata actually found the gap.

Renata Kowalska has spent four years building eval infrastructure for financial products, the last two of them at Aldergate. She is the person other PMs call when a metric looks too good to be true.

The second box is the one that didn't exist for the first six model versions Aldergate ever shipped.

For those first six versions, the process worked exactly as designed: run the golden set, check the blended score, ship at 95% or above. Every version cleared. Nobody had a reason to look closer, because nothing on the dashboard was telling them to.

Then a regulatory inquiry landed on Renata's desk. A 61-year-old customer had filed a complaint after a downturn wiped out nearly a fifth of his retirement savings, invested at an allocation the Advisor had recommended only four months before he planned to retire. The allocation would have made sense for someone with thirty years to recover. He had less than one.

Both numbers were true at the same time. Only one of them was being watched.

Renata pulled the golden set apart by segment for the first time since it was built. What she found was not a bug in the model. It was a hole in the eval spec: near-retirement cases made up a fifth of real customers and a fraction of that in the golden set, and inside that small slice, the real pass rate had been sliding for months, invisible under a blended average that had barely moved.

The two highest-harm segments sit in exactly the corner the golden set almost never reached.

We did not build a model that failed near-retirement customers. We built a spec that never had to notice if it did.

Renata rebuilt the spec around four pieces instead of one number: a golden set deliberately stratified so every harm-prone segment gets real weight regardless of how rare it is in daily traffic, a separate pass bar for each of those segments, herself named as the spec's owner with a standing quarterly review on her calendar, and every passing result pinned to the exact model version and date that earned it.

Four pieces. The old spec only ever had the first half of the first one.

The segment bar was already telling the story. The complaint count just hadn't caught up to it yet.

The next model version failed the new spec on its first run, on the near-retirement bar specifically, at 74%. It stayed home two extra weeks while the team retrained specifically on decumulation cases. The version after that cleared every segment bar at 90% or above, blended score included.

What I would tell myself a year earlier: a 96% pass rate was never proof of anything on its own. It was an average, and an average's whole job is to hide exactly the thing you actually need to see.

LEAD, the spec Renata actually builtNot "measure accuracy." LEAD forces you to find the number that would have warned you first.

The recap, one line per letter: link is a suitable allocation, not a model score, early signal is the segment pass rate falling months before any complaint, abuse is a random golden set quietly under-weighting the highest-harm group, and decision is a hard per-segment bar that blocks a release the blended score alone would have passed.

And if you want to be sure it really works, try it somewhere elseSame four letters, a staffing agency's resume screener instead of a wealth app. A completely different field.

Fenwick Staffing runs Fenwick Screen, an AI tool that ranks job applicants for recruiters before a human reviews the shortlist. Dmitri Kessler owns the eval spec that gates every new ranking model.

Mapped onto LEAD: link is not "ranking accuracy," it's whether qualified candidates from every demographic group actually reach the human-reviewed shortlist at a similar rate. Early signal is the shortlist rate broken out by protected group, tracked on its own, since it can drift for months while the overall "top candidates match hiring manager picks" score stays flat. Abuse: a golden set built from the agency's own historical placements quietly bakes in whatever imbalance existed in who got hired before, so a high match score can mean "matches old bias" as easily as "found the best candidate." Decision: if any group's shortlist rate falls more than 15 points below the overall rate, the release is blocked and the ranking model is re-audited before it ships to any client.

Same four pieces again. A hiring tool needs them for exactly the same reason a robo-advisor does.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "a stratified golden set, a bar per harm category, a named owner, a version pin, not one blended score," and stop.
Cost: there's no budget to build a fully custom stratified set this quarter. Say so honestly, and start by re-weighting the existing set so the highest-harm segment isn't under 5% of it, even before better data collection catches up.
The model gets better, for real: if the blended score climbs to 99%, that's still not a reason to drop the segment bars. A near-perfect average model can still be quietly wrong for the one group whose mistakes cost the most.

Where people run it wrong.
They chase one blended number because it's simpler to report to leadership, and simplicity quietly wins over honesty.
They build the golden set from whatever data is easiest to pull, instead of the data that actually represents who could get hurt.
They check the spec once at launch and never again, so a segment that passed at launch can fail two versions later with nobody watching.

How to use it live. If someone asks "isn't 96% good enough," ask back: 96% of what set, built how. That question alone usually tells you whether the eval spec in front of you is a real one.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0