What does an evals PM own, and should that be a distinct role?

Quillbranch searches a database of 42 million resumes so a recruiter can type a role brief and get back the candidates most likely to fit. Isbeth Loxley owns the golden set that decides whether a change to that search is good enough to ship. Eight months into her role, one slice of that golden set fell eighteen points, and not one recruiter ever filed a complaint about it.

How to answer this, stage by stage

Nobody is grading whether you can list what an evals PM does in a week. They're grading whether you can say what breaks when nobody owns it, and commit to when it needs its own seat.

Let's learn

Here is what happens when a search tool works fine for months, then quietly stops working for one kind of resume, while every dashboard keeps saying it is fine.

Quillbranch searches a database of forty two million resumes. A recruiter types a role brief, a title, the must have skills, the years of experience, and Quillbranch hands back the candidates most likely to fit, ranked top to bottom.

Every change to Quillbranch's matching model passes through this same run before it ships. The step that actually catches trouble is the third box, not the last one.

Before a change ships, it gets checked every night against a golden set: fourteen hundred pairs of a real role brief and the resume a panel of recruiters agreed was, or was not, a real match for it.

For most of a year, the blended score on that nightly check sat around ninety four out of a hundred. Good enough that nobody thought about it much.

Then the job market started calling the same job something new. Engineers at small AI companies began calling themselves "Founding Engineer." Some listed "Applied AI Engineer" or "Member of Technical Staff" instead of anything with the word "Senior" in it. Quillbranch's matching model had never learned to treat those words as close cousins of the older titles a recruiter still typed into the search box. So candidates who described themselves the new way started dropping off page one, even on searches they were the strongest fit for.

Here is the turn. The eighty four pairs that used the new titles, six out of every hundred in the golden set, were the whole story, and the other thirteen hundred were quiet enough to hide them.

A recruiter who never sees a good candidate does not know one was missing. They just see a harder search than usual, and blame the market.

What it costs at its worst: nobody complained, and nobody was going to. A recall miss is invisible to the person it happens to, because they only ever see what surfaced, never what got buried. Recruiters put the slower fills down to AI talent being scarce right now, which was also true, just not the whole truth.

What I would leave alone: the golden set's slice for skilled trades roles, forklift operators, HVAC technicians, warehouse leads. Those titles barely shift in wording year to year. Putting that slice on the same fast refresh as the AI native titles would spend recruiter panel time on a slice that was never moving in the first place.

The lesson: a model can look completely healthy on the number everyone glances at, and still be quietly failing the exact people it was built to help, if the failure only shows up in a slice small enough to disappear into an average. Watching the whole isn't the same as watching the parts.

Now here is the same thing as a story

The short version above is what you actually say in the room. Read this one for the fourteen months it actually took to build, and lose, the habit that would have caught this sooner.

Isbeth Loxley can read a nightly eval report the way some people read weather. Before Quillbranch, she spent six years grading search relevance for a job board nobody outside recruiting has heard of, and she says the tell is never the score itself. It's which slice of the score moved, and which slice stayed dead still while everyone was watching the wrong number.

Quillbranch created her role in month nine, right after the company shipped its second shared model, a resume parser sitting underneath the same golden set as the original matcher. For the first few months it was quiet, uneventful work. She reviewed the nightly run every Monday morning, sliced by segment, not just the blended total. For a while, three or four other people from the search team joined her review, curious about the new discipline.

By month four, the blended score had held steady near ninety four for so long that the Monday review started to feel like a formality. One teammate stopped coming because nothing ever looked different. Then another. By month six, it was Isbeth alone, at her desk, scrolling the same segment breakdown she'd scrolled every week since the role began.

Nobody decided to stop paying attention. It just got easier not to, one quiet Monday at a time, because a flat blended line looks exactly like nothing is wrong.

Nobody decided, on any single day, to let this run for eight weeks. It just never got a second pair of eyes on it anymore.

In week three of what would become the eight week drift, Isbeth's Monday slice for AI native titles read seventy eight, down from a steady eighty nine. She flagged it in her own notes as "watch," not "block," because a single week's dip could be noise, and a false alarm burns trust for the next real one. By week six she noticed something else in the recruiter activity logs, roles tagged with those newer titles were taking a week longer to fill than the same roles had six months earlier. By week eight the slice read seventy one, and the pattern stopped looking like noise.

I want to say the problem was that the model got worse. It did get worse, on that one slice. But that's not really the story. Isbeth never had a single number in her head that told her the loop was broken. She had a Monday habit, and the habit only kept running because she personally kept showing up to it, alone, after everyone else had quietly decided the flat line meant there was nothing left to watch.

The decision she'd take back sits in a meeting fourteen months earlier, when three people built the original golden set in eleven days flat, to hit a launch date of their own. They talked about how often to refresh it. Someone suggested a quarterly schedule. Someone else pointed out that quarterly labeling would cost real recruiter panel hours for a model that, at the time, barely had any users to search for anything unusual. They landed on refreshing it only when someone flagged a reason to. Reasonable, for a company with one model and forty resumes searched a day. Nobody ever came back to revisit it once that stopped being true.

Run the same eight weeks again, with a scheduled quarterly refresh in place instead. By week three, the "watch" flag on the AI native titles slice triggers an actual relabeling pass, not just a note. Forty new resume samples get pulled from live search traffic, including the new titles, and a recruiter panel checks them against the same brief format the golden set already uses. By week five, the matching model gets retrained against the refreshed set. The slice never sees seventy one. It bottoms out at seventy eight and climbs back to eighty six within the month, and the eighteen point fall that actually happened becomes an eleven point dip that got caught and fixed before a single extra week of hiring got slower.

What I would tell myself, back in that eleven day meeting: a refresh schedule that only fires when someone notices a problem is a schedule that never fires, because the whole point of a slow drift is that nobody notices in time to ask for one.

LEAD, the four checks that gave one number a veto over a launch date

Not a way to prove Isbeth is smarter than the applied PM under deadline pressure. LEAD is what forces you to say which number actually predicts trouble, and who gets to act on it before anyone else even sees a reason to.

A metric can be satisfied on paper while the thing it stands for quietly walks off the page.

Four different situations, four different moves. None of them is "add another meeting."

The recap, one line per letter: link the eval score to whether a recruiter actually finds someone worth hiring, not to a similarity number nobody outside engineering reads. The early signal is the eval score itself, sliced by segment, because complaints and hiring metrics both arrive too late or never at all. Name both ways the bar gets gamed, a rushed deadline and a golden set nobody schedules time to touch. And the decision step is what makes ownership real: different, specific moves at each threshold, not a single blanket rule.

Two things worth saying plainly, since this is where the real judgment sits. Quillbranch's leadership considered a different fix before creating Isbeth's role at all: fold eval ownership into the data platform team as a purely technical function, measured on uptime and pipeline health rather than product judgment. They rejected it, because setting a pass bar is a call about how much a recruiter will tolerate before they stop trusting the tool, and that is a product decision wearing a technical costume, not an engineering metric. The AI specific failure worth naming by name is a vocabulary drift the matching model was never trained to bridge, new job titles the golden set predates, and the guardrail that actually catches it is a segment sliced nightly eval paired with a scheduled quarterly refresh sourced from live query samples, not a refresh that waits for someone to notice. And the trade-off was real, and taken on purpose: Merrin's team shipped to one paying customer at a known, lower quality bar, eighty four instead of ninety, trading full parity for three fewer weeks of waiting, contained to a single relationship instead of every recruiter on the platform.

And if you want to be sure it really works, try it somewhere else

Same four letters, a property insurance claim instead of a resume search, and this time the honest verdict lands one notch short of a fully dedicated role.

Brackentide reads photos and a homeowner's own description of storm or water damage and drafts a repair cost estimate for a human adjuster to check and approve. Cerys Harlowe runs product for the estimating tool, and hit a smaller version of Isbeth's exact question eleven months into the product's life.

Brackentide sits on the middle branch. Not nothing, not yet a dedicated seat either.

Brackentide's golden set held nine hundred labeled pairs, a photo and description alongside the estimate a senior adjuster agreed was fair. Seventy of those pairs, about eight percent, covered newer building materials: engineered vinyl plank flooring, standing seam metal roofing, spray foam insulation. Over ten weeks, the blended score drifted from ninety one to ninety, nothing worth a second look. The modern materials slice fell from eighty five to sixty six in that same window, because the estimating model had learned older material names and treated the newer ones as unfamiliar, quietly undervaluing the repair.

Mapped onto LEAD: the link is the same shape, an estimate score has to track whether an adjuster approves it and a homeowner accepts it, not just how close the model's number sits to some internal target. The early signal is again the sliced eval score itself, not the dispute rate, which took two extra months to catch up. The abuse Cerys found wasn't a deadline ask, it was the plainer kind: a part-time, whoever's-free-that-month owner is the same as no owner. And her decision step matched Isbeth's, segment blocks and a scheduled refresh, just running on two shared models instead of three.

That's also where the verdict actually diverges, and it's worth saying honestly. Brackentide runs two models off one golden set, the estimator and a photo classifier, both on the same regional launch calendar. That's the middle branch, not the top one. Cerys's fix wasn't a dedicated evals PM. It was naming herself the explicit owner in writing, with a real refresh schedule and real veto power, something Quillbranch also had for its first eight months, before a third model made the job too big for anyone's spare time.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: name the golden set, the threshold, and the block call as one owner's job, and slice the eval by segment, never just the blended average.
Cost: no budget for a dedicated evals hire this quarter. Whoever already owns the feature writes down and defends the pass bar and the refresh cadence in public, this month, before anyone gets hired for it.
The model got better, for real: say the matcher's overall benchmark climbs to ninety eight. The golden set still needs a scheduled refresh, because a model getting smarter on old vocabulary says nothing about whether it understands this quarter's new one.

Where people run it wrong.
They watch the blended score and never slice it, so a real, narrow drift hides inside a number that looks fine.
They let "no complaints" stand in for "no problem," when the actual failure is invisible to the person living through it.
They answer a deadline pressure ask by adding a second reviewer instead of naming one owner with real veto power, and the two reviewers end up deferring to each other.

How to use it live. When an interviewer asks whether a role should be distinct, ask yourself one question before answering out loud: "how many things share the same golden set, and whose actual job is it to say no to a deadline." That's usually the exact distinction a question shaped like this one is testing.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0