Describe how you would build shared ownership of the eval set across PM, engineering and design.

Tarnholm Systems builds Proviso, an AI tool that checks a nonprofit's grant application against a funder's rules before anyone submits it. Nyoka Ostrell is the AI PM who owns Proviso's roadmap. Iskren Kirrick leads the engineering team that built and, until recently, alone maintained Proviso's eval set. Almendra Lundgrave leads design, and for most of Proviso's first year had never once opened it.

1How to answer this, stage by stage

Nobody is grading whether you can say "we should all care about quality." They are grading whether you can name the actual room, the actual question asked in it, and the actual line between contributing to an eval set and deciding how a model gets fixed.

2Let's learn

This is the gap the monthly session exists to close. Not a lack of care in the room. A lack of any road for what design or the PM already notice.

Proviso reads a nonprofit's grant application, checks it against a funder's rulebook, and marks every rule PASS or FAIL with the exact line it's flagging. Before Tarnholm built anything, a grant writer at a place like Ravensmoor Youth Trust checked her own application by hand against a rulebook of about forty items: page limits, required certifications, budget math, font size, section order. That took her close to three hours, usually the night before the deadline, the way it always got done.

Proviso now reads the same application in under two minutes. Extraction accuracy, whether it reads the right words off the right line, climbed from 89 percent to 98 percent over Proviso's first nine months. Iskren's team built that eval set, ran it in the same repo as the code, and kept it passing every release. By every number they tracked, Proviso was getting better every month.

Here's what that accuracy number never touched. Proviso's eval set tested the text it pulled out of a document. It never once rendered a page and looked at it, because nobody had asked it to. For eight straight months, zero of Proviso's cases tested how a compliance report actually looks once it's printed or exported, the thing a grant officer or a funder's reviewer actually opens.

Same failure, same product. What changes is only whether anyone was ever asked to look at the printed page.

Ravensmoor Youth Trust ran their application to the Underleigh Foundation's youth literacy grant through Proviso. Every field passed. But a page limit trim, a routine step that removes extra white space to fit a report inside its allowed pages, had pushed the signature line of the required non discrimination certification onto its own page, orphaned, with nothing above it to say what it belonged to. The extracted text was still whole and correct, so Proviso's eval set, which only ever checked extracted text, had nothing to flag.

Xenna Menzies, the grant writer, still prints a hard copy before every submission, an old habit from years of doing this by hand. At 9:40pm, with the portal closing at midnight, she opened page two and found a lone signature line with no context above it. To a tired reviewer at Underleigh, that reads like a missing certification, not a formatting accident, and a missing certification is grounds for a flat rejection. She fixed it by hand and resubmitted with about eighty minutes to spare.

Nothing in this gap was Proviso lying. Every word it read was correct. It was simply never asked whether the page still made sense.

What it costs at its worst: if a certification like this had reached Underleigh uncaught, Ravensmoor risks a flat rejection on a formatting accident, not a real compliance failure, on a grant they were otherwise fully eligible for. Worse, if it happened more than once, grant writers using Proviso would stop trusting its PASS mark and go back to checking every application by hand, the exact three hours a night Proviso was built to remove.

What I would leave alone: Iskren's team runs its own regression tests for PDF field extraction, whether a form field is read correctly at all, entirely separate from this session. That's a deterministic pass or fail with no judgment call in it. Handing it to a three person vote would slow down real engineering work for a decision that doesn't need three people in the room.

The lesson: a model's own scorecard only ever answers the questions somebody thought to ask it. If only one team ever writes the questions, whole categories of failure stay invisible, no matter how high the score climbs.

3Now here is the same thing as a story

The short version above is what you'd actually say in the room. Read this one for why the fix had to be a standing session, not a promise to "loop design in more."

Nyoka Ostrell has run product for compliance tools for six years, and she can usually tell within a week whether a rule engine is actually solid or just quiet because nobody's tested it hard yet. When she joined Tarnholm to lead Proviso, Iskren Kirrick had already built something genuinely good: a rules engine that read a grant application and caught the kind of mistakes that used to sink applications on technicalities. She trusted it, because the numbers backed her up. Every release, extraction accuracy climbed a little further.

For most of Proviso's first year, that trust felt earned. Nyoka watched the eval set's pass rate every Friday from a dashboard Iskren's team maintained. Ninety one percent. Ninety four. Ninety seven. She never opened the eval set itself, the actual pile of test cases underneath the number, because there was no reason to. It lived in a repo she didn't have much cause to visit, reviewed by engineers who plainly knew what they were doing.

Almendra Lundgrave, who led design, had it even further from her desk. She designed the report Proviso generates for a nonprofit to review before submitting, the layout, the page breaks, the print styling. She never once looked inside the eval set that was supposed to prove the tool worked, because as far as anyone had told her, that wasn't a design concern. It was a text accuracy number, and text accuracy wasn't her department.

This was the line Nyoka would eventually draw on purpose. Early on, it wasn't a line at all. It was just the whole eval set, sitting on the other side of a door nobody thought to open.

Then came the Tuesday morning Nyoka got a message from a member of Ravensmoor Youth Trust's staff, not angry, mostly confused. Their application to the Underleigh Foundation had passed every Proviso check the night before, and Xenna Menzies had still nearly missed her deadline fixing a certification page that looked, on paper, like it was missing entirely.

Nyoka pulled up the case herself. The extracted text was fine, word for word, exactly what Proviso's eval set would have graded correct. The printed page was not fine. A trim step meant to keep the report inside its page limit had pushed the certification's signature line onto page two, alone, with nothing above it. Proviso hadn't lied about a single word. It had simply never been asked whether the page still made sense once a human opened it.

She brought it to Iskren first, expecting frustration, and instead got something closer to relief. His team had been maintaining an eval set built for one kind of question, is the text right, for over a year. Nobody outside engineering had ever asked it a different kind of question, does the page still read the way it's supposed to, because nobody outside engineering had ever been in the room where that question would occur to anyone.

Nyoka's first instinct was the tempting one: ask Iskren's team to just add rendering checks to their own eval set, quietly, without changing who owned it. Almendra talked her out of it within a day. A rendering check written by an engineer who's never sat with a grant writer's real, panicked 9:40pm moment tests what an engineer imagines could go wrong on a page, not what actually goes wrong. The people who'd catch the next one weren't in the room where the eval set got written, and adding cases from the outside without adding the outside people wouldn't fix that.

They also floated an open ticket queue instead, anyone at Tarnholm could file a suspected eval case whenever they noticed one, no meeting required. Nyoka killed that idea inside a week of trying it: three tickets came in, none got triaged, and by the second week nobody was filing them, because writing up a case nobody would look at felt like shouting into a drawer.

The whole answer in one picture. A small fixed session, one case from each vantage point, a vote before anyone leaves the room.

What they built instead: once a month, forty five minutes, all three of them sit down with five real Proviso outputs from the last few weeks. Each person is expected to bring one case, sourced from wherever they actually sit. Nyoka's usually comes from a support ticket or a comment on a call. Almendra's comes from something that looked wrong the moment she opened the rendered report, a run of text too close to a page break, a checkmark sitting next to something that visually reads as broken. Iskren's comes from a production log, a pattern in what Proviso actually got flagged for by users who complained. Every case that survives the discussion gets voted into the shared eval set before the meeting ends, no exceptions, no "we'll circle back."

The first session ran two months after the Ravensmoor near miss. Almendra brought the exact shape of the page break problem, generalized: any certification statement sitting within four lines of a page limit trim. Nine other render cases followed over the next two sessions, plus nine cases Nyoka pulled from support tickets. By month nine, formatting and render coverage in Proviso's eval set had gone from zero to sixty one percent of the known failure patterns the team could name, without touching how engineering actually builds or ships a fix.

What I'd tell myself, sitting across from Iskren that Tuesday: the mistake was never that engineering owned the eval set too tightly on purpose. The mistake was that nobody ever asked whether the questions it answered were still the right questions, once the product outgrew the ones it started with.

4SPARK, five checkpoints for who's in the room

Not a script for sounding collaborative. SPARK forces you to name the one concrete session, and prove it survives both ways a shared eval set actually breaks.

Letting Ravensmoor's own staff submit suspected cases directly was tempting early. Nyoka deferred it: without a PM triaging first, most submissions would describe a symptom, not the actual page level cause.

Same finding, same team, same failed render. What changes the outcome is only whether the road to the eval set exists.

The recap, one line per letter: situation is an eval set only one role ever looked at, payoff is all three roles treating a failing case as shared, anchor is the monthly session and the vote that makes a case real, risk is either extreme costing months of blind spots or months of stalled shipping, and keep out draws the line at implementation, never at nomination.

5And if you want to be sure it really works, try it somewhere else

Same five letters, a textile mill instead of a grant checker, and this time the blind spot isn't a page break. It's a defect a camera model was never trained to name.

Elmswood Textiles builds Loomwatch, an AI tool that watches fabric on a conveyor belt through a mounted camera and flags defects before a bolt gets shipped to a garment maker. Praxton Yellin owns Loomwatch's roadmap. Loomwatch's eval set, like Proviso's, lived entirely inside the computer vision team's own test harness for its first year, built from thousands of labeled defect photos nobody outside engineering had ever reviewed.

Same ritual, a loom instead of a grant form. The eval set still only learns to see what someone was in the room to point at.

Mapped onto SPARK: the situation is a defect eval set that only ever learned from the defects a computer vision engineer already knew to label. The payoff is a quality floor supervisor and a line operator both contributing real cases the moment they see something Loomwatch missed, not months later in a formal incident report. The anchor is the same monthly session, one case each: the PM from a client complaint, the floor supervisor from a defect she caught by eye that never triggered a flag, the vision engineer from a pattern in near-miss confidence scores. The risk runs the same both ways: skip it and a whole class of defect, say a subtle sheen difference that only shows under certain mill lighting, stays permanently invisible to the model's own scorecard; run it as a design review and engineers spend the session defending model architecture instead of nominating cases. Keep out draws the same line: the floor supervisor helps decide what belongs in the eval set, never how the vision model itself gets retrained.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to the anchor, the monthly session and the vote, and what it protects against.
Cost: no budget for a standing monthly ritual across every product line. Run it quarterly instead, coverage still climbs off zero, just slower, and the PM's time cost drops to a quarter of the monthly version.
The model got better, for real: say Proviso's extraction accuracy hits 99.5 percent. The session still matters, because accuracy was never measuring the thing that broke. A better text score doesn't teach the eval set to check a printed page it was never asked about.

Where people run it wrong.
They let the session become a status update, cases get mentioned, nobody votes, and by the third month everyone quietly stops bringing anything, the way Tarnholm's open ticket queue died in two weeks.
They let a PM or a designer start dictating the technical fix inside the session, which turns a forty five minute nomination meeting into an hour and a half architecture debate nobody scheduled.
They add render or formatting cases without ever checking them against a broader held out set, and a fix built to patch one dramatic near miss quietly breaks a different, ordinary case nobody was watching.

How to use it live. Before answering a "how would you build shared ownership" question cold, ask yourself one thing: what's the smallest standing room where a real case, from any of the three roles, has somewhere to land and get voted on. Naming that room, not a value word like alignment or collaboration, is usually exactly what the question is listening for.

6Flashcards (tap any card to flip it)

7Check yourself Score: 0 / 0