Explain why an AI PM often owns the evaluation set while a traditional PM would not own a test plan.

Formshift is Kinetra Labs' app for people who lift at home: point a phone at a set of squats and it tells you, in a few seconds, whether your knees are caving in or your depth is off. Idelette Draeger owns the question of when a Formshift verdict is actually good enough to trust. Three releases in, her own eval set was already telling a story nobody was reading by slice.

How to answer this, stage by stage

Nobody is grading whether you know the words eval set and test plan. They are grading whether you can say, plainly, why one of them needs an owner and the other one just needs a runner.

Let's learn

Formshift is Kinetra Labs' app for people who lift at home. Point a phone at a set of squats and it tells you, in a few seconds, whether your knees are caving in or your depth is off, the same two things a trainer would watch for.

A test plan checks a rule that already stood still. An eval set checks a bar that has to keep moving.

Before Formshift, a home lifter had two real options. Pay a trainer sixty dollars a session to watch a set in person. Or film it on a phone propped against a shoe box and guess, which is hard to do about your own knee from an angle you can barely see. Most people picked guessing, or gave up watching their form at all.

With Formshift, the same lifter gets a verdict in under three seconds, on nearly every set they film, for free. The average user films four to five sets a week and gets specific feedback on each one, not a trainer's once-a-month glance.

Five steps. The fourth one, how sure the pose model actually is about where a knee sits, is the step this whole answer turns on.

Idelette's team retrained the model three times in five months. Formshift's blended squat accuracy, how often its knee-cave call matched a trainer's own label across the whole 1,180-clip set, climbed release over release: 82 percent, then 87, then 92. On paper, that is three clean wins in a row.

Here is the turn. Split the same eval set by camera angle, lighting, and clothing, and the side-on, dim-light, loose-clothing slice, the exact setup a lot of evening lifters actually use, moved the other way: 78 percent, then 68, then 55. Same eval set. Same three releases. Opposite direction. And the one number leadership watched, the blended 92, never once showed it.

What it costs at its worst: a caved-in knee that never gets flagged is not a small annoyance, it is a way to actually get hurt, repeated for weeks by a tool someone trusted specifically because it had never been wrong before. And even without an injury, the same slice of users just quietly trained less. Average completed sessions a week for that segment fell from 4.2 to 2.6 after the third release shipped.

What I would leave alone: the front-on, bright, fitted-clothing slice. It held at 94 percent through all three releases, and it covers most of Formshift's users. Gating that slice too would just slow down everyone the retrain never actually hurt.

The lesson: a test plan checks a rule that does not move. An eval set decides what good enough means today, for bodies and rooms that keep moving, and that decision has to sit with whoever owns whether the product is actually safe to trust, not whoever is asked to run the same 1,180 clips one more time.

Now here is the same thing as a story

The short version above is what you actually say in the room. Read this one when you want to feel why a new hire's one question mattered more than three straight releases of good news.

Idelette Draeger has run product for Formshift for two years, and before that she spent six months building its very first eval set herself, filming her own squats in her apartment in six different setups so the labelers would have something real to score against.

Formshift launched doing one thing: squats, filmed front-on, in daylight, because that is how the earliest users happened to train. For the first year, that was basically the whole business. QA reran the same 1,180 clips every release, blended accuracy climbed steadily, and Idelette signed off without a second look, because the number had never once lied to her.

Nobody at Kinetra decided any of this was happening. It just happened, the way a season does.

Kinetra kept growing the exercise list, deadlifts, lunges, push-ups, plank holds, overhead presses, hip hinges, and its users kept growing with it. A lot of newer lifters, people like Belmara Krenwick, an engineer who trains most evenings after work, started filming later in the day, propping a phone low near the couch instead of a tripod by a window, wearing whatever she had worn to the office instead of fitted gym clothes.

Three releases into that shift, a new applied scientist, three weeks into the job, asked a question in a routine eval review that nobody could answer: why did the 1,180-clip eval set only have about forty clips filmed after seven at night, when close to half of Formshift's daily sessions happen after eight? Nobody had an answer, because nobody had ever asked.

Idelette pulled the eval set apart by slice for the first time in months. The blended number was exactly what everyone remembered, climbing, 82, 87, 92. Split by camera angle, lighting, and clothing, the side-on, dim-light, loose-clothing slice, Belmara's exact setup, told a different story: 78, 68, 55.

Belmara never filed a complaint. She had no reason to; Formshift kept saying clear. She just trained less. Her weekly sessions slipped from about four to just over two and a half over six weeks, and she told herself it was just a busy season at work, not a tool that had quietly stopped seeing her knees.

The decision Idelette would take back happened in a launch review eighteen months earlier, back when Formshift did one exercise. Someone asked whether QA should own re-running the eval set every release as a straight pass or fail. The room said yes, reasonably: one exercise, one filming habit, a fixed check was fast and it worked. Nobody ever came back to ask whether that was still true once there were seven exercises and users training in the dark.

Run the same eighteen months again, with the eval set owned by product from day one instead of handed to QA as a fixed regression. Every slice gets its own floor, 75 percent, checked separately from the blended number. The side-on, dim-light slice trips that floor the same week the third release ships, not six weeks later in a cohort review nobody was running yet. Formshift adds a heavier, slower pose check just for that slice, 2.4 seconds slower, and pulls in real clips of users like Belmara to retrain against. Three weeks after that, the slice is back to 81 percent, and Belmara's segment is back to almost four sessions a week.

What I would tell myself, back in that first launch review: a checklist that never changes is fine for a button that never changes either. It was never going to be fine for anything that has to see a knee bend in the dark.

LEAD: the four things only an eval set's owner gets to decide

Not a way to make an eval set sound more official than a checklist. LEAD is what forces you to say which slice is the one worth losing sleep over, and what you would actually do about it.

Four checks, run on one eval set. Skip any one of them and a real regression can climb three releases in a row without anyone catching it.

The eval set's hard slice sits top left on purpose. It was measured, and it was early, weeks before anything else on this page.

A model can clear the same 1,180 clips every single release. That was never proof it could see Belmara's living room.

Three things worth saying plainly, since this is where the real judgment sits. Idelette's team considered two other fixes and rejected both. Leaving the eval set with QA as a fixed regression, which was the actual arrangement for eighteen months and the one that let this slide. And simply raising the blended pass bar, say to 96 percent, instead of scoring by slice, which would not have caught this either, since the blended score was already climbing past 90 while the real slice sat at 55. The AI-specific failure underneath both is a keypoint problem, not a bug: side angle, dim light, and loose fabric all make it harder for the pose model to see exactly where a knee sits, so its confidence drops in a way a blended average was never built to isolate. The guardrail is the per-slice floor itself, fed by real production misses, not a smarter blended threshold. And the fix is not free: the heavier check on that one slice costs 2.4 seconds of extra wait, on purpose, only there, because slow and honest beats fast and wrong on somebody's knee.

And if you want to be sure it really works, try it somewhere else

Same four letters, a drone instead of a phone, and this time a missed call means a farmer skips a field walk they actually needed to take.

Fieldglass Aerial builds LeafScope, which reads drone photos of a field and flags early blight before it spreads. Amoret Isibor owns LeafScope's eval set the same way Idelette owns Formshift's: not a QA checklist, a living judgment call about what good enough to trust means this season.

Different sensor, same shape of mistake. The blended number climbed while one real slice quietly went the other way.

LeafScope's own eval set is 2,400 labeled drone images across four crops, three growth stages, and both sunny and overcast light, since cloud cover changes how a leaf's color reads from four hundred feet up. Blended blight-detection accuracy climbed for two straight model versions, 88 to 93 percent. Split by slice, overcast days on late-stage soybean, when the canopy is thickest and the light is flattest, fell from 81 to 62 percent over the same two versions, because the model had far fewer late-stage, overcast clips to learn from than early-stage, sunny ones, the conditions most of Fieldglass Aerial's corn customers actually fly in.

Run the same four letters. Link: not the offline detection score, but whether a farmer who sees no blight detected can skip walking that field themselves without missing a real outbreak. Early signal: the eval set's own overcast, late-stage slice, falling while the blended score climbed, the same shape as Formshift's. Abuse: the easy slice, sunny, early-stage corn, is what most of the labeled set was built from first, because it is the biggest customer base and the clearest images to hand-label, so the hard slice stayed thin. Decision: Amoret set a slice floor at 75 percent, the same number Idelette landed on independently, added real farmer-flagged misses from overcast fields to the next eval cycle, and routed any overcast, late-stage flag through an agronomist's review instead of auto-clearing it, while every other slice ships as scored.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: own the eval set, score it by slice, hold a floor no slice can fall under, and feed real misses back in, because a test plan checks a rule that never moves and this one has to.
Cost: no budget this quarter to grow the eval set. Ship with the thin slice honestly labeled as thin, and grow it with every real miss instead of waiting to launch until it is bigger.
The model got better, for real: say the hardest slice climbs to 90 percent on its own next release. The floor does not go away, because 90 percent still is not the 94 the easy slice has held for a year, and slices keep drifting even when they look healthy today.

Where people run it wrong.
They watch the blended score because a segmented one feels like admitting the launch might not be a clean win.
They treat a fixed eval set as proof of quality forever, instead of a snapshot of whatever the world looked like the day it was built.
They fix the whole model when only one slice is actually broken, and lose real, honest gains everywhere else.

How to use it live. Ask yourself one question before trusting any climbing benchmark: what does this look like split by the slice that was always going to be hardest, not averaged across the slice that was always going to be easy? That question alone is usually the exact distinction a LEAD question is listening for.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0