What does it mean for a problem to be underspecified, and why does that break AI solutions?

Kestrelfeed is a photo and video app. Recourse is the tool inside it that reads an appeal, checks it against the policy, and decides whether a takedown should stand or reverse. Xochitl Marrone runs product for the team that built it. This is the quarter she learned that a shrinking backlog and a right answer are not the same thing.

How to answer this, stage by stage

Nobody is grading whether Xochitl can define a word. They are grading whether she can turn "underspecified" from something that sounds like an excuse into a test she can actually run, out loud, under a follow-up.

Let's learn

Kestrelfeed is an app where people post photos and short videos. Recourse is the tool inside it that decides what happens when someone appeals a takedown: it reads the appeal, checks it against the policy the post was flagged under, and either reverses the removal, upholds it, or hands the case to a person.

The confidence score is the quiet step. Everything before and after it is easy to picture. That one is where the judgment actually happens.

Before Recourse could auto-resolve anything, every appeal waited for a person. On a normal week, 12,400 appeals sat in the queue, and the average wait was 11 days. Once Recourse started auto-resolving the appeals it was confident about, at a confidence bar of 0.85, the queue moved fast. By week ten the backlog was down 43 percent, and most appeals got an answer the same day.

Here is the part that matters. The extra speed was never the problem. The problem showed up four weeks later, when a routine quality audit asked three senior reviewers to look at 240 of the appeals Recourse had already closed and decide, on their own, whether the call was right. They agreed with each other only 61 percent of the time.

Skip past that and keep shipping on the old directive, "resolve appeals fairly," and Recourse quietly becomes a machine that hands out confident, well-argued decisions that a third of the company would have made differently. Nobody notices until an appeal that should have gone one way goes the other, in public, and there is no shared definition anyone can point to that says the tool was wrong.

What I would leave alone: the clear-cut appeals. When a post breaks a rule in an obvious way and the appeal has no real argument in it, all three teams agree almost every time, 95 percent in the same audit. Automating those needs no new definition. The definition problem only lives in the borderline cases.

The lesson: a short instruction sounds specified because it is short, not because it is actually pinned down. Before you build toward one, hand it to two people who care about different things and see if they would point at the same case and give the same answer. If they would not, you do not have a target yet. You have a hope wearing a sentence.

Now here is the same thing as a story

Say the short version above out loud if you get cut off early. Read this one when there is room to actually feel why the backlog number lied.

Xochitl Marrone could read an appeal in under a minute and already know, before she reached the last line, whether it was the kind that never should have been removed or the kind where the poster was hoping nobody would check the rule twice. She had worked trust and safety appeals at Kestrelfeed for four years before Recourse existed. She knew the terrain the way a good waiter knows a menu, without having to think about it.

The directive came down in one line on a slide: resolve appeals fairly, cut the backlog. Nobody in the room argued with it. Why would they. Her team built Recourse to read each appeal, check it against the written policy, and act on its own once it was confident, past a bar of 0.85. It shipped in week six.

For a while, it was the best kind of quiet. Every morning the backlog number was smaller than the day before. By week ten it was down 43 percent, and someone put the chart on a slide in the all-hands. At launch the team had spot-checked a sample of Recourse's calls every single day. After a month of the sample looking clean, that dropped to once a week. By week ten, with the number that good, the daily check had quietly stopped being anyone's actual job. The quarterly audit would catch anything real. That was the plan, on paper.

The audit landed on schedule, unremarkable, the kind of thing that shows up as a line item on a calendar nobody dreads. Three senior reviewers were asked to re-judge 240 appeals Recourse had already closed, blind to what the tool had decided. Xochitl wasn't even in the room. Someone forwarded her the results on a Tuesday afternoon. Agreement across the three reviewers: 61 percent.

Five points on the same two quarters. The gap that mattered sits between the fourth mark and the fifth, where retraining changed nothing.

She pulled the raw disagreements herself and read them one by one at her desk after most of the floor had gone home. Her first guess was the model. She checked Recourse's own confidence scores on the disputed appeals; nearly all of them were high, the tool had been sure. Her second guess was the reviewers, maybe the newer ones hadn't caught the latest policy update. She checked that too: no. The disagreement wasn't lopsided toward one reviewer. All three disagreed with each other, in different directions, on different appeals.

Two guesses that would have been comfortable to blame, and one that meant the fix was never going to be a retrain.

She could picture the meeting where "resolve appeals fairly" became the entire spec, because she had been in it. Four people, one bullet point on a slide, a nod around the table. Nobody had asked, out loud, what a correct decision on a borderline appeal actually looked like, because everyone in the room already privately knew what they meant by fair. They just did not know they meant three different things.

Same appeal, same word, three different answers depending on which team was reading it.

To check it properly, her team pulled 40 real appeals, 20 clear-cut and 20 borderline, and asked the three teams to score each one blind, before anyone saw what Recourse had decided.

One test, run twice on two kinds of case, and only one of the two results matched what the audit had already shown.

Six weeks later, the same 20 borderline appeals got scored again, this time against a written rubric the three teams had actually agreed on, line by line, sentence by sentence, until they stopped disagreeing. Agreement came back at 91 percent. Recourse itself had not been touched since the last retrain.

What she would tell herself, the day they put that 43 percent drop on a slide: a backlog going down is not the same as a question getting answered. They had made the queue quieter. They had not made anyone agree.

TRACE, for telling a hard problem from an undefined one

Not a script for sounding careful about definitions. TRACE is what stops a falling backlog number from being mistaken for a solved question, the moment someone asks why the reviewers still disagree.

One alternative worth naming and rejecting: collecting all that disagreement as more training data and retraining on whichever answer the historical record leaned toward most. That was on the table. It got turned down, because training on the majority view just picks a winner among three real, unresolved positions and hides the argument instead of settling it; the first hard borderline case that leans the other way brings the whole disagreement straight back. The AI-specific risk worth saying plainly: a model trained on inconsistently labeled human decisions will still sound completely sure of itself, because fluent, confident language is not the same thing as a correct or defensible one, so any real fix checks whether the humans who produced the training labels agreed with each other before trusting an accuracy number measured against those labels at all. And the trade-off, accepted on purpose: pausing auto-resolve on the borderline slice for a few weeks, while the actual definition gets written down, costs Kestrelfeed a slower backlog number, and it is worth it, because a confident wrong answer nobody agreed to is a worse cost than a queue that moves a little slower.

And if you want to be sure it really works, try it somewhere else

Same five letters, a hiring tool instead of an appeals tool. The word this time is not "fair." It is "culture fit," and it breaks the same way.

Sorrelwood is a recruiting platform companies use to screen resumes before a person looks at them. Pieternel Aldecott runs people analytics at a firm that uses it. Asked the same question cold, in a fairness review rather than a content appeal, she finds the identical shape under a completely different tool.

Mapped onto TRACE: the timeline is a directive, months earlier, telling the screening tool to flag candidates who were a strong culture fit, shipped and praised for speeding up hiring. Recut is department by department: the flag looked reasonable on average and cratered specifically in two departments with the newest hiring managers. Assume nothing rules out "the model is biased" as the first, most flattering-to-nobody explanation, before it has been checked. Cause candidates line up the same three honest options: the model is biased, hiring managers score inconsistently, or "culture fit" was never actually defined. Evidence test: the hiring manager, the people analytics lead, and a team lead each score 15 borderline resumes blind, before seeing the model's flag. On measurable criteria like required certifications, agreement is 92 percent. On "culture fit," it is 29 percent.

The criterion furthest from agreed-on is also the one everyone kept treating as if it were obvious.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to the definition and the evidence test; the audit numbers are what you would add back if asked to go deeper.
Cost: there is no scheduled review to point to live. Say what you would check instead: "I would pull ten borderline cases and ask two people who care about different things to score them blind, today, not wait for a quarterly review."
The model got better, for real: say Recourse's accuracy against its old labels jumps to 99 percent. The evidence test still runs exactly the same way. A better model changes how confidently it repeats an answer, not whether that answer was ever agreed on.

Where people run it wrong.
They treat any low agreement number as proof the model is biased or broken, without first checking whether the humans agree with each other.
They fix it by adding more training examples, which only teaches the model to repeat whichever definition happened to win in the data, not the correct one.
They stop at the audit number and never run the blind test, so they are never sure whether they found a real gap or just two tired reviewers on a bad Friday.

How to use it live. The moment someone says a number "looks bad," buy two seconds by asking one question back: would two careful people, working blind, actually agree on the right answer here? That question is the whole method in one breath.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0