How do you define a task-level success criterion for a summarization feature?

How to answer this, stage by stage

Seven moves. Name the two checks before the story, or the answer sounds like a definition instead of something you'd actually write into a spec.

Let's learn

Before we built anything, Rutger read every manuscript that landed on his desk, start to finish, before he decided whether it earned a reviewer's time. Forty pages, most mornings, before his coffee went cold.

It took him close to fifty minutes a paper. On a heavy Tuesday, with nine new submissions waiting, that's most of his morning gone before he answers a single email.

Then we built Precis. It reads the manuscript and drafts a plain abstract, the research question, the method, the sample size, the headline result, in about ninety seconds. Rutger could read that instead of the paper and make the same call in under five minutes.

For a long stretch, that trade worked. The check that ran on every abstract, does it name the required fields, held steady, month after month, around 97 percent. Nobody had a reason to look twice.

Here's the turn. Hitting every field is not the same as keeping the fact that would actually change Rutger's mind. And the number that tracks that, whether the abstract kept the paper's one real caveat, was falling the entire time nobody was watching it. It dropped from 89 percent to 61 percent over eight weeks, while the field-check stayed at 97 the whole way down.

At its worst, that gap sends a flawed paper to reviewers as if it were clean, or tells an editor a paper is weaker than it is, because the one thing that would have argued for it never made it into the page he actually reads. Either way, the journal is trusting a summary that passed a checklist and missed the point.

Every field ticked. The one fact that mattered kept walking.

What I'd leave alone. A note about who funded the study, or which section of the journal a paper belongs in, doesn't need this kind of check. Nobody argues about facts like that. Save the checking for the parts where the summary has to make a judgment call about what mattered most in forty pages.

The lesson. A rubric that only checks whether something got mentioned will always look healthy, right up until the week it isn't. If a criterion can be satisfied by hitting a template, it was never really testing whether the summary did its job.

Now here is the same thing as a story

Use this version when you've got the time to sit with it. The short version is above. This is for when you want to feel why it mattered.

Rutger Van Dijk has handled submissions for the Journal of Coastal Hydrology for eleven years. Ask him and he'll tell you the real skill was never reading fast. It was knowing, by about page six, whether a paper had a real question in it.

For years, that meant every submission got the same forty minutes. He'd read the introduction slowly, skim the method, and slow down again for the results and discussion, because that's where a paper usually admits what it didn't manage to do.

Then Precis arrived. Thao Bui had built it to draft a plain abstract from every submission the moment it landed, the question, the method, the sample size, the headline result, so editors like Rutger could triage without reading each paper cover to cover.

For the first several months, it earned his trust honestly. He'd read the Precis summary, skim the discussion section himself just to check the tone matched, and move on. Fifteen minutes, not fifty.

Then the checking faded, in three small steps, none of them looking wrong at the time. First, the backlog grew, and skimming the discussion section on top of the summary started to feel like doing the job twice. Second, the field-check that ran automatically on every summary, does it name the method, the sample size, the result, never once failed, so there was no obvious reason to keep double-checking behind it. Third, nobody was tracking whether the summary's one caveat matched the paper's real one, only whether the summary existed at all.

Rutger stopped skimming the discussion section sometime that spring. He couldn't tell you the week. It just stopped feeling necessary.

Then a manuscript came in modeling coastal erosion across twelve monitoring sites, testing a new prediction method against the standard one everyone in the field already used.

Precis's summary said the new model "outperformed the baseline method across the test sites," and named the method, the sample size, twelve sites, forty months of data, and the headline result. Every required field, present and correct. Rutger read it in under two minutes and reached for the rapid-review track, the fast lane for papers whose findings were clean enough not to need the usual three reviewers.

Then, out of nothing but habit from years before Precis existed, he clicked through to page twenty-six anyway, not because anything looked wrong.

The new model had failed at two of the twelve sites. Badly. Not a footnote, a paragraph, the authors' own honest account of where their method broke down. It was true that the model outperformed the baseline overall. It was also true that a sixth of the sites told a different story, and that difference was exactly the kind of thing three reviewers needed to weigh in on, not wave through on the fast track.

It would be easy to say Precis got it wrong. It didn't, not by the rubric it was scored against. Every field was there. The summary was accurate, sentence by sentence. Nobody had ever written down that "accurate" and "keeps the fact that changes the decision" were two different bars, so the tool had never been asked to clear the second one.

So here's the decision Thao would take back.

Two years earlier, in the meeting where the success criteria first got written down, the team scored abstracts on whether a script could confirm the required fields were present, because that was the check a script could actually run without a person reading anything. Someone asked whether they should also check for the paper's stated caveats. It felt like something a good editor would just catch anyway. Thao remembers agreeing they'd add it if it ever became a real problem.

I'd write the decision-fact check in from day one instead. Same weekly sample, but every abstract gets compared against the paper's own stated limitation, by a person, and that agreement number rides on the same weekly report as the field-check.

Here's the replay. Same eight weeks, decision-fact capture tracked from week one. It crosses the 70 percent floor in week six, the same week it actually did, but this time somebody is looking, instead of the flag coming from Rutger's old habit on page twenty-six. Thao pulls the failing cases and finds the pattern, multi-site studies, where the model learned to summarize the average and drop the outlier. She adds one line to the prompt: always state where the result didn't hold, not just where it did. Decision-fact capture climbs back to 88 percent by week eight. In the two months after, papers sent to the wrong track drop from seven to one.

One success criteria section checks whether the abstract said the required things. The other checks whether it kept the one thing that would have changed what Rutger did next.

And the thing Thao would tell herself, back in that first meeting: the risk was never a summary that got a fact wrong. It was a summary that got every fact right and still, on the page that mattered, said nothing at all.

LEAD, run on a summary instead of a score

This reads like a question about writing style, does the summary sound right. Underneath it, it's still asking whether the editor can act on the summary and be right. That's LEAD, run on a paragraph instead of a single number.

And if you want to be sure it really works, try it somewhere else

An AI tool called Riser reads a technician's full inspection report after every elevator service call and drafts a two-line summary for the building's maintenance manager, so they don't have to read six pages of readings to know whether anything needs attention. Iveta Sokolova manages maintenance for a block of towers at Corrado Lift Services' biggest client and reads a Riser summary after almost every visit instead of the full report. Same shape of problem. A summary that checks every required field, cables, brakes, doors, tested and logged, can still drop the one reading that was actually creeping toward the line.

Every stop logged. The drifting reading is a separate question.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0