A traditional PM writes user stories. What is the AI equivalent artifact and why?

GrantLoom is Bellcroft Software's writing tool for small nonprofits. It reads a funder's RFP and a nonprofit's own program data, then drafts the sections a grant application actually needs. Frideric Obadele owns GrantLoom's product. Linnea Sedgwick runs Thrushwood Youth Mentoring, an after-school program for 180 kids across three sites, and she is the one who has to trust whatever GrantLoom hands her before she hits submit.

How to answer this, stage by stage

Nobody is grading whether you know the word "eval." They are grading whether you can say, specifically, what an AI feature needs instead of a user story, and why a story alone lets a wrong number ship clean.

Let's learn

GrantLoom reads two things: a funder's RFP, and a nonprofit's own program data, who it serves, what it spends, what it's achieved, then drafts the sections a grant application needs, a needs statement, a methods section, a budget narrative.

Before GrantLoom, Thrushwood wrote every application by hand. Linnea used to spend about fourteen hours on a single mid-size foundation application, most of it copying numbers out of three different spreadsheets into one clean document.

With GrantLoom, that fourteen hours became about ninety minutes: paste in the RFP, review the draft, fix the parts that needed her own ear. For most of a year, that was the whole story, and it was a good one.

Nobody typed 1,200 into anything. The model filled a gap the same quiet way it fills every gap.

Here's the turn. A wrong word here and there was never the real risk. The real risk is a wrong number that reads exactly as confident as a right one, sitting inside a document with a Friday deadline nobody has time to check line by line.

When Bellcroft ran the new eval spec back over ninety days of live drafts, after the near miss, 23 percent of them, about one in four, had at least one number that couldn't be traced to any real source field. The old check never caught a single one, because every one of those drafts still had all its sections in the right order.

What it costs at its worst: a nonprofit submits a number that isn't true to a foundation that fact-checks. Hawthorne's compliance team calls references and checks reported reach against tax filings on anything over 500. Had Thrushwood submitted 1,200, the mismatch alone could have flagged the whole application for a closer look, past the point where "we made a mistake" reads as an honest slip instead of an inflated claim.

What I would leave alone: established, data-rich nonprofits, the ones with several grant cycles of complete records already in the system, almost never hit this problem. Their own real numbers are already there, so the model has nothing left to guess at. I'd leave GrantLoom's drafting for those orgs exactly as it is.

The lesson: a feature that writes words for someone isn't done when the words show up in the right order. It's done when every fact inside those words can be checked against something real, automatically, not just by whoever happens to read it before the deadline.

Now here is the same thing as a story

Read the short version above when you're in the room. Read this one when you want to feel what six hours and a lucky cup of coffee actually saved.

Every grant cycle, Linnea Sedgwick used to spend about fourteen hours turning three separate spreadsheets into one clean narrative. She was good at it, careful, the kind of person who reads a sentence twice before she trusts it.

GrantLoom arrived in the spring, and for most of a year it was exactly the upgrade it promised to be. She'd paste in a funder's RFP, GrantLoom would draft the needs statement and the budget narrative, and she'd spend her ninety minutes checking tone, not arithmetic. The first few drafts, she checked every single number against her own spreadsheet out of habit. They were always right, because Thrushwood's early applications happened to go to funders with simpler forms, where every field was already filled in.

By the fourth or fifth cycle, she'd stopped opening the spreadsheet at all. GrantLoom had never once been wrong, so there was nothing left to double-check.

The Hawthorne Family Foundation application was the biggest ask Thrushwood had made in three years, forty-five thousand dollars, and it landed on a form with a field Thrushwood's own intake sheet had always left blank: exact annual youth served, broken out by site. Nobody at Thrushwood had ever needed that precise a number before. GrantLoom drafted the application Tuesday morning anyway, and it read beautifully. It said Thrushwood serves 1,200 youth a year.

This is the whole test a user story runs. Every box gets ticked, and nobody ever asks whether the words inside them are true.

Linnea read the draft twice for tone, the way the last dozen cycles had trained her to. Tone was fine. She scheduled the submission for Friday at five.

Reinholt Blythewood, Thrushwood's board treasurer, reads every application before it goes out, mostly out of habit from his day job auditing small budgets. Friday morning, coffee in hand, he got to the needs statement and stopped. Twelve hundred kids, across three sites, on a budget the size of Thrushwood's, didn't sit right with a number he half-remembered from a board meeting. He texted Linnea: "is this 1,200 right? feels big."

It wasn't. Thrushwood serves 180 kids a year, and everyone on the board knew that number cold. What nobody knew, until that Friday morning, was where GrantLoom's 1,200 had actually come from.

Linnea spent the next four hours doing what she used to do before GrantLoom existed: reopening every source document and rechecking every number in the application by hand, one at a time, with a real deadline six hours out. She caught two smaller numbers along the way that were also wrong, a budget line and a partnership date, neither one nearly as dangerous as the 1,200, both invented the same quiet way.

She never had a rule for which numbers to trust and which to check. She had a feeling, built out of a year of the tool never once being wrong, and Reinholt's text was the first thing to ever put a crack in it.

The decision Frideric Obadele would take back sits in a launch review from a year earlier. Someone on the team asked whether GrantLoom should refuse to write a sentence it couldn't source, forcing the writer to fill the gap by hand instead. The answer, at the time, was no: it would mean choppier drafts and more manual typing for nonprofits who already didn't have much staff time, for a case that seemed rare. Nobody pictured a $45,000 application riding on the one field Thrushwood had always left blank.

Run that Friday again, with the eval spec live. GrantLoom still doesn't have Thrushwood's real annual reach in its own data, that gap doesn't close on its own. But the moment the draft generates, Tuesday morning, the number gets flagged inline: "Unverified: confirm your annual reach here." Linnea fixes it in about ninety seconds, types 180, and the application ships Thursday, a full day early, with nothing left for Reinholt to catch by luck.

One design hands a nonprofit a fluent paragraph and hopes somebody happens to read it closely enough. The other tells the writer, the moment it happens, exactly which sentence it isn't sure of.

What I'd tell myself, sitting in that launch review a year back: a tool that fills every gap fluently isn't being helpful. It's just moving the moment someone finds out from before the deadline to after it.

SPARK, or what replaces a story when the output is a guess

Not a way to make "test it more" sound official. SPARK is what forces you to name a graded artifact engineering can actually build against, instead of a sentence that only checks whether a feature exists.

This is the whole answer to the question, in one picture. Four real parts, none of them a sentence starting with "as a user."

A user story's whole job stops at this arrow. It never asks the next question.

Same draft, same day. Only the second bar was ever built to catch what actually broke.

None of this is forgotten. It's just not gradable yet, and shipping a fake score for it would be worse than leaving it out.

And if you want to be sure it really works, try it somewhere else

Same five letters, an ambulance instead of a foundation, and the missing fact is a vital sign instead of a headcount.

Crestfallow Regional Ambulance runs RunNote, a tool that reads a paramedic's radio call and the truck's own telemetry, then drafts the incident report before the crew reaches the hospital. Euphemia Chukwuemeka has run calls for six years and reviews her own report before it locks. The user story that shipped it read: "As a paramedic, I want the app to draft my incident report from the call, so I can spend more time with the patient." It passed every structural check, every required field present, every time.

Same shape of fix, a completely different siren. The fact that goes missing here is a vital sign, not a headcount.

Same artifact, mapped straight onto RunNote: the eval spec is real incident transcripts paired with the truck's own telemetry, a rubric checking that every vital sign, medication, and timestamp traces to telemetry or an explicit voice note, and a pass bar before RunNote's auto-fill can lock a report without a paramedic's review. Euphemia's fix is the same shape as Linnea's: a missing vital gets flagged the moment the draft generates, not discovered by whoever happens to read closely before the report locks.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to it: real graded examples, a rubric checking every fact traces to a real source, and a pass bar before it ships clean, that's the artifact, and that's why it replaces the story.
Cost: no budget this quarter to build a full human-graded set. Ship the traceability check on the highest-risk fields first, dollar amounts, headcounts, dates, and grow the graded set as budget allows.
The model got better, for real: say GrantLoom's underlying model gets tuned and every draft reads noticeably better. Keep the eval spec anyway. A better model still fills a blank field with something plausible when it has nothing real to draw on, and reading better was never the same claim as being grounded.

Where people run it wrong.
They keep the user story and just add "should be accurate" as one more line nobody can actually test against.
They build the eval set from whichever past drafts were easiest to find, which usually means the biggest, most complete orgs, and misses the exact thin-data case that breaks in production.
They treat one clean demo draft as proof the feature works, the same mistake the old story's checklist already made at launch.

How to use it live. Ask this before agreeing a feature is done: "what's the graded, repeatable check that would have caught the worst thing this could plausibly say, and who grades it?" That question alone usually shows whether "it's tested" means a real eval spec or a demo that happened to go well.

Three things worth stating directly, since the real judgment sits here. Bellcroft's team considered a second model as an automatic judge instead of human graders, and rejected it, because a model trained to sound confident on thin data doesn't catch another model doing the same thing. The AI-specific failure worth naming is grounding failure under sparse input: asked to write fluent, persuasive text, a model fills a real gap with a plausible value borrowed from similar cases, and nothing about the sentence looks any different from a true one. The guardrail is the traceability check itself, tracing every generated fact to one real source field and flagging what can't be traced. And the trade-off is real and accepted on purpose: requiring field-level traceability slows the first draft down, most of all for the smaller, data-thin nonprofits who need GrantLoom the most, trading a faster first draft for one that doesn't put a false number in front of a funder who checks.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0