Describe how you would turn user edits into a quality signal.

Larkspell is a freelance marketplace. Larkspell Assist drafts a personalized outreach message the moment a freelancer applies to a job post. Rasheed Iqbal manages trust and quality for Assist, and checks its numbers from his phone most mornings.

How to answer this, stage by stage

Nobody is grading whether you'd log edits at all. They're grading whether you can tell a real pattern apart from one person's habit.

Let's learn

The draft message sits on a phone screen, one thumb away from being sent exactly as the AI wrote it, or rewritten first.

Larkspell is a freelance marketplace, and Larkspell Assist drafts a personalized outreach message the moment a freelancer applies to a job post.

Before Assist, a freelancer spent close to twenty minutes writing a pitch to a job post by hand. Assist drops a full draft in under ten seconds, and a freelancer can send it as-is or edit first.

At its worst: every edit gets treated as a vote against the draft, fed straight into a model that retrains in real time, and the drafting tool quietly overfits to a handful of the most active freelancers' personal writing style, while the real skill-mismatch problem, the one hurting many different freelancers, gets diluted among thousands of purely cosmetic tweaks and never gets fixed at all.

What I would leave alone: a freelancer who's used Assist for months and clearly has a signature style, a specific greeting, a specific sign-off, doesn't need their own repeated personal edit feeding into the shared quality signal at all. That's their voice, not a flaw in the model.

The lesson: the useful part of an edit was never the fact that it happened. It's whether a stranger, somewhere else, made the exact same fix without ever seeing the first person's draft.

Now here is the same thing as a story

The short version above is what you'd say defending this design to Larkspell's product council. Read this one for how the pattern actually got found.

For months, Larkspell Assist looked like a clean win. For months after that, it looked like nothing, because nothing about it was ever actually measured.

The second box used to take twenty minutes. Assist collapsed it to seconds, and nobody watched what happened after.

Rasheed Iqbal checks Assist's numbers most mornings from his phone, thumbing through a dashboard during his commute before he's even sat down at his desk. Adoption climbed fast. Freelancers applied to more jobs per week, since drafting no longer cost them real time.

There was no single bad day. Over months, win-rate for Assist-drafted proposals, whether a freelancer actually got hired, sat a few points below hand-written ones, without ever cratering hard enough to trip an alarm. It was just persistently, quietly worse, and nobody could say why, because the edited text itself was never kept anywhere.

Rasheed started informally asking a few freelancers what they usually changed. Several, without prompting, mentioned the same thing: "I always fix the skill Assist highlights, it's usually not the one that actually matters for this job."

One person's habit and ninety strangers converging on the same fix look nothing alike, once you're actually counting who they are.

That offhand pattern became the seed of a structured edit-capture system: every edit gets tagged by section, and the tool tracks how many distinct freelancers made the same kind of fix, not just how many times it happened.

Four parts, and only one of them, skill mentioned, turned out to be the systemic problem.

The pattern traced back to how Assist parsed job posts with several required skills listed: it grabbed the first skill mentioned in the post, not the one most relevant to that specific freelancer's profile. Reordering how the retrieval step ranked skills closed most of the gap within a few weeks, and win-rate for Assist-drafted proposals caught up to hand-written ones by the following month.

The old design asked win-rate alone to explain itself, with no view into what freelancers were actually fixing. The new one asks the edit itself what it means, by checking whether it repeats across people who've never spoken to each other.

I didn't build edit tracking into version one because it felt like a nice-to-have we could add anytime. It took months of a quietly underperforming number, with no single bad day I could point to, to see that anytime never has a deadline unless something forces it to.

SPARK, and the edit that finally countedNot a debate over whether to log edits. SPARK is what tells you which edits are actually worth trusting.

Only the top right corner, spread across many freelancers and a real structural change, is worth a weekly review.

Each of these is a real feature the signal could eventually power. None of them is safe to ship on day one.

The recap, one line per letter: situation is an edit that vanishes with no record, payoff is trusting a cross-freelancer pattern instead of one person's habit, anchor is section-level edit capture weighted by spread, risk is the single freelancer's personal tweak the anchor has to ignore, and keep out is holding off on automatic retraining and individual scoring.

And if you want to be sure it really works, try it somewhere elseSame five letters, a public library's catalog records instead of a freelance pitch. A completely different field, the same anchor.

Bellhaven Public Library uses an AI tool to draft the catalog description for a newly acquired book, before a cataloger reviews and publishes it to the library's public search system. Callista Wren is the cataloging systems librarian who owns that tool.

Mapped onto SPARK: situation is a cataloger today, editing the AI-drafted record silently before publishing, with no record kept of what changed. Payoff is trusting a pattern that repeats across many different catalogers and many different titles, instead of one cataloger's personal formatting preference. The anchor is structurally the same idea, aimed at library records instead of pitches: capture edits by section, subject headings, summary, age rating, and weight a fix by how many distinct catalogers make it across how many distinct titles, not by raw edit volume. Risk is a veteran cataloger who always reformats the summary a certain way out of personal style, the anchor ignores it because it never spreads to anyone else's records. Keep out is no automatic retraining in real time, no scoring individual catalogers, and no automated changes to controlled-vocabulary subject headings without a professional librarian's review, since those follow strict standards no model should touch unsupervised.

Two of the four branches lead to "ignore." That's by design, not a gap in the signal.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "capture what changed in an edit, then trust the pattern only once it repeats across strangers, not one person's habit," and stop.
Cost: there's no time to build the full structured capture this sprint. Say so honestly, and start by just logging the pre-edit draft alongside the sent message, since even that alone beats losing every edit completely.
The model gets better, for real: if Assist's overall drafting quality improves, that's still not a reason to drop edit tracking, a better average model can still carry one narrow, systemic gap that only shows up once enough strangers hit it.

Where people run it wrong.
They treat every edit as equally meaningful, instead of separating personal style from a real structural gap.
They wire raw edit volume straight into automatic retraining, risking a model that drifts toward whoever edits the most.
They score individual users by how often they edit, punishing engagement instead of measuring the draft itself.

How to use it live. When someone asks how to turn edits into a signal, ask yourself one question first: would a stranger, with no knowledge of the first person's edit, make the exact same fix. If the answer is yes across enough people, you've found a real signal. If not, you've found someone's personal style.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0