How do you design feedback that is useful for debugging rather than just scoring?

Halberd Systems is a platform team of twenty-four engineers. Revyn reads every pull request's diff and leaves inline comments flagging likely bugs, risky patterns, and style issues before a human reviewer looks at it. Dario Salgado is the engineering manager who has to decide what happens after someone clicks thumbs down.

How to answer this, stage by stage

Nobody's grading whether your feedback widget has enough buttons. They're grading whether a report it produces can actually get a bug fixed.

Let's learn

Halberd Systems' platform team used to review pull requests by hand. A reviewer opened the diff, read it top to bottom, and left inline comments where something looked off. Before Revyn, that took about forty-five minutes per pull request on average.

Revyn now reads the diff first, flagging likely bugs and risky patterns before a human reviewer even opens it. Review time dropped to about eighteen minutes. A thumbs-up or thumbs-down sits under every flag Revyn leaves.

This is the real workflow the feedback widget has to fit into, not a blank screen with a thumbs icon on it.

The turn: the eighteen minutes saved were never the real story. The real story is what happens after a thumbs-down. With no reason attached, every negative click starts a guessing session from zero, usually a Slack message to whoever owns Revyn asking "why did it flag this?"

At its worst: engineers tired of unexplained thumbs-down loops start blanket-dismissing Revyn's flags without reading them, and a real hardcoded-credential warning gets waved through with the rest of the noise.

What I would leave alone: a bare thumbs-down is still fine for a pure style nitpick, like trailing whitespace. Nobody needs a reason code for that, and forcing one there just adds friction with no debugging payoff.

The lesson: feedback that's supposed to help you fix something has to point somewhere. A single click that only says "no" is a mood, not a bug report, and the moment your feedback volume outgrows a Slack thread, that gap is what breaks first.

Now here is the same thing as a story

The short version above is what you'd say defending this design to Halberd's leadership. Read this one for how the gap actually got found.

Revyn's inline comment box sits right in the pull request view, a small thumbs-up and thumbs-down under every flag it leaves, the same interface any code review tool would have.

For the first few months, that plain widget was enough. Revyn flagged maybe thirty comments a week across the team, and if someone was confused by a thumbs-down, they just asked in the channel and got an answer in minutes.

A score tells you something is wrong. A diagnosis tells you what to go fix. The widget only gave the team the first one.

By month four, Revyn was flagging closer to six hundred comments a month as it rolled out to two more repositories. The habit that had worked fine at thirty comments a week started thinning in three beats: first, people stopped asking in the channel because the questions piled up faster than anyone could answer; then they stopped reading Revyn's reasoning line at all, since it rarely explained itself anyway; then a few engineers quietly started dismissing every Revyn flag on sight, just to keep moving.

The trigger was small: a new hire asked Dario in a one-on-one, "why does everyone just click no on these without reading them?"

Dario pulled a week of thumbs-down clicks and timed how long it took engineers to figure out what each one meant. Twelve minutes of guessing, on average, before anyone even started the actual fix.

Same wrong flag, two different reports. Only one of them tells you where to look.

Dario's team shipped the anchor: a required one-tap reason, five fixed categories, captured with the exact line and rule that triggered the flag. No free text yet. No cross-report clustering yet. Just enough structure to turn a mood into a location.

Run the same new-hire question forward under the new design: nobody blanket-dismisses Revyn anymore, because a thumbs-down now takes two minutes to understand instead of twelve, and the false-positive tag alone tells the platform team exactly which detector needs a look.

I built the plain thumbs widget because it shipped in a day and felt like enough. It took one new engineer's honest question to see that a feedback signal with nowhere to point isn't lightweight. It's just unfinished.

SPARK, the whole anchorNot a feature list. SPARK is what turns "add feedback" into one design decision worth defending.

Four small facts, and together they turn a click into something someone can actually go act on.

The recap, one line per letter: situation is the manual review workflow Revyn sits inside, payoff is engineers who stop guessing and start fixing, anchor is the required, located reason tag, risk is the false-positive spike that survives a worse model, and keep out is the free-text field and clustering held back for later.

And if you want to be sure it really works, try it somewhere elseSame five letters, an AI diagnosis assistant for a veterinary clinic instead of a code review tool. A different desk, the same anchor.

VetLumen suggests likely causes for a pet's symptoms to a veterinarian before they finalize a diagnosis. Chidinma Obi, a veterinarian at a busy clinic, rates each suggestion with a thumbs up or down before moving to the next patient.

Mapped onto SPARK: situation is a vet who already examines the animal and forms her own read before ever opening the suggestion. Payoff is wanting her to stop silently ignoring VetLumen's suggestions once she's seen a few bad ones, and instead flag exactly what kind of miss it was. Anchor is the same shape as Revyn's: a one-tap reason chosen from a short list, wrong species-typical range, missed a symptom in the notes, right condition wrong urgency, rather than a bare thumbs-down. Risk is a future model update that starts over-flagging rare conditions; the "wrong urgency" tag would spike on its own and point straight at the miscalibration, without needing a single vet to write an essay about it. Keep out is the same restraint: no free text yet, since five clinics don't yet produce enough tagged volume to make clustering useful.

The sweet spot is the one quadrant that's both fast enough to actually use and specific enough to act on.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "a required, located reason tag instead of a bare thumbs-down," and stop.
Cost: there's no engineering time to build a full tagging UI before next sprint. Say so honestly, and ship three reason tags instead of five first, since a thin version still beats a bare thumbs-down.
The model gets better, for real: if Revyn's flags get sharper and thumbs-down volume drops overall, the reason tags still earn their keep, since whatever's left over is now the hardest, most worth-debugging remainder.

Where people run it wrong.
They build a free-text box first because it feels more flexible, and end up with hundreds of comments nobody has time to read.
They track the overall thumbs-down rate and miss that one specific reason is spiking underneath a flat-looking average.
They require a reason tag on every single flag, including the trivial nits nobody needed a diagnosis for.

How to use it live. When someone asks how you'd design feedback for an AI feature, ask yourself one question first: if this exact feedback showed up on my desk tomorrow, would I know what to go do with it. If the answer is no, you've designed a score, not a debugging tool.

Day one ships the smallest version that still points somewhere. Everything after that waits for real data to justify it.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0