What consent do you need before using user feedback to improve a model?

Interviewer's question: "What consent do you need before using user feedback to improve a model?" Verity County runs AskVerity, a chatbot that answers questions about food assistance and unemployment benefits. Selam Getachew leads the product team behind it.

How to answer this, stage by stage

Nobody is grading whether you know the word "consent." They're grading whether you size the ask to the worst thing the data could be used for.

Let's learn

AskVerity is a chatbot on Verity County's benefits website. It answers questions about food assistance and unemployment claims, in plain language, any hour of the day.

Before AskVerity, a resident with a question called the county's benefits line and waited, on average, twenty two minutes. Or they drove to an office that shut at four, on a workday they'd already taken off to be there.

Now most people get an answer in under a minute, typed on a phone in a parking lot or a shared computer at the library. Every one of those chats gets saved, because Selam's team wants to know which answers actually worked.

Here's the turn: saving the chat is not the problem. What happens to it after is. One version of "using your feedback" is a thumbs-up count that tells the team which answers residents liked. A very different version is an engineer reading the actual words someone typed about an eviction notice or a kid's medical bills, and folding that into the next training run, because the same checkbox covered both.

One document, four different things it might mean. Only one of them was ever explained clearly.

At its worst: a resident who mentioned a disability, or an immigration detail, or a landlord's name, finds that detail sitting inside a training set nobody explained to them, with no way to know it's there and no way to take it back.

The bottom-left corner never needed a second ask. The top-right corner is the one that got treated the same as the bottom-left, and shouldn't have been.

What I would leave alone: the ordinary quality signal, whether an answer actually matched a resident's real question, stays exactly as useful as it always was. That number carries no name and no story, and slowing it down to match the transcript rule protects nobody.

The lesson: consent sized to the smallest risk in the pile is really consent to the largest risk in the pile. Ask at the size of the biggest thing you might actually do with someone's words, not the size of the most common thing.

Now here is the same thing as a story

The short version above is what you'd say defending this design to Verity County's privacy officer. Read this one for how the gap actually got found.

Every few weeks, Iveline Prosper sits at the third computer in the Verity County library, the one by the window, because the mouse on the others sticks. She's applying for food assistance while her hours at the warehouse get cut, and the chatbot is faster than the phone line and doesn't ask her to explain herself twice.

For months, AskVerity did exactly what it promised. It told her which forms she still owed, in one clear paragraph, no hold music.

Then a caseworker mentioned, almost in passing, that the model team reviews chat transcripts to see how the bot is doing.

Iveline didn't think much of it. Everyone had clicked past the same little checkbox at sign-up, "Help us improve AskVerity." She'd read it as a survey.

She kept using the chatbot the way she always had. She typed the real thing, not a summary. That her landlord had already filed to evict her before the assistance came through. That her son's asthma medicine was the actual reason the grocery money mattered so much that month.

Four steps. The last one is the one nobody built.

A friend, also a resident, mentioned that her own eviction detail had shown up, worded almost exactly, in a training example a volunteer had pulled up to illustrate "a hard case for the model," at a public meetup about the county's AI work.

Nobody meant harm by it. The volunteer had access because, technically, it was in the training set. And the training set existed because of one checkbox that never distinguished between "we count how many people liked this answer" and "we might quote your actual words to a room of strangers."

Selam's team hadn't built this to cause harm. When the checkbox was written, the plan was only ever to look at ratings and categories. The verbatim-transcript training use came later, added by an engineer trying to fix a stubborn edge case, and nobody circled back to ask again.

The meetup incident is what pushed the county's privacy office to actually look. They pulled two hundred saved transcripts and checked what the consent on file actually covered.

Nobody had lied about anything. Nobody had ever actually split the two risks apart, either.

Here's the decision I'd take back: we let one consent checkbox cover both today's small ask and tomorrow's bigger one. That was fine while the two stayed the same size. It stopped being fine the day someone decided a transcript was more useful raw than summarized.

Replayed with the split in place: Selam's engineer still wants that hard eviction-notice example. But now she has to check a separate flag, opt-in, before she can use verbatim text, and Iveline's transcript was never flagged, because Iveline never saw a second, clearer ask and never said yes to it. The engineer finds a different, already-consented example instead. Nothing about Iveline's exact words ever leaves the county's own systems.

The old design asked once and used the answer everywhere it could. The new one asks again, in plain terms, at the size of the actual risk.

I signed off on that first checkbox because it felt efficient, one ask instead of two seemed like good design. It took a stranger reading Iveline's own words back to her, secondhand, at a public meetup, to see that efficiency was the wrong thing to optimize.

GUARD, spelled out for one hard caseNot a privacy-policy exercise. GUARD is what tells you why one checkbox was never enough.

One of them decided the default. The other one lived inside it.

Only the middle one ever needed a second, clearer question.

The recap, one line per letter: groups is Selam's team against residents like Iveline, unequal is who has no other channel and no power to ask questions, ability to contest is that Iveline has no lever at all today, reduce is the two-layer consent split, and detect is a monthly sample checking the split actually holds.

And if you want to be sure it really works, try it somewhere elseSame five letters, a health plan's prior-authorization tool instead of a county benefits chatbot. A different consent risk, the same shape of decision.

Veridian Health Plan runs ClaraAuth, a tool that helps decide whether a treatment request needs a person's sign-off before it's approved. Booker Aldana is a patient whose specialist request got denied, and who appealed in writing, in his own words, about why the treatment mattered.

Mapped onto GUARD: groups are Veridian's claims team, who can read and use appeal text, and patients like Booker, who write it while they're already sick and already stressed. Unequal is that patients filing an appeal have no idea their words might train the next version of the denial-reasoning model, and no time or energy to find out. Ability to contest is close to zero, once an appeal is filed, a patient has no way to ask what happened to the specific sentences he wrote. Reduce is the same split: whether a claim got approved or denied, and which category it fell into, keeps training the model with no extra ask. The actual appeal text, in Booker's own words, needs a specific opt-in offered at the moment he files it, not buried in a fifty-page handbook. Detect is a monthly sample of the training set, checking that no verbatim appeal text shows up without a matching opt-in record.

Same shape of gate, a health plan instead of a county office.

Swap the trigger and it still runs.
Speed: an interviewer caps you at thirty seconds. Say "split by risk, not by source, aggregate flows free, verbatim needs its own opt-in," and stop.
Cost: if two separate consent flows are too expensive to build this quarter, ship the opt-in gate first, since that's the one that actually stops the exposure. The nicer wording can come later.
The model gets better, for real: if the model gets so good it barely needs new transcripts at all, that's not a reason to relax consent, it's a reason there's even less excuse for reaching into someone's raw words instead of the aggregate signal that was already enough.

Where people run it wrong.
They write one privacy-policy paragraph and treat it as consent for every future use of the data, including ones nobody had designed yet.
They ask once, at signup, and never again, even after the actual use of the data changes completely.
They treat "anonymized" as a finished job instead of a claim to keep checking, since a transcript with a landlord's name and a child's diagnosis isn't anonymous just because a name field got deleted.

How to use it live. When someone asks you a consent question, ask yourself one thing out loud: what's the single riskiest thing this data could be used for, and does the ask actually match that size, not the average one.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0