What disclosure obligations apply when users interact with an AI system?

Solace Health pairs patients with human therapists and gives them Coda, an AI companion they can message between sessions when a therapist isn't available. Thandiwe Okonjo leads the Coda product team, and keeps a folder of every support ticket that ever used the phrase "I didn't realize."

How to answer this, stage by stage

Nobody is grading whether you can quote a specific law's article number. They're grading whether you know the difference between showing a disclosure and it actually landing.

Let's learn

Coda messages back and forth with a patient between their scheduled therapy sessions, offering grounding exercises and check-ins when a human therapist isn't available.

At signup, every new patient sees a modal: "Coda is an AI companion, not a licensed therapist." They have to tap "I understand" before the chat unlocks. That one screen was, for a long time, the entire disclosure strategy.

Now, the turn: the modal was never really the problem. The problem is that "shown once" and "known now" quietly became the same thing in the team's mind, when they had never actually been the same thing at all.

What I would leave alone: Solace's appointment-reminder texts, sent by a simple automated system with no ongoing conversation, need no repeated disclosure at all. Nobody mistakes a reminder text for a person, and repeating a disclosure there would just be noise.

The lesson: a disclosure requirement isn't satisfied by a screen someone tapped through once. It's satisfied by what a person still knows, unprompted, on an ordinary Tuesday three months later.

Now here is the same thing as a story

The short version above is what you'd say defending Coda's disclosure design to Solace's clinical safety board. Read this one for how the gap actually got found.

Thandiwe Okonjo has led the Coda product team since it launched. She can usually tell which onboarding screens people actually read versus which ones they tap through on reflex, mostly from how fast users move past them.

Coda's signup modal had a strong completion rate, essentially everyone tapped "I understand" before their first conversation. For a long time, that number was the only evidence anyone tracked, and it always looked fine.

Four steps ran exactly as designed. The missing fourth one, a reminder somewhere in those thirty sessions, was the one nobody had built.

Then a new clinical hire, three weeks into the job, asked an ordinary question in a product review meeting: does Coda tell people mid-conversation that it's AI, or only once at the very start?

The team had built the left panel and quietly assumed it was the right one.

Nobody in the room had a confident answer. Thandiwe realized, saying it out loud for the first time, that the team had never actually measured whether users still knew, only whether they'd once agreed.

Coda sat in exactly the corner where a fading memory of disclosure matters most.

Thandiwe ran a quick comprehension check, a short, direct question inserted for a sample of active users: "Just to check, who do you think you're chatting with right now?"

Coda took the branch that needed the strongest version of disclosure, the one the team had actually built the weakest version for.

Users six months into using Coda answered correctly barely more than half the time, a sharp drop from the near-universal accuracy of users just a few weeks in.

The team had two of these four solidly in place. The third one, repetition, was the one that had quietly gone missing.

Thandiwe's team designed a small, persistent badge in the chat header, always visible, reading "Coda, an AI companion" in plain type, refreshed at the top of every session.

Small, constant, and quiet, exactly the opposite of a modal that interrupts once and is never seen again.

Nine months from the new hire's question to a redesigned disclosure, measured again the same honest way.

The old approach asked users to remember something told to them once, months earlier, in a completely different emotional state. The new one puts the fact quietly back in front of them, every single time, without ever interrupting the conversation.

I built one disclosure modal because it satisfied every checklist we reviewed at launch, and Coda was new enough then that nobody used it for more than a few weeks anyway. It took one new hire's plain question, asked without any sense that it was a hard one, to see that the checklist and the actual promise had quietly drifted apart.

LEAD, the signal that moved firstNot a compliance recap. LEAD is what forces you to find the number that would have warned you before anyone complained.

The recap, one line per letter: link is calibrated trust in Solace, early signal is comprehension by cohort age falling well before any complaint, abuse is a leading survey question inflating the number falsely, and decision is redesigning toward a persistent badge once comprehension drops below a real threshold.

And if you want to be sure it really works, try it somewhere elseSame four letters, an insurance claims portal instead of a therapy companion. This time the early signal is a click, not a survey answer.

Harrowgate Mutual runs an AI assistant that helps policyholders file and track home-insurance claims through chat. Desmond Achebe manages that product, and the disclosure question there looks different once the relationship is transactional rather than emotional.

Link: policyholders trusting the claims process enough to actually use the chat instead of calling and waiting on hold, which is the whole reason the feature exists. Early signal: the share of chat sessions where a policyholder clicks "speak to a human" within the first minute, specifically citing confusion about who or what they're talking to, tracked as a rate over each week. Abuse: policyholders might click "speak to a human" for entirely unrelated reasons, like an urgent claim, so the metric only counts sessions where the transcript shows an explicit "wait, is this a person" moment. Decision: if that rate crosses roughly 5 percent of sessions in a week, add a firmer identity statement at the top of the chat window, since the claims context is lower-stakes than a therapy companion, but a policyholder still deserves to know who's handling their claim.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "measure whether people still know, not just whether they were once told, and watch for that number by cohort age," and stop.
Cost: there's no engineering budget this quarter for a redesigned badge. Say so honestly, and start with a lightweight monthly in-chat reminder message, since even a repeated text line beats a disclosure that only exists once, on day one.
The model gets better, for real: if Coda's responses get warmer and more helpful, that's exactly when disclosure matters more, not less. A more convincing companion is more likely to be mistaken for a person, not less.

Where people run it wrong.
They treat a high tap-through rate on a signup modal as proof the disclosure is working, when it only proves the modal was shown.
They measure disclosure once, at launch, and never check whether comprehension holds up months later.
They add a repeated disclosure everywhere uniformly, including low-stakes single-use interactions that never needed it in the first place.

How to use it live. When asked about disclosure obligations, don't start with what the law technically requires. Start with what you'd actually measure to know whether anyone still remembers, and let the design follow from that number.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0