ConceptAdvancedResponsible AI & Advanced Practice / Compliance and legal partnership / #11
What is a data protection impact assessment and when does an AI feature trigger one?
ORDER the product is Kindling Health, a wellness app that profiles users' health data for AI coaching
Kindling Health gives users daily coaching based on sleep, heart rate, cycle tracking, and short mood check-ins. Leandra Vos leads the product team building Kindling's AI coaching features, and had never had to explain the difference between "we have consent" and "we assessed the risk" until a partnership deal made her.
The direct answer
A data protection impact assessment, a DPIA, is a written, structured check of whether an AI feature's benefit to users is worth the privacy risk it creates, done before launch, not after. It's required, not optional, whenever a feature does large-scale profiling of sensitive data like health or biometric information, or makes an automated decision with a real effect on someone's life. Run the cheap screening check first. Only commit to the full assessment once that screening says the stakes actually call for one.
Do this, in order
Run a DPIA before launch on any feature that scores or profiles sensitive health data.Why: a privacy harm discovered after launch is far harder to undo than a design changed before anyone's data was ever exposed.
Map what data the feature actually touches before assessing anything else.Why: you can't judge necessity or risk for a data flow nobody has actually written down.
Use a quick screening check before committing to the full assessment.Why: not every feature needs the full process, and the screening tells you fast which ones do.
Rank which existing features need a retroactive DPIA first.Why: the ones combining sensitive data with real-world consequence are the ones most likely to cause harm if left unassessed.
Leave low-stakes, non-sensitive features off the list entirely.Why: applying the full process everywhere wastes the scrutiny the genuinely risky features actually need.
How to answer this, stage by stage
Nobody is grading whether you can recite a regulation's article number. They're grading whether you can say, for a real feature, whether it needs one and why.
Stage 1
Scope it to one real feature
Say it like this
"I'll answer this against Kindling's mood-inference feature, since 'does this need a DPIA' only becomes answerable once you're looking at one real, specific thing the AI actually does."
Why this works
Grounds an abstract compliance term in a concrete, describable feature.
Stage 2
Say your structure out loud
Say it like this
"I'll use ORDER. Outcome, what we're protecting. Reversibility, which mistake is hardest to undo. Dependency, what blocks what. Evidence, what's cheap to check first. Rank, the actual order."
Why this works
Signals a prioritization framework is coming, not a definition recited from memory.
Stage 3
Reframe the question
Say it like this
"A DPIA isn't legal homework bolted onto a launch. It's the moment you actually write down whether the feature's benefit is worth the specific privacy risk it creates, instead of assuming a general consent checkbox already answered that."
Why this works
Separates the real purpose of a DPIA from the paperwork reputation it often gets.
Stage 4
Give the one decision
Say it like this
"Run a quick screening check on any new feature that touches health data. If it profiles at scale or drives a real decision about the user, that screening triggers a full DPIA before launch, not after."
Why this works
This is deliverable 0, stated as a concrete process step, not a definition.
Stage 5
Prove it with a failure
Say it like this
"Before a partner insurer's privacy officer asked for it, we had four features that clearly met the trigger criteria, and only one, our oldest, had ever actually gone through a formal assessment."
Why this works
A real, countable gap beats an assertion that "we take privacy seriously."
Stage 6
Say what you'd measure
Say it like this
"I'd track how long a DPIA takes to complete, once the process is routine. If it's still taking eleven weeks a year in, the process itself, not just the paperwork, needs fixing."
Why this works
Shows the assessment itself gets measured and improved, not just performed once and forgotten.
Stage 7
Close on the one line
Say it like this
"Consent tells you someone agreed. A DPIA tells you whether they should have had to. Those are two different documents, and only one of them was ever written."
Why this works
Leaves the interviewer with the exact distinction the whole answer turns on.
Let's learn
Kindling's AI coaching reads a user's sleep, heart rate, cycle data, and short daily mood check-ins to suggest what to focus on that day, sometimes flagging a pattern worth a doctor's attention.
For years, Kindling treated its terms-of-service consent and privacy policy as the whole answer to "are we allowed to build this." Every new feature got a legal review of its consent language and nothing more structured than that.
Knowledge spark: what actually goes inside a DPIA?
A written record answering four things: what data flows through the feature and where, whether collecting that much data is actually necessary for the feature to work, what could realistically go wrong for the people whose data it is, and what specific steps reduce that risk. It's not a form you fill in once and file away. It's a document that has to hold up if someone skeptical reads it later.
Now, the turn: consent had always answered "did the user agree." It had never once answered "did we check whether this specific use of their health data creates a real risk worth taking seriously."
Features requiring a DPIA versus features that actually had one
Three of four features that clearly met the criteria had launched with only a consent-language review behind them.
The decision I would take back
We treated a solid privacy policy and clear consent language as the whole compliance answer for every new AI feature, since that was the process the earliest, simplest version of the app had actually needed. That made sense when Kindling tracked step counts and sleep hours. It stopped making sense once the app started inferring mood, flagging cycle irregularities, and feeding data into a coaching model that shaped real daily suggestions, because consent was never built to answer whether that specific inference was worth the specific risk it created.
What I would leave alone: Kindling's step-count reminder feature, a simple daily nudge with no profiling and no sensitive inference behind it, genuinely doesn't need a DPIA. Running the full process on it would spend real assessment time on the one feature in the app carrying almost no privacy risk at all.
Consent had always answered whether someone agreed. It had never once answered whether we'd actually checked what agreeing to this specific thing could cost them.
The lesson: a DPIA isn't a form that proves you're compliant. It's the actual moment someone sits down and asks, in writing, what could go wrong for the person whose health data this is, before the feature ever reaches them.
Now here is the same thing as a story
The short version above is what you'd say defending Kindling's readiness to a partner's own data protection officer. Read this one for how the gap actually surfaced.
Leandra Vos has led Kindling's AI coaching team for two years, mostly focused on making the daily suggestions feel genuinely useful rather than generic.
A promising partnership with a health insurer, offering Kindling's coaching as a covered wellness benefit, reached its final stage of due diligence. The insurer's own privacy officer sent one document request: the DPIA for the mood-inference feature.
Kindling's mood-inference feature matched at least two of these four criteria on its own. Nobody had ever checked the list against it.
Nobody on the team could produce one. The feature had a consent screen, a privacy-policy paragraph, and a legal sign-off on its wording. It had never had anything resembling a structured risk assessment.
Three features sat in the exact corner a DPIA exists for. Only one of them had ever gotten one.
Leandra pulled together a fast screening pass across every feature in the app, using the same four trigger questions the insurer's officer had clearly been checking against.
Three of Kindling's features took the same branch as the mood-inference tool. Only the oldest one had actually walked it.
The mood-inference feature, the stress-scoring feature, and a data-sharing integration built specifically for insurer partnerships all matched the criteria. Only the cycle-tracking coaching feature, built earliest and reviewed ad hoc by a since-departed team member, had anything close to a real assessment on file.
The launch decision had always been made without ever passing through the second box.
Leandra's team spent the next two quarters writing real DPIAs for the three missing features, starting with the mood-inference tool the insurer had actually asked about.
Four honest questions, answered in writing, for a feature that had never had to answer any of them before.
Three quarters to close a gap that had been open since the very first sensitive feature shipped.
The old approach asked whether a user had agreed to something, once, at signup. The new one asks, feature by feature, whether the agreement was ever actually informed by a real look at the risk.
Time to complete a DPIA, per feature, as the process matured
The process itself got faster every time. By the fourth assessment, what once took eleven weeks took three.
I let consent stand in for a real risk check because it was the process the earliest version of Kindling had genuinely needed. It took a partner's own privacy officer asking for a document that had simply never been written to see that consent and assessment had quietly been treated as the same requirement, when they had never actually meant the same thing.
ORDER, the four features rankedNot a compliance checklist. ORDER is what forces you to say which gap actually had to close first.
O
Outcome. What we're protecting.
Users' health data being used in ways they'd genuinely be comfortable with, and the insurer partnership closing without a compliance gap surfacing late.
Without a stated outcome, ranking the four features is just opinion.
R
Reversibility. The hard step.
A privacy harm discovered after launch, like a mood inference reaching someone it shouldn't have, is nearly impossible to undo. Running the DPIA before launch is cheap and fully revisable.
The asymmetry that decides which features get assessed first.
The whole ranking follows from this asymmetry. A DPIA is a redo. A discovered harm isn't.
D
Dependency. What blocks what.
A full DPIA can't be written without a data-flow map first, and the launch decision can't be finalized without the DPIA.
Shows the order is forced by what depends on what, not just judgment.
E
Evidence. What's cheap to check first.
A quick four-question screening pass across every feature, done in an afternoon, showed exactly which three needed the full process.
A cheap check that sized the real gap before committing months to closing it.
R
Rank. The actual order.
Mood inference first, since it was the one the insurer had actually asked about, then stress scoring, then the data-sharing integration, with the step-count feature left off the list entirely.
Defends the top pick in one line: closest to real, current consequence first.
The recap, one line per letter: outcome is a partnership closing clean and users genuinely protected, reversibility is a cheap pre-launch check against a hard-to-undo post-launch harm, dependency is the DPIA needing a data-flow map first, evidence is the fast four-question screening pass, and rank is mood inference first, the step-count feature never.
And if you want to be sure it really works, try it somewhere elseSame five letters, a municipal library system instead of a wellness app. This time the trigger isn't health data at all.
The Faircross Public Library system is piloting an AI tool that flags which patrons might be at risk of losing library access due to overdue fines, so staff can proactively offer a payment plan. Femi Adisa manages digital services there and ran into the exact same question from the city's legal office.
Outcome: catching patrons before they lose access, without building a system that quietly profiles low-income patrons in a way the city couldn't defend if challenged. Reversibility: a flagged patron who's contacted awkwardly, or whose fine history gets shared somewhere it shouldn't, is a real, hard-to-undo harm to someone already in a vulnerable spot. Running the screening check before the pilot expands citywide costs almost nothing by comparison. Dependency: the DPIA can't assess risk without first mapping exactly which patron data feeds the flagging model, something the pilot team had never fully documented. Evidence: a screening pass found the flagging model was, in fact, correlating overdue patterns with zip code in a way that closely tracked income level, a red flag worth investigating before scaling further. Rank: pause the citywide rollout, complete the full DPIA first, and only then decide whether zip code stays in the model at all.
Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "large-scale sensitive data plus a real decision triggers a DPIA, run the screening check first to size it," and stop.
Cost: there's no budget this quarter for a full assessment on every flagged feature. Say so honestly, and start with the screening pass alone, since even a quick check beats assuming consent already covered the question.
The model gets better, for real: if the mood-inference model gets more accurate, that changes nothing about whether the underlying use of the data was ever actually assessed. A more accurate inference is still an inference that needs the same real look.
Where people run it wrong.
They treat a privacy policy and consent screen as automatically satisfying what a DPIA is actually meant to check.
They run the full, heavy assessment on every feature regardless of stakes, burning time the genuinely risky ones need most.
They wait for a partner or regulator to ask for the document instead of screening new features for the trigger criteria as they're built.
How to use it live. When asked what a DPIA is, don't lead with the legal definition. Lead with the question it actually answers: not whether someone agreed, but whether anyone ever checked what they were agreeing to.
Flashcards (tap any card to flip it)
1 · THE FRAMEWORK
What framework fits "what is a DPIA, and when does an AI feature trigger one"?
Tap to flip
ANSWER
ORDER: outcome, reversibility, dependency, evidence, rank. Reversibility explains why a pre-launch DPIA beats a post-launch scramble.
2 · THE PEOPLE
Who is this answer about?
Tap to flip
ANSWER
Leandra Vos, who leads Kindling Health's AI coaching product team and had to explain the DPIA gap to a partner insurer's privacy officer.
3 · WHAT A DPIA IS
What does a DPIA actually check, in plain words?
Tap to flip
ANSWER
Whether a feature's benefit to users is worth the privacy risk it creates, written down before launch, not just whether users technically agreed to it.
4 · THE GAP
What did the screening pass find at Kindling?
Tap to flip
ANSWER
Three of four features meeting the DPIA trigger criteria had never had one, only a consent-language review.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Treating a privacy policy and consent screen as the whole compliance answer for every AI feature, which made sense for the earliest, simplest version of the app.
6 · THE NUMBER
Fill in the blank: the first DPIA took 11 weeks to complete. By the fourth one, it took only ___ weeks.
Tap to flip
ANSWER
3 weeks. The process itself got faster every time it was run.
7 · THE REPLAY
Same insurer request, redesigned process. What changes?
Tap to flip
ANSWER
The mood-inference feature already has a completed DPIA on file, ready to hand over the same day it's requested, instead of triggering a scramble.
8 · CROSS PRODUCT TRANSFER
Section 4 answers this again for a different organization. Which one, and what triggered the DPIA question there instead of health data?
Tap to flip
ANSWER
Faircross Public Library. There, the trigger was a fine-flagging model that turned out to closely track patrons' zip codes and income level.
Check yourself Score: 0 / 0
Fill in the blank
1. Fill in the blank: before the audit, ___ out of 4 features meeting the DPIA trigger criteria actually had a completed assessment.
Show hint
Look at the bar chart comparing required versus completed DPIAs.
Show answer
1. Only the oldest feature, cycle-tracking coaching, had anything close to a real assessment on file.
Multiple choice
2. Why doesn't a solid consent screen and privacy policy satisfy what a DPIA is meant to check?
A. Because consent screens are always written poorly.
B. Because consent answers whether someone agreed, while a DPIA answers whether the specific risk was actually assessed before launch.
C. Because DPIAs replace the need for consent entirely.
D. Because privacy policies are not legally binding.
Show hint
Look at the closing line of the walkthrough.
Show answer
B. The two documents answer genuinely different questions, and Kindling had quietly treated them as the same one for years.
True or false
3. True or false: this answer recommends running a full DPIA on Kindling's step-count reminder feature too.
True
False
Show hint
Look at "what I would leave alone."
Show answer
False. That feature has no profiling and no sensitive inference behind it, so it stays off the DPIA list entirely.
Short answer, name the reversal
4. What old decision does this answer take back, and why did it make sense when it was made?
Show hint
Look at "the decision I would take back."
Show answer
Model answer: Treating consent and a privacy policy as the entire compliance process for every feature. It made sense when Kindling only tracked step counts and sleep hours, with little sensitive inference involved.
Short answer, where it wouldn't matter
5. Name a feature at Kindling where a DPIA genuinely isn't needed.
Show hint
Look at the quadrant sorting Kindling's features by sensitivity and impact.
Show answer
Model answer: The step-count reminder feature. It carries no profiling and no sensitive inference, sitting in the lowest corner of both sensitivity and impact.
Short answer, apply it yourself
6. Think of an app on your phone that uses your health, location, or financial data to make a suggestion. Do you think anyone ever wrote down, in detail, what could go wrong if that suggestion were misused?
Show hint
Think about the difference between a privacy policy you scrolled past and an actual written risk assessment.
Show answer
Model answer: Most people realize they've only ever seen the consent side of the equation, never any evidence of the actual risk assessment behind it, the same gap Kindling had.
Before you close the answer
Why this works
Tests whether you understand a DPIA as a real, written risk check distinct from consent, and whether you can name a concrete trigger instead of reciting a definition with no way to apply it.
Follow-up traps
"Couldn't Kindling have just added a DPIA-shaped clause to its existing privacy policy instead?" Response: no, a clause in a policy is a promise, not an assessment. A DPIA has to actually map the data flow and name the real risk, which a policy clause never does on its own.
"Isn't running four separate DPIAs at once wasteful, compared to one big review?" Response: the features carry different risks, mood inference and a step-count reminder aren't the same problem, so a single combined review would blur exactly the distinctions a DPIA exists to draw out.
If pressed
Kindling's real screening check includes a fifth, informal question beyond the four standard trigger criteria: whether the feature's output could plausibly be requested by a third party, like an employer or insurer, later. That question alone is what first flagged the data-sharing integration as needing a full DPIA, even though its data volume was smaller than the others.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
