ConceptAdvancedResponsible AI & Advanced Practice / Compliance and legal partnership / #2
Explain risk categorization under the EU AI Act in product terms.
GUARD the product is LoanSight, an AI credit-decisioning tool at Aurelia Credit Union
Aurelia Credit Union serves members across Ireland and Germany. LoanSight is the tool that scores every loan application before a human ever sees the file. Priya Nakamura runs model risk for the lending team, and keeps a running list, taped inside a drawer, of every AI feature the credit union has shipped.
The direct answer
Sort every AI feature by one question: what happens to the person on the receiving end if it's wrong? Manipulate or score people as citizens and it's banned outright. Decide credit, jobs, health, or safety and it's high-risk, which means human oversight, a log of every decision, and a way to appeal, all built before launch, not bolted on after a regulator asks. A chatbot just has to say it's a chatbot. A spam filter needs nothing extra at all.
Do this, in order
Classify by consequence to the person, not by how the model works.Why: two features using the same model can sit in different tiers if their outcomes differ.
Build the appeal path for anything high-risk before it ships, not after a complaint.Why: it's the one piece regulators check for first, and the one most teams skip.
Give the human reviewer a real reason to look, not just a rubber stamp.Why: oversight that never overrides anything isn't oversight.
Log every score, even the ones nobody disputes.Why: you can't prove fairness later without a record from day one.
Leave the low-stakes features alone.Why: piling high-risk paperwork onto a spam filter wastes effort the real risk needs.
How to answer this, stage by stage
Nobody is grading whether you recite every article number. They're grading whether you can sort a real feature into a tier and say why, out loud, in one breath.
Stage 1
Pick one concrete feature
Say it like this
"I'll answer this against LoanSight, a real credit-decisioning tool, instead of talking about the Act in the abstract."
Why this works
A law with four tiers stays abstract until it's pinned to one product a listener can picture.
Stage 2
Say your structure out loud
Say it like this
"I'll use GUARD. Groups affected, where the harm lands unevenly, who can actually push back, the design fix, and how we'd detect it in production."
Why this works
Tells the interviewer this isn't a legal recitation, it's a product framework applied to a legal question.
Stage 3
Reframe the question
Say it like this
"The Act isn't really sorting technologies. It's sorting how much power an AI decision has over one person's life, and whether that person ever finds out."
Why this works
This is the line that separates a candidate who read the Act from one who understands why it exists.
Stage 4
Give the one decision
Say it like this
"LoanSight decides credit, so it's high-risk under Annex III. That means before launch, not after: a human who can override it, a log of every score, and an appeal path for anyone denied."
Why this works
Names the tier, names the trigger, and names the three obligations, all in one breath. That's deliverable 0 in spoken form.
Stage 5
Prove it with the gap
Say it like this
"When we launched LoanSight we had the override and the log. We didn't have the appeal path, and it took an internal audit finding a thin-file applicant with no way to challenge her denial before we built one."
Why this works
A real gap, found and fixed, is worth more than a clean recitation of the rule.
Stage 6
Say what you'd watch for
Say it like this
"I'd track appeal requests by applicant segment. If thin-file, self-employed applicants file appeals at three times the rate of everyone else, that's the tier's whole reason for existing, showing up in a number."
Why this works
Shows the classification isn't paperwork, it's a thing you monitor after launch too.
Stage 7
Close on the one line
Say it like this
"Sort by what happens to the person, not by what the model does. That's the whole test, for any feature you'll ever ship."
Why this works
Restates the decision so the interviewer's last impression is the answer, not the story.
Let's learn
LoanSight is Aurelia Credit Union's AI tool for scoring loan applications. It reads a member's income, spending, and credit history and hands the loan officer a recommendation before the officer ever opens the file.
Before the EU AI Act, Aurelia treated every AI feature the same way: ship it, watch the accuracy number, fix bugs as they came in. Nobody sorted features by how much power they held over a person's life.
Knowledge spark: what does "high-risk" actually mean under the Act?
It's not a warning label. It's a specific list of uses in Annex III, credit, hiring, health, education, policing, that trigger real obligations: a human who can overrule the system, a log of every decision, testing for bias, and a way for the affected person to contest the outcome. Miss any of the four and the feature isn't compliant, no matter how accurate the score is.
Now, under the Act, every feature needs a tier before it ships. LoanSight lands in the highest tier that still allows the product to exist at all: high-risk, not banned. A tool that scored members for a social credit-style ranking would be banned outright. LoanSight decides who gets a loan, which is serious, but not manipulation.
Loan denial rate by applicant credit-file thickness
Thin-file applicants, mostly self-employed or new to the country, are denied more than four times as often. Nobody built them a way to ask why.
The turn: the extra denials among thin-file members are not really the problem the Act is aimed at. The real problem is that none of those members has any way to see the reason or challenge it. LoanSight's score just arrives, final, in the loan officer's queue.
The high-risk obligations weren't optional reading. Aurelia had a fixed date to be ready by, same as every other lender using AI to decide credit.
The decision I would take back
We shipped LoanSight with a human override and a full decision log, because those were the two obligations we knew about at launch. We left out the applicant-facing appeal path, since no member had ever asked for one and the compliance list we were working from was two years old. That made sense before the Act's Annex III rules bound us. It stopped making sense the day an internal audit pulled ten denied files and found a thin-file member with a strong repayment history and zero way to ask what went wrong.
What I would leave alone: Aurelia's internal spam filter for the member-support inbox needs none of this. Nobody's life changes if a filter misfires. Piling human oversight and appeal paths onto it would waste effort the real high-risk features actually need.
The Act was never really sorting technologies. It was sorting how much power a decision holds over one person, and whether that person ever gets to ask why.
The lesson: a risk tier isn't a compliance label you attach after the model works. It's a design brief. High-risk means build the appeal path on day one, the same day you build the score itself.
Now here is the same thing as a story
The short version above is what you'd say defending LoanSight's classification to a regulator. Read this one for how the gap actually got found.
Priya Nakamura has run model risk at Aurelia for six years. She can tell a genuinely risky feature from a cosmetic one almost on sight, mostly by asking one question: who loses something if this is wrong.
LoanSight launched with two of its three high-risk obligations built in from day one. A loan officer could always override the score. Every score, override or not, got logged with a timestamp. Priya was proud of both. The appeal path sat lower on the list, filed under "eventually," because nothing about it felt urgent.
Every AI feature at Aurelia needed one of these four labels. LoanSight, deciding credit, only ever fit in one of them.
Then came the internal audit, six months after launch. Someone from Aurelia's own risk committee pulled ten denied files at random, the ordinary kind of spot check nobody expects to find anything in.
The loan officer always had a lever. The applicant never did. That gap is exactly what the Act's high-risk rules are built to close.
One file belonged to a member who ran a small bakery, paid every bill on time for four years, and got denied anyway. She had called the branch twice asking why. Both times, the answer was some version of "the system flagged it," because nobody at the branch could say more than that.
Four steps ran smoothly. The fifth one, the missing appeal step, was the whole gap the audit found.
Priya sorted every feature Aurelia had shipped against the Act's four tiers, using the test she'd been using informally for years: what happens to the person if this is wrong.
LoanSight and a resume screener land in the same corner: high impact, no way to push back. That corner is the Act's high-risk tier, drawn as a picture.
She built the appeal path in three weeks: a plain-language denial reason, a form to request human review, and a ten-day service commitment on every request.
Two of the four were already live. The bakery owner's file was missing the other two.
The bakery owner, told about the new process, filed the first appeal under it. A human reviewer looked at her file properly for the first time and approved the loan nine days later.
Appeal requests filed per month, since the appeal path launched
Once members knew they could ask, they asked. Every one of these 45 monthly appeals was invisible before the path existed, not because nobody needed it.
The old process gave the loan officer a lever and gave the applicant nothing. The new one gives the applicant a lever too, sized for how much the decision actually costs her.
I filed the appeal path under "eventually" because it felt like the least urgent of the three obligations, and nothing in our metrics ever flagged it as missing. It took one bakery owner's two unanswered phone calls to see that "nobody's asked yet" and "nobody can ask" are very different things, and we'd quietly built the second one.
GUARD, sorted by tierNot a compliance checklist. GUARD is what forces you to name who has the lever and who doesn't.
G
Groups. Who is affected.
The loan officer, who operates LoanSight, and the applicant, the subject on the receiving end of its score.
Names both, not just the operator.
U
Unequal. Where the harm lands.
Thin-file applicants, self-employed or new to credit, get denied four times as often, with thinner data behind each score.
Points at the specific group the tier exists to protect.
A
Ability to contest. The hard step.
The applicant had no way to see her denial reason or ask for a human look, until the audit forced the question.
The whole reason a feature lands in the high-risk tier instead of a lighter one.
R
Reduce. The design fix.
A plain-language denial reason, a request-review form, and a ten-day response commitment.
A concrete build, not a training session or a policy memo.
D
Detect. How you'd know.
Track appeal rate by applicant segment. A gap between segments, not just the total, is the signal worth watching.
Turns the tier from a launch requirement into an ongoing watch.
The recap, one line per letter: groups is the officer and the applicant, unequal is thin-file members carrying four times the denial rate, ability to contest is the missing appeal path the audit found, reduce is the plain-language reason and review request, and detect is watching the appeal rate by segment going forward.
And if you want to be sure it really works, try it somewhere elseSame five letters, a factory floor instead of a credit union. This time the tier is employment, and the appeal question looks completely different.
Kestrel Manufacturing uses an AI resume screener to shortlist assembly-line applicants before a recruiter ever reads a resume. Dov Elazar runs HR operations there, and the classification question is the same one Priya faced, sorted through the same test.
Kestrel's resume screener and Aurelia's LoanSight take different branches on the way in, hiring versus credit, and land in the exact same tier.
Groups: the recruiter who reads the shortlist, and the rejected applicant who never learns why they didn't make it. Unequal: applicants who worked gig jobs list scattered short-term roles that the screener reads as instability, when it's really just how gig work looks on paper. Ability to contest: Kestrel's rejected applicants get a form email with no reason attached, so nobody can even name what to appeal. Reduce: Dov's fix is a short, specific rejection reason, "screened out for missing a required certification," not a generic no. Detect: track how often rejected applicants who later reapply, with no resume changes, get accepted anyway, since that's a sign the first screen was noise, not signal.
Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "sort by what happens to the person if it's wrong, high-risk gets oversight and an appeal path, built before launch," and stop.
Cost: there's no budget this quarter for a full appeal system. Say so honestly, and start with a plain-language denial reason and a single email inbox a human actually reads, since even a manual appeal beats none.
The model gets better, for real: if LoanSight's accuracy improves, the tier doesn't change. A more accurate high-risk system is still a high-risk system, and it still needs the same oversight and appeal path.
Where people run it wrong.
They classify by the technology, "it's just a scoring model," instead of by what happens to the person on the other end.
They treat the appeal path as paperwork to add after launch instead of a design decision made before it.
They build oversight that never actually overrides anything, which satisfies the letter of the rule while missing its whole point.
How to use it live. When asked to explain the tiers, don't start with the four names. Start with the one question that sorts everything: what happens to the person if this is wrong, and can they do anything about it. The tier falls out of the answer to that question every time.
Flashcards (tap any card to flip it)
1 · THE FRAMEWORK
What framework fits "explain risk categorization under the EU AI Act in product terms"?
Tap to flip
ANSWER
GUARD: groups affected, unequal harm, ability to contest, reduce, detect. The ability-to-contest step is the one the four tiers are really built around.
2 · THE PEOPLE
Who is this answer about?
Tap to flip
ANSWER
Priya Nakamura, who runs model risk for Aurelia Credit Union's lending team and keeps a running list of every AI feature they've shipped.
3 · THE GROUPS
Who are the two people GUARD names in LoanSight's story?
Tap to flip
ANSWER
The loan officer, who can override the score, and the applicant, who couldn't see her denial reason or contest it.
4 · THE GAP
What was missing from LoanSight at launch?
Tap to flip
ANSWER
The appeal path. A human override and a decision log existed at launch. A way for the applicant to contest a denial did not, until an internal audit found the gap.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Shipping LoanSight with only two of the three high-risk obligations, filing the appeal path under "eventually" since no member had ever asked for one.
6 · THE NUMBER
Fill in the blank: thin-file applicants get denied about ___ percent of the time, versus 8 percent for thick-file members.
Tap to flip
ANSWER
34 percent. More than four times the thick-file rate, with no way for those applicants to ask why.
7 · THE REPLAY
Same bakery owner, redesigned process. What changes?
Tap to flip
ANSWER
She gets a plain-language denial reason, files an appeal, a human reviewer looks properly, and her loan is approved nine days later.
8 · CROSS PRODUCT TRANSFER
Section 4 answers this again for a different product. Which product, and what's different about its ability-to-contest gap?
Tap to flip
ANSWER
Kestrel Manufacturing's resume screener. There, rejected applicants get a generic form email with no reason at all, so they can't even name what to appeal.
Check yourself Score: 0 / 0
Multiple choice
1. Why does LoanSight land in the high-risk tier rather than the banned tier?
A. Because it's a credit union, not a bank.
B. Because it decides credit outcomes for real people, which is serious but not manipulation or social scoring.
C. Because it only affects a small number of members.
D. Because it was built before the Act existed.
Show hint
Look at the labeled parts diagram of the four tiers.
Show answer
B. Banned covers manipulation and social scoring. Deciding credit is Annex III's high-risk category, serious enough for real obligations, not serious enough to prohibit outright.
True or false
2. True or false: once LoanSight had a human override and a decision log, it fully met its high-risk obligations.
True
False
Show hint
Look at the icon list of what high-risk status requires.
Show answer
False. It was still missing the applicant's appeal path, the third obligation, which is what the internal audit surfaced.
Fill in the blank
3. Fill in the blank: within six months of launching the appeal path, monthly appeal requests rose to about ___.
Show hint
Look at the line chart of appeal requests by month.
Show answer
45. Up from just 3 in the first month, showing demand that had been invisible, not absent, before the path existed.
Short answer, name the reversal
4. What old decision does this answer take back, and why did it make sense when it was made?
Show hint
Look at "the decision I would take back."
Show answer
Model answer: Filing the appeal path under "eventually" at launch. It made sense because no member had ever asked for one, and the compliance checklist in use at the time was two years old.
Short answer, where it wouldn't matter
5. Name a place at Aurelia where this same tiering discipline genuinely doesn't matter.
Show hint
Look at "what I would leave alone."
Show answer
Model answer: The member-support inbox's spam filter. Nobody's life changes if it misfires, so it needs no oversight or appeal path at all.
Short answer, apply it yourself
6. Pick an AI feature you've used yourself. If it made a wrong call about you, could you find out why, and could you push back? What tier does that put it in?
Show hint
Think of an app that scored, ranked, or decided something about you, and whether it ever explained itself.
Show answer
Model answer: Most people land on something like a loan, insurance, or job-matching tool that gave a flat result with no visible reason, the same shape as LoanSight before the appeal path existed.
Before you close the answer
Why this works
Tests whether you can turn a four-tier legal framework into a real product decision, sorted by consequence to a specific person, rather than reciting the Act's structure from memory.
Follow-up traps
"Couldn't you just make every AI feature meet the high-risk bar, to be safe?" Response: no, that wastes real effort. Aurelia's spam filter needs nothing extra, and piling oversight onto it steals attention from features where someone's life is actually on the line.
"Isn't a human override enough oversight on its own?" Response: not without the appeal path. An override that only the loan officer can trigger protects the officer's judgment, not the applicant's ability to be heard.
If pressed
Aurelia's real appeal path routes any request tied to a protected characteristic, age, disability, national origin, to a senior reviewer automatically, not the original loan officer, so the same person who denied the loan never rules on the appeal.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
