What governance does an internal AI tool need that a customer product does not?

Interviewer's question: "What governance does an internal AI tool need that a customer product does not?" Kestrel Mill Textiles runs a fabric mill with about six hundred floor and office employees. Grace Mbeki works as an HR generalist there.

How to answer this, stage by stage

Nobody is grading whether you can name "bias" as a risk. They're grading whether you can name who specifically can't push back, and what design gives them a lever.

Let's learn

AskHR answers employee questions about benefits, leave eligibility, and pay, drawing on the company handbook and the union's collective agreement.

Grace uses it daily to draft answers faster than she could by cross-referencing both documents herself. On a routine question, it's usually right, and she signs off on its draft in under a minute.

The turn: the low overturn rate among new employees was read, for a while, as proof AskHR was more accurate for them. It wasn't. It meant they were less likely to know an answer could be wrong at all, let alone how to challenge one.

Both people are looking at the same kind of answer. Only one of them has a lever to pull if it's wrong.

At its worst: a worker's protected leave gets miscalculated because two policies interacted in a way nobody caught, and the worker, unaware anything was ever uncertain, gets flagged for unauthorized absence and terminated over a mistake they never had a chance to contest.

The fourth step is empty. That's not a missing feature, it's the actual governance gap.

What I would leave alone: routine, low-stakes questions, like how many vacation days remain this quarter, don't need this level of scrutiny. A wrong answer there is annoying, not life-altering, and gets corrected the next time someone checks their balance.

The lesson: a customer who's wronged can walk away and tell someone. An employee who's wronged by a company system often has nowhere obvious to walk to, and designing as if they do is how the harm stays invisible.

Now here is the same thing as a story

The short version above is what you'd say to Kestrel Mill's HR director. Read this one for how the case actually unfolded.

Grace Mbeki has worked HR at Kestrel Mill for three years. She's good at spotting when a leave request touches more than one policy, the kind of case that takes real judgment, not a form.

For most of her time using AskHR, she asked it about a whole case at once, the same way she'd think through it herself. It handled routine cases well. Then came a case with two overlapping leave provisions, medical leave and a separate accommodation policy, and AskHR's answer, drafted for the whole case at once, missed how the two interacted.

Grace caught that particular miss by luck, cross-referencing the two policies herself out of habit. Rattled, she changed how she used AskHR afterward: instead of asking about a whole complex case, she started asking about one clause at a time, wanting to double-check each piece carefully.

The second milestone felt like extra caution. It's exactly what let the third one happen.

Months later, another complex case came through, this time while Grace was managing a busy week. She asked AskHR about each clause separately, confirmed each piece looked right, and never re-assembled the full picture the way she had, by instinct, the first time. The interaction between two policies got missed again, quietly, and this time nobody caught it before the decision went out.

Here's the decision I'd take back: never building a way to handle a complex case in structured pieces safely. The all-or-nothing choice, ask about everything at once or ask about one clause at a time, forced Grace into fragmenting her own view the moment a case got too complex to hold in one query. That made sense when most cases were simple enough not to need it. It stopped making sense the day a case wasn't.

Replayed with a structured breakdown that shows every clause and their interactions on one screen, confirmed by a person before anything is final: the same overlapping-policy case gets flagged automatically, since the design forces the interaction to be visible, not something Grace has to reconstruct from memory. The correct, longer leave period gets approved, and the termination never happens.

I let the query stay all-or-nothing because building structured, cross-referenced views felt like solving a problem nobody had reported yet. It took a wrongful termination to see that nobody reporting it wasn't the same as nobody being harmed by it.

GUARD, run against one leave caseNot a compliance checklist. GUARD is what tells you who has no lever to pull when the answer is wrong.

Two of these four callouts are missing entirely. Both of them are governance, not features.

All three exist specifically because the harmed party often never files anything themselves.

The bottom-left corner is exactly who a governance plan has to be built around, not an edge case to handle later.

The recap, one line per letter: groups is Grace and the employee, unequal is newer and less-connected employees carrying more risk, ability to contest is a worker who never knew to appeal, reduce is mandatory sign-off with a visible flag, and detect is auditing without waiting for a complaint.

And if you want to be sure it really works, try it somewhere elseSame five letters, a community bank's loan-modification desk instead of a textile mill's HR office. Nothing about the two jobs is alike.

Corrigan Trust Bank gives loan officers an internal assistant that evaluates whether a struggling borrower qualifies for a loan modification. Malik Osei is a loan officer there.

Mapped onto GUARD: groups are Malik, who operates the tool and can escalate or override it, and the borrower facing default, who receives its determination. Unequal: borrowers with less financial literacy or English fluency are least likely to know a modification denial can be appealed at all. Ability to contest is the strongest parallel: a borrower who's denied and never told a human hadn't confirmed the decision has no reason to think there's anything left to ask for. Reduce is the same design change: mandatory human sign-off on any denial, with a visible confirmed-or-not flag. Detect is structurally identical too: random audits of denials against lending policy, since a borrower who quietly accepts a wrong denial and walks away never generates a complaint about it.

A different shape of picture than Section 2 used: a branching path instead of a labeled document. The missing branch is the same kind of gap either way.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "require human sign-off on anything touching pay or leave, flag whether a person confirmed it, and audit without waiting for a complaint," and stop.
Cost: if a full structured cross-reference view is too expensive to build immediately, start by simply forcing any multi-policy case to route to a human, no exceptions, until the tooling catches up.
The model gets better, for real: even as AskHR's accuracy improves overall, the subject's inability to contest a wrong answer doesn't improve with it, so the governance requirement doesn't shrink just because errors get rarer.

Where people run it wrong.
They treat a low complaint rate as proof a tool is working well for everyone, without checking whether the quiet group even knows how to complain.
They build governance around the confident, well-connected employee who will ask questions, and miss who actually needs the protection.
They add a review step that's optional or easy to skip under time pressure, instead of one that's mandatory for anything touching pay or protected leave.

How to use it live. When someone asks what governance an internal tool needs, ask yourself first: if this tool is wrong about someone, will that person even know to say so? If the honest answer is no, that's exactly where the mandatory human check has to go.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0