The direct answer
Never let one person sign off alone. Name a second signer, the person who actually reads what goes wrong after launch, not just the one reading the eval dashboard before it. Make their sign-off block the ship, not a note they read afterward. And watch the outcome for the specific case type most likely to break, every month, because the overall number will look fine right up until it doesn't.
Do this, in order
Require a second signer for any change to what the tool can approve on its own.Why: this is the whole reversal. One person alone will always miss the failures a dashboard can't show them.
Pick that second signer for what they see, not their title.Why: it has to be whoever reads the actual corrections and complaints first, not a more senior version of the first signer.
Make the sign-off a real gate that blocks the ship, not a summary sent afterward.Why: being told about a bad call after it shipped gives someone knowledge with no lever, which is the same as never asking them.
Watch the outcome rate for the case type most likely to break, every month.Why: the overall rate can look completely healthy while the harm sits entirely inside one small slice.
Leave single sign-off in place for changes that don't touch what ships unreviewed.Why: a second signer's time is limited, and not every change is a fairness risk worth spending it on.
Ask out loud, before you ship, who can't contest this decision if they disagree with it.Why: that one question finds a missing gate faster than any policy review would.
How to answer this, stage by stage
Seven moves. The first one says what's actually hard about this question before any framework shows up.
1
Say what's actually hard about this question
Say it like this
"When people ask who signs off, they usually mean one name on a launch checklist. For an AI feature that's not enough, because the person reading the eval dashboard and the person who'd see it break for one real case are almost never the same person. So the honest answer names both of them, not just whoever has the launch button."
Why this works
It shows you understand why this question is different before you reach for a framework, instead of jumping straight to a checklist.
2
Name who's in the room, and who isn't
Say it like this
"There's the product lead who reads the score and decides the bar is met. There's the case reviewer who reads the complaints after it ships. And there's the person the form is actually about. Usually only the first one is in the sign-off meeting."
Why this works
This is GUARD's G step, and it sets up the whole answer: only one of three people affected has a vote.
3
Turn the overall number into the slice that actually breaks
Say it like this
"Overall, forms the tool approved on its own only needed fixing after the fact about one percent of the time. But the ones flagged for a name that didn't quite match across documents needed fixing one time in six. The overall number was true. It was also hiding the entire problem."
Why this works
This is GUARD's U step, and a real number lands harder than saying "it's unfair" in the abstract.
4
Say who can't push back, and why that's the hard part
Say it like this
"The case reviewer read every one of those correction tickets. She wasn't asked before the bar moved, and once she saw the pattern, she had no way to pause it. Knowing about a bad call and having a lever on it are two different things. Right now she only had the first one."
Why this works
This is GUARD's hardest and strongest step. It's what separates a real risk answer from a rollout plan with a fairness slide bolted on.
5
Give the one decision, not a policy
Say it like this
"I'd put a real gate back in. Any change to what the tool can approve without a person needs her sign-off too, before it ships, not a note she reads after. And the name-mismatch flag blocks auto-approval on its own, it doesn't just get outvoted by a high overall score."
Why this works
A build someone can ship on Monday beats a review board nobody can point to a year later.
6
Say how you'd catch a bad sign-off without waiting for a complaint
Say it like this
"Every month I'd pull the fix rate for flagged forms on its own, not folded into the overall number. And I'd have someone re-read a sample of the ones that shipped without a person, blind, no score shown, and see how many they'd have caught."
Why this works
This is the detect step, and it's the only thing that catches a bad sign-off before an applicant, a journalist, or a regulator does.
7
Close on the line that isn't hedging
Say it like this
"So: name a second signer who sees what the dashboard can't, make their sign-off block the ship, and track the outcome for the case type most at risk every month, not the number that already looked fine for a whole quarter."
Why this works
Restating the decision in one breath is the part an interviewer actually remembers.
Let's learn
What happens when the person best placed to say "this is ready" is the one person who will never see it go wrong?
Say a small company called Compass Immigration Aid builds a tool that reads someone's answers, fills in a government form, and checks it before they send it in. About eighteen hundred forms a month go through it.
Before the tool, every one of those forms got read in full by a person on the review team, about twenty minutes each. At eighteen hundred forms a month, that's six hundred hours of reading, every month, done by a team of four.
Now the tool checks most of the work itself. It gives each form a score out of a hundred for how sure it is the form is right. Score ninety two or higher, and it tells the applicant, right then, "this looks ready to submit," with nobody reading it first. About seven in ten forms clear that bar. The review team's workload drops from six hundred hours a month to under two hundred.
Here is the part that matters. It isn't that the tool makes mistakes. Every process makes mistakes, a tired reviewer misses a typo too. The real change is in who decides where that ninety two line sits, and how that decision gets made. For the first year, the product lead never moved that bar without asking the review lead to check a sample by hand first. Then, slowly, she stopped asking.
At its worst, this ends up behind the old way, not ahead of it. The old process was slow, but a person read every form once, so a mismatch got caught before it ever reached the government. This one can auto-approve the exact same kind of mismatch for months, and the applicant never finds out a machine decided their form was fine, because that's the entire promise the tool made them.
One of them can move the number. Neither of them can stop what happens next.
The decision I would take back
We let the review lead's sign-off shrink from a full sample of every proposed change, down to a two-line summary, and then down to nothing for small moves. It made sense at the time. The full review took a day to turn around, and the last several moves had all come back clean. It also meant nobody with a reason to slow it down ever got asked again.
What I would leave alone. Not every change needs a second signature. If the product lead wants to reword a question on the intake form, or redesign the applicant's dashboard, that doesn't touch what ships without a person reading it. Single sign-off is fine there. Save the second signer's time for changes that move which forms go out unread.
Nobody signed off on shipping it wrong. Nobody had the power to sign off on stopping it, either.
The lesson. I used to think "who signs off" meant finding the most senior person free that week. It doesn't. It means finding whoever will actually see the form that breaks, and giving them a real vote before it ships, not a summary after. A sign-off that only one kind of evidence can reach is a sign-off that's already missing half the picture.
Now here is the same thing as a story
The short version is above. Read this one when you want to feel why the fix matters, not just know what it is.
Anke has shipped things at Compass Immigration Aid for three years. She's good at the part of the job most people find boring: reading a dashboard full of numbers and knowing, fast, whether a launch is actually ready or just looks ready.
When the auto-confirm feature first went out, eighteen months ago, she never moved the approval bar alone. She'd pull a sample of the borderline cases, the ones sitting right near the cutoff, and send them to Grace, who ran the review team and had read immigration forms for eight years. Grace would read the sample by hand and say "these are fine" or, sometimes, "wait, look at this one." It became the two of them, every time, before anything shipped.
For the first year, that ritual held. Six months in, to save Grace the day it took to read a full sample, Anke started sending a two-line summary instead: what moved, what the pass rate did. Grace kept saying "looks fine" to that too. A few months after that, Anke started shipping the smaller moves, a point or two on the bar, without sending anything at all. The last four had all come back clean. Asking about a fifth started to feel like a formality neither of them had time for.
This quarter's move was small. Ninety five down to ninety two, three points, meant to clear a few more forms a week off the review queue. It went out the way the last few had: without asking.
Knowledge spark: what's a Request for Evidence
It's the government's way of saying: not yet, send more proof. It isn't a no. It's a pause, and it means gathering documents again, sometimes for months, while whatever the form was supposed to unlock waits too.
One of the forms that went out that way belonged to a woman named Halina Wojcik. Her passport carried an accent mark in her name, Wójcik, that an earlier translated document had dropped. The tool's own check flagged the mismatch. But a single flagged field barely moved her overall score, which cleared ninety two anyway, so the message she got was the same one everyone gets: "this looks ready to submit." She trusted it, the way the product asked her to. Months later, a request for evidence arrived over the mismatch, and the paperwork gap cost her several weeks of unpaid leave while her employer sorted out whether she could keep working through it.
Grace found the pattern the way she finds everything: reading the correction queue, which was always her job, nobody had to hand it to her. A cluster of forms needing fixes, all carrying that same name-mismatch flag, all auto-approved anyway. She opened a ticket. Nobody with the power to pause the bar saw it for ten days. In that stretch, about thirty more flagged forms shipped the same way Halina's had.
The ten days weren't the real cost. The real cost was that nobody had ever built her a way to stop it, only a queue to complain into after.
It was never really about the model's accuracy. Grace never had a number in her head for "when do I get asked." She had a feeling, and it only had two settings: either she was part of the decision before it shipped, or she read about it afterward with nothing to do about it. There was no version where she was a little bit part of it.
I remember the meeting, eight months before any of this, where we decided the two-line summary was enough. It felt efficient. Grace was busy, we were busy, and the full review cost a whole day every time. Cutting it down saved everyone real time, every single week it worked.
Run that same quarter again, but Grace has to sign off on any move to the bar before it ships, not read about it after. The ninety five to ninety two proposal sits in her queue two days, about as long as it takes her to read the sample Anke already has ready. She flags three cases like Halina's. The bar goes out at ninety three instead of ninety two, and the name-mismatch flag now blocks auto-approval on its own, whatever the overall score says. Same tool. Two extra days on a launch that used to ship the same afternoon. None of that quarter's flagged forms go out unread.
One design gave Anke a dial only she could turn. The other gives the two of them a lock neither can open alone. And the thing I'd tell myself, back in that first meeting: we asked whether the summary was fast enough. We never asked what happens the one time it's wrong and nobody's watching for it.
The five letters, run on a sign-off meeting instead of a support case
This is a risk question, so the framework is GUARD. "Who signs off" sounds like a process question. It's actually asking who holds the lever and who has to live with how they use it, which is exactly what GUARD is built to find.
G, groups. Who has the vote, and who lives with a bad one? Anke has always had it, alone, since the two-line summary replaced the full review. Grace and Halina live with what she decides, and only Grace even knows a decision got made.
U, unequal. Where does a bad call land hardest? Not on a random slice of applicants. It lands on the ones whose name doesn't type the same way twice, an accent mark here, a hyphen there. A single mismatched field barely dents the overall score. The outcome it causes is nowhere near small.
Forms needing a fix after filing, by whether the name flag showed
Same quarter, same approval bar. All of these scored above ninety two and went out with nobody reading them first.
Carried the name-mismatch flag
17%
Seventeen times the failure rate, and the overall fix rate never showed it. It sat at about one percent the whole quarter, because it was always an average of a small flagged group and a much bigger group that almost never needed a second look.
A, ability to contest. Who can't push back on a call they'd disagree with? Grace saw the flag first, every single time, because reading the correction queue is her actual job. She never got a vote before a change shipped, only a ticket to file after it already had.
Her box exists on the org chart. It was never wired into the chain that ships.
R, reduce. What's the real build, not the policy? Any move to the approval bar needs Grace's sign-off before it ships. And the name-mismatch flag blocks auto-approval by itself, it can't be outvoted by a high overall score anymore.
D, detect. How would you know a bad sign-off happened, before someone outside tells you? The fix rate on flagged forms, watched on its own, every month. Not the blended rate. That one stayed calm the entire time this was happening.
Where this answer would fail
If the fix here were a review committee, a launch checklist item that just says "risk reviewed," or a training deck on bias, none of it counts. "Her sign-off blocks the ship" is a build someone can turn on this week, and you can check afterward whether they did.
And if you want to be sure it really works, try it somewhere else
A school district's tool drafts goals for a student's special-education plan, and a coordinator signs off before it goes to the family. Different sector, same five letters, same trap.
G, groups. The special-education coordinator who approves the AI-drafted goals, and the specific student the plan is actually for.
U, unequal. The tool drafts strong, specific goals for common, well-documented conditions. For a student with more than one qualifying condition at once, it tends to produce generic, boilerplate goals that don't fit either one well, and those are exactly the students who need the plan to fit.
A, ability to contest. The speech therapist who works with that student twice a week sees the goal fail in the room within days. She isn't part of sign-off at all. Only the coordinator's stamp is required.
R, reduce. Any AI-drafted plan for a student with more than one qualifying condition needs sign-off from the specific specialist working with that student, not just the coordinator's approval.
D, detect. How often a goal gets rewritten within the first grading period, split by whether the student has one condition or several. Not the overall rewrite rate, which stays low because most students have just one.
Swap the trigger and it still runs
- Speed: intake triples during a filing-deadline surge and nobody revisits who actually has time to review anything. The second-signer step gets skipped "just this once," repeatedly, until it's just skipped.
- Cost: leadership trims the review team because the model's overall accuracy looks strong. That removes the exact people who would ever notice the pattern in the first place.
- The model gets better: overall accuracy climbs, so fewer cases route to a person at all. The reviewer who used to see forty cases a week now sees four, and stops building the pattern-recognition that let her catch it the first time.
Where people run it wrong
- Counting "we looped them in on a monthly summary" as the same thing as giving them a vote before anything ships.
- Testing the overall pass rate and calling it fairness testing, when the entire problem lives inside one slice the average never shows.
- Writing a policy that names a second reviewer, without ever wiring their sign-off to actually block the release.
If this question hits you cold
Ask who currently can't stop a launch they think is wrong, specifically, by name or by role. That question buys you ten seconds of thinking time, and it usually answers itself, because most teams never gave that person a real lever, only a way to complain once it's too late.
Flashcards (click a card to flip it)
1 · THE FRAMEWORK
Which framework fits a question about who signs off on an AI feature, and why?
Tap to flip
ANSWER
GUARD, for risk, safety and fairness. Sign-off is really asking who holds the lever and who has to live with how they use it, which is exactly what GUARD is built to find.
2 · THE PEOPLE
Who does this answer name, and who actually has a vote on the launch?
Tap to flip
ANSWER
Anke Voormann, the product lead who sets the approval bar. Grace Manalo, the review lead who reads the corrections after they happen. Halina Wojcik, an applicant whose flagged form went out unread. Only Anke had a vote.
3 · THE HABIT
What did Anke quietly stop doing, and over how long?
Tap to flip
ANSWER
Sending Grace a full sample of borderline cases before every move to the bar. That shrank to a two-line summary after about six months, then to nothing for small moves, over roughly a year.
4 · THE FLIP
What's the two-setting switch in this story?
Tap to flip
ANSWER
Grace goes from being asked before a change ships, to reading about it after, in a ticket, with no lever to pause anything. There's no setting where she's a little bit part of the decision.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Letting Grace's review shrink from a full sample to a two-line summary, then to nothing for small moves. It made sense because the full review took a day and the last several moves had all come back clean.
6 · THE NUMBER
Fill in: forms with the name-mismatch flag needed a fix after filing ______ percent of the time, against ______ percent for forms with no flag.
Tap to flip
ANSWER
Seventeen percent, against one percent. The overall fix rate stayed near one percent the entire quarter, because flagged forms were a small slice of the total.
7 · THE REPLAY
Same quarter, Grace has to sign off first. What changes, and by how much?
Tap to flip
ANSWER
The move from ninety five to ninety two sits in her queue two days instead of shipping the same day. She flags three cases like Halina's. The name flag now blocks auto-approval on its own. None of that quarter's flagged forms ship unread.
8 · TRANSFER
Section four runs GUARD again on a different product. Which one, and what does the reduce step become?
Tap to flip
ANSWER
A school's tool for drafting special-education plans. Reduce: any AI-drafted plan for a student with more than one qualifying condition needs sign-off from the specific specialist working with that student, not just the coordinator's stamp.
Check yourself Score: 0 / 0
Short answer
1. What old decision does this answer take back, and why did it make sense when Compass first made it?
Show hint
Look for the meeting that decided a two-line summary was good enough.
Show answer
Model answer: "Letting Grace's review shrink from a full hand-read sample to a two-line summary, and later dropping it entirely for small moves to the bar. It made sense at the time because the full review took a day to turn around, and the last several moves had all come back clean, so asking started to feel like a formality that was slowing down real, useful launches."
Multiple choice
2. A teammate says the approval bar is safe to lower again because the overall pass rate looks fine. Per the reduce step, what's the actual fix?
- A. Lower the bar a little less than planned this time.
- B. Require the review lead's sign-off before any move to the bar ships, and let the name flag block approval on its own.
- C. Add a banner telling applicants a machine checked their form.
- D. Retrain the model on more forms and check again next quarter.
Show hint
The reduce step is a specific build, not a smaller or bigger version of the same automatic step.
Show answer
B. A and D just turn the same dial a little differently. C is honest, but a banner gives nobody an actual lever to stop a bad form before it ships. Only giving the review lead a real vote, and letting the flag matter on its own, changes what actually goes out.
True or false
3. True or false: because the overall fix rate on auto-approved forms stayed near one percent the whole quarter, there wasn't really a problem.
Show hint
Where would a problem in one small, badly-affected slice hide inside a number that averages everyone?
Show answer
False. Forms carrying the name-mismatch flag needed a fix seventeen times more often than forms without it. An average is exactly the place that gap hides, because flagged forms were a small slice of the total, so their much worse rate barely moved the overall number.
Fill in the blank
4. Auto-approved forms carrying the name-mismatch flag needed a fix after filing about ______ percent of the time, against about ______ percent for forms with no flag.
Show hint
It's the pair of numbers behind the two-bar chart in the GUARD section.
Show answer
Seventeen percent, against one percent. That gap sat inside the data for the whole quarter, hidden by an overall rate that looked completely calm.
Short answer, apply it yourself
5. Pick an AI feature you've used or built. Who signs off on it today, and who sees its worst failures first but has no vote on it?
Show hint
Look for whoever reads the complaints or corrections, not whoever reads the dashboard.
Show answer
Model answer: "A tool that auto-replies to simple customer emails at a company I worked with. The support lead signed off on the auto-reply bar alone. The agents handling escalations saw every reply that went out tone-deaf to an angry customer, and they had no say in where that bar sat, only a shared doc to log complaints nobody read weekly." Any real answer works if it names someone who sees the failure and has no vote on the setting that caused it.
Multiple choice
6. A team says: "Grace already sees this, she gets a monthly summary of flagged forms." What's wrong with counting that as her sign-off?
- A. Nothing, a monthly summary is frequent enough for a review like this.
- B. A summary lets her notice a pattern eventually, but it gives her no way to stop or change one form before it ships.
- C. The summary should be sent weekly instead of monthly.
- D. Grace needs more training on how the tool scores forms.
Show hint
Ask what she can actually change, not how often she gets to look.
Show answer
B. A monthly report is real information, but it arrives after the forms already went out. B is the only option that gives her an actual lever on one decision before it happens. A, C, and D just adjust the same after-the-fact summary; none of them hand her a vote.
