Explain the tension between privacy commitments and model improvement.

What happens when the safest thing for your privacy policy is the worst thing for the people your model is already getting wrong? Almbrook Rental Screening runs LeaseGuard, a tool that scores rental applicants for landlords. Ines Baptiste is the AI PM who has to explain, in one meeting, why keeping a privacy promise and improving the model can't both be fully true at once.

How to answer this, stage by stage

Nobody is scoring whether you can define privacy. They're scoring whether you'll name the person who has no way to push back.

Let's learn

Here is what happens when the safest-looking privacy decision quietly becomes the worst decision for the people already being misjudged the most.

Before LeaseGuard, a leasing agent reviewed each rental application by hand: pay stubs, references, a gut read, about twenty minutes a file. With LeaseGuard, a risk score comes back in seconds, built from credit history, rental history, and income documents. The tool started as a small pre-filter agents could override freely. Two years later, most agents just accept the score.

Both people are affected by the same score. Only one of them has any way to touch it.

Here's the turn: Almbrook's privacy policy promises to delete applicant data 30 days after a decision. That's a real, meaningful promise. It also means the one record of every case the model got wrong disappears before anyone can learn anything from it, and it disappears fastest for the applicants whose cases were hardest to score correctly in the first place.

At its worst, a policy meant to protect people from misuse can quietly widen the exact gap it was never designed to touch, and nobody notices because the applicants losing the most are the ones with the fewest ways to be heard in the first place.

What I would leave alone: I wouldn't change the 30-day deletion window for applicants who were approved and never contested anything. There's no tension to resolve there, since nothing about their case needs revisiting.

The lesson: a privacy commitment and a model-improvement need don't cancel each other out just because they pull in different directions. The job is building the one narrow exception that serves both, not picking whichever one sounds better in a press release.

Now here is the same thing as a story

The short version above is what you'd say defending a data-retention decision to legal. Read this one for how a fear of one headline created a different, quieter one.

Before the panic, LeaseGuard denied and approved applicants the same way for two years: a score came back, a leasing agent read it, and denial letters went out with a form paragraph nobody thought much about.

Petra Vondrak has worked leasing for a mid-size property group for five years. Part of her job, quietly, without anyone asking her to, was stapling a short disclosure sheet to every denial letter: which factors the score weighed, and a phone number to call with questions. Almost nobody called. She kept doing it anyway, because it felt like the right thing to hand someone on their way out the door.

The gap in the middle is the whole tension. Nothing was ever built to sit there.

Then a rival screening vendor got hit with a regulatory fine for keeping applicant data years past any reasonable need. Almbrook's leadership, understandably rattled, ordered every data-retention policy tightened within the month: 30-day deletion, no exceptions, no manual holds.

Petra kept stapling her disclosure sheets for a few more weeks. Then a corporate memo, following the same fear that drove the retention change, told leasing staff to stop mentioning that a model was involved in scoring at all, out of concern that saying so created legal exposure. Petra stopped stapling the sheet. Applicants started receiving a denial letter that said only "does not meet our leasing criteria."

Neither object is wrong on its own. The tension is that both are true for the same case at the same time.

Nobody at Almbrook decided, on any single day, to make the model worse for thin-file applicants specifically. The deletion policy applied to everyone equally. But applicants with thin credit files, gig workers, recent immigrants, people paid irregularly, were exactly the cases the model handled worst, the ones a human reviewer used to catch and Petra used to explain. Once those cases got deleted before anyone reviewed them, the model kept making the same mistake on the next thin-file applicant, and the one after that.

Each step made sense on its own. Together, they removed every way an applicant could have been heard.

It took a tenant advocacy group, running their own numbers from public eviction and rental records, to notice that denial rates for applicants with thin credit files had climbed nearly 13 points over two quarters while approvals for everyone else barely moved.

GUARD, the lever nobody checked who was holdingNot a compliance checklist. GUARD is what tells you exactly which product decision quietly took someone's appeal away.

The recap, one line per letter: groups is the leasing team holding the lever against applicants who don't, unequal is thin-file applicants absorbing nearly all of the harm from an equally-applied policy, ability to contest is applicants never even being told a model was involved, reduce is a narrow, consent-gated record of contested cases only, and detect is a standing segment-level denial-rate check that catches drift before an outside group does.

And if you want to be sure it really works, try it somewhere elseSame five letters, a ticketing platform's fraud screen instead of a rental application. Different flip family entirely, the same missing appeal.

Gatewell Ticketing runs a fraud-risk model that flags buyers likely to be reselling tickets in violation of platform rules, quietly capping their purchase limits with no notice. Mapped onto GUARD: groups are Gatewell's fraud team, who can review any flagged account, and buyers, who are never told they've been flagged at all. Unequal lands on buyers who happen to purchase in patterns common among families buying for large groups, exactly the pattern the model can't yet tell apart from resale behavior. Ability to contest is the same gap: a flagged buyer sees a purchase limit with no explanation and no path to ask why. Reduce is the same fix, adapted: a visible, disputable flag with a human review path, not a silent cap. Detect is tracking false-flag rate by purchase-pattern segment. The flip here is an input one, not a concealment one: Simeon Drury, a fraud-review analyst, used to write plain notes explaining why he'd cleared a flagged account. Once he noticed which phrases the model's own retraining seemed to weight most, he started wording his notes to match those phrases instead of describing what he actually saw, contorting his own language to feed a system he no longer fully trusted to read him straight.

The same four parts fix the gap in a rental application and a ticketing fraud flag alike.

Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "keep the contested cases, delete the rest, and tell people a model was involved so there's something to contest in the first place," and stop.
Cost: no budget for a full appeals system this quarter. Say so honestly, and start with plain disclosure alone, since that costs almost nothing and is the precondition for everything else.
The model gets better, for real: if disclosure and a real appeal path actually shrink the denial-rate gap over two quarters, that's the fix working as intended, and the honest move is to say so, not assume the tension is permanent.

Where people run it wrong.
They treat "delete everything" as the automatically safe choice, without checking who that choice actually harms.
They build a review process for the operator's convenience and call it an appeal, when the subject never even knows there's something to appeal.
They wait for a segment-level gap to show up in a lawsuit or a news story instead of a monthly check they built themselves.

How to use it live. The moment someone asks about this tension, ask yourself: who here can push back on a wrong decision, and who has no idea one was even made? Name both, and the rest of the answer follows.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0