InterviewAdvancedAI Opportunity & Model Strategy / Build vs buy vs fine-tune decisions / #23
Walk me through a build-versus-buy decision you would make for a company I name.
GUARDthe applicant never got a lever, and buying the vendor's score is exactly what took it away
Say the interviewer names Cassowick Property Analytics, a company that sells tenant-screening software to property managers. Its risk score reads a rental applicant's credit history, eviction records, and income, and tells a leasing agent whether to approve, deny, or send the file for manual review. Simone Havelka is the AI PM who has to decide what Cassowick buys from a vendor and what it builds itself.
The direct answer
Buy the commodity data layer, the credit bureau and eviction-record feeds every screening company needs the same way. Build in-house the one piece that decides whether an applicant ever gets a fair shot: a match-confidence check on eviction records, and a dispute flow that catches a wrongful match before it becomes a denial nobody can contest. That's the actual seam, and it's not where the cost sits, it's where the power to push back sits.
Do this, in order
Ground the decision in Cassowick's actual risk score, not a generic build-vs-buy lecture.Why: the method only works pinned to something real, an interviewer can tell the difference immediately.
Buy the commodity data layer: credit and eviction-record feeds every screening company needs the same way.Why: rebuilding solved data infrastructure wastes the exact budget the sensitive piece needs most.
Build the record-matching confidence layer in-house, since that's where a wrong call costs someone their home.Why: a vendor's black-box match has no visibility into false-positive eviction hits.
Give the applicant a real way to see and dispute a flagged record before a denial is final.Why: right now they're the one person in this decision with no lever at all.
Track denial rate and override rate by match confidence in production, not just overall accuracy.Why: it's the signal that would show the gap before a wrongful denial becomes a lawsuit.
Name the trade-off plainly: this costs more and launches slower than a pure buy.Why: in exchange, you get a defensible, contestable decision instead of a fast, opaque one.
How to answer this, stage by stage
Nobody is scoring whether you can name five build-vs-buy criteria from memory. They're scoring whether you notice, unprompted, that someone in this decision has no way to push back.
Stage 1
Scope it to the named company's actual decision
Say it like this
"Let's say you name Cassowick Property Analytics, tenant-screening software for property managers. I'd ground the build-vs-buy call in their actual risk score, whether to buy a vendor's full scoring pipeline or build the sensitive piece ourselves, not a generic architecture answer."
Why this works
Shows you can take an arbitrary company name and immediately find the one real decision worth walking through.
Stage 2
Say the structure out loud
Say it like this
"I'll run this as GUARD. Groups, who's the operator and who's the subject. Unequal, where the harm actually lands. Ability to contest, who never gets a lever. Reduce, the specific design change. Detect, how I'd know in production."
Why this works
Signals a repeatable way to reason about build-vs-buy in a sensitive domain, not a generic cost comparison.
Stage 3
Reframe: it isn't "which is cheaper," it's "who can push back"
Say it like this
"This isn't really a question of which option costs less or launches faster. It's a question of who can push back if either one gets it wrong, and right now, for this product, the answer is nobody."
Why this works
This is where a strong answer separates from a candidate who only compares price and speed.
Stage 4
Give the one decision
Say it like this
"Here's the actual split: buy the credit-bureau and eviction-record data feeds, that's commodity infrastructure every screening company uses the same way. Build in-house a match-confidence layer that checks whether an eviction record really belongs to this applicant, plus a real dispute flow before a denial goes final."
Why this works
This is the direct answer, stated as an actual architecture split, not a vague sense that "fairness matters."
Stage 5
Prove it with the compressed evidence
Say it like this
"A new compliance hire asked what happens if an eviction record isn't actually the applicant's. We pulled 500 recent auto-denials and had someone manually verify each eviction match. Seventy, 14 percent, belonged to someone else, usually a person sharing a common surname. None of those seventy people had any way to find out why they were denied."
Why this works
Compresses the whole case into the one audit that turned "probably rare" into a real, counted number.
Stage 6
Name the AI-specific reasoning and the trade-off
Say it like this
"The honest reason this isn't a generic build-vs-buy call is that the vendor's score treats a same-name record match as certain instead of probabilistic, and it's confidently wrong about it. We accepted a slower launch and a real six-figure engineering cost to build the matching layer ourselves, in exchange for a decision an applicant can actually understand and contest, instead of a faster, cheaper one that quietly denies the wrong person."
Why this works
This is the load-bearing, AI-specific judgment. A normal software feature doesn't produce a confidently wrong identity match with no visibility into why.
Stage 7
Say what wouldn't change, then close
Say it like this
"I wouldn't rebuild the credit-score portion of the pipeline, since credit bureau data ties to a real, verified financial identity with far fewer mismatch problems than county eviction records. For Cassowick, the split holds: buy the data, build the one decision that decides whether someone gets to keep their home."
Why this works
Closes with real judgment about where the concern doesn't apply, and restates the direct answer in one breath.
Let's learn
Say we build a tool that reads a rental application and tells a leasing agent whether to approve it.
Cassowick's risk score reads a rental applicant's credit history, eviction records, and income verification, and returns approve, deny, or manual review to a leasing agent. Before any scoring tool, an agent manually pulled a credit report and called a previous landlord, about 40 minutes per applicant, approving roughly 70 percent of applicants.
With the vendor's tenant-screening score, an agent gets a recommendation in under 10 seconds, and the overall approval rate holds around 68 percent, close to the old manual number.
The five checks, held up as one page. Ability to contest is the one most build-vs-buy conversations skip entirely.
Here's the turn: the overall approval rate barely moving is exactly what hides the real problem. The extra mistakes aren't spread evenly. They land almost entirely on applicants whose eviction-record match is wrong, and once denied, those applicants have no way to find out why, or to fix it before the apartment is gone.
500 audited auto-denials, by whether the eviction record actually matched the applicant
Seventy real people, denied on a stranger's eviction record, with no way to know that's what happened.
"Rare" is not a number. The day someone counted it, rare turned out to mean 1 in 7.
At its worst, real people lose a home over a stranger's eviction record, and Cassowick never finds out unless a lawsuit or a news story tells them.
The choice I would take back
Cassowick's original decision to buy the vendor's full scoring pipeline as one opaque black box, with no visibility into how any single denial was reached. That made sense when launching fast mattered most and eviction-record mismatches seemed like a rare edge case nobody had actually measured. It stopped making sense once the audit found the real mismatch rate was 14 percent.
What I would leave alone: I wouldn't touch the credit-score portion of the pipeline, since credit bureau data is tied to a real, verified financial identity with far fewer mismatch issues than county eviction records, which are often matched by name alone.
The lesson: the applicant who can't see or contest the record that denied them isn't a side effect of buying a vendor's score. It's a design decision, and it's the one worth building around first.
Now here is the same thing as a story
The short version above is what you'd say in the interview room. Read this one for what it felt like the week a new hire's plain question exposed a gap nobody had measured.
For six months, Cassowick's leasing agents trusted a single number. Then a new hire asked what it was actually based on.
Simone Havelka had spent years building fraud-detection systems before joining Cassowick, where a wrong call used to cost a company money, not somebody's home. The vendor's score launched clean: fast, cheap, and leasing agents loved not having to manually pull reports anymore.
The habit thinned in three beats. At first, agents still skimmed the underlying credit report themselves before confirming a denial. Within two months, most agents just clicked deny the moment the score said deny, without opening the report at all. By month five, almost nobody opened it.
Same decision, same outcome, and only one of these two people ever had a lever in their hand.
A newly hired compliance associate, reviewing a stack of denials for an unrelated audit, asked Simone plainly: "What happens if the eviction record on file isn't actually this person?" Nobody in the room had a ready answer.
Knowledge spark: why would a "confident" score be wrong about someone's identity?
Many county eviction-court systems record cases by name only, with no unique ID like a social security number. A vendor's score built on that data treats a same-name hit as a real match, without any way to check date of birth, middle name, or address history. It doesn't hedge. It just answers, and sometimes it's confidently wrong.
Simone pulled 500 recent auto-denials and had a paralegal manually verify the eviction record against the applicant's full name, date of birth, and address history. Seventy of the 500, 14 percent, were mismatched: someone else's eviction record, often a person sharing a common surname.
We didn't just deny 70 applications. We took 70 people's next home, based on a record that was never actually theirs.
The gap sits exactly where a real process would have a dispute step. There isn't one.
Simone realized the real question was never "how accurate is the score." It was whether an applicant, once flagged, ever got a chance to say "that's not me" before the apartment was gone. Right now, they didn't, not once, not ever.
When the vendor contract was first signed, someone in the room said, "eviction mismatches must be rare, most people don't share exact identities," and it sounded reasonable, since nobody had actually measured it yet.
Wrongful denial rate, by eviction-record match confidence
Almost all of the wrongful denials sit in the low-confidence bucket, exactly where a manual review step would catch them before they became a denial at all.
Rerun the same 500 applicants with a match-confidence layer in place: of the 70 mismatched records, 59 get flagged for manual review instead of an automatic denial, and a human confirms the mismatch and approves the applicant within two business days, well before the unit is gone.
One version denies a stranger's mistake to 70 real people with no way to object. The other catches 59 of those 70 before the denial ever goes out, and gives the applicant a real, working way to say "that's not me."
What I'd tell myself, hearing that new hire's question land in the room: "rare" is not a number. The day someone actually counts it, rare turns out to mean 1 in 7.
GUARD, the five checks a build-vs-buy call can't skip in housingNot a script for refusing every vendor score. GUARD is what tells you exactly which piece of the pipeline earns the extra cost of building.
G
Groups. Who's the operator, who's the subject?
The leasing agent, who can override the score with one click, is the operator. The rental applicant, who never sees or can contest the record that denied them, is the subject.
Naming both people is what stops "the applicant" from staying an abstract statistic.
U
Unequal. Where does the harm land unevenly?
Mismatched eviction records cluster on applicants with common surnames, since county court systems often match by name alone, with no unique identifier.
An 86-percent overall accuracy number hides exactly whose 14 percent it is.
A
Ability to contest. Who never gets a lever?
Applicants are never told which record triggered a denial, and by the time a dispute could work its way through, the apartment is already rented to someone else. Buying a vendor score with no visibility into individual matches is the decision that took their lever away.
This is the hardest, most important step, and the one most build-vs-buy answers skip entirely.
R
Reduce. The specific design change.
Buy the commodity credit-and-eviction data feed. Build in-house a match-confidence layer that routes low-confidence eviction hits to manual review instead of automatic denial, plus an applicant-facing "that's not me" flow.
A real design change, not a review board or a policy document nobody in the pipeline ever touches.
D
Detect. How you'd know in production.
Track denial rate broken out by match confidence, and the override rate on manual reviews, watching for the gap between high- and low-confidence denials to widen without an explanation.
This is what tells Cassowick about a growing problem before an applicant's lawyer does.
The recap, one line per letter: groups names the leasing agent as operator and the applicant as subject, unequal shows the mismatch clustering on common surnames, ability to contest is the applicant's total lack of a lever before the unit is gone, reduce is the match-confidence layer and dispute flow, and detect is watching the denial-rate gap by match confidence before it becomes a lawsuit.
And if you want to be sure it really works, try it somewhere elseSame five letters, a translation office instead of a leasing office. The subject changes, the missing lever doesn't.
Casimir Bilodeau runs product for Dubrocante Translate, which provides AI-assisted translation of legal documents for asylum applicants' cases. Mapped onto GUARD: groups are the caseworker, who can flag a segment for a certified human translator, and the asylum applicant, who often can't read the target language and has no way to check whether their own testimony was translated correctly. Unequal: mistranslation risk concentrates on lower-resource language pairs, the ones with fewer training examples, which also tend to be spoken by the most vulnerable applicants. Ability to contest: an applicant can't verify their own translated testimony before it becomes part of the official case record, and by the time an error surfaces, a hearing may have already happened. Reduce: buy the general translation engine for common, well-covered language pairs, build a confidence-flagging layer for lower-resource pairs that routes uncertain segments to a certified human translator instead of shipping raw machine output into a legal filing. Detect: track flag rate and post-hoc correction rate by language pair, watching for the languages where corrections spike.
A different subject, a different industry, and the exact same missing lever.
The actual design change, drawn as a real branch a case can take, not a policy statement.
Swap the trigger and it still runs.
Speed: an interviewer caps you at sixty seconds. Say "buy the commodity data, build the layer that decides whether the person on the other end of a wrong call ever gets to push back," and stop.
Cost: no time to run a real audit before deciding. Say so honestly, and commit to a small, fast sample check, even fifty cases, before trusting a vendor's black-box match at all.
The vendor's model got better, for real: if a future vendor version claims improved identity matching, that's still worth testing against your own known-mismatch cases before trusting it, since "improved" and "improved on your specific population" aren't the same claim.
Where people run it wrong.
They treat a build-vs-buy call as purely a cost and speed question, with fairness as an afterthought bolted on later.
They assume a mismatch problem is rare without ever actually counting it.
They build a review board or a policy document instead of an actual design change a real applicant benefits from.
How to use it live. The moment an interviewer names any company and asks for a build-vs-buy walkthrough, ask yourself: is there a person on the receiving end of this decision who can't push back if it's wrong? If yes, that's exactly the piece worth building, no matter what else you'd happily buy.
Flashcards (tap any card to flip it)
1 · THE FRAMEWORK
What framework fits a build-vs-buy call where the decision affects someone who can't push back?
Tap to flip
ANSWER
GUARD: groups, unequal, ability to contest, reduce, detect. It forces you to name who never gets a lever before deciding what to build or buy.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Simone Havelka, the AI PM at Cassowick Property Analytics, who audited 500 auto-denials after a new hire asked a question nobody could answer.
3 · THE HABIT
What habit did leasing agents fall into?
Tap to flip
ANSWER
Clicking deny the moment the vendor's score said deny, without opening the underlying eviction record at all, by month five almost nobody checked it.
4 · GROUPS
Who's the operator and who's the subject in this decision?
Tap to flip
ANSWER
The leasing agent, who can override the score, is the operator. The rental applicant, who never sees or can contest the record that denied them, is the subject.
5 · THE OLD DECISION
What decision would you take back?
Tap to flip
ANSWER
Buying the vendor's full scoring pipeline as one opaque black box, assuming eviction-record mismatches were rare before anyone had actually measured them.
6 · THE NUMBER
Fill in the blank: of 500 audited auto-denials, ___ percent, or ___ people, had an eviction record that belonged to someone else.
Tap to flip
ANSWER
14 percent, 70 people.
7 · THE REPLAY
Same 500 applicants, match-confidence layer live. What changes?
Tap to flip
ANSWER
59 of the 70 mismatched records get flagged for manual review instead of an automatic denial, and the applicant is approved within two business days, before the unit is gone.
8 · CROSS PRODUCT TRANSFER
Section 4 runs this again for a different product. Which one, and who's the subject there?
Tap to flip
ANSWER
Dubrocante Translate's legal-document translation tool. The subject is an asylum applicant, who often can't verify whether their own testimony was mistranslated before it becomes part of their case record.
Check yourself Score: 0 / 0
True or false
1. True or false: the overall approval rate staying near 68 percent after the vendor's score launched proved nothing was wrong.
True
False
Show hint
Careful, read the claim exactly as written before picking.
Show answer
False. The overall rate hid the real problem completely. The mismatches were concentrated in a small share of denials, not spread evenly, so a flat overall number told you nothing about the harm underneath it.
Multiple choice
2. Why is the rental applicant considered the "subject" with no lever in this decision?
A. Applicants don't want an apartment badly enough to contest a denial.
B. They're never told which record triggered a denial, and by the time a dispute could work, the unit is already rented to someone else.
C. Leasing agents always deny automatically, with no exceptions.
D. Applicants don't have the legal right to see their own credit report.
Show hint
Look at the "ability to contest" step in the GUARD recap.
Show answer
B. The lever arrives too late to matter, which is functionally the same as never having one at all.
Fill in the blank
3. Fill in the blank: of 500 audited auto-denials, ___ had eviction records that actually belonged to someone else, most often sharing a common ___.
Show hint
Look at the bar chart in "Let's learn."
Show answer
70, or 14 percent; surname. A number that only became real once someone actually audited a sample instead of assuming it was rare.
Short answer, where it wouldn't matter
4. Name a part of Cassowick's pipeline where this same concern would NOT apply, and say why.
Show hint
Look at "what I would leave alone."
Show answer
Model answer: The credit-score portion of the pipeline. Credit bureau data ties to a verified financial identity with a much lower real mismatch rate than county eviction records, which are often matched by name alone.
Short answer, apply it yourself
5. Think of an automated decision you've been on the receiving end of. Did you have a real way to contest it before it mattered? What would that lever have looked like?
Show hint
Think about whether you could act before the decision's consequence was already final.
Show answer
Model answer: A flight rebooking algorithm that assigned a middle seat with no visible reason. A real lever would have been a one-tap "explain this seat assignment" option shown before check-in closed, not after boarding.
Short answer, work the number
6. If the match-confidence layer only caught 40 of the 70 mismatched records instead of 59, would it still be worth building?
Show hint
Compare 40 out of 70 to catching none at all.
Show answer
Model answer: Likely yes, though less decisively. Catching 40 of 70, about 57 percent, still cuts wrongful denials by more than half, a real improvement over catching none, even short of the 84 percent catch rate in the base case.
Before you close the answer
Why this works
Tests whether you'll notice that a build-vs-buy call in a sensitive domain isn't just about cost or speed, but about who gets a chance to push back if the automated decision is wrong.
Follow-up traps
"Isn't a phone number for disputes enough of a lever?" Response: not if the unit is usually rented out before a dispute call gets resolved. A lever that arrives after the harm is done isn't really a lever.
"Doesn't building this slow down the launch and cost more?" Response: yes, about six weeks and roughly 95 thousand dollars up front, and that's the honest trade-off: a slower, costlier launch for a defensible, contestable process instead of a fast one that quietly produces wrongful denials.
If pressed
The match-confidence threshold itself was calibrated against a labeled set of 200 known-mismatch and known-correct-match cases, re-checked quarterly as county court data-sharing formats change, not a one-time setting left alone after launch.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
