ConceptAdvancedShipping & Model Lifecycle / Model migration and version changes for users / #3
Explain why a strictly better model can still be a bad migration.
The direct answer
Because a strictly better model is measured against an aggregate score, and an aggregate can rise while it quietly gets worse on the one task your highest-stakes users depend on most, especially once its new mistakes stop announcing themselves. Before migrating a task type, pull a stratified, blind-scored sample of that task specifically and check whether the model's confidence on its own errors got worse too, not only whether its accuracy did.
Do this, in order
Run a stratified, blind-scored test on the task your aggregate score can't see, not just the aggregate.Why: an average that's up 8 points can still be hiding a task-level regression the leaderboard never sampled.
Lay the real timeline: when the benchmark win was reported against when the first real complaint surfaced.Why: it shows how many weeks a hidden regression can run clean before anyone was watching the right slice.
Rule out nostalgia before trusting one complaint.Why: a second, unbriefed reviewer and a check for real cost separate a genuine regression from a preference for the old model.
Name three specific ways an aggregate win can hide a real loss, and test which one actually happened.Why: an underweighted task, a downstream format break, and a shift from obvious mistakes to confident silent ones are different problems with different fixes.
Check the model's confidence-flag rate on its own errors in the high-stakes segment, not only its accuracy there.Why: a model that's wrong less often but silent about the wrongness that remains can be a worse migration even while every dashboard number improves.
Leave the aggregate-only gate alone for tasks the benchmark already represents well.Why: high-volume, low-stakes tasks are already covered by the old gate, so rebuilding it everywhere would be motion, not a fix.
How to answer this, stage by stage
Seven moves. This question tempts a vague answer about "benchmarks don't capture everything," so most of these stages exist to show exactly what an aggregate score can't see, and the one check that would have caught it before every client was moved over.
1
Name the tool, the person, and the tension before anything else
Say it like this
"Let me put this on one product. Say a legal-tech company, Statute Analytics, builds Covenant, a tool that reads a contract and pulls out the clauses a lawyer has to check by hand: who can end the deal early, who pays if it goes wrong, and how much. Vikram Sethi is the PM. Covenant just migrated to a new model that beat the old one on every published benchmark and on Statute Analytics's own 900-contract eval suite. Five weeks later, a client's in-house lawyer almost signs off on a liability cap that's ten times too high."
Why this works
A question about a strictly better model that still fails stays abstract forever unless you name the product, the person, and exactly which two numbers said "ship it."
2
Say your structure, then give the direct decision straight away
Say it like this
"I want to run this as a diagnosis, T-R-A-C-E: timeline, recut, assume nothing, cause candidates, evidence test. Because the real question isn't whether the new model is actually better. It clearly is, on average. It's that 'on average' is exactly the kind of number that can hide a real regression in one place while looking great everywhere else. So here's what I'd do. Before migrating a task, pull a stratified, blind-scored sample of that task specifically, and check whether the model's confidence on its own mistakes got worse, not just whether its accuracy did."
Why this works
Naming the plan and the direct answer in the same breath means nobody has to wait for the ending to know what you'd do.
3
Lay the timeline before touching the near miss itself
Say it like this
"Vikram's team reported the aggregate win in the first week of March: exact-match extraction up from 84 to 91 percent, clause classification F1 up from 0.81 to 0.89, across all clause types, on both the published benchmark and Statute Analytics's own eval suite. A two-week shadow run on live traffic held above 90 percent the whole time. Full cutover happened the last week of March. The first real complaint didn't land until the third week of April, five weeks after the benchmark result, four weeks after every client was already on the new model."
Why this works
Separating when the benchmark win was reported from when the real problem surfaced is what stops this from sounding like "the numbers were secretly bad the whole time," which they weren't, on average.
4
Recut the aggregate by clause type, not by another aggregate filter
Say it like this
"So I recut it, not by client or region, by clause type. Confidentiality clauses went from 86 to 93 percent. Termination-for-convenience went from 79 to 88. Assignment restrictions went from 75 to 85. Every one of those got better, which is exactly what the aggregate number was reporting. Then I got to liability caps, the line that sets the dollar ceiling on a company's risk, and exact-match accuracy went from 92 percent under the old model to 68 percent under the new one. The aggregate never showed it, because liability-cap clauses are 36 of the 900 contracts in the eval set, about 4 percent, buried under a much bigger pile of NDAs and vendor agreements that all got better."
Why this works
A regression that's invisible in one clean aggregate number and glaring in one clause-type recut is the strongest single move a diagnosis can make.
5
Rule out the complaint being nostalgia before trusting it
Say it like this
"Before I act on that gap, I check two things. First, is this rose-tinted memory: I ran the old model against the exact contract that tripped up the new one, and it read the clause correctly, five million dollars, no issue. So it's not nostalgia. Second, is it one person's hunch: a second contracts specialist, not told about the near miss, reviewed 40 fresh liability-cap extractions blind and flagged the same pattern on her own. And every flagged error changed the actual dollar figure, not just the wording, so it's a real, costly gap."
Why this works
Ruling out nostalgia and confirmation bias is what separates a real regression from a team getting attached to the old model's quirks.
6
Name three reasons a model that wins on average can still ship a bad migration
Say it like this
"Three candidates. One, liability-cap clauses are a task type the eval barely weighted, 4 percent of the contracts, even though they're the single clause type our highest-value clients care about most. Two, the new model changed how it structures a 'greater of X or Y' clause in its output, which could break a client's downstream risk spreadsheet even where the extracted text is more accurate. Three, and this is the one I'd bet on, the kind of mistake changed. The old model would say 'cap value uncertain, needs review' on a nested clause like this. The new model states a specific number with full confidence, right or wrong, so nothing tells the reviewer to double-check."
Why this works
Naming three separate, named causes instead of one vague "the new model has issues" is what makes the next step a real test instead of a guess.
7
Run the evidence test, say what you'd measure, then close on the one line
Say it like this
"The evidence test: score the same 40 cases not just for accuracy, but for whether each error carried a low-confidence flag. The old model flagged 76 percent of its liability-cap mistakes as uncertain. The new model flagged 9 percent. That's the real migration risk, not that it's wrong more, it's wrong about as rarely as before on most clauses and far more confidently silent about the mistakes it still makes here. Going forward, any clause-type migration needs its own stratified, confidence-scored test before it ships, not just a passing aggregate score. Leave the confidentiality-clause migration on the old gate, it's most of the real volume and already well covered. Covenant's benchmark win was real. It just never got asked whether the model still knew when to doubt itself on the clause that mattered most."
Why this works
Ending on the decision, not a recap, is the line an interviewer actually remembers.
Let's learn
Covenant reads a contract, a merger agreement, a supply deal, an NDA, and pulls out the handful of clauses a lawyer actually needs to check by hand: who can walk away from the deal early, who pays if something goes wrong, and how much money is on the hook if it does.
Before this migration, Covenant ran on an older model. It caught the right clause 84 times out of 100, and its classification score, how well it labeled what kind of clause it found, sat at 0.81. That was enough to cut a paralegal's contract review from about three hours to forty minutes, because they only had to check what Covenant flagged instead of reading the whole thing cover to cover.
Knowledge spark: what's exact-match accuracy?
A strict score. The model doesn't get partial credit for finding the right clause with the wrong number in it. It's counted right only if the whole extraction, the clause and its value, matches the source exactly.
The new model beat the old one on every published benchmark and on Statute Analytics's own 900-contract eval suite. Exact-match accuracy went from 84 to 91 percent. Classification F1 went from 0.81 to 0.89. A two-week shadow run across live traffic held above 90 percent the whole time. Vikram signed off on a full migration.
Covenant's migration dashboard, week by week
Aggregate exact-match accuracy, across all clause types, weeks after full cutover
Here is the turn. Those eight extra points of accuracy are not the real story. The real story is one clause type the aggregate number was never built to protect: the liability cap, the line that sets the dollar ceiling on how much a company can lose if the deal goes bad. On that one clause type, specifically, the new model got worse, and it stopped saying when it wasn't sure.
Covenant did not get less accurate. It got quieter about the one mistake that mattered.
At its worst, this costs a client the one number a due-diligence lawyer relies on Covenant for the most, with no signal that it might be wrong. Tariq Nasser, in-house counsel at Bellwood Industrial, nearly signed a risk memo listing a $50 million liability cap on a $6 million acquisition. The real clause said $5 million. Covenant stated the wrong figure with full confidence, nothing to make him look twice. This is silent degradation: a real capability getting worse in exactly the place nobody was watching, while every number on the migration dashboard stayed green. The guardrail that catches it is a stratified, confidence-scored test on the segment the aggregate can't see, not a bigger aggregate sample.
The gap between when Covenant's benchmark win was reported and when anyone actually caught the real mistake
The choice I would take back. The migration gate Statute Analytics wrote a year earlier, when Covenant mostly handled a narrow, fairly uniform mix of vendor contracts and NDAs, was simple: ship the new model once it beats the old one on the aggregate eval, full stop. That was a sensible default when the contract mix was narrow. It stopped being sensible once Covenant's client base grew to include M&A due diligence, where the aggregate can look great while the one clause type carrying the most financial risk quietly gets worse.
The decision that mattered
Migrate a clause type once it clears a threshold on a stratified, blind-scored sample of that clause type specifically, checked against source text, most of the time, by design, not once the aggregate score alone beats the old model.
That fix costs something too. Every liability-cap or indemnification extraction below a confidence threshold now waits for a human reviewer before a client ever sees a number, which adds review time back on exactly the clause type Covenant was built to speed up. That's the trade Vikram's team accepted: slower and more expensive on the highest-stakes clause, in exchange for never shipping a confident wrong number silently.
What I would leave alone. The confidentiality-clause migration doesn't need any of this rebuilt. NDAs are most of both the benchmark and Statute Analytics's real contract volume, so the aggregate number is already a good stand-in there, and a mismatched confidentiality clause rarely carries the same financial exposure as a misread liability cap.
The lesson. A benchmark only proves the model is better where the benchmark actually looked. If your real users' highest-stakes task is a sliver of that benchmark, a model that's strictly better on paper can still be a worse migration in the exact place it costs the most to be wrong.
The morning Tariq almost signed a ten-times mistake
Read the short version above if you're short on time. This is the long version, for the part where you feel exactly how a clean topline number can talk a whole team out of checking the one place that mattered.
Vikram Sethi has run Covenant's product for three years. He knows the drill for a model migration by heart: aggregate benchmark, then a two-week shadow run on live traffic, then full cutover.
His first two migrations at Statute Analytics, he personally pulled a hand-scored sample broken out by clause type before signing off on anything, because the process was new and nobody trusted the aggregate number alone yet. Every clause type checked out both times. The process worked.
By his fourth migration, four clean rollouts in a row had a way of doing what clean rollouts do. He stopped pulling the clause-type breakdown himself and started trusting the shadow period's own topline report. He'd skim it, see a green number above 90, and move on.
By this fifth migration, Covenant's biggest one yet, he didn't even open the shadow run's raw numbers by clause type. He watched for the same green line in the weekly migration review, same as always, and it was there.
Four clean migrations don't prove the fifth one is safe. They just make it easier to stop checking.
Then, on a Thursday in the third week of April, five weeks after the benchmark win, an email came in from Tariq Nasser at Bellwood Industrial. Bellwood was closing a small bolt-on acquisition, a $6 million deal, and Tariq had pulled Covenant's extraction of the target company's indemnification clause to draft the risk memo. Covenant reported the liability cap at $50 million, high confidence, no flag.
Tariq almost used it. He'd stopped independently checking every dollar figure Covenant reported months ago, once it had been reliable enough for long enough that re-reading the source page for every number felt like the thing Covenant was supposed to save him from doing. What made him pause wasn't a warning from the tool. It was that a $50 million cap sitting inside a $6 million deal felt structurally wrong, the kind of number that doesn't belong in a bolt-on acquisition that size. He scrolled back to the actual clause: "aggregate liability under this Agreement shall not exceed the greater of five million dollars ($5,000,000) or the fees paid in the preceding twelve (12) months." Five million. Not fifty.
He emailed Vikram that afternoon. Not a complaint, mostly a question: had anything changed in how Covenant reads these clauses.
Before trusting that one email, Vikram's team checked two things. They re-ran the old model against the same Bellwood contract, and it read $5,000,000 correctly, no issue, so this wasn't the old model looking better in hindsight than it actually was. Then they handed 40 fresh liability-cap extractions, unlabeled, to a second contracts specialist who'd never heard about Tariq's near miss. She flagged the exact same pattern of errors, on her own, and every flagged case changed the actual number, not just the phrasing.
The team also weighed rolling every Bellwood contract straight back to the old model. They ruled it out. The old model was worse everywhere else, on assignment clauses, termination language, plain confidentiality clauses, so a blind rollback would have traded one hidden problem for a bigger, more visible one. The fix wasn't picking a model. It was building a migration gate that could actually see the one clause type the aggregate couldn't.
And the part I'd want to tell myself, if I could go back: we built a gate that could prove the new model was better on average. We never built one that could prove it hadn't gotten worse anywhere specific, and specific was exactly where the money was.
What the clause-type recut actually showed
Before deciding anything, Vikram's team checked whether the new model was even landing on roughly the right numbers company-wide. Across the full 900-contract eval set, both models agreed with the confirmed source text on the underlying clause type at nearly the same rate. The new model wasn't guessing badly across the board. It was failing narrowly, and confidently, in one place.
Exact-match accuracy by clause type, old model against new model
86%
93%
79%
88%
75%
85%
92%
68%
Confidentiality
Termination-for-convenience
Assignment restrictions
Liability cap
Old model
New model, improved
New model, this clause type only
Three clause types got better, exactly as the aggregate number reported. Liability caps got worse, and at 36 of 900 contracts, about 4 percent of the eval set, that one clause type barely dents an aggregate that's dominated by NDAs and vendor agreements.
That left the question of what the wrong answer actually cost. Vikram's team split the liability-cap errors by whether they carried a low-confidence flag, and checked the same for a clause type that hadn't regressed, to see whether the new model had simply gotten quieter about mistakes everywhere or just here.
Old model, liability-cap errors
Errors on 8 of 100 cases, sampled
76%flagged low-confidence, caught before anyone had to notice
New model, liability-cap errors
Errors on 32 of 100 cases, same sample basis
9%flagged low-confidence, stated as fact instead
New model, confidentiality-clause errors
A clause type that got more accurate overall
68%flagged low-confidence, close to the old rate
The quiet-mistake problem wasn't the new model in general. It was the new model on this one clause type, which is exactly why the aggregate score never caught it, and exactly why a second, unrelated clause type wasn't a reliable place to go looking for it either.
Three reasons a winning benchmark can still ship a bad migration
Not because anyone was careless. Each of these, on its own, looks like a small, forgivable rough edge in a genuinely better model. Together, they're why a migration that won every published score could still cost a client a ten-times mistake on the one number that mattered most.
Three separate, checkable causes, only one of them confirmed by the evidence test
Cause 1
A task type the eval barely weighted, but real users lean on hardest.
Liability-cap and indemnification clauses inside complex, nested agreements are 36 of the 900 contracts in Statute Analytics's own eval set, about 4 percent, and a similarly small slice of the published benchmark. NDAs and vendor agreements make up most of both. But liability caps are the exact clause type a client doing M&A due diligence, like Bellwood, cares about most.
How you'd check it: count what share of the eval set each clause type actually is, then separately rank clause types by real financial exposure per contract. A task type can be rare in the benchmark and still be the one thing the highest-value use case depends on.
Cause 2
An output change that breaks a downstream tool, even where the text is more accurate.
The new model sometimes represents a "greater of X or Y" clause as two separate structured fields instead of one flat number, closer to how the clause actually reads. A client's own risk-flagging spreadsheet, built to read one field, could silently grab the wrong one when the shape changes, even on a case the model itself got right.
How you'd check it: diff the structured output schema between old and new model versions on a sample of nested clauses, and check whether any client-side integration reads a field that changed shape. A schema break shows up in integration logs, not in an accuracy score.
Cause 3, confirmed
The kind of mistake changed: from obvious-and-catchable to subtle-and-confident.
The old model flagged 76 percent of its liability-cap mistakes as low-confidence, prompting a manual check before anyone relied on the number. The new model flagged only 9 percent of its liability-cap mistakes. It's about as accurate overall on this clause type as the old model was wrong just as rarely, but when it is wrong, it states the number as fact instead of asking for a second look.
How you'd check it: for a sample of confirmed errors in the high-stakes segment, score what share carried a low-confidence flag, old model against new. A falling flag rate on real errors is the signature of this exact cause.
TRACE, once the leaderboard has already said yes
This reads like a question that wants a general answer about benchmarks not being everything. The real job is diagnosis: work out exactly which real gap an aggregate score structurally cannot show, and prove it before it costs a client a wrong number.
T, timeline. Covenant's aggregate benchmark win was reported the first week of March: exact-match extraction up from 84 to 91 percent, classification F1 up from 0.81 to 0.89. A two-week shadow run held above 90 percent. Full cutover happened the last week of March. The first real complaint, Tariq's near miss at Bellwood, didn't surface until the third week of April, five weeks after the benchmark result and four weeks after every client was already on the new model.
R, recut. Not by client or region, by clause type. Confidentiality, termination, and assignment clauses all improved, matching the aggregate story. Liability-cap exact-match accuracy dropped from 92 percent under the old model to 68 percent under the new one. The aggregate never showed it, because liability caps are only about 4 percent of the eval set, buried under a much bigger pile of clause types that all got better.
A, assume nothing. Before trusting one lawyer's near miss, rule out two things. First, is it nostalgia: the old model re-run against the same contract still read the clause correctly, so the old model wasn't secretly bad the whole time. Second, is it one person's hunch: a second, unbriefed specialist reviewed 40 fresh cases and flagged the identical pattern, and every flagged error changed the actual dollar figure, not just the wording. Both checks confirmed a real, costly gap.
C, cause candidates. Three, named and separate: a task type the eval barely weighted, liability caps at 4 percent of the contracts, even though they're the clause type highest-value clients depend on most; an output-format change that could break a downstream risk spreadsheet even on cases the model got right; and a shift in what kind of mistake the model makes, from an obvious "not sure, please check" to a confident, unflagged wrong number.
E, evidence test. Score the same sample not just for accuracy but for confidence-flag rate on the errors. The old model flagged 76 percent of its liability-cap mistakes as low-confidence. The new model flagged 9 percent. That confirmed cause three: the new model is about as rarely wrong here as the old one, and far more confidently silent about the mistakes it still makes, which is exactly the setup that lets a wrong number reach a signed contract instead of a reviewer's desk.
Why the evidence test is the hard step
Anyone can say a benchmark doesn't capture everything. The evidence test turns that into a number: what share of real errors, in the segment that actually matters, still come with a warning attached. That's the difference between a hunch and proof a client would accept.
The same blind spot, three floors up in pediatric radiology
Loxwood Health pilots Corvidae, a tool that drafts the first-pass findings on a chest X-ray for a radiologist to confirm. Ilyana Deverell leads the product. A new model version beats the old one on every published sensitivity benchmark, 88 to 93 percent overall, and a three-week canary on live scans holds above 92 percent the whole time.
T. The benchmark win was reported, the canary ran three weeks clean at the aggregate level, then in week three a pediatric radiologist flagged a missed rib fracture on a child's film that the old model would have caught. R. Recut by patient age band, not by scan type. Adult-finding sensitivity rose across the board, matching the aggregate. Pediatric fracture-adjacent findings dropped from 81 to 54 percent. Pediatric films are only about 3 percent of the training and eval data, both dominated by adult chest X-rays. A. A second radiologist, unbriefed, reviewed 30 pediatric films the new model had marked clear and found 9 real missed findings against confirmed diagnoses, not a training artifact and not one reviewer's bad day. C. Three candidates: pediatric cases are a task type the eval barely weighted; a report-template field the model stopped populating for pediatric studies, breaking the hospital's records system; and a shift toward confident, unflagged misses instead of the old model's low-confidence tags. E. Check confidence-flag rates on the confirmed misses in both groups: the new model flagged pediatric misses at almost the same rate as adult ones, ruling out cause three this time. What confirmed cause one instead was a direct comparison of training-image proportions: pediatric skeletal proportions differ enough from the adult-dominated training set that the model's shortcut cues for adult rib fractures simply don't transfer, a different confirmed cause than Covenant's, reached the same way.
Swap the trigger and it still runs
Speed: Statute Analytics could have skipped the two-week shadow run entirely to hit a client's contract deadline. TRACE still starts by asking when the benchmark win was reported and when the real problem actually surfaced, not by how fast the rollout ran.
Cost: the team could have decided a stratified test for every clause type was too expensive to run before every migration. The task-level gap still surfaces eventually, just after a client signs something instead of before.
The model really did get better everywhere: say the next version fixes liability caps too, and every clause type improves. TRACE still finds whatever task remains underweighted in the eval, because the recut checks the segment, not the average, even when the average is genuinely good news.
Where people run it wrong
Treating a published benchmark win and a clean shadow run as proof the model is safe everywhere, without ever recutting by the task that actually carries the risk.
Rolling straight back to the old model on one complaint, instead of checking whether the old model is worse somewhere else, which it usually is.
Fixing the visible symptom, retraining the confidence classifier alone, instead of checking whether the underlying task is represented in the eval at all.
How to use it live
Buy yourself ten seconds by saying the gap out loud before answering. "There's what the benchmark measured, and there's what our highest-stakes users actually need, and those aren't automatically the same set of tasks. Let me tell you the one check that tells them apart." That's not stalling. That's where the real answer starts.
Flashcards (click a card to flip it)
This is a question about a strictly better model still causing a bad migration, worked as a diagnosis, so these eight test the TRACE moves and the real numbers behind them.
1 · THE FRAMEWORK
Which framework fits "explain why a strictly better model can still be a bad migration," and why?
Tap to flip
ANSWER
TRACE. It sounds like it wants a general answer about benchmarks being incomplete, but the real job is diagnosis: find exactly which real gap an aggregate score structurally can't show, and prove it.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Vikram Sethi, PM for Covenant at Statute Analytics. He's run the product for three years.
3 · THE HABIT
What did Vikram stop doing across his first four migrations, even though it kept working out fine?
Tap to flip
ANSWER
Personally pulling a hand-scored sample broken out by clause type before signing off. After four clean rollouts, he started trusting the shadow run's topline number and stopped checking the breakdown himself.
4 · THE THREE CAUSES
Name the three reasons a model that wins on average can still ship a bad migration.
Tap to flip
ANSWER
A task type the eval barely weighted, an output-format change breaking a downstream tool, and a shift from obvious-and-catchable mistakes to confident, unflagged ones.
5 · THE NUMBER
The old model got liability caps right 92 percent of the time. Reading the same clause type after migration, the new model was only right ______ percent of the time.
Tap to flip
ANSWER
68 percent. The aggregate score never showed this gap, because liability caps are only about 4 percent of the eval set.
6 · THE CHECK
Name the one test that confirmed which cause was real.
Tap to flip
ANSWER
Scoring the sampled errors for confidence-flag rate, not just accuracy. The old model flagged 76 percent of its liability-cap mistakes as uncertain; the new model flagged only 9 percent.
7 · THE FIX
What should change about Covenant's migration gate going forward?
Tap to flip
ANSWER
A clause type migrates once it clears a threshold on a stratified, blind-scored sample of that clause type specifically, checked against source text, most of the time by design, not once the aggregate score alone beats the old model.
8 · CROSS-PRODUCT TRANSFER
Section 4 runs TRACE again on a different product. Which one, and what's the confirmed cause?
Tap to flip
ANSWER
Loxwood Health's Corvidae radiology tool, led by Ilyana Deverell. The confirmed cause: pediatric films are an underweighted task type in the training and eval data, a different confirmed cause than Covenant's confidence-flag gap.
Check yourself Score: 0 / 0
Fill in the blank
1. The old model caught the right liability cap value 92 percent of the time. Reading 40 of the new model's blocked cases by hand, the accuracy was only ______ percent.
Show hint
Look at the clause-type chart, the last group of bars, not the aggregate line above it.
Show answer
68. The gap only showed up once the eval was recut by clause type, instead of read as one aggregate score across all 900 contracts.
True or false
2. True or false: because Covenant's new model beat the old one on every published benchmark and the two-week shadow run held above 90 percent the whole time, that proves it was safe to migrate every clause type over.
True
False
Show hint
Check what an aggregate score across 900 contracts can and can't show about one clause type that's 4 percent of the set.
Show answer
False. Both the benchmark and the shadow run were aggregates dominated by clause types that improved. The clause-type recut showed liability caps regressed sharply, a gap the aggregate was never built to catch.
Multiple choice
3. Why couldn't Vikram just trust Tariq's near miss and roll every Bellwood contract straight back to the old model?
A. Because one near miss could be a fluke, and the old model was worse on every other clause type, so a blind rollback would trade one hidden problem for a bigger one.
B. Because Bellwood's contract with Statute Analytics didn't allow a rollback.
C. Because the old model had already been deleted from production.
D. Because Tariq wasn't authorized to report a modeling issue.
Show hint
Ask what the old model's own clause-type recut looked like, not just its liability-cap number.
Show answer
A. The old model was correct on liability caps but worse everywhere else. A rollback based on one anecdote, without checking the second reviewer and the real cost, would have been just as unfounded as ignoring the near miss entirely.
Short answer
4. Name a place in Covenant's migration where the old aggregate-only gate is still fine, and say why.
Show hint
Think about a clause type that's both high-volume and low financial exposure per contract.
Show answer
Model answer: "Leave the confidentiality-clause migration on the aggregate gate. NDAs are most of both the benchmark and Statute Analytics's real volume, so the aggregate is already a good stand-in there, and a mismatched confidentiality clause rarely carries the financial exposure a wrong liability cap does."
Short answer, apply it yourself
5. Think of an AI tool you've used that got upgraded to a "better" version. What's one narrow task inside it that a new aggregate score could be hiding a regression in, and how would you check?
Show hint
Look for a task that's rare in general use but high-stakes for you specifically, then check just that slice, blind, against a source you trust.
Show answer
Model answer: "A coding assistant upgrade could raise average code quality while getting worse specifically at a less common language or framework I actually use. I'd pull a stratified sample of just that framework's tasks, score it blind against known-correct answers, and check whether its errors there still come with a caveat, not trust the overall pass rate." Any honest answer works if it names a real, checkable slice, not a general sense that the new version "feels better."
Fill in the blank
6. Liability-cap clauses are about 4 percent of the 900-contract eval set. That's roughly ______ contracts.
Show hint
Take 4 percent of 900.
Show answer
About 36. 4 percent of 900 is 36. That's how few examples of the highest-stakes clause type the whole migration decision actually rested on.
Before the interviewer pushes back
Why this works
Tests whether you understand that "strictly better" is a claim about an average, and averages are exactly where a real regression hides. Most candidates stop at "benchmarks don't capture everything" without naming the check that would have actually caught it.
Follow-up traps
"What if the segment is real but tiny, does a 4 percent slice really justify slowing the whole migration down?"Size it by financial exposure, not by row count: liability caps were 4 percent of the eval set but the highest-stakes clause type for the biggest clients, so the segment's size in the benchmark and its size in real risk are two different numbers, and the second one decides.
"Isn't a stratified test just slicing the data until you find a story that fits?"No, the slice was chosen before seeing the data, because liability caps were already flagged internally as the highest-stakes clause type. Nobody went looking for whichever clause happened to look bad after the fact.
If pressed
The confidence-flag gap wasn't uniform across the whole model either, it was worst specifically on clauses using a "greater of X or Y" formula structure. That points to the new model's handling of nested numeric clauses specifically, not a general drop in legal-language quality, a narrower root cause than "the model got worse at contracts" would suggest.
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
