Explain how you would investigate a complaint you cannot reproduce.

How to answer this, stage by stage

Nobody is grading whether you can say "I would try to reproduce it." They are grading whether you can tell a real bug from a number that was never as certain as it looked. Six moves get you there.

Let's learn

Here is what happens when eleven hundred resumes get read the exact same way, until one afternoon, one of them does not.

Keystone reads a resume and a job post and gives the pair a match score, out of 100. Anything at or above 50 gets sorted into "possible match" or better. Anything under 50 gets sorted into "does not meet requirements," and a hiring manager may never open it at all.

Before Keystone, Bellcourt Freight's hiring team read every resume for a role by hand. For the warehouse operations manager posting, that meant about 80 resumes over two weeks, roughly six hours of Concepcion's close reading. Now Keystone screens all 80 in under a minute and puts the top two buckets in front of her first.

Say plainly: the extra mistake Concepcion found is not the real problem. The real problem is what one loud complaint reveals about forty five quiet ones already closed the same way, without anyone checking whether they were the same bug wearing a different resume.

The gap between the two middle points is the trap. A model update sitting right between the complaint and the replay looks like the obvious cause before anyone has checked anything.

At its worst, this pattern runs quietly for months. A steady share of borderline candidates get auto-rejected on a run that happened to land low, while an identical resume on a different day, or a different second, would have cleared the bar. Nobody sees a broken feature. They see a clean pass rate, a clean dashboard, and forty six closed tickets that say "could not reproduce."

The choice I would take back. When we built the admin console's replay button, we made it always run against Keystone's current live model, not the exact snapshot that produced the original score. That was the sensible default at the time, because most complaints really were the model getting something wrong, and testing against whatever is live catches whether a fix already shipped. It stopped being sensible the day a real case needed the exact old snapshot just to see the bug at all.

What I would leave alone. The three-bucket screen Concepcion actually sees: strong match, possible match, does not meet requirements. Showing a plain bucket instead of a raw number to a hiring manager is fine on its own and does not need to change. The problem was never that a person reads a bucket instead of a number. It is that one score, sitting right at the edge, was allowed to decide the bucket completely on its own.

The lesson. A ticket that says "I cannot make it do that again" is not the same claim as "nothing is wrong." Sometimes it means the opposite. It means whatever happened only needed to happen once, and it will happen again, to someone else, on a different resume, at the exact same edge.

Now here is the same thing as a story

Read the short version above if you want it fast. Read this one if you want to feel exactly how close the fix came to never happening.

Anand Trivedi has owned Keystone's screening model for a year and a half. He built the eval set it launched on. He can tell you, without looking it up, that the model agrees with a real recruiter's call about 84 times out of 100 on the current test set. He is not careless. He is, if anything, the person on the team who checks things twice.

For most of that year and a half, "could not reproduce" tickets were a known, boring category. A hiring manager flags a score. Anand opens the admin console, re-runs the exact resume against the current model, sees a normal-looking result, and writes back: score confirmed accurate on replay. Nine times out of ten, that was true. The model had genuinely gotten something wrong on a resume like this before, a fix had already shipped, and the replay proved it.

Somewhere over the last few months, without anyone deciding it on purpose, Anand's habit thinned in three small beats. First, he ran a full replay twice, once against the current model and once through a slower manual check of the parsed text. Then he dropped the manual check, because the parsed text always matched. Then he started glancing at whether the new bucket roughly matched the complaint before closing the ticket, because a replay had never once turned up anything else.

Concepcion's ticket was not louder than the others, at first. It read like every other one: a resume, a screenshot, a request to look again.

What made Anand actually stop was one line she added at the bottom. "I have done hiring at Bellcourt for eleven years. I know what a distribution-floor resume looks like. If your tool cannot tell the difference, I need to know that, not a form letter saying it looks fine now."

So this time Anand did not run the shortcut. He pulled the original log entry: Tuesday, 9:14am, score 49.6, cutoff 50.0, bucket "does not meet requirements." He replayed it once against the current model, got 51.2, "possible match," and felt the old relief start to arrive. Then he made himself check the calendar instead of closing the ticket. Keystone's nightly snapshot had rolled out at 2am Wednesday, hours after the original score and hours before his replay. Version skew looked like the obvious answer.

He tested it properly instead of trusting the story. He pinned the exact old snapshot, the one live when Concepcion first saw the number, and ran the identical resume through it twenty times. Nine runs came back above 50. Eleven came back below. The same file, on the exact model that produced the original complaint, landed on both sides of the line almost half the time. He ran the same twenty-times test on the new snapshot too, in case the update had genuinely fixed something: eleven above, nine below. Almost the same split. The deploy had changed nothing. The number had never been stable, on either version.

Anand went back through the quarter's backlog of 46 closed "could not reproduce" tickets, across every client Marchwood serves, and checked how close each original score had sat to its own cutoff. Nineteen had landed within one point. Nine more sat within two points. Twenty eight of forty six, sitting close enough to the line that ordinary run-to-run noise could have flipped the bucket on its own.

The decision that got made in a meeting years earlier, when Keystone first shipped, was to treat the replay button as a fix-verification tool: test today's model against yesterday's complaint, see if it's still broken. Nobody in that meeting was wrong. Nobody imagined a bug that only existed as instability, one that any single replay would erase just by running it again.

The fix Anand shipped puts a review band around the cutoff, from 47 to 53, and routes any score that lands inside it straight to a person instead of an automatic bucket. Re-run against the closed backlog, that band catches 28 of the 46 old tickets outright, the exact ones sitting closest to the coin flip. Concepcion's candidate gets an interview. The 18 tickets outside the band stay closed, because those really were something else.

What Anand would tell his past self: a bug you cannot reproduce is not a bug that vanished. It is a bug that only needs to happen once to matter, and the fact that it will not happen twice on command is not evidence, it is exactly what an unstable number looks like from the outside.

TRACE, run on exactly one ticket

This is a diagnosis question about a single case, not a design or estimation question, so TRACE fits: rule things out in order, then narrow to the real cause, rather than build or size anything.

T, timeline. Pin the exact clock. The score logged Tuesday 9:14am. The nightly snapshot deployed Wednesday 2am. Concepcion escalated Wednesday 10am. Anand replayed it Thursday. The model update sitting between the complaint and the replay looked like the obvious cause before anything was checked.
R, recut. One complaint has no segment to slice, so the input gets recut across conditions instead: the exact old snapshot, the exact new snapshot, the console's clean upload path versus Bellcourt's real integration path. Each recut isolates one thing that could have changed.
A, assume nothing. Check the log and the parser before the model. Is the file byte-identical to what Concepcion saw? Yes. Did the resume's two-column layout parse cleanly, no scrambled text, nothing that reads like a hidden instruction? Yes. Instrumentation cleared, which means the cause lives inside the model's own judgment.
C, cause candidates. Three named suspects, not a shrug at "it's probabilistic." One, a rare input pattern scrambling the parse (ruled out in the A step). Two, model or snapshot version skew, since a deploy sat right in the gap. Three, ordinary sampling noise at a score that landed close enough to the cutoff to flip on replay.
E, evidence test. Pin the exact old snapshot and replay the identical file twenty times: 9 above the cutoff, 11 below. Run the same test on the new snapshot: 11 above, 9 below, almost the same split. That single check kills version skew and confirms the third suspect, because the instability existed on both versions of the model, not just the newer one.

One more thing the arithmetic alone does not show: the review band's width, 47 to 53, was not a guess. It came from a small held-out set of past near-cutoff resumes with a known correct call, checking how wide a band it actually took to catch the cases that genuinely flipped on repeated replay, without dragging in every resume that merely sat somewhere in the middle of the scale.

Suspect one was cleared in the assume-nothing step. Suspect two looked strongest because of the timing, but the evidence test cleared it too. Suspect three is the one left standing.

And if you want to be sure it really works, try it somewhere else

Corvus Veterinary runs Pinion, an AI that reads a dog or cat's x-ray and flags whether it likely shows a fracture, sorted the same way: clear, needs a look, likely fracture.

Tansy Marchbane, a vet there, reported that Pinion read one dog's leg x-ray as clear, but she spotted a hairline fracture herself on manual review. When a technician re-ran the same image later that day, Pinion flagged it correctly. Nobody could get it to miss the fracture a second time.

T, timeline. No model deploy sat between the two reads this time, ruled out inside an hour, a genuinely different shape from Bellcourt's case.
C, cause candidates. One, the clinic's viewer software had quietly re-exported the image at lower resolution before the second read. Two, a monitor calibration difference between two workstations. Three, the confidence score sat close enough to Pinion's own cutoff that its internal averaging flipped the call.
E, evidence test. Pinning the exact original DICOM file and running it twenty times on the same workstation split 12 flagged, 8 clear, the same boundary-noise pattern as Keystone's resume, on a completely different kind of input.

Swap the trigger and it still runs.
Speed: an interviewer caps the answer at ninety seconds. Skip straight to the pin: pin the model and the input, replay enough times to see a spread, check whether the number sat near a cutoff.
Cost: extra replay passes are not free at Keystone's volume. Do not add extra passes to every resume; add them only inside the review band, where the extra compute is actually buying certainty.
The model got better: say the new snapshot genuinely fixed the instability. The pinned-replay test still earns its place, because it is the only way anyone would know that for certain rather than assume it from one clean re-run.

Where people run it wrong.
They replay once against today's model, see a normal result, and call the ticket resolved, without ever checking which model produced the original number.
They treat "I could not make it happen again" as proof the system is fine, instead of as a specific, checkable claim about instability near a cutoff.
They fix the one loud ticket and never look at the quiet backlog of tickets closed the exact same way.

How to use it live. Say the pin before naming a single cause: "the first thing I would check is whether we are even testing the same model that produced the complaint, because a clean replay against a different snapshot proves nothing." That buys you room to ask what actually got logged with the original score, instead of guessing at a cause with no way to confirm it.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0