Explain how rollout differs when quality varies by user segment.

How to answer this, stage by stage

Eight moves. This question tempts a broad speech about "testing more languages," so most of these stages exist to show exactly how one healthy blended number hid a segment that was failing outright, and to name the test that would have caught it before it cost anyone a job.

Let's learn

Profila is a tool Harrowmere built to read an uploaded resume and turn it into a structured candidate profile: work history, education, skills, and dates, all sorted into the right fields.

Before Profila, a recruiter working a posting on Harrowmere had to open every resume by hand and copy the details into their own tracking system. For a typical posting with 150 applicants, that was about three minutes a resume, seven and a half hours a week just typing out what was already on the page.

Profila cut that manual step to almost nothing. Company-wide, it launched with a blended parse accuracy of 88 percent in its first week, above the 85 percent bar Harrowmere had set to go live, and it only climbed from there.

Here is the turn. That climb from 88 to 91 percent was never really the story, and the two or three points it moved most weeks were never the problem either. The real problem showed up the moment somebody split that one number into two.

At its worst, this costs Harrowmere about 900 French-format candidate profiles a week that come back with the work-history field left entirely blank, out of roughly 2,500 French-format resumes Profila handles in an average week. A blank field isn't a low score a recruiter can see and shrug off. Most client hiring filters, including Kesterline Logistics's own, read a blank work-history field as zero years of experience, so a genuinely qualified candidate gets filtered out before a person ever opens the resume, and nobody, not the candidate, not the recruiter, ever finds out why.

The gap between when Profila's blended number first read healthy and when anyone actually checked it by resume format

The choice I would take back. When Naila's team built the holdout set to certify Profila for launch, they pulled a random sample from Harrowmere's resume history, the same way you'd sample anything else. French-format resumes are a minority of that history, so a random sample barely included any, and their accuracy problem never had a chance to show up before go-live. I would take that back. I would stratify the holdout by resume format and language first, then sample inside each one, so a rare segment can't hide just because it's rare.

What I would leave alone. The English-format pipeline doesn't need any of this. It's held 97 percent since week one, the week Profila launched. Rebuilding anything there would just spend effort on a gap that isn't happening.

The lesson. A blended number that clears the bar every single week isn't proof a tool is ready for everyone using it. It's proof the group carrying the average is doing fine, and the room reading one number has no way to tell whether that's one group or two.

The morning the filter came back empty

Read the short version above if you're short on time. This is the long version, for the part where you feel exactly how close a good number came to hiding a real one.

Naila Aubarede has run product for Harrowmere's resume tools for four years, and she's usually the person recruiters call when a filter behaves strangely, because she's usually the one who can tell them why in under a minute.

Profila was her idea. She'd watched recruiters open resume after resume by hand, three minutes each, and she wanted one button that did the sorting for them. It launched on May 4th, and for the first week, then the second, the number on her dashboard did exactly what a launch number is supposed to do. It sat above 85 percent, then crept a little higher.

By week three she'd stopped checking it every morning. It kept being fine. She had other launches to watch.

Then, on a Thursday in week six, Wilhelmina Feringer called her.

Wilhelmina runs bilingual hiring for Kesterline Logistics, a client that posts forklift and warehouse roles across Quebec and Ontario. She'd posted a bilingual forklift-operator role, gotten dozens of applicants, and pulled up Harrowmere's candidate list to filter for three or more years of experience. Almost none of the French-language applicants showed up. Not low scores. Not weak matches. Just gone, as if they'd never applied.

Naila's first thought was that Wilhelmina's filter was set wrong. It wasn't. So she pulled Profila's dashboard back up. Blended parse accuracy, week six: 90 percent. Comfortably above the bar. Exactly what it had read every week since launch.

So she split it.

English-format resumes: 97 percent. French-format, Quebec-style resumes, about 18 percent of everything Profila processes: 58 percent.

She pulled up the deck from Profila's go-live review. One line on slide four: "Holdout accuracy: 91 percent, ready to launch." Nobody in that room had asked what the holdout actually contained.

So here is the decision I would take back. When Naila's team built that holdout set, they sampled at random from Harrowmere's resume history. French-format resumes are a minority there, so the sample barely touched them, and a segment that was quietly failing never had a chance to be seen before real candidates started disappearing from a real filter.

And the part I'd want to tell myself, if I could go back: we tested Profila against the resumes that would never once make it look bad. We never once tested it against the fifth of the resumes it was actually going to meet.

What splitting the number by segment actually showed

Before trusting the segment gap, Naila's team checked whether Profila's own math was even right. Engineers pulled twenty real French-format extractions and checked whether each one made sense given what the model was fed at that exact moment. Nineteen of twenty did. The model wasn't computing garbage. That left the resumes themselves.

That left the question of what, exactly, about a French-format resume was breaking. Naila's team ran three tests on the same batch of resumes, changing one thing at a time.

Fixing the layout closed almost the whole gap. Fixing the language on its own barely moved it. That's what pointed at the two-column reading order, not the French words themselves, as the real cause.

Three reasons the French segment broke, and the one the test confirmed

Not because anyone was careless. Each of these, on its own, looks like a sensible reason a new format would score a little lower. Together, they're why a blended number that never once looked shaky could still be filtering real people out of a real search.

Three separate, checkable causes, only one of them confirmed by the reflow test

TRACE, run over a resume shaped like two columns

This reads like a question that wants a general rule about testing more languages, but the real job is diagnosis: work out why a rollout that read healthy every single week could still filter real candidates out of a real search, and prove exactly where that promise broke.

The same gap, on a building permit instead of a resume

A city permits office pilots ClearScope, a tool that reads a submitted building-permit application and pulls out the project scope, square footage, and contractor license number. Bartholomew Osterhagen manages the rollout across every permit type at once. Blended field-extraction accuracy read 92 percent in week one, climbing to 93, 93, and 94 percent by week four, comfortably above the office's 88 percent go-live bar.

T. ClearScope went live on March 10th. The blended number read fine every week through week four. In week five, a permit clerk flagged that homeowner-submitted applications kept coming back with the square-footage field empty, backing up the manual review queue.
R. Recut by applicant type. Contractor-submitted permits, typed CAD cover sheets, about 76 percent of volume: 98 percent accurate. Homeowner-submitted permits, handwritten and photographed on a phone, about 24 percent of volume: 63 percent accurate.
A. Same frozen model the whole five weeks. A hand check of twenty homeowner-tagged permits confirmed the tag was real: genuine handwritten, phone-photographed applications, not a labeling mistake.
C. Three candidates: homeowner submissions were under 5 percent of training data; phone-camera photos have inconsistent lighting and skew next to a clean PDF; and homeowners often write square footage as a range, "about 400 sq ft," while the model only recognized an exact number as a valid value.
E. Take homeowner permits that wrote square footage as a range, and manually normalize just that field's wording to an exact number, same handwritten scan otherwise: accuracy jumps to 91 percent, close to the contractor rate. Take a separate batch, keep the approximate wording, but correct the scan's lighting and skew: accuracy barely moves, 66 percent. The exact-versus-approximate mismatch, not the scan quality, was driving the gap, a different cause than Profila's, confirmed the same way.

Flashcards (click a card to flip it)

This is a question about rollout quality varying by user segment, worked as a diagnosis, so these eight test the TRACE moves and the real numbers behind them.

Check yourself Score: 0 / 0