Critique a pilot that was run entirely by the team that built the feature.

How to answer this, stage by stage

Seven moves. The number that matters shows up in stage five, and the whole answer is really about who gets to hold the rubric.

Let's learn

What happens when the people grading a pilot are the same people who'd be embarrassed if it failed?

Say a devtools company builds a feature that reads every comment left on a pull request and scores it, from zero to a hundred, for whether it's specific enough to be worth a reader's time. A comment that says exactly what's wrong and how to fix it scores high. A comment that just says "looks off" scores low.

Before the feature, a team lead had no way to tell a good comment from a lazy one except by reading every single one by hand. On a busy week, one senior engineer read maybe thirty comments closely out of two hundred left across the team.

The company ran its own pilot, three rounds over five months. Round one, self-graded, 74 of every 100 flagged comments held up. Round two, 85. Round three, 93. Good enough, the team decided, to tell the customer running the pilot, Ferrovia Systems, that Lintline was ready for their other 46 engineers.

Here's the turn. The extra points between round one and round three were real. Nobody faked a number. The problem is who was doing the checking. When a customer's engineer checked a batch of round three's flagged comments himself, against the actual code, only 31 of 50 held up.

The self-graded score kept climbing. What the team asked someone outside to check only ever had one setting.

At its worst, this costs more than an awkward gap between two numbers. It costs the thing the tool was supposed to build in the first place, trust that a comment marked "good" is actually worth reading. Ferrovia paused the rollout to its other 46 engineers and asked for four weeks of manual spot checks before they'd trust the score again.

Who grades a pilot is a switch, not a dial

What I would leave alone. A huge share of what Lintline flags is cosmetic, a suggestion to rename a variable, or a note catching a typo. There's no fact to get wrong in a comment like that, so it doesn't need an independent check. Only the comments making a real claim about how the code behaves need someone outside the room.

The lesson. A number that only the people who wrote the rubric ever checked isn't proof the tool works. It's proof they agree with themselves.

Now here is the same thing as a story

Pull this one out when there's more time, and you want the interviewer to feel the gap, not just note it down.

Sabine Tavares can read a pull request thread and tell, before she reaches the third comment, whether the reviewer actually looked at the diff or just skimmed it. She built Lintline over eight months, the feature inside Anvilcrest's review platform that reads every comment on a pull request and scores it for whether it's specific enough to be worth a reader's time.

Ferrovia Systems, a company that builds routing software for freight rail networks, agreed to pilot it across a slice of their engineering team. Round one ran eight weeks, twelve engineers. Sabine and two teammates from the Lintline team pulled a sample of flagged comments each week and checked them by hand. To be sure they weren't fooling themselves, they also asked Priti Nandy, a researcher from Anvilcrest's developer experience team who'd never touched Lintline's code, to check a separate sample of twenty on her own, blind. Round one: the team's own check said 74 of every 100 flagged comments held up. Priti's blind check said 71. Close enough that nobody worried.

Round two, ten weeks, eighteen engineers. The self-graded number climbed to 85. Priti checked again, ten comments this time instead of twenty, since round one had matched so closely. Her number: 82. Still close.

Round three, nine weeks, twenty-four engineers. Self-graded, 93. Nobody asked Priti to check anything at all this time. Sabine figured, reasonably, that a team who'd read thousands of these threads by now knew a good comment when they saw one, and Priti would only be relearning a rubric they'd already spent six months writing.

Then came a Tuesday, three weeks after round three's report had already gone out.

Rashad Sundberg, a senior engineer at Ferrovia, was chasing down a strange bug in the dispatch scheduler. Nothing had broken in production, just a race condition caught in staging. He traced it back to a queue that had been marked, in review, "no lock needed, this queue is only touched by one thread." Lintline had scored that comment 97, specific and actionable. The queue had picked up a second consumer weeks later, and nobody had flagged the comment as something to revisit.

Curious, and a little unsettled, Rashad pulled fifty of the round three comments Lintline had scored 90 or higher and read every one against the actual code, not just the words. Only thirty one held up.

I want to say the problem was that Lintline's model needed more training. It didn't, really. It was already good at spotting a specific, well-written comment, three rounds running. That's not the same as spotting a true one, and nobody on the team had ever been asked to separate the two. Sabine never had a rule for who was allowed to grade the pilot. She had a habit, and the habit had two settings. Check it against someone outside the room, or don't. Round one, she did. Round three, she didn't, because round one and round two had never once disagreed with her own team, so asking felt like a formality nobody needed anymore.

So here is the decision I would take back.

At the pilot's first planning meeting, someone had actually raised giving Priti's team a permanent seat on the grading panel, every round, not just a spot check. Sabine said no, gently. Priti would need weeks to learn a rubric they'd already spent six months building. It was a pilot. They could always add her later if the numbers looked shaky. Nobody's numbers ever looked shaky, so later never came.

I would put Priti on the panel from round one, with one job only, not tone, not clarity, just whether the technical claim in a comment was actually true, checked against the code. Run round three again with that in place. The self-graded number still lands at 93, because the team is still right about what a well-written comment sounds like. But by week four of round three, not three weeks after it shipped, Priti's spot checks catch that "no lock needed" comments are getting scored on how confident they sound, not on whether anyone rechecks the claim later. The rubric gets one line added: a claim about behavior needs a note on when it was last true. By the time the report goes to Ferrovia, the independently checked number is 89, not 62, and it's the number that actually sits next to the self-graded one in the report.

If I'm honest, saying no to Priti at that first meeting wasn't the mistake. Nobody hands a stranger a rubric they haven't finished writing yet. The mistake was never going back, three rounds and five months later, to ask whether "we'll add her if the numbers look shaky" still meant anything once the numbers never looked shaky at all.

FLIPS, run against the report Sabine actually sent

The letters matter less than which one breaks first. Here's the same five steps, mapped onto Lintline's round three report.

FLIPS, five rows

"They should have double-checked more carefully" is a diagnosis anyone can offer after the fact. The harder part is naming the exact thing missing from day one, a checker with no stake in the feature, and showing there was no smaller fix once someone outside the room had already found the gap.

And if you want to be sure it really works, try it somewhere else

Loomwright Vision is nowhere near code or comments. It makes a camera system that spots defects in fabric as it runs down a factory line, and it piloted the camera entirely with its own engineers on Aldergrove Textiles' floor. Same question, a different flip this time. Nobody stops checking. Somebody starts quietly choosing what gets checked.

F. Cosima Petrenko, quality engineer at Loomwright Vision, ran the pilot on Aldergrove's line.
L. In the first two weeks, whatever roll of fabric queued up next on the line went straight past the camera, dark denim and busy prints included.
I. A different flip from Sabine's. Cosima doesn't stop checking, she starts choosing. When a hard batch, heavy or dark or textured, came up on a day the pilot was measuring, she'd ask the floor to run it after the pilot's shift instead, "so we don't waste a good reading on a batch we haven't tuned for yet."
P. At kickoff, the team decided pilot days would run whatever the line already had scheduled, since building a separate, blind batch queue felt like extra setup for a camera that was still unproven.
S. Have Aldergrove, not Loomwright, draw a fixed queue of batches for every pilot day before anyone at Loomwright sees what's coming. Round two: the honest number lands low at first, 73% of known defects caught across every batch, easy and hard. But the miss shows up by week two instead of after rollout, the camera's exposure setting gets fixed for dark fabric by week five, and round two finishes at 94%, on batches nobody picked in advance.

Flashcards (click a card to flip it)

Eight fixed slots, pulled straight from the answer above.

Check yourself Score: 0 / 0