Describe the on-call runbook entry for a sudden quality drop.

How to answer this, stage by stage

Nobody is grading whether you can say "we'd have a runbook." They are grading whether you can design the one line in it that actually decides what happens at 3am. Seven moves get you there.

Let's learn

Say we build a tool that reads every pull request the moment it opens and leaves comments on it: a missing test, a null check about to bite someone, a function that already exists somewhere else in the repo. Patchwright is Lintbrook's version of that tool. Before it existed, a human reviewer read the diff line by line to catch that class of issue, about 22 minutes for an average pull request, and still missed close to six in ten of them by eye.

With Patchwright live, comments show up the moment the pull request opens. Authors got in the habit of fixing the flagged issue before asking a human to look at all, and review rounds dropped from an average of 2.3 back-and-forths to 1.1. On a normal week, about 58 percent of Patchwright's comments get resolved by the author actually applying the fix, its trailing baseline acceptance rate.

Then came a Tuesday. A different team shipped a prompt template edit at 1:20am, meant to nag harder about missing unit tests. It had a small bug: it broke how Patchwright cited line numbers in some comments, so a chunk of them pointed at the wrong line or rendered as broken text. By 2:00am, fleet-wide acceptance rate had fallen from 58 percent to 21 percent.

Say plainly: the 37 point drop is not the real problem. The real problem is what a developer does the third time a Patchwright comment points at the wrong line. They stop reading the fourth one. Then they stop reading Patchwright at all, and tell their team lead to turn it off for the repo.

At its worst, Patchwright keeps posting comments all night, the coverage dashboard still shows 100 percent of pull requests reviewed, and nobody is reading a single one of them. The product looks fine on volume and is functionally off, which is a worse place to be than if it had crashed outright, because a crash pages someone.

The choice I would take back. Lintbrook logged model version deploys in one system and prompt template edits in a separate feature flag system, because the two changes ship from different teams on different schedules. That split felt tidy at the time. It's exactly why the first change log check that night came back clear when the real cause was sitting in the other system the whole time.

What I would leave alone. Patchwright also sends team leads a weekly digest of comment stats. If that digest is a day late or a little off, nobody's behavior changes, because nobody reads it in real time. It doesn't need a 3am entry of its own.

The lesson. A runbook that lists everything to check teaches on call nothing about what to check first. The whole job of a good entry is to replace a scan with one branch, and the branch is only as good as the log it's built on.

Five screens, no order between them. By the time anything useful turns up, three teams have already muted Patchwright.

Now here is the same thing as a story

The short version is above. Read on if you want to feel how close a merged log came to arriving one incident too late.

Kaveh Nazari has been a senior platform engineer at Lintbrook for five years, three of them owning the pager for Patchwright. Ask him what makes a good pull request comment and he'll tell you it's not the finding, it's whether the line number is right, because a developer forgives a wrong opinion faster than a comment that points at code that isn't there.

For most of that Tuesday, nothing about the night suggested trouble. Kaveh checked the on-call dashboard at handoff, saw acceptance sitting at its usual 57, 58 percent, and went to bed. Patchwright had been steady for months. He'd stopped opening the fleet health page every hour the way he used to in his first year on call. It kept being fine, so he checked it less.

At 1:20am, a totally different team, the one that owns Patchwright's prompt templates, shipped a small edit meant to make the missing-test nag a little firmer. Nobody paged Kaveh. Why would they. It wasn't his system, and it looked like a wording change.

At 1:52am, PagerDuty went off. Acceptance rate had fallen from 58 to 41 percent in half an hour. Kaveh opened his laptop, saw the number, and did what the old runbook told him to do: check everything. Grafana for latency, clean. The GPU queue, clean. The model host's status page, green across the board. He typed "patchwright" into the internal deploy log, the one his team owned, filtered to the last 24 hours. Nothing there either.

By 2:00am the number was 21 percent and still falling.

What Kaveh didn't know yet, because it lived in a different team's feature flag dashboard he had no reason to open at 2am, was that the prompt edit from 1:20am had a formatting bug. Comments that used to read "line 214, missing test for this branch" now sometimes read "line [object Object], missing test," or cited a line three files away from the actual change. Developers didn't need a runbook to know that was broken. They just stopped trusting it, one pull request at a time, and by 2:10am two teams had disabled Patchwright review on their repos entirely, a setting that takes one click to turn off and, Kaveh would learn later, a full sprint planning meeting to turn back on.

He found the real cause at 2:24am, twenty two minutes after the page, by finally scrolling through the engineering Slack channel and seeing someone mention the prompt update almost as an aside. He rolled back the template in four minutes. Acceptance rate started climbing by 2:35am.

The post-incident review asked one question that mattered: what would have gotten Kaveh to the fix faster. Not more monitoring, Lintbrook already had plenty. Not a smarter engineer, Kaveh knew the system better than almost anyone. The gap was that the one log he checked first, the one the old runbook told him to trust, didn't cover the surface that actually broke.

The redesigned entry does two things the old one didn't. It merges the model deploy log and the prompt flag system into one queryable history, so "did anything change" means everything, not just the half Kaveh's own team owns. And it puts that single question first, before latency, before GPU, before Slack, so the very first thing on call does is the thing most likely to actually answer the page.

Run that Tuesday again with the new entry, and the change log shows the prompt edit at 1:20am inside the 24 hour window on the first check, not the twenty second one. Kaveh rolls back by 1:58am, six minutes after the page, before a second team has even noticed anything wrong enough to disable review over.

What Kaveh would tell his past self, the one who trusted a log because it was the one he owned: a change log that only watches your own team's surface will tell you, correctly and unhelpfully, that nothing changed.

SPARK: the shape this runbook is built on

This is a design question, describe the artifact you'd build before the failure happens, not a diagnosis of a habit that already flipped, so SPARK fits and TRACE doesn't.

S, situation. Without a runbook entry, whoever's on call for Patchwright when acceptance rate craters opens every dashboard in reach, in whatever order panic suggests, hoping something looks red.
P, payoff. The habit worth building is checking one thing first, the thing that tells on call which of two branches they're on, instead of scanning a dozen screens with no order between them.
A, anchor. The entry's first line: open the unified change log for the last 24 hours. A change in the window means roll back. No change means flip the circuit breaker and escalate. This is the answer to the question.
R, risk. The first time the log is wrong, because it only watches half the surfaces that can change, on call trusts a false "all clear" and burns twenty extra minutes while trust in the product erodes team by team.
K, keep out. Root-causing why a change hurt quality, an auto-rollback bot with no human check, and per-language dashboards all wait past day one. None of them are needed for the branch to work.

One more thing worth naming: the rejected alternative. Early drafts of this entry were a fifteen item checklist, latency, then GPU, then cost, then status page, in a fixed order. It was rejected because a checklist with no priority just recreates the panic scan with extra steps, and by item eight, teams have already started disabling Patchwright.

The circuit breaker branch has a real cost, and it's worth saying out loud instead of pretending the fix is free. Raising the confidence threshold means fewer comments post at all while the breaker is up, so some real bugs pass through completely unflagged for those hours. That's a deliberate trade of coverage for safety, cheaper than letting a broken comment train more teams to stop trusting Patchwright, but a cost all the same, not a free win.

One question, two branches. Everything else in the entry exists to make this question answerable fast.

There's an AI-specific failure mode worth naming here too: hallucinated citations. That prompt bug didn't just make Patchwright wrong, it made some comments cite a file or line number that plainly didn't match the diff, a different kind of failure than an ordinary bad opinion. The guardrail Lintbrook added afterward is a citation validity check: before a comment posts, its cited file and line get checked against the real diff, and anything that fails validation gets suppressed instead of shown. That suppression rate now feeds the same quality metric that pages on call, so a citation bug shows up as a page, not just a slow trickle of Slack complaints.

Same question, two different logs behind it. Only one of them was ever going to catch that Tuesday's actual cause.

And if you want to be sure it really works, try it somewhere else

Husklane Cooperative runs Kernscope, a camera on the grain intake belt that grades incoming truckloads for kernel damage, standing in for the manual probe check a human grader used to do on every load. Halvdan Kristoffersen is the night-shift ag-tech engineer on call during harvest.

S, situation. Without a runbook entry, Halvdan's first move when Kernscope's numbers look wrong is to physically walk out to the intake belt and start checking hardware, one camera at a time, while trucks queue up behind him.
P, payoff. Same habit as Lintbrook: check the one thing that tells you which branch you're on before touching hardware.
A, anchor. The identical first line. Open the change log for the last 24 hours. A software or calibration change in the window means roll back. No change means flip the circuit breaker, in this case reverting to full manual grading, and escalate.
R, risk. Here the anchor is tested by the opposite failure. The log correctly says no change happened, because nothing did, and that's exactly right: three weeks of harvest dust had been slowly fouling one camera's lens, and a wet, high-volume morning finally pushed its readings past the threshold. Rolling back would have reverted to nothing and fixed nothing.
K, keep out. Dust and moisture self-diagnostic sensors on every camera, and automatic lens wipers, both real fixes, both hardware investments that wait for the off-season, not a fix on a Tuesday during peak intake.

Agreement between Kernscope's grade and the elevator's official manual grade normally sits at 91 percent. That morning it fell to 53 percent across 60 trucks in about 45 minutes. The runbook's first check came back clear, correctly, because the real cause was a slowly dirtying lens, not a shipped change. Halvdan flipped the circuit breaker, put manual grading back on for that camera's lane, and the hardware team replaced the lens by evening. A model retrain was considered and rejected for day one: training Kernscope to tolerate a smeared lens is a weeks-long eval cycle, not a same-morning fix, and the operational fallback, manual grading, was already sitting there ready to use.

Judgment shows up in what you don't build yet as much as in what you do.

Swap the trigger and it still runs.
Speed: an interviewer caps the answer at ninety seconds. Skip straight to the anchor: check the unified change log first, roll back if something changed, circuit breaker and escalate if nothing did.
Cost: the on-call team can't staff a 24 hour ML rotation. Don't skip the circuit breaker to save headcount, that's the one piece that buys safe time until daylight without needing anyone awake to investigate.
The model got better: the new prompt template turns out to be a genuine improvement once the citation bug is fixed. That doesn't make the runbook entry wasted work, it's still the only way anyone would have caught the citation bug fast enough to keep the improvement instead of reverting it for good out of caution.

Where people run it wrong.
They write a checklist instead of a branch, so on call still has to decide what order to check things in under pressure.
They build the change log around whichever team is writing the runbook, and miss the surfaces owned by everyone else.
They set a rollback as the automatic first move for any quality drop, even when nothing changed, which turns a drift problem into a wasted rollback and a still-unsolved incident.

How to use it live. Say the anchor before naming a single dashboard: "the entry opens with one question, did something change, because that's what decides whether you roll back or wait." That buys the room to talk about what this product's own change surfaces actually are, instead of reciting a generic incident-response checklist.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0