What percentage would you start a canary at, and how do you decide?

How to answer this, stage by stage

Seven moves. The first one puts a number on the table so you are not dodging. The rest earn it.

Let's learn

Every morning the county benefits office gets about 10,000 applications for housing and heating help. A caseworker reads each one, decides whether the paperwork stands up, and either approves it or writes back asking for proof.

Before the tool, that was 60 files a day per person and a lot of Fridays that ended at six. So we built a pre-screener. It reads the form and the uploaded documents and marks each application one of three ways: clear, check this, or likely not eligible. Anything marked likely not eligible drops into the second pile, where the letter asking for more proof goes out on the next Tuesday post run instead of the same day.

It works. Fridays end at four now. It catches missing pages a tired person misses at 4:45.

Then you roll it out. Somebody says 1 percent, because 1 percent is what you say, and everyone nods, and it goes in the plan.

Four terms. Most people only argue about the second one.

Here is the sum nobody did in that meeting.

The mistakes are not the problem. Seventy wrong calls out of 700 is a bad week, and a bad week can be fixed. The problem is who the seventy are, and what happens to them next.

They are not a random seventy. The tool learned from years of approved claims, and almost all of those claims came with payslips. So the applications it cannot read are the ones with no payslips: people paid weekly in cash, agency cleaners, seasonal farm work, a first month of self-employment. That is about 8 percent of applicants and it is where nearly all of the error sits.

And the deadline is fourteen days from the date of application, not from the date the letter goes out. A file that sits in the slow pile for six days has eight days left to produce proof that a person keeps in an envelope. Plenty of them do not make it, the claim closes for no reply, and they start again.

Now the part that makes this a risk question rather than a rollout question. Those seventy people can appeal the decision. There is a form and a phone number and it works, slowly. What they cannot appeal, or even see, is being in the test group at all. They did not choose this office, they cannot go to a different one, and nobody told them there were two versions of it that week.

What I would leave alone. If a change can only ever move a case up the queue, ship it wide. Say the model gets better at spotting a household about to be evicted and pulls those forward. Being wrong costs a caseworker twenty minutes on a file that could have waited. That is a real cost and it is a cheap one, so a 0.5 percent canary on that change is theatre. Fifty percent, day one, watch the review time. Save the small slices for changes that can make somebody wait.

The lesson. I used to think the canary percentage was the safety decision. It is not. It is the rate dial on a decision that was already made somewhere else, in whichever meeting settled how quickly a human being would look at the results. If that meeting says "we review it on Thursdays," then no percentage is small enough to save you, because the harm is the rate times the wait. Set the wait first. The percentage is what is left over.

The long version, for when you want to feel why it matters

You don't need this to answer the question. Read it when you want to know why the answer is right, not just what it is.

Adaeze has worked the benefits desk at the county office for nine years. She reads the free-text box first, before the income section, because that is where people say the real thing. "My hours got cut in January." "I moved out in a hurry." She can tell from the second page whether a household is going to lose their flat, and she is almost never wrong about it.

Sixty files a day. She has a system: the ones she can decide on the spot, the ones that need a letter, and a small third stack she keeps by her elbow for the ones she wants to think about.

In April the pre-screener arrives. Nobody makes a fuss about it. It just starts appearing at the top of each file: clear, check this, or likely not eligible. And it is good. It spots a missing page 14 before she does. It notices when an employer's name on the payslip does not match the one on the form. By June her Fridays end at four and she has stopped dreading the first week of the month.

The canary starts in the second week of September, at 1 percent, reviewed at the Thursday quality meeting. Adaeze is not told, and there is no reason she would be. Rollout settings are not caseworker business.

The first thing she notices is nothing at all. The second thing is Marek, two desks over, saying on the Wednesday that he has had two reconsiderations come back in the same week, which is unusual, and both of them were people paid in cash. He says it the way you mention weather. She says huh, and goes back to her screen.

Two ways to get the same 500 cases

Priya cleans offices at night. Three buildings, five nights a week, paid every Friday in cash by an agency that gives her a handwritten slip and an envelope. She applied on the second of September, uploaded a photograph of six of those slips, and wrote in the free-text box that her landlord had already sent one letter.

The pre-screener found no wage evidence it recognised. Likely not eligible. Slow pile. The letter asking for employer confirmation went out on the Tuesday post run, six days after she applied. Her fourteen days were counted from the second. She had eight left to get a signature out of an agency that answers the phone about half the time.

It closed for no reply on the sixteenth. She reapplied on the twenty-second. The money reached her in the second week of October, five weeks after she first asked for it, and the eviction notice arrived before it did.

Adaeze never saw Priya's file. It was in the slow pile, and the slow pile is a different rota.

What she did see, by the end of the month, was that the second pile had gone strange. More cash-income cases in it than she would expect. She said so at the Thursday meeting, in the way you say something you are not sure about yet. Someone wrote it down as a possible seasonal effect, which was a completely reasonable thing to write down, because September is when the agency contracts turn over.

That was a real decision, made in a room, by people being careful. If a caseworker can see that the new tool touched a file, they might treat it differently, and then your test is measuring the caseworker instead of the model. That is a genuine problem and it is the standard answer to it. It also meant that the one person in the building who noticed something was wrong had no way to check, and the audit that could have checked was six days away.

Seven hundred applications went through that window. About seventy of them looked like Priya's.

Now run September again, the other way. On the first Monday the model is switched on at 100 percent, in shadow. It reads every application, writes down what it would have said, and moves nothing. By eleven that morning there are more than 500 files where the model's answer and the caseworker's decision can be lined up next to each other. The split by income type comes out of that comparison the same day: on payslip cases the model agrees with the caseworker almost always, and on cash-income cases it says likely not eligible about ten times more often than a person does.

Nobody's letter goes out late. Nobody's fourteen days start ticking against a queue they were never told about. You find the exact same problem, one week earlier, and Priya is just an applicant instead of a data point.

Then you go live, at half a percent, with a next-day audit, and the model only allowed to move a case up the queue for the first month. And you write "screened by the new tool" on the file, and accept that it might bias a caseworker slightly, because the alternative is what happened to Adaeze in September.

I was in the meeting where we picked 1 percent. It took maybe forty seconds. I remember thinking it was a conservative number.

GUARD, and where each letter bit

This is a risk question, so the framework is GUARD. A canary is a risk decision wearing an engineering hat, which is exactly why "1 percent" gets said in meetings without anyone doing the sum.

GUARD, for risk, safety and fairness questions

And if you want to be sure it really works, try it somewhere else

A bank is rolling out a new fraud model. When it fires, the account is frozen straight away and a human looks at it within two working days. Different industry, same five letters, same trap.

G, groups. The fraud analyst who clears the queue, and the account holder whose card just stopped working in a shop.
U, unequal. The freezes land on people whose money moved in an unfamiliar shape: a first freelance invoice, a first salary from a new job, someone sending money home the week they arrived in the country. Long, boring account histories almost never trip it.
A, ability to contest. There is a phone line, and it works, in about forty minutes. But the freeze happens before any person is involved, so the appeal starts after the rent has already bounced. The past decision: freeze first, review later, because the team was measured on money out the door.
R, reduce. During the canary the model cannot freeze anything. It can only put the account at the top of the review queue. Worst case is an analyst spends four minutes on a clean account.
D, detect. The freeze reversal rate on accounts less than 90 days old, checked daily. Not the overall reversal rate, which barely moves.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0