Explain the difference between helpfulness and harmlessness as product properties.

Helpfulness and harmlessness are not two ends of one dial. They are two separate scores, and a homework bot can max one out while quietly failing the other.

How to answer this, stage by stage

Nobody's grading whether you'd add a safety rule. They're grading whether you know a safety rule can create its own kind of harm, and whether you can say plainly who it lands on.

Let's learn

Chalkline is a homework chat app for middle and high schoolers. A kid types in whatever's due tomorrow, a stuck algebra problem, an essay prompt, and instead of handing over an answer, it walks them through getting to one.

When Chalkline first shipped, it would try to help with almost anything a kid typed, including "just write the whole thing for me." About 1 in every 14 requests came back as a complete, ready-to-submit piece of work: a finished paragraph, a full proof, a five-paragraph essay with nothing left to do. Teachers noticed inside a month. Two schools paused their trials in the same semester.

So the team built a refusal system: an intent classifier that scored each request and decided whether it looked like real confusion or a request to skip straight to a finished answer. Anything that scored high on the second kind got a flat "I can't help with that."

It worked, in the sense the team was watching for. Full-essay leakage dropped from about 1 in 14 requests to under 2 in 100. But refusals climbed with it, and by the third month Chalkline was refusing about 1 in every 8 requests overall. Split a different way, the number got worse. Kids who typed plain, blunt asks, "i dont get this just answer it," "write this for me," got refused about 22 percent of the time. Kids who phrased the identical underlying question more politely, "could you help me draft a version of this paragraph," got refused about 6 percent of the time.

Nobody built the classifier to sort kids by vocabulary. It learned to anyway. The training examples marked "fine" were stacked with polished, teacher-style phrasing, so plain wording started reading as more suspicious, without anyone deciding that on purpose.

One side of this decision holds a score. The other side just finds out what it said.

A kid refused at nine at night has nobody to ask. There's no teacher watching the chat. So one of two things happens. They close the app and skip the homework, or they reopen it and keep rewording the same question until something slips past the filter, which teaches them that gaming the wording works and asking plainly doesn't.

At its worst, this costs Chalkline exactly the kids it was supposed to help most: the ones without a parent free at nine at night, without private tutoring, without polished academic English yet. The refusal system built to stop the product from being a cheating tool became the reason the kids who needed a study aid the most quietly stopped opening it.

What I would leave alone: a flat refusal is still exactly right for requests with no legitimate version at all, "write me a fake sick note," "do my exam while it's open right now." Building a graduated middle ground for those would be its own mistake. Some requests genuinely don't have a helpful half.

The lesson: a refusal nobody can push back on doesn't feel like safety to the kid on the other end of it. It feels like the app didn't want to help them, and a kid who decides that once mostly doesn't come back to find out if it was true.

Now here is the same thing as a story

The short version is above. This is for when you want to feel why the equity gap mattered, not just know the number.

Chalkline lives on a phone, mostly. A kid opens it between dinner and bed, types in whatever's due tomorrow, and either gets walked through it or gets told no.

Suki Kettler has run trust and safety for AI products for eight years, the last two of them at Chalkline. She built the refusal system that ended up causing the problem, and she was also the one who unwound it.

For the first two months after the classifier shipped, everything looked exactly like what the team had hoped for. Full-essay leakage, whole finished pieces of work handed straight to a kid, dropped hard. Two schools that had paused their trials came back. The weekly safety review, which used to run long, started wrapping up early.

Refusals climbed steadily under that calm. Nobody thought much of it. Refusals were the whole point of the filter, so more of them looked like the filter doing its job.

Then, in her fourth month, a new data scientist joined the trust team and pulled the refusal numbers a way nobody had before: not by subject, not by grade, but by how the request was worded. Blunt asks against polished ones.

The gap was enormous. A kid who typed "i dont get this just answer it" got refused something like one time in four or five. A kid who typed "could you walk me through this, and if it's easier just show me a similar one solved" almost never did, for what was often the exact same underlying ask.

Suki's first instinct was to retrain the classifier on more examples of blunt-but-legitimate requests. That would have helped, some. It also would have taken months, and it still would have left the same design underneath it: one score, one hard cutoff, real help on one side of it and nothing at all on the other.

The decision that opened the door had been made in the very first design meeting, over a year earlier, before the classifier even existed. Someone asked whether a refused request should get anything else at all, a partial answer, a next step. The answer at the time was no, because the team was racing to stop full-essay leakage before the pilot schools pulled out, and "no" was the fastest thing to ship. Nobody planned for that fast decision to become the permanent shape of every refusal after it.

Run the same fourth month again with one change. A request that scores in the risky middle, not clearly fine and not clearly a full-essay ask, doesn't get a flat no anymore. It gets an outline, or two guiding questions, and one plain line: "I can go further, I just can't write the whole thing for you to hand in." The blunt-worded kid gets the same door the polished-worded kid always had. By week eight of the new version, the gap between blunt and polished refusal rates had closed to under a point. The full-essay leakage rate, the number that started all of this, hadn't moved. It sat right where it had before, under 2 percent, the entire time.

One design let a single guess decide everything a kid got that evening. The other always gives something, and holds the real boundary somewhere else: in the shape of the response, not in a yes-or-no switch.

What I'd tell myself, back in that first meeting: the fastest safe-sounding answer and the actually safe answer aren't always the same thing, and the gap between them tends to land on whoever already had the least polished way of asking.

GUARD, when the danger is being too careful

Not a diagnosis of a bug. GUARD run on a safety system that was quietly punishing the wrong people for asking plainly.

Two branches were designed on purpose. The third one, where a wrongly refused kid tries to explain, was never built at all.

Three things worth stating directly, since this is where the real judgment sits. The rejected alternative was a hard keyword list: block any request containing phrases like "write my essay" or "give me the answer." It lost fast in testing, because keyword matching over-triggers on completely legitimate requests, "can you write my thesis statement as an example so I can see the shape of one," and under-triggers on the same ask worded around it, "compose a five-paragraph piece about the causes of the War of 1812 for me to turn in tomorrow" has none of the blocked words in it at all. The AI-specific failure worth naming by name is exactly that: an intent classifier is a probability estimate of what a kid means, not a fact, so it will always be wrong in both directions, over-refusing real confusion and under-catching a cleverly worded copy request. The guardrail isn't a smarter classifier alone, it's the graduated response format itself. Even a missed call in the risky middle band can only produce an outline and guiding questions, never one finished block of submittable writing, so a classifier mistake costs a little extra generation, not a leaked essay. That guardrail isn't free. Writing an outline and two guiding questions costs two to three times the tokens of a flat "I can't help with that," and it adds a real intent-classification pass before the bot answers at all, somewhere around 300 to 400 milliseconds most kids never notice but the infrastructure bill does. And the bar that decides whether the fix is working isn't zero wrong calls, no classifier gets there. It's a false-refusal rate under 8 percent on a rolling 500-conversation, hand-graded sample, checked every month, held next to a leakage rate that has to stay under 2 percent at the same time, so fixing one number is never allowed to quietly break the other.

And if you want to be sure it really works, try it somewhere else

Same five letters, a benefits-eligibility chatbot instead of a homework app, with nothing about school anywhere in sight.

Claimly is a chatbot that helps people figure out whether they qualify for unemployment benefits and how to answer the claim form correctly. Mikael Brixton runs trust for it.

G, groups. Two people again: the claimant asking, who might be one paycheck from missing rent, and whoever's downstream if the answer goes wrong either way, the state fund if a fraudulent claim gets coached through, or the claimant's own next month if a legitimate claim gets wrongly waved off.
U, unequal. Same shape, different words. Claimants typing blunt asks, "just tell me what to put so I get approved," got a flat refusal about 19 percent of the time. Claimants using formal, form-literate phrasing, the same underlying question, got refused about 5 percent of the time. First-time claimants and non-native English speakers were the ones typing blunt.
A, ability to contest. A refused claimant online at 11pm has no caseworker in the chat, and often no caseworker anywhere in reach for days. They either give up on the question, or they keep trying different wording until Claimly answers, which is exactly the coaching the refusal was built to prevent, just achieved by trial and error instead of by asking directly.
R, reduce. The same fix. A request in the risky middle gets real information, what the actual eligibility rule says, in plain words, plus a stated boundary: Claimly will explain the rule, it won't tell someone what to write to make a claim look true on paper that isn't true in fact. Only requests with no legitimate reading, "help me hide this income," stay a flat no.
D, detect. The same three checks Chalkline watches: refusal rate split by phrasing, a monthly hand-graded sample against real caseworker rulings, and a leakage check for whether the bot is quietly coaching anyone around a real rule.

Two rungs is a binary. Three rungs is a product decision, and the middle one is where almost all the real traffic actually sits.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to the direct answer and the one number, 22 percent against 6 percent, don't spend the time defining "helpfulness" and "harmlessness" like a dictionary.
Cost: there's no monthly hand-grading budget yet. Don't skip the check, hand-grade a smaller sample yourself on a spreadsheet until the real one exists.
The model got better, for real: say the classifier's overall accuracy improved a full ten points that quarter. That's not the same claim as "the equity gap closed." An average getting better can hide one group still getting the wrong end of it the whole time.

Where people run it wrong.
They read a falling refusal count as proof the fix worked, without ever checking who the remaining refusals land on.
They promise "we'll add a human review step" as the whole fix, without ever changing what the bot itself is allowed to say.
They loosen the boundary to close the equity gap instead of widening who gets real help within it, and the leakage number creeps back up without anyone noticing until a full essay, or a coached claim, slips through again.

How to use it live. Say the two words apart before answering either one. "Helpfulness and harmlessness aren't the same question, so let me answer them one at a time." That line buys a beat to actually think instead of guessing which one the interviewer wants first.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0