What is the difference between a threshold criterion and a distributional criterion?

How to answer this, stage by stage

Seven moves. Say what each rule protects before you compare them, or the answer sounds like two definitions instead of one real distinction.

Let's learn

Say a university uses a tool called Sourceline that reads every essay a student turns in and gives it a score for how likely it is that a machine wrote it.

Before Sourceline, an instructor with 90 essays a term read every one by hand and caught the ones that felt off by instinct, maybe two or three cases a term, the kind so obvious they read like nobody had touched them at all.

Now Sourceline scores every essay the moment it's turned in. Anything scoring above 85 gets flagged for a human to look at. For ten weeks in a row, that flag rate held steady around 3 percent, week after week. The dashboard looked fine the entire time.

Here's the turn. That flat 3 percent wasn't proof nothing had changed. It was the exact place the real story was hiding. Underneath the flag count, the share of essays scoring 75 to 84, sitting just under the line, was climbing every single week.

At its worst, the tool ends up worse than never building it. Instructors trusted the cleared label completely and stopped reading flagged-clear essays themselves, so work that had quietly learned to sit just under the line traveled all the way to a final grade, in some cases into a transcript, before anyone caught it by hand.

Same cutoff, same clean score. The real thing still walks off the page.

What I'd leave alone. Low stakes work, a weekly reading response nobody grades hard, doesn't need this. A slightly stale cutoff there barely matters, because no real grade or degree depends on catching every one of them fast.

The lesson. A number that never moves is not proof nothing is wrong. Sometimes it's proof everyone already learned exactly where it sits. The only way to know a cutoff still means something is to keep checking the shape behind it, not just the number it produces.

Now here is the same thing as a story

Use this version when you've got the time. The short version is above. This is for when you want to feel why it mattered.

Baris Yildirim can read a false positive complaint from a professor and tell inside a minute whether the model got it wrong or the professor did. He's run product for Sourceline for two years, and he set the original 85 cutoff himself, after testing it against a stack of sample essays nobody had graded yet.

For the first two terms, that number earned its trust the honest way. Every month, Baris pulled the flag rate next to what the integrity office actually found in random spot checks, and the two agreed. A flagged essay was almost always a real problem. A cleared one almost always was fine.

Then the checking faded, in three small steps, none of them looking wrong at the time. First, because the flag rate held so steady, Baris stopped pulling the raw score list every month and only checked it once a term. Second, seeing the same steady number, the integrity office quietly dropped its extra random audits between terms. Third, instructors, trusting the cleared label completely, stopped even skimming the essays that came back under the line.

Then a teaching assistant, grading a stack of essays late one night, sent Baris an offhand message. A run of essays in one course all sounded the same, oddly polished, in a way she couldn't quite name.

Baris didn't pull the flag count. He pulled the raw scores instead, every single one, plotted by week. The band sitting just under 85 had grown from a sliver to more than a third of the class.

It would be easy to say the model got worse. It didn't, not really. A class had spent ten weeks learning exactly where 85 sat, one small edit at a time, and the flag count had no way to show that, because none of them individually crossed the line.

So here's the decision Baris would take back.

Two years earlier, in the meeting where the monitoring plan got set, someone had asked whether they should also chart the full spread of scores every week, not just the flag count. It sounded like an extra dashboard for a problem the flag count already seemed to answer. Baris remembers agreeing they could add it later.

I would build that chart from day one instead. Same 85 cutoff, same weekly runs, but with a real floor on the band underneath it, one someone actually has to act on.

Here's the replay. Same ten weeks, new chart running the whole time. The band crosses the 15 percent floor in week five, and that crossing is the trigger, not a teaching assistant's late night message. Baris pulls a fresh batch of that week's actual essays and retrains the cutoff region on them. The band falls back under 15 percent by week seven. Finals close with 2 confirmed cases, the normal number for a term, instead of 19.

One design watches whether today's essay passed. The other watches whether passing still means anything.

And the thing Baris would tell himself, back in that first meeting: the flag count was never the risk. Trusting it with nothing to check it against was.

What LEAD looks like when you're comparing two rules instead of picking one metric

This reads like a definitions question. Underneath it, it's still asking what tells you a pass or fail rule is working, before someone outside the tool finds out it isn't. That's LEAD, run on two rules at once.

And if you want to be sure it really works, try it somewhere else

A circuit board plant runs a camera over every solder joint on the line, scoring how likely each one is to fail, and rejects anything over a set score before the board ships. Same shape of question. A pass rate that holds perfectly steady at that reject line can hide the same thing a flat flag rate hid at Bellhaven: something drifting, just under where anyone's watching.

Same idea, a solder line

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0