What eval would you build first with only one week?

With five real days, build the golden set that catches a wrongly assigned action item, not the eval that polishes a sentence someone was going to reread anyway.

How to answer this, stage by stage

Nobody is grading whether you know what a golden set is. They are grading whether you can pick the one mistake worth a scarce week, and defend leaving the rest for later.

Let's learn

Notchbrief listens to a client call, and about ninety seconds after it ends, everyone on the call gets a summary and a list of who owes what by when.

Before Notchbrief, an associate at Fenshaw and Corbeau spent about twenty five minutes after every client call writing the notes and the action items up by hand, across roughly thirty five client meetings a week, team wide.

With Notchbrief running but an associate still reading every draft before it went out, the draft was ready in ninety seconds, and the read-through added back about eight minutes, catching a wrong name or a misassigned task in a couple of lines. That eight minutes felt like plenty of safety margin, so nobody built a real eval. Why would you, when a person reads it every single time.

Now leadership is cutting that read-through step, starting Monday, one week from today, to keep a sales promise that action items land in your inbox before you're back at your desk. The extra mistakes Notchbrief makes on its own are not really the problem. Most of them are typos and clunky sentences, and someone notices and fixes those the moment they open the email. The real problem is the mistake nobody is specifically looking for: an action item assigned to the wrong side of the table, or invented outright, sitting in a list that reads exactly as confident as the correct ones.

The golden set is the one box that has to exist before anything downstream of it can happen, and it's buildable with what Beckett already has on day one.

At its worst, Notchbrief without a gate is worse than never building it at all. Two people each think the other one owns a task. Nobody does it. By the time anyone notices, three weeks have gone by, a deal has gone quiet, and nothing in the notes was ever flagged as unusual.

What I'd leave alone: the summary's tone and phrasing don't get the same scrutiny in week one. A slightly stiff sentence costs a raised eyebrow, not a missed deliverable.

The lesson: a mistake with no complaint attached to it is not a small mistake. It's a mistake nobody's counting yet, and a week is exactly enough time to start counting the one that matters.

Now here is the same thing as a story

Read the short version above for the two minute answer. Read this when you want to feel why waiting for "enough usage" felt like the reasonable call at the time.

Before Notchbrief existed, Beckett Prentice was one of the associates at Fenshaw and Corbeau who wrote up call notes by hand, every evening, for years. Nobody's action item ever went to the wrong person on Beckett's watch, because Beckett had sat in the room and knew exactly who'd said they'd send what.

Notchbrief launched as an internal pilot, and the good months were good. A draft landed ninety seconds after a call ended instead of twenty five minutes later. An associate still read every one before it went to a client, caught the odd wrong name, fixed it, sent it. Nobody thought twice about skipping a formal eval. The reading step was the eval, in everyone's head.

Then the habit thinned, in three small beats. First, reading every line against the actual call became skimming the top summary and trusting the action item list, since it had been consistently fine for weeks. Then, associates rushing between back to back calls started forwarding drafts straight to clients without the full read, just this once, just today. Then, the daily standup stopped asking "did you check the action items specifically," and it quietly stopped being a step at all.

The trigger was small. A new associate Beckett was training asked a question nobody had a real answer for: "How would we even know if Notchbrief assigned a task to the wrong person, since nobody's going to complain about a task they don't know is theirs?" Beckett didn't have a good answer. Said he'd get back to her.

Before Beckett could build anything, leadership set the date: the read-through was being cut starting Monday, to keep the sales promise that action items land in your inbox before you're back at your desk. That left five work days. Beckett spent day one hand checking ten recent drafts against their real transcripts, the same check that produced the eighty eight versus nine percent gap. That told him where to point the rest of the week: not at wording, at ownership.

By Thursday, the full golden set was forty real transcripts, graded by Beckett and one associate who'd sat in each call. Recut by the kind of task, a pattern jumped out. Action items involving a hand-off, one side sending the other side data or a document, were misattributed a third of the time, thirty three percent, wrong owner or invented outright. Every other kind of action item sat at four percent. The blended number across all action items had looked fine. The hand-off slice alone did not.

The decision that opened the door went back to that first launch meeting. The team agreed eval could wait until Notchbrief had "real usage" behind it. Nobody wrote down what "real usage" actually meant, or set a date to revisit it, because at the time it felt like a sensible thing to defer, not a live risk.

Run the same week again with one change: Beckett doesn't wait for a full rebuild of the review step, and doesn't cut it entirely either. He keeps a short human check only on action items that involve a hand-off, roughly one in three of the week's drafts, and lets everything else go straight out. Review time drops from eight minutes on every one of the week's thirty five drafts to about ninety seconds on the third that actually carry the risk. On the following Monday, that narrow check catches four hand-off items that would have gone to the wrong side, before a single client ever sees one.

One design trusted a single person's eyes on every draft to speak for a mistake type nobody had actually measured. The other design measures the one category that matters and keeps a person only there.

What I'd tell myself, back when the new hire asked her question: the day someone asks how you'd know, that's the day to start counting, not the day to promise you'll get back to them.

ORDER, when the week is the whole budget

Not a checklist for a planning meeting. Five questions that build toward the one that actually decides what gets built first: what's hardest to undo.

One of these gets caught by a stranger rereading it. The other one never gets reread at all.

Three things worth stating directly, since this is where the real judgment sits. The rejected alternative was a broad, automated eval scoring every kind of thing Notchbrief writes, tone, structure, and action items alike, in the same week. It lost because a five day budget spread evenly across everything ends up an inch deep on the one output where being wrong actually costs someone a client. The AI specific failure worth naming by name is the invented action item: a task that reads exactly as confident as a real one but was never agreed to on the call, which is a different and worse failure than a task just sent to the wrong person. The guardrail is scoring invented tasks as their own tagged category inside the golden set, not blended into a general accuracy number. And the trade being accepted on purpose is real: keeping a short human check only on hand-off type action items, instead of cutting the human step to zero or keeping it on every draft, costs real minutes on roughly a third of the week's output, traded against a category that had shown thirty three percent misattribution with nobody watching for it.

The five, in one line each:
O: whether the output can be trusted enough to act on unread.
R: a wrong sentence gets reread for free. A wrong action item doesn't.
D: real transcripts plus two people who were in the room, buildable inside the week itself.
E: a cheap ten-draft hand check on day one points the rest of the week at the real risk.
R: action item correctness first, hand-off tasks flagged on their own, tone waits.

Same five letters, a vet clinic instead of a client call

Not every silent mistake sits in a client inbox. Sometimes it sits on a kitchen counter, in a bowl labeled twice a day instead of once.

Coastwell Veterinary Partners runs Scriblatch, a tool that listens to a vet visit and, once the appointment ends, texts the pet owner a summary and a list of home care tasks, medication, dose, how often, for how long. Griffen Askew runs eval on it, also with one week before the clinic's front desk stops reading every text before it sends.

Griffen's day one hand check, ten recent visits against the vet tech's own written notes, found something close to Beckett's split. A wrong clinic detail or a stiffly worded line got caught by an owner calling in to ask, almost every time. A wrong medication dose or frequency didn't get caught by anyone, not until the follow up visit, sometimes two weeks later.

Same method, different shape: a wrongly assigned meeting task and a wrongly stated medication dose look nothing alike, but both are a model output that gets acted on by someone who has no reason to double check it, before anyone outside the company would ever know it was wrong.

Swap the trigger and it still runs.
Speed: an interviewer caps you at ninety seconds. Skip straight to the one line: find the mistake that doesn't get reread, then spend the week building the eval that catches that one first.
Cost: there's no budget this quarter for a full labeling team. Grade fifteen transcripts instead of forty, with the two people you already have. The sample shrinks; the category you're protecting doesn't change.
The model got better, for real: say the newest version is measurably more accurate on average. That's exactly when this matters most, since a more accurate model still makes its rarest mistakes with full confidence, and a rare, confident mistake is the hardest kind for a person to catch by eye.

Where people run it wrong.
They build one broad eval across every output instead of ranking by which mistake actually can't be undone.
They grade against a clean, written up transcript instead of the messy real recording, and miss the exact failure that only shows up in overlapping, real speech.
They wait for "enough real usage" before starting the golden set, instead of using the handful of real transcripts already sitting in a folder.

How to use it live. Say the ranking rule out loud before you say anything else: "rank by what's hardest to undo." That buys a beat of thinking time, and it turns the rest of the answer into naming the one mistake that fits that description.

Flashcards (tap any card to flip it)

Check yourself Score: 0 / 0