What leading indicator would catch a prompt regression before an eval run does?

A check that runs once a day only tells the truth once a day. Everything that happens in between still happens to someone.

How to answer this, stage by stage

Nobody is grading whether you can name a pile of possible signals. They are grading whether you can rank them and say which one gets built first. Eight moves get you there.

Let's learn

What do you watch to catch a bad prompt before the harm is already done, not after the next scheduled check confirms it?

Rowcast is a tool built by Ironwood Analytics. Type a question in plain English, something like "what was our net revenue by plan tier last month," and it writes the SQL, runs it against the warehouse, and hands back a table.

Before Rowcast, an analyst wrote that kind of question by hand. A question that touched three or four tables took about 12 minutes, longer if the tables had moved since the last time anyone looked.

With Rowcast, the same question comes back in about 15 seconds. Analysts stopped writing raw SQL for anything routine. So did the customer success team, who used to wait on an analyst for every number they needed on a call.

Here is the turn. On a Wednesday, Odunayo shipped a new version of Rowcast's prompt, meant to make it better at questions that touch several tables at once. The new prompt worked. It also had a bug nobody had caught yet. For one kind of question, net revenue questions, it started joining against an old table that had already been folded into a newer one. That old table subtracted returns a second time. Every net revenue number Rowcast gave back for that kind of question came out about 9 percent too low.

At its worst, this does not stay inside engineering. At 3:40 that afternoon, a customer success manager pulled "net revenue by plan tier, trailing 30 days" through Rowcast for a renewal call starting in twenty minutes, and read the wrong number straight off the screen to the customer. Nobody flagged it. Rowcast does not know when it is wrong, it only knows when a query fails to run, and this one ran fine.

The nightly eval did catch the bug, eventually. It runs once a day, at 1 in the morning, against 240 hand checked questions, and it usually lands at 96 percent right. By the time it ran that night, the mistake was already 11 hours old, and the customer had already heard the wrong number.

The fast signal is not free. It is a small build, in order, before it protects anyone.

The choice I would take back is not building Rowcast, and it is not the nightly eval either. It is that we never asked how long a bad number gets to live before someone acts on it. We assumed a day was fine because a day was fine for something else, a while ago.

What I would leave alone: Rowcast's "explain this query" feature, the one that just walks an analyst through SQL someone already wrote. A prompt change there can produce a confusing explanation. It cannot produce a wrong number, because it never touches the warehouse. That one can stay on the slower nightly check.

The lesson: a check that runs once a day is not slow because someone was lazy. It is slow because nobody ever put a real number on how long a bad answer gets to live before someone uses it. Go find that number first. Then build the check that beats it.

Now here is the same thing as a story

The short version is above. Read on for the Wednesday this cost a renewal call twenty minutes of trust nobody got back.

Odunayo Larrimore has owned the prompt behind Rowcast for two years. Hand her a slow query and she can usually tell you which join is the problem before she opens the plan. That instinct is most of the job.

For most of those two years, Rowcast earned its trust the plain way, one right answer at a time. Analysts asked it questions all morning. Customer success reps started pulling numbers straight from it on live calls, no analyst in the loop at all, because it was almost always right, and fast. Nine in the morning, a rep would type a question, read the table back to a customer by nine oh two, and move on.

The double checking thinned out the same quiet way it always does. At first, a rep would glance at Rowcast's number and then ping an analyst to confirm anything over a few thousand dollars. A few months in, that shrank to just the really big ones, a new deal, a churn save. By spring, nobody was pinging anyone. Rowcast's number was the number.

On the Wednesday in question, Odunayo shipped a prompt change meant to make Rowcast better at questions with several joins in them, the ones analysts complained took two or three tries to get right. It shipped at two in the afternoon. Nothing about the release looked unusual.

At three forty, a customer success manager pulled "net revenue by plan tier, trailing 30 days" for a renewal call starting in twenty minutes. Rowcast answered in fifteen seconds, the way it always did. The number was 9 percent lower than it should have been, because the new prompt had quietly started joining that question against an old, already retired refunds table. The rep read it straight to the customer. Nobody had a reason to check it.

It was never really about 9 percent. Nobody on that call had a number in their head for how far off Rowcast could be before they would double check it. They had a feeling, and the feeling had two settings: trust it, or check it. Months of right answers had switched everyone to trust it. One bad Wednesday could not switch them back in the moment, because nothing told them it needed to.

One door still opens either way. The other one closed the moment the customer heard the number.

Months earlier, when the team set up Rowcast's launch checklist, the decision was simple. Every prompt change gets reviewed by the nightly eval before the team calls it done. At the time, prompt changes shipped about once a month, so waiting a day for the nightly run to confirm nothing broke felt more than careful enough. Nobody in that meeting pictured a week where three prompt changes would ship, each one trusted by people who no longer double checked anything.

Run the same Wednesday through the fixed design. The golden set catches the drop in the match rate five minutes after the two o'clock deploy, twenty one right answers out of thirty instead of thirty out of thirty. The page goes out at two fifteen. Odunayo rolls the prompt back by two forty five. The three forty call still happens, and the number on the screen is the real one.

One design waits for morning to find out what happened at lunch. The other one catches it before the next call starts.

What I would tell myself, before any of this: we built a fast way to answer a question and a slow way to find out when the answer was wrong, and we never once compared how fast each one actually was.

ORDER, in one screen

This is a prioritization question, so ORDER does the ranking, not FLIPS. Each letter gets a name, a question, and where it shows up in Rowcast's Wednesday.

Two things are worth naming directly, since this is where the real judgment lives. First, the join that broke was not a syntax error, it was a confident, wrong choice of table, the same kind of overconfidence you would call a hallucination anywhere else in the model's output. The guardrail is not "ask the model to be careful." It is the golden set itself, built specifically around the join patterns Rowcast has gotten wrong before, checked every five minutes instead of once a day. Second, we looked at just running the full 240 question eval more often instead of building a smaller live check, and rejected it. It is too expensive to run that often against the warehouse, and it was built for broad coverage, not for catching one bad join fast. The real trade is a small one: a bit of extra warehouse spend and a page someone has to own, running that thirty question set continuously, in exchange for not finding out about a wrong number from a customer instead of from an alert.

And if you want to be sure it really works, try it somewhere else

Same five moves, an insurance claims tool instead of a query tool, and this time the hardest to undo failure is a paid claim that should not have been paid.

Cobblegate Insurance runs ClaimNote, a tool that drafts a claim summary from a recorded phone call, for an adjuster to read before approving or denying a payout. Bellamy Petrosky owns the prompt behind it.

O, outcome. Catching a bad summary before an adjuster acts on it, not before the weekly QA sample gets pulled.
R, reversibility. A summary that leaves out a liability detail and gets used to approve a payout is much harder to undo than one that just reads awkwardly. Once the money is paid, getting it back is its own project.
D, dependency. A small set of past calls with verified correct summaries has to be written and wired to score every new prompt version's draft continuously, the same build cost as Rowcast's golden set.
E, evidence. Cheap to check first: does the draft include every required field, the date of loss, the injury described, any third party named. A field completeness rule catches an obvious drop in days, before anyone builds a hand graded set.
R, rank. Build the field completeness check first, it ships in days. Then the golden set live check. Keep the weekly QA sample as the slow confirmation layer, same role as Rowcast's nightly eval.

Swap the trigger and it still runs.
Speed: an interviewer caps the answer at ninety seconds. Skip straight to the rank: build the small always-on check against a known-answer set first, everything else after.
Cost: engineering says the live golden set check cannot ship for two months. Do not lean on the nightly eval alone in the meantime and call it fine. Ship the cheap real time error check today. It is not a full answer, but it is not nothing.
The model got better, for real: say Rowcast's accuracy genuinely improves next quarter. That does not retire the golden set. A better model just means fewer real alerts, not zero risk of one bad join slipping through unnoticed.

Where people run it wrong.
They treat "we have an eval" as the same claim as "we would catch this fast," without asking how long the gap actually is.
They build the most thorough check first, instead of the fastest one, because thorough feels more responsible.
They wait for a customer to notice, instead of charting the golden set's match rate every few minutes whether anyone remembers to look or not.

How to use it live. Say the outcome before naming a single signal: "the question isn't which check is most thorough, it's which one would have paged someone before a customer heard the wrong number." That buys you room to actually rank something, instead of listing every monitoring idea you know.

Flashcards (click a card to flip it)

Check yourself Score: 0 / 0