ConceptFoundationalEval-Driven Specification / Writing an eval spec / #1
What is an eval spec and who is its audience?
The direct answer
An eval spec is the written definition of what counts as a right answer for an AI feature, plus the test cases and the pass bar it has to clear before it ships. It has more than one reader, but write it for the engineers who build against it first: precise enough that two people grading the same answer agree with each other. Every other version of the spec, the ops team's escalation rule, the leadership summary, the legal sign-off, only means anything once that first version is real.
The ranking, by what breaks first if skipped
Write the spec's pass bar precise enough that two engineers grading the same real answer land on the same verdict.Why: every other version of this document is a translation of this one. Get this one wrong and the tool ships against a target nobody agreed on.
Pull the day-to-day escalation rule straight from that same precise bar, for whoever handles a wrong answer in real time.Why: this is the audience closest to the actual cost of a mistake, and their rule needs to trace back to the real criteria, not a separate guess.
Test the draft spec cheaply before building the full test suite: have two people grade the same 10 to 20 answers on their own.Why: catches a still-vague spec while it costs an afternoon, not after weeks of building against it.
Write the one-page summary for leadership only after the precise version exists.Why: a summary of a spec that isn't precise yet is a summary of nothing. It just moves the vagueness up a level and gets signed off anyway.
Get legal or compliance sign-off on which topics the spec covers before a wide rollout.Why: cheap to add once the core spec exists, and it catches gaps, like a sensitive topic, that precise engineering criteria alone can miss.
How to answer this, stage by stage
Seven moves. The trap in this question is treating "who is the audience" as a list you recite: engineers, leadership, legal, done. An interviewer wants to see you rank those readers by whose version, if it's wrong, breaks the document first.
1
Ground it in one real product
Say it like this
"Let me make this real. Say it's Thackeray Pump and Valve, a company that builds industrial pumps, and there's an AI tool called Signpost that reads their internal wiki, safety steps, parts specs, HR policy, and answers employee questions instead of making them search for it. Signpost gets tested against an eval spec before every version ships. I'll answer against that."
Why this works
Grounds "who is the audience" in one real product, so the ranking that follows isn't abstract.
2
Reframe from "list the readers" to "rank whose needs come first"
Say it like this
"I can name everyone who reads this document. But the real question is whose version has to be right before anyone else's version is worth writing. Most of these readers you can write for whenever. One of them, if you get it wrong, means the tool already shipped against the wrong idea of correct."
Why this works
Tells the interviewer a method is coming, not a list you memorized the night before.
3
Name the outcome every version is protecting
Say it like this
"Every version of this document exists to protect one thing: that 'Signpost gave a right answer' means the same thing no matter who's reading that claim. An engineer, a support lead, and a VP signing off all have to be talking about the same bar."
Why this works
Without naming what's being protected, any ranking that follows is a preference dressed up as a method.
4
Find the gap that can't be undone
Say it like this
"If the version for engineers is vague, they don't build something a little rough, they build against a target that doesn't actually exist yet. By the time anyone notices, real employees have already gotten real answers, graded against criteria nobody agreed on. You can't go back and re-grade a workday that already happened."
Why this works
This is the reversibility test, made specific instead of "be precise, it's best practice."
5
Show the dependency underneath the other versions
Say it like this
"A one-page summary for leadership only means something if there's a precise version underneath it to summarize. Write the summary first and you're not simplifying anything. You're describing a bar that doesn't exist yet."
Why this works
Shows the readers aren't a flat list. One version depends on another, which is what makes this a ranking instead of an opinion.
6
Prove it with the failure it prevents
Say it like this
"Here's what happens without the precise version. At Thackeray, the first eval spec just said answers should be 'helpful, accurate, and safe.' An engineer testing Signpost decided 'accurate enough' meant citing any page that mentioned the pump model, even for a lockout and tagout question. Six weeks later, a technician followed Signpost's answer and reached toward a valve that was still live. A coworker caught it before anyone got hurt."
Why this works
A concrete near miss carries more weight than any amount of asserting that precision matters.
7
Close with the ranked list, defended
Say it like this
"So, in order: a precise, testable spec for engineers first, since nothing else means anything without it. The ops escalation rule second, pulled from that same bar. A cheap two-person agreement check third, to catch a still-vague spec early. Then the leadership one-pager, and legal sign-off, both cheap to add once the first three exist."
Why this works
Ends on the literal list the question asked for, with a defended order behind it instead of a memorized sequence.
Let's learn
Say a company builds a tool that reads its own internal wiki and answers employee questions instead of making them search for it.
Thackeray Pump and Valve makes industrial pumps for water treatment plants and factories. Its internal wiki holds everything: safety steps, parts specs for two hundred pump models, HR policy, the works. Before Signpost, an employee spent about 11 minutes on average hunting through the wiki's keyword search for one answer, and plenty just gave up and messaged a coworker instead, which pulled two people off the job instead of one.
Knowledge spark: what an eval spec actually is
It's the answer key plus the rules for using it. A set of real questions, the correct answer to each one written down, and a bar for how close a new answer has to be to count as right. Before a new version of a tool ships, someone runs it against that set and checks the results against the bar.
Signpost's first version answered most questions in under 20 seconds, citing the wiki page it pulled from. To get budget signed off fast, the team had written a one-page eval spec: answers should be "helpful, accurate, and safe." Everyone nodded. Nobody turned that into something an engineer could actually test against.
The leadership one-pager and the ops rule are both just a translation of the first box
Here is the part that matters. The extra wrong answers Signpost gave were never the real problem. The real problem was that nobody had written down, precisely, what "accurate" meant for a safety-tagged question versus a trivia one, so the engineer building it had no fixed target, and neither did anyone meant to catch it.
Signpost didn't get a wrong answer. It got graded by a bar nobody had actually agreed on.
An engineer testing Signpost against a lockout and tagout question read "accurate" as "cites a page that mentions the pump model," which is a much lower bar than "gives the exact required step." Two engineers grading the very same test answer under that one-page spec agreed on pass or fail only 61 percent of the time. They weren't careless. The spec simply didn't tell them what to agree on.
How often two engineers agreed on pass or fail, grading the same real answers
This is the cheap check that would have shown the gap before Signpost ever shipped. Two people, twenty real answers, an afternoon.
The choice I would take back
The team wrote the one-page "helpful, accurate, safe" spec because the VP sponsoring the project wanted proof of progress by a set date, and a page felt faster than a full test suite. That was a fine call for getting the green light. Nobody circled back to write the precise version for engineers once the light turned green. I would take that back: write the precise, testable version first, and let the one-pager for leadership come from it, not instead of it.
What I would leave alone. For a question like "what's the WiFi password" or "when's the cafeteria open," a loose "helpful" bar is genuinely fine. Nobody needs a formal, testable pass bar with an agreement check for that. The cost of Signpost being a little off there is close to zero, so don't gate everything behind heavy process.
The lesson. An eval spec that reads well to leadership and an eval spec that actually tells an engineer what to build are not the same document. Writing only the first one feels like progress. It quietly guarantees you'll build against a target that was never really there.
Now here is the same thing as a story
The short version is above. Read on if you want to feel why the engineer's version has to come first, not just take it on faith.
Radhika Suresh could turn a two-line ask from a VP, "employees want stuff answered without emailing IT for it," into a working prototype in about a month. That was her whole reputation at Thackeray. She'd built three internal tools before Signpost, and all three shipped on time.
The first eight months of Signpost were the kind of good that makes a team stop asking questions. Technicians asked it about pump specs mid-repair instead of walking back to the parts room. HR fielded fewer benefits emails. Every Friday, Radhika pulled twenty of that week's safety-tagged answers and read them herself, just to see if anything looked off. It never did.
So she went from reading twenty to reading ten. Then a handful, on the weeks she remembered. Model updates started shipping every couple of weeks instead of every couple of months, and her Friday read felt more and more like a habit left over from when the team was smaller, not something the tool still needed.
Then came an ordinary Tuesday afternoon on the plant floor.
One of these you can fix whenever you get to it. The other one already shipped answers to real people.
A technician asked Signpost how to isolate a pump before opening its housing. Signpost cited the right-sounding page and answered fast, the way it always did. The technician reached for the valve it pointed to. It was still live. A coworker grabbed his arm before his hand got there. Nobody was hurt. Everybody on that floor thought about it for weeks anyway.
We didn't build a broken tool. We built a tool that nobody had actually told what "right" meant, and then we were surprised when it disagreed with us about a valve.
Radhika pulled the eval spec that afternoon and read it properly for the first time in months. "Helpful, accurate, and safe." Three words doing the job of a hundred test cases. She found the case Signpost had actually been graded against for lockout and tagout questions: the engineer's note just said "cites the correct pump manual section," with no line about matching the exact required step. Nobody had lied. Nobody had cut a corner on purpose. The spec had simply never told anyone where the real line was.
A year earlier, in a planning meeting nobody minuted, someone had asked whether Signpost needed a real test suite before launch or whether the one-pager was enough to start. With a VP waiting on a date and a small team, "enough to start" won, reasonably. Nobody put a date on the calendar to go back and finish it.
I would take that back. With a precise version written for engineers first, safety-tagged questions get a zero-tolerance bar: the exact required step, not just a page that mentions the topic. Two engineers grading the same twenty answers agree 96 percent of the time instead of 61. The leadership one-pager gets written from that real bar, not instead of it. The technician's question either gets the exact step or gets handed straight to a person, and nobody's hand goes anywhere near a live valve on the strength of a citation.
What I'd tell myself, back in that meeting: skipping the precise version felt like saving three weeks. It cost the team three weeks anyway, spent freezing Signpost on anything safety-tagged and re-grading everything it had already answered, and it nearly cost someone a hand.
Who gets served first, and why that's not a guess
LEAD would fit if the question were about which metric to watch. Nothing here is a metric question, it's about ranking readers of a document before one of them gets skipped, which is ORDER's job.
O, outcome. Every version of this spec, the engineer's, the ops team's, leadership's, is competing to protect one thing: that "Signpost gave a right answer" means the same thing no matter who is reading that claim.
R, reversibility. A late leadership summary is fixable any time, you just write it. A missing legal sign-off is fixable before a wide rollout. A vague spec that engineers already built and shipped against is different: real employees have already gotten real answers graded against criteria nobody agreed on, and there's no going back to re-grade a day that already happened.
D, dependency. An ops escalation rule and a leadership one-pager are both, in the end, someone's attempt to describe what the precise spec says, in a form their own reader can use. Write either one first and you're not simplifying the precise version. You're guessing at a version that doesn't exist yet.
E, evidence. Cheap to check: hand two engineers the same ten to twenty real answers and the draft criteria, and see how often they land on the same verdict without talking to each other. At Thackeray, the first draft produced agreement only 61 percent of the time.
R, rank. Precise spec for engineers first, since nothing else means anything without it. The ops escalation rule second, pulled from that same bar, since it protects someone in real time. A cheap agreement check third, to catch a still-vague spec before weeks get spent building against it. The leadership one-pager fourth. Legal or compliance sign-off fifth, cheap once the rest exists.
The check that keeps this ranking honest
Swap the outcome and the order should move. If a wrong answer from Signpost only ever cost someone a re-read and a shrug, the precise spec could sit lower on this list. It doesn't rank first because precision is rigorous. It ranks first because a technician's hand was inches from a valve that a citation said was safe to touch.
Rank it again, where the wrong answer costs a deadline, not a hand
Dunmore County Library System built Deskline, an AI assistant that answers patron reference questions from the library's own resource guides: local housing programs, small business permit steps, and which weeks a free legal aid clinic is actually staffed. Deskline gets tested against an eval spec before any update ships.
O. Every version of this spec protects one thing: that a patron who asks Deskline a question gets the same answer a trained reference librarian would give, not a plausible-sounding one shaped by whichever draft spec was easiest to write that week.
R. A late library-board summary is recoverable, write it next meeting. A spec that's vague about time-sensitive information, like which week a clinic is staffed, is not: a patron who's told the wrong week and misses the one clinic date they could get to has no way to get that week back.
D. A plain "when in doubt, hand it to a person" rule for the front desk only works once there's a precise spec defining "in doubt," specifically, any date-sensitive or eligibility question. Without that, the rule has nothing real to check itself against.
E. Cheap to check: two reference librarians independently grade the same 15 real patron transcripts against the draft criteria. Dunmore's first draft produced agreement of just 54 percent, mostly on date-sensitive questions.
R. Same order: precise spec for the engineers first, so "in doubt" has a real definition. The front-desk escalation rule second. A cheap agreement check third. The board's one-pager fourth. A privacy-office review of patron-data handling fifth, since it's cheap once the first three exist.
Deskline answers later found wrong, caught only after a patron already acted on them, by week
This is what a precise spec buys back: not a perfect tool, just fewer answers a patron acts on before anyone catches the mistake.
Swap the trigger and it still runs
Thackeray starts shipping Signpost updates weekly instead of quarterly. The order doesn't move. The precise spec matters more, not less, since faster shipping leaves less time to catch a vague criterion before it's live.
The AI team shrinks to two people and can't afford a dedicated escalation lead. Doesn't reorder anything. The escalation rule still needs the same precise criteria underneath it, it's just written by whoever's left.
The underlying model gets noticeably more accurate on its own. Doesn't reorder either. A more accurate model still needs a precise bar to be judged against, or nobody can tell "better" from "wrong in a different way."
Where people run it wrong
Writing the leadership summary first because it's the version someone's actually asking for by Friday.
Treating "the tool passed our eval" as proof it's safe, when the eval's own pass bar was never precise enough to trust.
Using one flat bar for the whole tool, instead of a strict, zero-tolerance bar for safety or eligibility questions and a looser one for trivia.
If you're asked this cold
Say the outcome out loud before naming a single audience. "Every version of this document exists to protect one thing: that a right answer means the same thing to everyone reading that claim." Then rank from there. Naming the outcome first is what turns a list into an argument.
Flashcards (click a card to flip it)
1 · THE FRAMEWORK
Which framework fits "what is an eval spec and who is its audience," and why not LEAD?
Tap to flip
ANSWER
ORDER, for ranking which reader's needs the document has to serve first. LEAD is for finding the metric that moves before the outcome does. This question is about ranking readers, not naming a signal to watch.
2 · THE PERSON
Who is this answer about?
Tap to flip
ANSWER
Radhika Suresh, the product manager who built Signpost, the internal-wiki answer tool at Thackeray Pump and Valve.
3 · THE HABIT
What habit let a too-vague eval spec go unnoticed for months?
Tap to flip
ANSWER
Radhika used to personally read twenty safety-tagged answers every Friday. As updates shipped more often, that read shrank to ten, then a handful, then nothing formal replaced it.
4 · THE DEPENDENCY
Which version of the eval spec has to exist before a leadership summary actually means anything?
Tap to flip
ANSWER
The precise, testable version written for engineers. A summary of a spec that isn't precise yet just describes a bar that doesn't exist, and gets signed off anyway.
5 · THE OLD DECISION
What decision would you take back, and why did it make sense at the time?
Tap to flip
ANSWER
Writing a one-page "helpful, accurate, safe" spec to get the VP's sign-off fast, and never circling back to write the precise, testable version for engineers once the launch date was hit.
6 · THE NUMBER
Under the vague spec, two engineers grading the same real answers agreed on pass or fail only ___ percent of the time. After the rewrite, agreement rose to ___ percent.
Tap to flip
ANSWER
61 percent, then 96 percent. That gap is the cheap, checkable proof that the spec, not the engineers, was the problem.
7 · THE REPLAY
Same lockout and tagout question, spec rewritten for engineers first this time. What changes?
Tap to flip
ANSWER
Safety-tagged questions get a zero-tolerance bar: the exact required step, not just a page that mentions the topic. Signpost either gives the exact step or hands the question to a person, and nobody's hand goes near a live valve on the strength of a citation.
8 · THE TRANSFER
Section 4 runs ORDER again on a different product, in a different setting. Which one, and what plays the role of the precise engineering spec there?
Tap to flip
ANSWER
Dunmore County Library System's patron-question tool, Deskline. The equivalent is a precise definition of "in doubt," specifically flagging any date-sensitive or eligibility question, so the front desk's escalation rule has something real to check itself against.
Check yourself Score: 0 / 0
True or false
1. True or false: since Signpost's overall accuracy score still looked fine after the near miss, the eval spec didn't really need a precise, testable version for engineers. Say why.
True
False
Show hint
Ask whether one overall accuracy number can tell the difference between a good answer and a dangerous one graded on too loose a bar.
Show answer
False. A single overall score can't isolate a small, high-stakes category like safety questions. It stayed healthy precisely because the criteria that should have flagged the gap were never precise enough to catch it.
Multiple choice
2. Which two readers does this answer rank above every other reader of the eval spec, and in what order?
A. Legal or compliance, then leadership
B. Engineers (the precise spec), then the ops team's escalation rule
C. Leadership, then engineers
D. The ops team, then legal
Show hint
One of these has to exist before the other one has anything real to work from, per the dependency step.
Show answer
B. Engineers come first because nothing downstream means anything without their precise bar. The ops escalation rule comes second because it's the version that protects someone in real time.
Fill in the blank
3. Before Signpost, an employee spent about ______ minutes hunting for an answer by hand. After Signpost's first version shipped, most questions were answered in about ______ seconds.
Show hint
It's the number that makes the whole tool worth building, and worth writing a real eval spec for.
Show answer
11 minutes and 20 seconds. That's the value an eval spec exists to protect. A vague spec doesn't threaten this number directly, it threatens whether the fast answer is also the right one.
Multiple choice
4. What does the dependency step (D) argue in this answer's ORDER?
A. A leadership summary can't mean anything until a precise engineering spec exists for it to summarize
B. Legal sign-off should happen before any test cases get written
C. Every audience should get the exact same one-page document
D. The ops escalation rule and the leadership summary are the same document
Show hint
Ask what a summary actually needs to exist in order to summarize anything real.
Show answer
A. A summary written before the precise version exists isn't simplifying anything real, it's guessing at a bar that hasn't been defined yet.
Short answer, apply it yourself
5. Pick an AI feature you use or are building that gets judged against some kind of spec or rubric. Whose needs does that spec actually serve first right now, and whose should it?
Show hint
Look for whichever reader's version, if it's wrong, means real people already got a real answer graded against nothing solid.
Show answer
Model answer: "A customer-support chatbot's eval spec, written mostly as a demo script for a sales team. It should serve the support engineers first, with a precise bar for when the bot has to hand off to a human, since that's the decision a customer's actual problem depends on, not how good the bot looks in a five-minute walkthrough."
Short answer, the number question
6. If Thackeray's wiki covered only casual questions, like the WiFi password or cafeteria hours, and nothing safety-tagged, would a precise spec for engineers still need to rank first? Say what changes and what doesn't.
Show hint
Reversibility is about whether a wrong answer can be undone, not about how many topics the wiki happens to cover.
Show answer
Model answer: "No, not at that level of stakes. If nothing on the wiki can hurt someone or cost them a real decision, the outcome itself is lower stakes, so a looser spec might genuinely be fine. What doesn't change: the moment anything with a real consequence enters the wiki, like a benefits deadline or a safety step, the precise-spec-first ranking snaps right back, because the dependency between the versions never went away."
From U2xAI Academy
From answering questions to owning outcomes.
A live workshop where you ship a working AI agent, defend a launch decision, and walk away with a portfolio recruiters can't wave off, not just more questions to study.
