AI Evals 101: How to Prove Your AI Feature Actually Works

"How do you know it works?" This is one of the simplest questions an interviewer can ask about an AI feature, and one of the most consistently poorly answered. Most candidates respond with something like "I tested it and it seemed pretty accurate," which sounds reasonable until an interviewer asks the obvious follow up: tested how, accurate compared to what, and how often.

Evals, short for evaluations, is the discipline of answering that question with actual evidence rather than an impression. It means defining what "correct" actually means for a specific task, building a real set of test cases with known right answers, and measuring a specific score, rather than relying on a general sense that a feature seems to work well most of the time.

This is, as of 2026, one of the least common skills among AI job candidates, precisely because it requires more discipline than simply using a tool and forming an opinion about it. This guide builds that discipline from scratch, with no math beyond counting and dividing.

Why "it seems to work" stopped being an acceptable answer

As AI features have moved from experimental add ons to core parts of real procurement and operations workflows, the cost of a feature that "seems to work" but actually fails on a meaningful share of real cases has grown substantially. A spend categorization tool that is right 95 percent of the time sounds impressive until you realize the 5 percent it gets wrong might be concentrated in exactly the highest value transactions, the ones a leadership report actually depends on getting right.

Employers building or evaluating AI features have learned, often the hard way, that a general impression of quality is not a substitute for a real measurement. This has turned evals thinking into a specific, testable interview topic, particularly for product manager and analyst roles that involve deciding whether an AI feature is ready to ship, or whether one AI approach is genuinely better than another rather than just differently flawed.

The employers asking about this are not expecting a candidate to have built a formal, automated evaluation pipeline. They are checking whether a candidate's instinct, when facing an AI feature, is to ask "how would I actually measure whether this works" rather than to form an impression and stop there.

This instinct matters beyond the interview itself, in a very concrete way. Someone who defaults to measuring rather than impressioning is the person a team turns to when two AI approaches genuinely need to be compared, or when leadership asks whether a feature is actually ready to expand to a new use case. That is a real position of influence inside a team, and it is earned by demonstrated rigor, not by confidence alone.

The three parts of a real evaluation

Every real evaluation, no matter how simple or sophisticated, is built from the same three parts. Understanding these three parts is the entire foundation of evals thinking.

Success criteria. A precise definition of what counts as a correct result for this specific task, written down before you run any tests, not decided case by case as you look at results. For a spend categorization tool, success criteria might be "the category assigned matches the category a human analyst would assign, for at least 90 percent of transactions, with 100 percent accuracy required on transactions over $10,000."

A test set. A real collection of example inputs with known correct answers, ideally pulled from actual past cases rather than invented from scratch, since real data contains real messiness that invented examples often miss. A good test set deliberately includes some easy, obvious cases and some genuinely hard or ambiguous ones, since a feature that only gets tested on easy cases will look better than it actually is.

A score. The actual, calculated result of running your feature against your test set and checking each result against your success criteria: 47 out of 50 correct, 94 percent. Not a feeling. A number you can state, defend, and compare against a previous version.

Building a real test set, step by step

A test set does not need to be large to be useful. Twenty to fifty real, well chosen cases catch a meaningful share of problems, and a small, real test set beats a large, artificial one every time.

1. Pull twenty to fifty real past examples relevant to the task, actual vendor invoices, actual contract clauses, actual spend transactions, from data you have permission to use.
2. Deliberately include a mix: mostly ordinary, representative cases, plus a smaller number of genuinely tricky ones, an unusual vendor name, an ambiguous category, a formatting quirk that has caused problems before.
3. Write down the correct answer for each case yourself, or confirm it with someone who would know, before you run anything through your AI feature. Deciding the correct answer after seeing the AI's output is the single most common way evals go wrong, since it is very easy to unconsciously rationalize a plausible sounding wrong answer as acceptable.

A worked example: evaluating a spend categorization feature

Here is a complete, simple evaluation, built by hand, for a real feature: an AI tool that assigns a spend category to each line in a vendor invoice file.

Success criteria: the assigned category matches the category a trained analyst would assign, for at least 90 percent of all transactions, with no tolerance for miscategorizing anything over $10,000, since large transactions carry outsized reporting impact.

Test set: 40 real transactions from a past quarter, 30 straightforward ones and 10 deliberately picked because they had caused confusion before, ambiguous vendor names, transactions that plausibly fit two categories, and one transaction over $10,000.

Running the test and scoring it: the tool correctly categorized 36 of the 40 transactions, a 90 percent overall score. Critically, it also correctly categorized the single transaction over $10,000, meeting the stricter requirement on high value transactions specifically. Of the 4 misses, all fell within the deliberately ambiguous group, none were in the straightforward group, which is itself a useful, specific finding: the tool is reliable on ordinary cases and weaker specifically on genuinely ambiguous ones, a much more actionable insight than a single overall percentage alone would have given.

Measuring hallucination rate specifically

Hallucination, covered in the earlier guide on how LLMs work, deserves its own specific measurement, separate from general accuracy, because a feature can score well on easy cases while still hallucinating on the specific cases where it matters most.

The method is a direct extension of the same eval structure: ask the feature a set of questions where you already know the correct, grounded answer, then count how many responses were invented rather than genuinely grounded in the source material you provided. A tool asked to extract payment terms from ten real contracts, where you already know the correct terms from reading them yourself, either correctly reports each term, correctly reports "not found" when a term genuinely is not present, or hallucinates a plausible sounding term that is not actually in the document. Counting that third category directly gives you a real hallucination rate for that specific task, not a general, unmeasured impression.

Practice these interview questions

Evals questions separate candidates who've actually shipped something from candidates who've only talked about AI in the abstract. The questions below test whether you have a real, repeatable way to prove a feature works, not just a claim that it does. Work through your own answer first, then compare with the sample.