LLM Evaluation Interview Questions
How to tell whether an LLM feature is good enough to ship: eval specs, eval design, golden datasets and acceptance criteria for output that varies. These questions come up for AI engineers, data scientists and AI product managers alike. The answers focus on the decisions behind an eval, not on code. 104 questions across 4 subtopics, each with a model answer.
Writing an LLM Eval Spec
How to define what good looks like for an AI feature: test cases, human baselines, how often evals run and how to stop them being gamed by prompt tuning. Asked of AI PMs, AI engineers and data scientists.
- What is an eval spec and who is its audience?
- List the components of a complete eval spec.
- How do you define a task-level success criterion for a summarization feature?
- Write a scoring rubric for the quality of a generated customer support reply.
- Describe the difference between an eval spec and a test plan.
- How many examples belong in a first eval set and how do you choose them?
- Explain how you would build an eval set that includes adversarial cases.
- What is the role of a human baseline in an eval spec?
LLM Eval Design
How to design evaluations a product team can run: what makes an eval relevant to users, how to know a suite is good enough, and how to evaluate safety on a small labelling budget. Asked of AI PMs, AI engineers and data scientists.
- What makes an eval product-relevant rather than research-relevant?
- Design an eval for a feature that drafts email replies.
- How do you decide between automated evals and human review?
- Explain the tradeoffs of LLM-as-judge for a product team.
- How do you validate that your judge model agrees with human raters?
- Describe a rubric that a non-technical reviewer could apply consistently.
- What is inter-rater reliability and why should a PM care?
- How do you design evals that catch regressions rather than just measuring level?
Golden Datasets and Test Sets
What a golden dataset is, how big it should be, who owns it and how it leaks into development. A core topic for AI PMs, data scientists and ML engineers.
- What is a golden dataset and why does the PM usually own it?
- How do you construct a first golden set with no production traffic?
- Describe the composition of a golden set: what proportion should be edge cases?
- How do you keep a golden set representative as your user base changes?
- Explain the risk of a golden set that engineering can see during development.
- What is a holdout set and when would you use one for an AI product?
- How do you handle labelling disagreement inside a golden set?
- Describe a process for adding newly discovered failure cases to the golden set.
Acceptance Criteria for LLM Output
How to decide when an AI feature is ready to ship when its output varies: pass bars, comparison with human performance, and criteria that can be met on paper while the product is still bad.
- Rewrite this criterion to be testable: the model should not hallucinate.
- How do you express an acceptance criterion as a rate rather than an absolute?
- What is the difference between a threshold criterion and a distributional criterion?
- Write acceptance criteria for an AI feature that extracts fields from an invoice.
- How do you set a pass bar when human performance on the same task is 92 percent?
- Describe acceptance criteria that account for the severity of different error types.
- How would you write criteria for a feature where the worst case matters more than the average?
- What acceptance criteria would you set for latency at the 95th percentile and why not the mean?