U2xAI
…

LLM Evals and AI PRDs

Writing an LLM Eval Spec Interview Questions

How to define what good looks like for an AI feature: test cases, human baselines, how often evals run and how to stop them being gamed by prompt tuning. Asked of AI PMs, AI engineers and data scientists.

  • 27 questions with model answers
  • 2 foundational · 9 intermediate · 16 advanced

Foundational Writing an LLM Eval Spec Questions

  1. What is an eval spec and who is its audience?Concept
  2. List the components of a complete eval spec.Concept

Intermediate Writing an LLM Eval Spec Questions

  1. How do you define a task-level success criterion for a summarization feature?Case
  2. Describe the difference between an eval spec and a test plan.Concept
  3. How many examples belong in a first eval set and how do you choose them?Concept
  4. What should the eval spec say about who breaks a tie when scores are ambiguous?Concept
  5. What is the difference between an offline eval and an online eval in the spec?Concept
  6. Critique an eval spec that measures only accuracy on a single benchmark.Artifact critique
  7. Describe the cadence at which evals should be re-run and by whom.Concept
  8. What is the minimum eval you would accept before a first internal release?Concept
  9. Why should a PM own the eval spec rather than delegating it entirely to engineering?Interview

Advanced Writing an LLM Eval Spec Questions

  1. Write a scoring rubric for the quality of a generated customer support reply.Artifact critique
  2. Explain how you would build an eval set that includes adversarial cases.Case
  3. What is the role of a human baseline in an eval spec?Concept
  4. Describe how you would specify an LLM-as-judge eval and its known weaknesses.Concept
  5. How do you keep an eval set from being gamed by prompt tuning?Concept
  6. Write the pass criteria for a feature where partial credit is meaningful.Artifact critique
  7. How do you specify evals for a multi-step agent rather than a single response?Concept
  8. Describe the sampling strategy for building an eval set from production traffic.Case
  9. How do you handle evals for subjective qualities like tone or brand voice?Case
  10. Explain how you would version and archive eval sets over time.Concept
  11. What does the eval spec say about statistical significance of a score difference?Calculation
  12. How do you write an eval for a feature where there is no single right answer?Case
  13. How do you specify evals for safety and policy compliance separately from quality?Concept
  14. Explain how an eval spec should handle cost and latency alongside quality.Concept
  15. Write the section that defines what constitutes a regression.Artifact critique
  16. Design an eval spec live for a feature I describe.Interview

Also in These Skill Collections