U2xAI
…

LLM Evals and AI PRDs

Acceptance Criteria for LLM Output Interview Questions

How to decide when an AI feature is ready to ship when its output varies: pass bars, comparison with human performance, and criteria that can be met on paper while the product is still bad.

  • 26 questions with model answers
  • 2 foundational · 8 intermediate · 16 advanced

Foundational Acceptance Criteria for LLM Output Questions

  1. Rewrite this criterion to be testable: the model should not hallucinate.Artifact critique
  2. How do you express an acceptance criterion as a rate rather than an absolute?Concept

Intermediate Acceptance Criteria for LLM Output Questions

  1. Write acceptance criteria for an AI feature that extracts fields from an invoice.Artifact critique
  2. What acceptance criteria would you set for latency at the 95th percentile and why not the mean?Calculation
  3. Explain how to write acceptance criteria that a QA team can actually execute.Case
  4. What does done mean for a feature whose quality will keep changing?Concept
  5. How do you write acceptance criteria for cost per interaction?Calculation
  6. What is the relationship between acceptance criteria and your eval set?Concept
  7. Critique criteria that specify a benchmark score without specifying the benchmark's relevance.Artifact critique
  8. Who signs off on acceptance for an AI feature, and why is that question harder than usual?Interview

Advanced Acceptance Criteria for LLM Output Questions

  1. What is the difference between a threshold criterion and a distributional criterion?Concept
  2. How do you set a pass bar when human performance on the same task is 92 percent?Calculation
  3. Describe acceptance criteria that account for the severity of different error types.Concept
  4. How would you write criteria for a feature where the worst case matters more than the average?Case
  5. Write criteria covering refusal behaviour for an assistant.Artifact critique
  6. How do you set acceptance criteria for a feature launching to an unknown distribution of users?Case
  7. Describe criteria for consistency: should identical inputs produce identical outputs?Concept
  8. Explain the role of a canary cohort in validating acceptance criteria.Concept
  9. Write criteria for a feature that must degrade gracefully when confidence is low.Artifact critique
  10. How do you avoid acceptance criteria that are technically met but produce a bad product?Concept
  11. What criteria would you set for an agent's tool-calling accuracy?Concept
  12. Describe how to phrase criteria when the model must never do something.Concept
  13. How do you handle acceptance when different customer segments need different bars?Case
  14. How do you write criteria that hold across a model upgrade?Concept
  15. Explain when you would accept a lower quality bar in exchange for lower latency.Case
  16. Write the acceptance criteria for a feature I describe, live.Interview

Also in These Skill Collections