U2xAI
…
Eval-Driven Specification

Acceptance criteria for non-deterministic output Interview Questions

26 questions. All 26 carry a written answer.

  1. #1Rewrite this criterion to be testable: the model should not hallucinate.Artifact critiqueFoundational
  2. #2How do you express an acceptance criterion as a rate rather than an absolute?ConceptFoundational
  3. #3What is the difference between a threshold criterion and a distributional criterion?ConceptAdvanced
  4. #4Write acceptance criteria for an AI feature that extracts fields from an invoice.Artifact critiqueIntermediate
  5. #5How do you set a pass bar when human performance on the same task is 92 percent?CalculationAdvanced
  6. #6Describe acceptance criteria that account for the severity of different error types.ConceptAdvanced
  7. #7How would you write criteria for a feature where the worst case matters more than the average?CaseAdvanced
  8. #8What acceptance criteria would you set for latency at the 95th percentile and why not the mean?CalculationIntermediate
  9. #9Explain how to write acceptance criteria that a QA team can actually execute.CaseIntermediate
  10. #10Write criteria covering refusal behaviour for an assistant.Artifact critiqueAdvanced
  11. #11How do you set acceptance criteria for a feature launching to an unknown distribution of users?CaseAdvanced
  12. #12Describe criteria for consistency: should identical inputs produce identical outputs?ConceptAdvanced
  13. #13What does done mean for a feature whose quality will keep changing?ConceptIntermediate
  14. #14How do you write acceptance criteria for cost per interaction?CalculationIntermediate
  15. #15Explain the role of a canary cohort in validating acceptance criteria.ConceptAdvanced
  16. #16Write criteria for a feature that must degrade gracefully when confidence is low.Artifact critiqueAdvanced
  17. #17How do you avoid acceptance criteria that are technically met but produce a bad product?ConceptAdvanced
  18. #18What criteria would you set for an agent's tool-calling accuracy?ConceptAdvanced
  19. #19Describe how to phrase criteria when the model must never do something.ConceptAdvanced
  20. #20How do you handle acceptance when different customer segments need different bars?CaseAdvanced
  21. #21What is the relationship between acceptance criteria and your eval set?ConceptIntermediate
  22. #22Critique criteria that specify a benchmark score without specifying the benchmark's relevance.Artifact critiqueIntermediate
  23. #23How do you write criteria that hold across a model upgrade?ConceptAdvanced
  24. #24Explain when you would accept a lower quality bar in exchange for lower latency.CaseAdvanced
  25. #25Write the acceptance criteria for a feature I describe, live.InterviewAdvanced
  26. #26Who signs off on acceptance for an AI feature, and why is that question harder than usual?InterviewIntermediate