U2xAI
…
Eval-Driven Specification

Writing an eval spec Interview Questions

27 questions. All 27 carry a written answer.

  1. #1What is an eval spec and who is its audience?ConceptFoundational
  2. #2List the components of a complete eval spec.ConceptFoundational
  3. #3How do you define a task-level success criterion for a summarization feature?CaseIntermediate
  4. #4Write a scoring rubric for the quality of a generated customer support reply.Artifact critiqueAdvanced
  5. #5Describe the difference between an eval spec and a test plan.ConceptIntermediate
  6. #6How many examples belong in a first eval set and how do you choose them?ConceptIntermediate
  7. #7Explain how you would build an eval set that includes adversarial cases.CaseAdvanced
  8. #8What is the role of a human baseline in an eval spec?ConceptAdvanced
  9. #9Describe how you would specify an LLM-as-judge eval and its known weaknesses.ConceptAdvanced
  10. #10How do you keep an eval set from being gamed by prompt tuning?ConceptAdvanced
  11. #11Write the pass criteria for a feature where partial credit is meaningful.Artifact critiqueAdvanced
  12. #12What should the eval spec say about who breaks a tie when scores are ambiguous?ConceptIntermediate
  13. #13How do you specify evals for a multi-step agent rather than a single response?ConceptAdvanced
  14. #14Describe the sampling strategy for building an eval set from production traffic.CaseAdvanced
  15. #15What is the difference between an offline eval and an online eval in the spec?ConceptIntermediate
  16. #16How do you handle evals for subjective qualities like tone or brand voice?CaseAdvanced
  17. #17Explain how you would version and archive eval sets over time.ConceptAdvanced
  18. #18Critique an eval spec that measures only accuracy on a single benchmark.Artifact critiqueIntermediate
  19. #19What does the eval spec say about statistical significance of a score difference?CalculationAdvanced
  20. #20How do you write an eval for a feature where there is no single right answer?CaseAdvanced
  21. #21Describe the cadence at which evals should be re-run and by whom.ConceptIntermediate
  22. #22What is the minimum eval you would accept before a first internal release?ConceptIntermediate
  23. #23How do you specify evals for safety and policy compliance separately from quality?ConceptAdvanced
  24. #24Explain how an eval spec should handle cost and latency alongside quality.ConceptAdvanced
  25. #25Write the section that defines what constitutes a regression.Artifact critiqueAdvanced
  26. #26Design an eval spec live for a feature I describe.InterviewAdvanced
  27. #27Why should a PM own the eval spec rather than delegating it entirely to engineering?InterviewIntermediate