Interview prep
Eval-Driven Specification Interview Questions
4 subtopics, 105 questions. All 105 carry a written answer.
Writing a PRD for an AI feature
- What sections does an AI PRD need that a standard PRD does not?
- Write the problem statement section for an AI meeting-summary feature.
- How do you specify expected behaviour when the output is generated text?
26 questions · 26 answered
Writing an eval spec
- What is an eval spec and who is its audience?
- List the components of a complete eval spec.
- How do you define a task-level success criterion for a summarization feature?
27 questions · 27 answered
Acceptance criteria for non-deterministic output
- Rewrite this criterion to be testable: the model should not hallucinate.
- How do you express an acceptance criterion as a rate rather than an absolute?
- What is the difference between a threshold criterion and a distributional criterion?
26 questions · 26 answered
Golden datasets and test set ownership
- What is a golden dataset and why does the PM usually own it?
- How do you construct a first golden set with no production traffic?
- Describe the composition of a golden set: what proportion should be edge cases?
26 questions · 26 answered