AI PM interview questions
LLM Evals and AI PRDs Interview Questions
How to specify an AI feature so it can be tested: the AI PRD, the eval spec, acceptance criteria for output that varies, and the golden dataset behind them.
- 105 questions
- 4 subtopics
- Model answer for every question
LLM Evals and AI PRDs Subtopics
Writing a PRD for an AI Feature
What an AI PRD needs that a normal PRD does not: quality bars, failure behaviour, eval plans and whether to name a model. Many interviews ask you to write part of one live, so practise the sections here.
26 questions
Writing an LLM Eval Spec
How to define what good looks like for an AI feature: test cases, human baselines, how often evals run and how to stop them being gamed by prompt tuning. Asked of AI PMs, AI engineers and data scientists.
27 questions
Acceptance Criteria for LLM Output
How to decide when an AI feature is ready to ship when its output varies: pass bars, comparison with human performance, and criteria that can be met on paper while the product is still bad.
26 questions
Golden Datasets and Test Sets
What a golden dataset is, how big it should be, who owns it and how it leaks into development. A core topic for AI PMs, data scientists and ML engineers.
26 questions
