How to Become an AI Evaluation Engineer
An AI evaluation engineer owns the defensible quality number on an AI product. This four-week hands-on course takes QA engineers, analysts and junior devs into that role with the full eval stack, real numbers and four portfolio artifacts you can show in an interview.
30 chapters, 100 lessons
14-day refund on the yearly plan. Real pricing on the plans page, no surprises.
100
lessons, yours to run
Easy to pick up, built to get you the next job
No coding background needed, and you write code with Claude as you go.
Every section below turns into something a hiring manager recognises, not just notes.
Understand the job no classic QA role covers
- You will explain in plain language why assert output == expected breaks the moment software stops being deterministic, and what replaces the answer key.
- You will map the four roles (QA, AI Engineer, Data Scientist, Eval Engineer) to the single question each owns, and route any ambiguous task to the right seat without hesitation.
- No prior coding background needed: every concept lands on a concrete Trellis CRM scenario before you write a single line, so the logic is clear before the syntax is.
Build a golden dataset and failure taxonomy a hiring manager can read
- You will hand-build a golden dataset from the Trellis CRM email-writer case study, with real pass and fail examples a rubric can score.
- You will write a failure taxonomy that names the exact ways an AI email can go wrong, turning vague 'it felt off' feedback into categories an engineer can act on.
- These are portfolio artifacts with real numbers, not slides, so interviewers can see your judgment in writing.
Automate scoring and measure the measurer
- You will wire rule-based checks and a calibrated LLM-as-judge into a single scoring stack that produces a defensible pass rate, not a gut feeling.
- You will measure your own judge's accuracy against human review so you can tell a sceptical VP exactly how much to trust the automated score.
- These are skills listed by name on AI Quality Engineer job descriptions, and you will have working code, not just knowledge of the concept.
Gate releases and monitor production drift
- You will wire your eval suite into CI as a release gate so a regression in email quality fails the build before it ships.
- You will set up production drift monitors that surface quality degradation after a model or prompt change, the capability that separates a junior eval role from a senior one.
- Every step is explained in plain language first: if you can read a checklist, you can follow the build, no matter where you are starting from.
Finish this course and you can do all of this, no prior background required:
- You will be able to build a hand-curated golden dataset and failure taxonomy from scratch, the foundation artifact every AI quality role asks candidates to demonstrate.
- You will be able to design and run a calibrated LLM-as-judge scoring pipeline and report its accuracy against human review, a capability listed by name on AI Eval Engineer job descriptions.
- You will be able to apply the right statistical tests to decide whether a score change between two model versions is meaningful, so you never ship a regression disguised as noise.
- You will be able to wire an eval suite into a CI pipeline as a release gate, the skill that moves you from 'ran some evals' to 'owns the quality bar at release'.
- You will be able to monitor a live AI feature for quality drift and surface degradation before users report it, the production-side capability that separates mid-level from senior eval roles.
- You will be able to walk an interviewer through four portfolio artifacts with real numbers, answering the hardest eval screen questions with evidence rather than theory.
100 lessons, 30 chapters
Built from real implementation work
Every agent, skill file, and dataset is drawn from 25+ years and 50+ Oracle ERP implementations across pharma, manufacturing, semiconductor, and distribution.
Kept current with the stack
Claude Code, skills, and MCP move fast. Your licence includes every system update for as long as you're subscribed, so what you build keeps working as the tools change.
You know software QA inside out, but an AI Eval Engineer job spec asks for someone who can grade a language model's output as well as confirm a button renders. Your current test suite can't tell you whether an AI-generated email is any good. You can't yet point to a golden dataset you built, a rubric you calibrated, or a release gate you wired into CI. The full job means building the taxonomy, running the stats, setting the gate and defending the number to a VP. That is a specific skill set you can learn in four weeks, and this course teaches it step by step.
What you'll be able to do, module by module
Every module leaves you able to build and run something real. Here is the syllabus, in the order you work through it.
Curriculum
30 chapters · 100 lessons
Orientation (Pre-Week 1)
You can explain in one breath why non-determinism kills classic QA, map the four roles to the single question each owns, and describe what an Eval Engineer actually does hour by hour, so you arrive at Week 1 with the mental model already in place.
0.1 What Is an Eval Engineer, Really?4 items
- 0.1.1: Why the role emerged: what broke about QA when software stopped giving the same answer twice🔒
- 0.1.2: Eval Engineer vs. QA vs. Data Scientist vs. AI Engineer, where the lines actually are🔒
- 0.1.3: A real day in the life, hour by hour🔒
- 0.1.4: Why 'I test software' and 'I've used ChatGPT' are each half the skill🔒
0.2 Meet the Case Study: Trellis CRM's AI Has a Vibes Problem4 items
- 0.2.1: The VP's stated ask and the support queue behind it🔒
- 0.2.2: The email-drafting feature and eight months of shipping blind🔒
- 0.2.3: The stakeholders you'll deal with🔒
- 0.2.4: What 'better' could even mean, and why no two stakeholders agree🔒
0.3 The 4-Week Roadmap3 items
- 0.3.1: Skill map: what is added each week and why in that order🔒
- 0.3.2: The 4 portfolio artifacts you'll walk away with🔒
- 0.3.3: How much time to budget per week🔒
0.4 Setting Up Your Toolkit4 items
- 0.4.1: Accounts and access (model API key, git, Python, an eval harness, a spreadsheet)🔒
- 0.4.2: Creating THE repository and loading the Trellis pack🔒
- 0.4.3: A first variance test: the same prompt run 10 times, differences logged🔒
- 0.4.4: Troubleshooting common setup failures🔒
Week 1: Defining Quality, the Judgment Half of the Job
You can write a defensible failure taxonomy for an AI feature and build a hand-curated golden dataset with clear pass and fail examples, the two artefacts that sit at the foundation of every eval system.
1.1 The Anatomy of 'The AI Is Bad'3 items
- 1.1.1: Reading Trellis's complaint sample: sorting 40 raw complaints into what went wrong🔒
- 1.1.2: Building a failure taxonomy: named, defined, mutually exclusive failure modes🔒
- 1.1.3: A real dissected example: a shipped product's failure taxonomy and how it shaped the roadmap🔒
1.2 What to Measure: From Taxonomy to Quality Criteria4 items
- 1.2.1: Translating each failure mode into a checkable criterion (and which resist checking)🔒
- 1.2.2: The criteria hierarchy: blockers, quality bars, and nice-to-haves🔒
- 1.2.3: Practice pass: writing the Trellis quality spec, one page, every criterion testable🔒
- 1.2.4: Common trap: measuring what's easy instead of what users are angry about🔒
1.3 Golden Datasets: The Core Craft4 items
- 1.3.1: Why hand-built beats generated at the start🔒
- 1.3.2: Building the first 25 Trellis cases by hand: inputs, expected properties, difficulty tags, rationales🔒
- 1.3.3: Coverage thinking: easy, hard, and adversarial cases, and the ratio that catches regressions🔒
- 1.3.4: Dataset hygiene: versioning, provenance, and why a golden set without rationales rots🔒
1.4 Your First Scoring Pass: Human Baseline3 items
- 1.4.1: Scoring 25 cases by hand against the quality spec, and timing how long it takes🔒
- 1.4.2: The disagreement exercise: two humans score the same 10 cases🔒
- 1.4.3: Turning disagreements into rubric fixes: tightening criteria until two humans converge🔒
1.5 Case Study Checkpoint: Trellis Gets Its First Real Baseline3 items
- 1.5.1: Running the current production prompt against the full golden set, scored by hand🔒
- 1.5.2: The baseline report: failure rates per taxonomy category, three worst offenders named🔒
- 1.5.3: Presenting the number to the VP: 'better' now has a definition and a starting point🔒
1.6 Week 1 Portfolio Artifact: A Quality Spec + Golden Dataset3 items
- 1.6.1: Packaging the taxonomy, quality spec, and 25-case dataset with rationales🔒
- 1.6.2: The methodology note: how cases were sourced, tagged, and validated🔒
- 1.6.3: Checklist: what makes this artifact interview-ready🔒
Week 2: Automating the Scoring, and Measuring the Measurer
You can build and run a full scoring stack that combines rule-based checks with a calibrated LLM-as-judge, and you can measure your judge's own accuracy against human review so the number you report is one you can defend.
2.1 Rule-Based Scorers: Cheap, Fast, and Underrated4 items
- 2.1.1: The failures code can catch: placeholders, length, banned phrases, missing fields, format🔒
- 2.1.2: Implementing the Trellis rule pack and running it across all 200 pairs in seconds🔒
- 2.1.3: What the rules caught that eight months of eyeballing missed🔒
- 2.1.4: Common trap: skipping rule-based checks because LLM-as-judge feels more sophisticated🔒
2.2 LLM-as-Judge: Writing Rubrics That Constrain4 items
- 2.2.1: The naive judge fails first: 'rate this email 1-10' and the useless scores it returns🔒
- 2.2.2: Rubric engineering: explicit criteria, anchored scale points, exemplars, required reasoning🔒
- 2.2.3: Building the Trellis judge and scoring the golden set automatically🔒
- 2.2.4: Judge mechanics that matter: model choice, position bias, and self-preference🔒
2.3 Judge Calibration: Does the Judge Agree With Humans?4 items
- 2.3.1: The agreement run: judge scores vs. the Week 1 human baseline, case by case🔒
- 2.3.2: Agreement metrics without the statistics degree: percent agreement and per-category splits🔒
- 2.3.3: The calibration loop: fixing the rubric where the judge diverges, re-running, converging🔒
- 2.3.4: Knowing when to stop trusting the judge: the case types that stay human-only🔒
2.4 The Layered Scoring Stack3 items
- 2.4.1: Composing rules, judge, and human review: what runs always, on samples, and escalates🔒
- 2.4.2: Cost and speed of the stack: scoring 200 cases for under a dollar🔒
- 2.4.3: The full automated run: baseline reproduced automatically, deltas documented🔒
2.5 Case Study Checkpoint: The Judge Disagrees, Confidently3 items
- 2.5.1: The twist: the calibrated judge scores long emails 2 points higher than humans🔒
- 2.5.2: The investigation: finding the verbosity bias, proving it with a split analysis🔒
- 2.5.3: The fix and the lesson: a rubric patch, a re-calibration, a known-limitations section🔒
2.6 Side Quest: Same Stack, Different Product3 items
- 2.6.1: Building a 10-case eval and mini-judge for a RAG support bot (grounding and citations)🔒
- 2.6.2: What transferred untouched, what needed new criteria, what needed a different judge🔒
- 2.6.3: Reflection: the one-page 'eval pattern vs. eval instance' note🔒
2.7 Week 2 Portfolio Artifact: A Calibrated Scoring Pipeline3 items
- 2.7.1: Packaging the rule pack, judge rubric, calibration report, and known-limitations doc🔒
- 2.7.2: The agreement numbers, presented honestly: overall, per category, human-only zones🔒
- 2.7.3: Checklist: what makes this pipeline credible to a technical reviewer🔒
Week 3: Statistical Honesty and Release Gating
You can apply the statistics needed to know whether a score change is signal or noise, and you can wire your eval suite into CI as a release gate that fails the build when quality regresses.
3.1 Variance: The Enemy You Run First3 items
- 3.1.1: The same eval, five runs, five different scores: measuring your pipeline's noise floor🔒
- 3.1.2: Sample size intuition: why 25 cases can't detect a 3% improvement🔒
- 3.1.3: Practical rules: multiple runs, score ranges, and the honesty phrase 'within noise'🔒
3.2 Comparing Two Versions Without Fooling Yourself4 items
- 3.2.1: The A/B eval: current prompt vs. candidate, same golden set, same judge, paired🔒
- 3.2.2: Reading the comparison: wins, losses, ties per category, better overall vs. where it matters🔒
- 3.2.3: Significance without ceremony: a simple resampling check anyone can run🔒
- 3.2.4: Common trap: cherry-picking the eval run that supports the change you already like🔒
3.3 Evals as Release Gates3 items
- 3.3.1: Wiring the pipeline into CI: every prompt change scored automatically before merge🔒
- 3.3.2: Gate design: blocker criteria that hard-fail, quality thresholds, and who can override🔒
- 3.3.3: The regression suite grows forever: every production bug becomes a permanent case🔒
3.4 Evaluating Agents: Trajectories, Not Just Answers3 items
- 3.4.1: Why answer-only evals miss agent failures: a perfect final message, a wrong action🔒
- 3.4.2: Trajectory scoring: expected tool calls, gate triggers, end states, step budgets🔒
- 3.4.3: Hands-on: building a 10-scenario trajectory eval for a small provided agent🔒
3.5 Case Study Checkpoint: The Improvement That Wasn't3 items
- 3.5.1: The twist: the new version scores +4 overall and everyone wants to ship today🔒
- 3.5.2: The honest analysis: +4 is within noise overall, but one category improved and one regressed🔒
- 3.5.3: The recommendation memo: ship/don't-ship with the evidence, defended against pushback🔒
3.6 Red-Teaming the Eval Itself3 items
- 3.6.1: Gaming the metric: three ways the system under test can score well while getting worse🔒
- 3.6.2: The eval audit: coverage gaps, stale cases, and criteria drift, checked quarterly🔒
- 3.6.3: Practice: finding the two planted weaknesses in a provided eval suite🔒
3.7 Week 3 Portfolio Artifact: A Release-Gating Eval in CI3 items
- 3.7.1: Packaging the CI integration, gate config, and the A/B comparison report🔒
- 3.7.2: The ship/don't-ship memo as the centerpiece: judgment, backed by numbers, caveated🔒
- 3.7.3: Checklist: what makes this artifact credible to an engineering team🔒
Week 4: Production, Drift, and Getting Hired
You can set up production drift monitors that catch quality degradation after a model or prompt change, and you can present four portfolio artifacts with real numbers to a hiring manager in a way that makes the eval role yours to lose.
4.1 Production Monitoring: Evals Don't Stop at Ship3 items
- 4.1.1: Offline vs. online: why a passing golden set and a healthy product are different claims🔒
- 4.1.2: Sampled live scoring: running the judge on a slice of production traffic, with cost math🔒
- 4.1.3: Drift detection: score trends, category shifts, and the alert thresholds worth paging on🔒
4.2 The Feedback Loop: Production Failures Become Golden Cases3 items
- 4.2.1: Mining flags, thumbs-downs, and support tickets for new eval cases🔒
- 4.2.2: The triage habit: which production failures earn a permanent case, which are one-offs🔒
- 4.2.3: Closing the loop: the golden set as a living record of every way the product has failed🔒
4.3 Case Study Checkpoint: The Regression the Eval Missed3 items
- 4.3.1: The final twist: a model upgrade passes every gate, then live complaints climb🔒
- 4.3.2: The investigation: finding the new failure mode in production, sizing it, emergency cases🔒
- 4.3.3: The retro: why the suite missed it, the coverage rule that prevents the class of miss🔒
4.4 Turning the Case Study into a Portfolio Piece3 items
- 4.4.1: Structuring the write-up: the vibes problem, the system built, the calls made, caught and missed🔒
- 4.4.2: Recording a short walkthrough video: the noise-vs-signal call and the missed regression🔒
- 4.4.3: What to leave out for confidentiality, even in a fictional case study🔒
4.5 The Eval Engineer Interview: What They're Actually Testing For4 items
- 4.5.1: The real skills behind typical questions ('design an eval for X', 'critique this rubric')🔒
- 4.5.2: Common formats: take-home eval design, live rubric critique, dataset review, behavioral🔒
- 4.5.3: How to present the Trellis system in an interview, leading with the judgment calls🔒
- 4.5.4: Practice round: a brand-new, unseen product, eval design sketched under time pressure🔒
4.6 Your 30-Day Post-Course Plan3 items
- 4.6.1: Where to find eval roles, the titles they hide under, and the QA-to-evals entry path🔒
- 4.6.2: Tailoring your portfolio and resume around the 4 artifacts🔒
- 4.6.3: A week-by-week outreach and application plan🔒
The eval stack has five parts: failure taxonomy, golden dataset, calibrated judge, statistical release gate and production drift monitor. You build all five and finish with four artifacts and real numbers behind each one, which gives an interviewer something concrete to question you on.
From subscribing to running your first artifact
Subscribe and get instant access
The moment you subscribe you unlock the full course, including a free preview lesson, the Trellis CRM case-study materials, and the workbook you will fill in across all four weeks. No setup gatekeeping, no waiting for a cohort to start.
Open Orientation and start the case study today
Begin with the three-chat exercise in Lesson 0.1.1: send the same prompt to a free AI tool three times, paste the outputs side by side, and name one difference a real customer would notice. You have just produced the raw material of every eval, and you have not written a single line of code yet. Daily live sessions and mentor review are available from day one.
Build your first scored artifact and add it to your portfolio
By the end of Week 1 you will have a written failure taxonomy and the start of a golden dataset grounded in the Trellis email-writer. By Week 4 you will have four portfolio artifacts with real numbers: a dataset, a taxonomy, a working scoring stack, and a CI-wired release gate, ready to walk through in an interview.
Catching a prompt-change regression before it ships: the Trellis email writer
The change lands on your desk
Devin the AI engineer rewrote the Trellis email-writer prompt to make emails shorter. He ran it on five examples and they looked fine to him. Your job is to find out whether quality actually held across the full range of cases, not just the ones he remembered building for.
Run the golden dataset through both prompt versions
You feed your hand-built golden dataset, fifty diverse Trellis contacts across different deal stages and tones, through the old prompt and the new one. Your scoring stack returns a pass rate for each version on every dimension in your rubric: clarity, correct call to action, no hallucinated pricing, appropriate length.
Apply the statistical check
The new prompt scores higher on brevity but lower on call-to-action completion. You run the statistical test you learned in Week 3 to confirm the drop is signal, not noise caused by a small sample. It is signal. You now have a number you can defend to a VP, not a feeling.
The release gate fires in CI
Because your eval suite is wired into CI as a release gate, the build fails automatically on the call-to-action regression. The change does not reach staging. Devin gets a report with the exact failure category from your taxonomy, so he knows what to fix rather than guessing.
You write the quality report
You document the finding: the new prompt improves brevity but introduces a measurable regression on call-to-action completion across enterprise-tier contacts. You recommend a targeted fix, not a rollback. That report, with real numbers and a named failure category, is the kind of artifact you add to your portfolio and walk through in an interview.
The toolkit you'll build and run in this program
You leave with working assets, not notes.
Real things you can demo in an interview.
A hand-built golden dataset grounded in the Trellis CRM case study, with documented pass and fail examples you can show as a portfolio piece.
A written failure taxonomy that names and categorises the specific ways an AI email writer can go wrong, ready to drop into any future eval project.
A working scoring stack combining rule-based checks and a calibrated LLM-as-judge, with accuracy metrics against human review.
A CI-integrated release gate that fails a build when your eval suite detects a quality regression.
A production drift monitor wired to the Trellis case study, demonstrating post-deployment quality tracking.
A completed interview workbook with four artifacts and real numbers, built entry by entry across the four weeks, ready to walk through in a screen.
You'll get the most from this if
QA engineers who want to move from testing deterministic software to owning the quality bar on AI features.
Analysts or junior devs who can follow a process step by step and want to retrain into a hard-to-fill role.
Anyone told they need 'AI experience' for a promotion who wants a portfolio of real artifacts to prove it.
Professionals without a coding or data science background who need every step explained in plain language before they build anything.
Yearly plan: 14-day keep-the-system guarantee
14-day refund on the yearly plan. Finish the first two modules, and if you haven't run your first live pipeline, take a full refund.
Straight answers before you subscribe
What does an AI evaluation engineer do?
An AI evaluation engineer measures whether an AI product's output is good enough to ship, and keeps measuring after it ships. In this course that means a failure taxonomy, a hand-built golden dataset, rule-based checks, a calibrated LLM-as-judge, human review, the statistics to tell when a number means something, CI release gates and production drift monitors.
I have never written code professionally. Will I be lost from the start?
No. The course is designed for QA engineers, analysts and junior devs, with no coding background assumed. Every concept is explained in plain language and grounded in the Trellis CRM case study before you touch any syntax. The first exercise uses a free AI chat tool, with nothing to install. Hands-on practice starts in the first lesson, and mentor review is available daily so you're never stuck alone.
Does it work with the tools and setup I already have?
The course is built around widely available tools and a self-contained case study, so you don't depend on a specific company tech stack. The Trellis CRM scenario gives you everything you need to practise on. If you hit a compatibility question with your own environment, daily live sessions and mentor review are there to help you work through it.
What if I start and it is not the right fit?
Refunds depend on the plan. The yearly plan has a 14-day full refund. In India every plan is a one-time pass that isn't refunded, but it never renews without you, and you can try a free lesson before you buy. The refunds page has the full policy.
How do updates work? Will the course go stale?
The course content is updated when the underlying tools, techniques or best practices change in a meaningful way. As a subscriber you get those updates as part of your enrolment, so the scoring stack and CI examples reflect current practice.
Can I use the artifacts and code I build here in my own work or show them to employers?
Yes. Everything you build (the golden dataset, the failure taxonomy, the scoring stack, the release gate) is yours to use, show in interviews and apply in your own role. The licence covers personal and commercial use in your own projects, so your portfolio goes with you.