Turn your QA or analyst background into an AI Eval Engineer role, with a portfolio that proves it
Built for QA engineers, analysts, and junior devs who want to own the defensible quality number on AI products, this four-week hands-on course gives you the full eval stack, real numbers, and four portfolio artifacts you can show in an interview.
30 chapters, 100 lessons
14-day refund on the yearly plan. Real pricing on the plans page, no surprises.
100
lessons, yours to run
Easy to pick up, built to get you the next job
No coding background needed, and you write code with Claude as you go.
Every section below turns into something a hiring manager recognises, not just notes.
Understand the job no classic QA role covers
- You will explain in plain language why assert output == expected breaks the moment software stops being deterministic, and what replaces the answer key.
- You will map the four roles (QA, AI Engineer, Data Scientist, Eval Engineer) to the single question each owns, and route any ambiguous task to the right seat without hesitation.
- No prior coding background needed: every concept lands on a concrete Trellis CRM scenario before you write a single line, so the logic is clear before the syntax is.
Build a golden dataset and failure taxonomy a hiring manager can read
- You will hand-build a golden dataset from the Trellis CRM email-writer case study, with real pass and fail examples a rubric can score.
- You will write a failure taxonomy that names the exact ways an AI email can go wrong, turning vague 'it felt off' feedback into categories an engineer can act on.
- These are portfolio artifacts with real numbers, not slides, so interviewers can see your judgment in writing.
Automate scoring and measure the measurer
- You will wire rule-based checks and a calibrated LLM-as-judge into a single scoring stack that produces a defensible pass rate, not a gut feeling.
- You will measure your own judge's accuracy against human review so you can tell a sceptical VP exactly how much to trust the automated score.
- These are skills listed by name on AI Quality Engineer job descriptions, and you will have working code, not just knowledge of the concept.
Gate releases and monitor production drift
- You will wire your eval suite into CI as a release gate so a regression in email quality fails the build before it ships.
- You will set up production drift monitors that surface quality degradation after a model or prompt change, the capability that separates a junior eval role from a senior one.
- Every step is explained in plain language first: if you can read a checklist, you can follow the build, no matter where you are starting from.
Finish this course and you can do all of this, no prior background required:
- You will be able to build a hand-curated golden dataset and failure taxonomy from scratch, the foundation artifact every AI quality role asks candidates to demonstrate.
- You will be able to design and run a calibrated LLM-as-judge scoring pipeline and report its accuracy against human review, a capability listed by name on AI Eval Engineer job descriptions.
- You will be able to apply the right statistical tests to decide whether a score change between two model versions is meaningful, so you never ship a regression disguised as noise.
- You will be able to wire an eval suite into a CI pipeline as a release gate, the skill that moves you from 'ran some evals' to 'owns the quality bar at release'.
- You will be able to monitor a live AI feature for quality drift and surface degradation before users report it, the production-side capability that separates mid-level from senior eval roles.
- You will be able to walk an interviewer through four portfolio artifacts with real numbers, answering the hardest eval screen questions with evidence rather than theory.
100 lessons, 30 chapters
Built by practitioners, not influencers
Every agent, skill file, and dataset is drawn from 25+ years and 50+ Oracle ERP implementations across pharma, manufacturing, semiconductor, and distribution.
Kept current with the stack
Claude Code, skills, and MCP move fast. Your licence includes every system update for as long as you're subscribed. Your system gets better, not obsolete.
You know software QA inside out, but the moment a hiring manager sees 'AI Eval Engineer' on the job spec, they want someone who can grade a language model's output, not just confirm a button renders. Your existing test suite has nothing to say about whether an AI-generated email is actually good. You cannot point to a golden dataset you built, a rubric you calibrated, or a release gate you wired into CI, so your application sits in the same pile as every other QA candidate who added 'AI' to their LinkedIn headline. The role is real, the demand is growing, and right now there is almost no one who can do the full job, build the taxonomy, run the stats, set the gate, and defend the number to a VP, which means the gap between your CV and the offer is a specific, learnable skill set, not years
What you'll be able to do, module by module
Every module leaves you able to build and run something real. Here is the syllabus, in the order you work through it.
Curriculum
30 chapters · 100 lessons
Orientation (Pre-Week 1)
You can explain in one breath why non-determinism kills classic QA, map the four roles to the single question each owns, and describe what an Eval Engineer actually does hour by hour, so you arrive at Week 1 with the mental model already in place.
0.1 What Is an Eval Engineer, Really?4 items
- 0.1.1: Why the role emerged: what broke about QA when software stopped giving the same answer twice🔒
- 0.1.2: Eval Engineer vs. QA vs. Data Scientist vs. AI Engineer, where the lines actually are🔒
- 0.1.3: A real day in the life, hour by hour🔒
- 0.1.4: Why 'I test software' and 'I've used ChatGPT' are each half the skill🔒
0.2 Meet the Case Study: Trellis CRM's AI Has a Vibes Problem4 items
- 0.2.1: The VP's stated ask and the support queue behind it🔒
- 0.2.2: The email-drafting feature and eight months of shipping blind🔒
- 0.2.3: The stakeholders you'll deal with🔒
- 0.2.4: What 'better' could even mean, and why no two stakeholders agree🔒
0.3 The 4-Week Roadmap3 items
- 0.3.1: Skill map: what is added each week and why in that order🔒
- 0.3.2: The 4 portfolio artifacts you'll walk away with🔒
- 0.3.3: How much time to budget per week🔒
0.4 Setting Up Your Toolkit4 items
- 0.4.1: Accounts and access (model API key, git, Python, an eval harness, a spreadsheet)🔒
- 0.4.2: Creating THE repository and loading the Trellis pack🔒
- 0.4.3: A first variance test: the same prompt run 10 times, differences logged🔒
- 0.4.4: Troubleshooting common setup failures🔒
Week 1: Defining Quality, the Judgment Half of the Job
You can write a defensible failure taxonomy for an AI feature and build a hand-curated golden dataset with clear pass and fail examples, the two artefacts that sit at the foundation of every eval system.
1.1 The Anatomy of 'The AI Is Bad'3 items
- 1.1.1: Reading Trellis's complaint sample: sorting 40 raw complaints into what went wrong🔒
- 1.1.2: Building a failure taxonomy: named, defined, mutually exclusive failure modes🔒
- 1.1.3: A real dissected example: a shipped product's failure taxonomy and how it shaped the roadmap🔒
1.2 What to Measure: From Taxonomy to Quality Criteria4 items
- 1.2.1: Translating each failure mode into a checkable criterion (and which resist checking)🔒
- 1.2.2: The criteria hierarchy: blockers, quality bars, and nice-to-haves🔒
- 1.2.3: Practice pass: writing the Trellis quality spec, one page, every criterion testable🔒
- 1.2.4: Common trap: measuring what's easy instead of what users are angry about🔒
1.3 Golden Datasets: The Core Craft4 items
- 1.3.1: Why hand-built beats generated at the start🔒
- 1.3.2: Building the first 25 Trellis cases by hand: inputs, expected properties, difficulty tags, rationales🔒
- 1.3.3: Coverage thinking: easy, hard, and adversarial cases, and the ratio that catches regressions🔒
- 1.3.4: Dataset hygiene: versioning, provenance, and why a golden set without rationales rots🔒
1.4 Your First Scoring Pass: Human Baseline3 items
- 1.4.1: Scoring 25 cases by hand against the quality spec, and timing how long it takes🔒
- 1.4.2: The disagreement exercise: two humans score the same 10 cases🔒
- 1.4.3: Turning disagreements into rubric fixes: tightening criteria until two humans converge🔒
1.5 Case Study Checkpoint: Trellis Gets Its First Real Baseline3 items
- 1.5.1: Running the current production prompt against the full golden set, scored by hand🔒
- 1.5.2: The baseline report: failure rates per taxonomy category, three worst offenders named🔒
- 1.5.3: Presenting the number to the VP: 'better' now has a definition and a starting point🔒
1.6 Week 1 Portfolio Artifact: A Quality Spec + Golden Dataset3 items
- 1.6.1: Packaging the taxonomy, quality spec, and 25-case dataset with rationales🔒
- 1.6.2: The methodology note: how cases were sourced, tagged, and validated🔒
- 1.6.3: Checklist: what makes this artifact interview-ready🔒
Week 2: Automating the Scoring, and Measuring the Measurer
You can build and run a full scoring stack that combines rule-based checks with a calibrated LLM-as-judge, and you can measure your judge's own accuracy against human review so the number you report is one you can defend.
2.1 Rule-Based Scorers: Cheap, Fast, and Underrated4 items
- 2.1.1: The failures code can catch: placeholders, length, banned phrases, missing fields, format🔒
- 2.1.2: Implementing the Trellis rule pack and running it across all 200 pairs in seconds🔒
- 2.1.3: What the rules caught that eight months of eyeballing missed🔒
- 2.1.4: Common trap: skipping rule-based checks because LLM-as-judge feels more sophisticated🔒
2.2 LLM-as-Judge: Writing Rubrics That Constrain4 items
- 2.2.1: The naive judge fails first: 'rate this email 1-10' and the useless scores it returns🔒
- 2.2.2: Rubric engineering: explicit criteria, anchored scale points, exemplars, required reasoning🔒
- 2.2.3: Building the Trellis judge and scoring the golden set automatically🔒
- 2.2.4: Judge mechanics that matter: model choice, position bias, and self-preference🔒
2.3 Judge Calibration: Does the Judge Agree With Humans?4 items
- 2.3.1: The agreement run: judge scores vs. the Week 1 human baseline, case by case🔒
- 2.3.2: Agreement metrics without the statistics degree: percent agreement and per-category splits🔒
- 2.3.3: The calibration loop: fixing the rubric where the judge diverges, re-running, converging🔒
- 2.3.4: Knowing when to stop trusting the judge: the case types that stay human-only🔒
2.4 The Layered Scoring Stack3 items
- 2.4.1: Composing rules, judge, and human review: what runs always, on samples, and escalates🔒
- 2.4.2: Cost and speed of the stack: scoring 200 cases for under a dollar🔒
- 2.4.3: The full automated run: baseline reproduced automatically, deltas documented🔒
2.5 Case Study Checkpoint: The Judge Disagrees, Confidently3 items
- 2.5.1: The twist: the calibrated judge scores long emails 2 points higher than humans🔒
- 2.5.2: The investigation: finding the verbosity bias, proving it with a split analysis🔒
- 2.5.3: The fix and the lesson: a rubric patch, a re-calibration, a known-limitations section🔒
2.6 Side Quest: Same Stack, Different Product3 items
- 2.6.1: Building a 10-case eval and mini-judge for a RAG support bot (grounding and citations)🔒
- 2.6.2: What transferred untouched, what needed new criteria, what needed a different judge🔒
- 2.6.3: Reflection: the one-page 'eval pattern vs. eval instance' note🔒
2.7 Week 2 Portfolio Artifact: A Calibrated Scoring Pipeline3 items
- 2.7.1: Packaging the rule pack, judge rubric, calibration report, and known-limitations doc🔒
- 2.7.2: The agreement numbers, presented honestly: overall, per category, human-only zones🔒
- 2.7.3: Checklist: what makes this pipeline credible to a technical reviewer🔒
Week 3: Statistical Honesty and Release Gating
You can apply the statistics needed to know whether a score change is signal or noise, and you can wire your eval suite into CI as a release gate that fails the build when quality regresses.
3.1 Variance: The Enemy You Run First3 items
- 3.1.1: The same eval, five runs, five different scores: measuring your pipeline's noise floor🔒
- 3.1.2: Sample size intuition: why 25 cases can't detect a 3% improvement🔒
- 3.1.3: Practical rules: multiple runs, score ranges, and the honesty phrase 'within noise'🔒
3.2 Comparing Two Versions Without Fooling Yourself4 items
- 3.2.1: The A/B eval: current prompt vs. candidate, same golden set, same judge, paired🔒
- 3.2.2: Reading the comparison: wins, losses, ties per category, better overall vs. where it matters🔒
- 3.2.3: Significance without ceremony: a simple resampling check anyone can run🔒
- 3.2.4: Common trap: cherry-picking the eval run that supports the change you already like🔒
3.3 Evals as Release Gates3 items
- 3.3.1: Wiring the pipeline into CI: every prompt change scored automatically before merge🔒
- 3.3.2: Gate design: blocker criteria that hard-fail, quality thresholds, and who can override🔒
- 3.3.3: The regression suite grows forever: every production bug becomes a permanent case🔒
3.4 Evaluating Agents: Trajectories, Not Just Answers3 items
- 3.4.1: Why answer-only evals miss agent failures: a perfect final message, a wrong action🔒
- 3.4.2: Trajectory scoring: expected tool calls, gate triggers, end states, step budgets🔒
- 3.4.3: Hands-on: building a 10-scenario trajectory eval for a small provided agent🔒
3.5 Case Study Checkpoint: The Improvement That Wasn't3 items
- 3.5.1: The twist: the new version scores +4 overall and everyone wants to ship today🔒
- 3.5.2: The honest analysis: +4 is within noise overall, but one category improved and one regressed🔒
- 3.5.3: The recommendation memo: ship/don't-ship with the evidence, defended against pushback🔒
3.6 Red-Teaming the Eval Itself3 items
- 3.6.1: Gaming the metric: three ways the system under test can score well while getting worse🔒
- 3.6.2: The eval audit: coverage gaps, stale cases, and criteria drift, checked quarterly🔒
- 3.6.3: Practice: finding the two planted weaknesses in a provided eval suite🔒
3.7 Week 3 Portfolio Artifact: A Release-Gating Eval in CI3 items
- 3.7.1: Packaging the CI integration, gate config, and the A/B comparison report🔒
- 3.7.2: The ship/don't-ship memo as the centerpiece: judgment, backed by numbers, caveated🔒
- 3.7.3: Checklist: what makes this artifact credible to an engineering team🔒
Week 4: Production, Drift, and Getting Hired
You can set up production drift monitors that catch quality degradation after a model or prompt change, and you can present four portfolio artifacts with real numbers to a hiring manager in a way that makes the eval role yours to lose.
4.1 Production Monitoring: Evals Don't Stop at Ship3 items
- 4.1.1: Offline vs. online: why a passing golden set and a healthy product are different claims🔒
- 4.1.2: Sampled live scoring: running the judge on a slice of production traffic, with cost math🔒
- 4.1.3: Drift detection: score trends, category shifts, and the alert thresholds worth paging on🔒
4.2 The Feedback Loop: Production Failures Become Golden Cases3 items
- 4.2.1: Mining flags, thumbs-downs, and support tickets for new eval cases🔒
- 4.2.2: The triage habit: which production failures earn a permanent case, which are one-offs🔒
- 4.2.3: Closing the loop: the golden set as a living record of every way the product has failed🔒
4.3 Case Study Checkpoint: The Regression the Eval Missed3 items
- 4.3.1: The final twist: a model upgrade passes every gate, then live complaints climb🔒
- 4.3.2: The investigation: finding the new failure mode in production, sizing it, emergency cases🔒
- 4.3.3: The retro: why the suite missed it, the coverage rule that prevents the class of miss🔒
4.4 Turning the Case Study into a Portfolio Piece3 items
- 4.4.1: Structuring the write-up: the vibes problem, the system built, the calls made, caught and missed🔒
- 4.4.2: Recording a short walkthrough video: the noise-vs-signal call and the missed regression🔒
- 4.4.3: What to leave out for confidentiality, even in a fictional case study🔒
4.5 The Eval Engineer Interview: What They're Actually Testing For4 items
- 4.5.1: The real skills behind typical questions ('design an eval for X', 'critique this rubric')🔒
- 4.5.2: Common formats: take-home eval design, live rubric critique, dataset review, behavioral🔒
- 4.5.3: How to present the Trellis system in an interview, leading with the judgment calls🔒
- 4.5.4: Practice round: a brand-new, unseen product, eval design sketched under time pressure🔒
4.6 Your 30-Day Post-Course Plan3 items
- 4.6.1: Where to find eval roles, the titles they hide under, and the QA-to-evals entry path🔒
- 4.6.2: Tailoring your portfolio and resume around the 4 artifacts🔒
- 4.6.3: A week-by-week outreach and application plan🔒
Eval Engineers who can build the full stack, taxonomy, golden dataset, calibrated judge, statistical release gate, and production drift monitor, are rare enough that most teams have not yet found one. Finishing this course with four artifacts and real numbers puts you in a category most applicants cannot reach by reading documentation alone.
From subscribing to running your first artifact
Subscribe and get instant access
The moment you subscribe you unlock the full course, including a free preview lesson, the Trellis CRM case-study materials, and the workbook you will fill in across all four weeks. No setup gatekeeping, no waiting for a cohort to start.
Open Orientation and start the case study today
Begin with the three-chat exercise in Lesson 0.1.1: send the same prompt to a free AI tool three times, paste the outputs side by side, and name one difference a real customer would notice. You have just produced the raw material of every eval, and you have not written a single line of code yet. Daily live sessions and mentor review are available from day one.
Build your first scored artifact and add it to your portfolio
By the end of Week 1 you will have a written failure taxonomy and the start of a golden dataset grounded in the Trellis email-writer. By Week 4 you will have four portfolio artifacts with real numbers: a dataset, a taxonomy, a working scoring stack, and a CI-wired release gate, ready to walk through in an interview.
Catching a prompt-change regression before it ships: the Trellis email writer
The change lands on your desk
Devin the AI engineer rewrote the Trellis email-writer prompt to make emails shorter. He ran it on five examples and they looked fine to him. Your job is to find out whether quality actually held across the full range of cases, not just the ones he remembered building for.
Run the golden dataset through both prompt versions
You feed your hand-built golden dataset, fifty diverse Trellis contacts across different deal stages and tones, through the old prompt and the new one. Your scoring stack returns a pass rate for each version on every dimension in your rubric: clarity, correct call to action, no hallucinated pricing, appropriate length.
Apply the statistical check
The new prompt scores higher on brevity but lower on call-to-action completion. You run the statistical test you learned in Week 3 to confirm the drop is signal, not noise caused by a small sample. It is signal. You now have a number you can defend to a VP, not a feeling.
The release gate fires in CI
Because your eval suite is wired into CI as a release gate, the build fails automatically on the call-to-action regression. The change does not reach staging. Devin gets a report with the exact failure category from your taxonomy, so he knows what to fix rather than guessing.
You write the quality report
You document the finding: the new prompt improves brevity but introduces a measurable regression on call-to-action completion across enterprise-tier contacts. You recommend a targeted fix, not a rollback. That report, with real numbers and a named failure category, is the kind of artifact you add to your portfolio and walk through in an interview.
The toolkit you'll build and run in this program
You leave with working assets, not notes.
Real things you can demo in an interview.
A hand-built golden dataset grounded in the Trellis CRM case study, with documented pass and fail examples you can show as a portfolio piece.
A written failure taxonomy that names and categorises the specific ways an AI email writer can go wrong, ready to drop into any future eval project.
A working scoring stack combining rule-based checks and a calibrated LLM-as-judge, with accuracy metrics against human review.
A CI-integrated release gate that fails a build when your eval suite detects a quality regression.
A production drift monitor wired to the Trellis case study, demonstrating post-deployment quality tracking.
A completed interview workbook with four artifacts and real numbers, built entry by entry across the four weeks, ready to walk through in a screen.
You'll get the most from this if
QA engineers who want to move from testing deterministic software to owning the quality bar on AI features.
Analysts or junior devs who can follow a process step by step and want to retrain into a role that is genuinely hard to fill.
Anyone who has been told they need 'AI experience' for a promotion and wants a portfolio of real artifacts, not a certificate, to prove it.
Professionals who do not have a coding or data science background and need a course that explains every step in plain language before asking them to build anything.
Yearly plan: 14-day keep-the-system guarantee
14-day refund on the yearly plan. Finish the first two modules, and if you haven't run your first live pipeline, take a full refund.
Straight answers before you subscribe
I have never written code professionally. Will I be lost from the start?
No. The course is designed for QA engineers, analysts, and junior devs with no coding background assumed. Every concept is explained in plain language and grounded in the Trellis CRM scenario before you touch any syntax. The first exercise uses a free AI chat tool, nothing to install. Hands-on practice starts in the very first lesson, and mentor review is available daily so you are never stuck alone.
Does it work with the tools and setup I already have?
The course is built around widely available tools and a self-contained case study, so you are not dependent on a specific company tech stack. The Trellis CRM scenario gives you everything you need to practise on. If you hit a compatibility question with your own environment, daily live sessions and mentor review are there to help you work through it.
What if I start and it is not the right fit?
There is a 14-day refund policy. If you have gone through the material and it is not working for you, you can request a refund within 14 days of purchase, no lengthy justification required.
How do updates work? Will the course go stale?
The course content is updated when the underlying tools, techniques, or best practices change meaningfully. As a subscriber you get access to those updates as part of your enrolment, so the scoring stack and CI integration examples reflect current practice rather than last year's tooling.
Can I use the artifacts and code I build here in my own work or show them to employers?
Yes. Everything you build, the golden dataset, the failure taxonomy, the scoring stack, the release gate, is yours to use, show in interviews, and apply in your own role. The licence covers personal and commercial use in your own projects, so the portfolio you build here travels with you.