Choose this when
Your AI feature already has users, stakeholders, or enough risk that mistakes matter.
All learning paths
AI learning path
Evals and reliability
Stop judging AI quality by vibes and start building repeatable checks.
- Best for
- AI product teams
- Level
- Intermediate
- Time
- 8-16 hours
You should be able to
You can define task examples, expected behavior, graders, traces, regressions, and review workflows.
Checkpoint
Move on when quality discussions point to examples and metrics, not taste.
Do
Learning sequence
Work through the material inside each step. Videos are embedded where they fit; tutorials and references sit next to the task they support.
Step 1
Collect examples
Turn real user tasks, edge cases, and failures into a small eval set.
Watch here
►
LLM evaluation with W&B
Introduces evaluation workflows and measurement for LLM apps.
Open here
LLM Evals
Your AI app needs quality checks before users see it.
Open resourceStep 2
Choose graders
Combine exact checks, human review, model grading, and trace inspection.
Watch here
►
AI evals with Phoenix
Use this when moving from examples to traces and debugging.
Open here
AI Evals for Engineers & PMs
You are shipping AI features and need a serious evaluation workflow.
Open resourceStep 3
Run regressions
Compare prompts, models, retrieval changes, and releases before users see them.
Watch here
►
Promptfoo red teaming
Regression testing and adversarial checks for prompt and model changes.
Open here
OpenAI Working with evals
You need API-level guidance for testing outputs, comparing models, and catching regressions during upgrades.
Open resourcePromptfoo Intro
You need regression tests for prompts, models, and LLM outputs.
Open resourcePractice task
Create a 20-row eval set for one AI workflow and run two prompt versions against it.
Reference
All resources in this path
Step 1
LLM Evals
Your AI app needs quality checks before users see it.
Step 2
AI Evals for Engineers & PMs
You are shipping AI features and need a serious evaluation workflow.
Step 3
OpenAI Working with evals
You need API-level guidance for testing outputs, comparing models, and catching regressions during upgrades.
Step 3
Promptfoo Intro
You need regression tests for prompts, models, and LLM outputs.
Educators to follow
Hamel Husain
Intermediate to advanced
Read the evals guide and build a small test set for your own app.
View educator
Shreya Shankar
Intermediate
Review the course outcomes and pair it with a real feature you can evaluate.
View educator
Chip Huyen
Intermediate to advanced
Use the book page and related essays as a production engineering path.
View educator
Josh Pigford
Beginner to intermediate
Read the public notes and examples before deciding whether the paid material matches your business.
View educator
Peter Yang
Intermediate
Review the Maven syllabus and compare it to your current product workflow.
View educator
Lenny Rachitsky
Beginner to intermediate
Browse the How I AI interviews and copy the workflows that match your role.
View educator