Search across educators, skills, and resources.

Learn first

• AI UX patterns
• Use-case selection
• Prototyping
• Measuring quality

Good matches

Josh Pigford Peter Yang Lenny Rachitsky Swyx

Open next

• AI Skills for Real Engineers
• 5 Agent Skills I Use Every Day
• grill-with-docs: Align Before You Build
• AI Engineering

Learn first

• Test sets
• Human review
• Regression checks
• Quality metrics

Good matches

Hamel Husain Shreya Shankar Chip Huyen Matt Pocock

Open next

• AI SDK v6 Crash Course
• The AI Engineer Roadmap
• AI Engineering
• LLM Evals

LLM Evals

Guide · Hamel Husain · Intermediate

Your AI app needs quality checks before users see it.

evals, quality, llm apps

Building and Evaluating Advanced RAG Applications

Short course · DeepLearning.AI · Intermediate

You already know basic RAG and need better retrieval, evaluation, and production-quality patterns.

rag, evals, retrieval, llm apps, ai engineering

AI Product Management Specialization

Specialization · Duke University · Beginner to intermediate

You want a structured product-management route for scoping, evaluating, and shipping AI products.

ai product, product management, ai strategy, product quality

Promptfoo

Open-source eval and red-team framework · OpenAI / Promptfoo · Intermediate to advanced

Declarative regression tests and side-by-side comparisons of prompts, models, RAG systems, and agent configurations, especially when red teaming is also required.

evals, llm evaluation, ai quality, open-source frameworks

DeepEval

Pytest-style LLM evaluation framework · Confident AI · Intermediate to advanced

Engineering teams that want LLM and agent evaluations to behave like software unit tests, with thresholds, assertions, and CI-friendly failures.

evals, llm evaluation, ai quality, open-source frameworks

Ragas

RAG and AI-application evaluation library · Vibrant Labs · Intermediate to advanced

Evaluating retrieval quality, grounded generation, agent tool use, and production-aligned test data for RAG applications.

evals, llm evaluation, ai quality, open-source frameworks

Inspect AI

Model and agent evaluation harness · UK AI Security Institute · Intermediate to advanced

Rigorous, reproducible model and agent capability or safety evaluations involving tools, multi-turn interaction, coding, or sandboxed environments.

evals, llm evaluation, ai quality, open-source frameworks

Arize Phoenix

AI observability and evaluation platform · Arize AI · Intermediate to advanced

Teams that want evaluation connected to OpenTelemetry traces, datasets, experiments, prompt iterations, and production troubleshooting.

evals, llm evaluation, ai quality, evaluation platforms

OpenEvals

Reusable evaluator library · LangChain · Intermediate to advanced

Python or TypeScript developers who want composable evaluator functions without adopting a full evaluation platform.

evals, llm evaluation, ai quality, open-source frameworks

Giskard

Agent testing and red-team framework · Giskard AI · Intermediate to advanced

Behavioral tests and adversarial scans of multi-turn agents, chatbots, and RAG systems from a pytest-compatible workflow.

evals, llm evaluation, ai quality, open-source frameworks

Braintrust

AI evaluation and observability platform · Braintrust Data · Intermediate to advanced

Teams that want a feedback loop from production traces and failures into datasets, regression experiments, CI gates, and continuous scoring.

evals, llm evaluation, ai quality, evaluation platforms

LangSmith

Agent evaluation and observability platform · LangChain · Intermediate to advanced

Agent teams, especially LangChain and LangGraph users, that need tracing, datasets, experiments, human review, and production feedback in one system.

evals, llm evaluation, ai quality, evaluation platforms

Langfuse

Open-source evals and observability platform · ClickHouse · Intermediate to advanced

Teams prioritizing open-source data control and one workflow across tracing, prompts, datasets, experiments, annotation, and evaluation.

evals, llm evaluation, ai quality, evaluation platforms

W&B Weave

LLM and agent evaluation platform · Weights & Biases · Intermediate to advanced

Teams already using Weights & Biases or wanting versioned datasets, scorers, traces, and evaluation comparisons in an experiment-oriented workflow.

evals, llm evaluation, ai quality, evaluation platforms

Opik

Open-source agent evaluation platform · Comet · Intermediate to advanced

Teams wanting hosted or self-hosted observability with failure-driven regression suites, experiments, metrics, and human review.

evals, llm evaluation, ai quality, evaluation platforms

MLflow GenAI evaluation

Open-source GenAI evaluation and monitoring · MLflow Project · Intermediate to advanced

Teams extending an existing MLflow or MLOps stack to LLM and agent tracing, evaluation-driven development, human feedback, and monitoring.

evals, llm evaluation, ai quality, evaluation platforms

Patronus AI

AI reliability and evaluation platform · Patronus AI · Intermediate to advanced

Safety- and reliability-sensitive RAG or agent systems needing specialized hallucination, retrieval, PII, bias, policy, and custom-criteria evaluators.

evals, llm evaluation, ai quality, evaluation platforms

OpenAI Evals

Hosted LLM and agent evaluation API · OpenAI · Intermediate to advanced

Existing OpenAI API teams maintaining dataset-based prompt or model regression tests during the remaining service window.

evals, llm evaluation, ai quality, cloud and provider evals

Vertex AI Gen AI evaluation service

Managed model and agent evaluation service · Google Cloud · Intermediate to advanced

Vertex AI teams evaluating model outputs, prompts, RAG responses, function calls, and agent trajectories with managed or custom metrics.

evals, llm evaluation, ai quality, cloud and provider evals

Microsoft Foundry evaluations

Cloud evaluation portal and SDK · Microsoft · Intermediate to advanced

Microsoft Foundry teams evaluating models, RAG systems, and single- or multi-turn agents for quality, safety, task completion, and tool use.

evals, llm evaluation, ai quality, cloud and provider evals

Amazon Bedrock Evaluations

Managed model and RAG evaluation jobs · Amazon Web Services · Intermediate to advanced

AWS teams comparing Bedrock models or assessing knowledge bases and external RAG sources with automated metrics, LLM judges, or human reviewers.

evals, llm evaluation, ai quality, cloud and provider evals

Anthropic evaluation guide

Evaluation methodology guide · Anthropic · Intermediate to advanced

Teams designing task-specific evaluation suites for Claude applications that need criteria, datasets, edge cases, grading patterns, and code examples.

evals, llm evaluation, ai quality, cloud and provider evals

Harbor

Agent evaluation and optimization harness · Harbor Framework Team · Intermediate to advanced

Running coding and computer-use agents against reproducible, sandboxed task suites at local or cloud scale.

evals, llm evaluation, ai quality, benchmarks and learning resources

Language Model Evaluation Harness

Language-model benchmark runner · EleutherAI · Intermediate to advanced

Reproducible few-shot and zero-shot evaluation of base or instruction-tuned language models on established academic benchmarks.

evals, llm evaluation, ai quality, benchmarks and learning resources

Lighteval

Multi-backend LLM evaluation toolkit · Hugging Face · Intermediate to advanced

Teams evaluating local, distributed, or API-served models across a large multilingual task catalog while retaining sample-level results.

evals, llm evaluation, ai quality, benchmarks and learning resources

SWE-bench

Software-engineering agent benchmark · SWE-bench team · Intermediate to advanced

Measuring whether coding agents can resolve real GitHub issues by producing repository patches that pass executable tests.

evals, llm evaluation, ai quality, benchmarks and learning resources

Terminal-Bench

Terminal-agent benchmark · Harbor Framework and Laude Institute · Intermediate to advanced

Comparing agents on difficult, verifiable coding, systems, security, and scientific work inside sandboxed terminals.

evals, llm evaluation, ai quality, benchmarks and learning resources

Demystifying evals for AI agents

Agent-evaluation learning resource · Anthropic · Intermediate to advanced

Product and engineering teams designing practical evaluations for multi-turn, tool-using agents.

evals, llm evaluation, ai quality, benchmarks and learning resources

LLM Evals: Everything You Need to Know

Application-evaluation learning resource · Hamel Husain and Shreya Shankar · Intermediate to advanced

Practitioners building product-specific evals from traces, domain-expert judgments, and observed failures instead of relying on generic benchmarks.

evals, llm evaluation, ai quality, benchmarks and learning resources

Ember-1 explained

Reasoning-efficient coding model explainer · Fireworks AI · Intermediate to advanced

Fireworks released Ember-1 on September 23, 2026 as a Kimi K3-based model trained to use fewer reasoning tokens. Use this explainer to evaluate the quality, cost, latency, and context-growth claims on your own agent workload.

ember-1, kimi k3, reasoning models, reasoning tokens, cost optimization

Project HydraFusion

Model orchestration research preview · GitHub · Intermediate to advanced

You want GitHub's September 4, 2026 technical explanation of runtime model orchestration for coding tasks, including plan decomposition, draft-critique-revise patterns, model cascading, evaluation design, and quality-versus-cost tradeoffs.

github copilot, model orchestration, model selection, coding agents, evals

Agent and Model Evaluations in Gemini Enterprise Agent Platform

Evaluation guide · Google AI for Developers · Intermediate to advanced

You want Google's practical July 31, 2026 guide to running the same agent and model evaluations during development and on production traffic, with metrics for quality, safety, grounding, tool use, and trajectories.

google, gemini, agents, evals, model selection

Multi-Vector Embedding Models with Sentence Transformers

Practical guide · Hugging Face · Intermediate to advanced

You want a runnable guide to ColBERT-style late-interaction retrieval, including MaxSim scoring, retrieve-and-rerank, indexing, visual document retrieval, evaluation, and the storage-quality tradeoff versus dense embeddings.

hugging face, sentence transformers, rag, retrieval, embeddings

OpenAI eval design guide

Guide · OpenAI · Intermediate

You need practical guidance for designing representative eval datasets, choosing graders, and turning model testing into an engineering loop instead of ad hoc spot checks, especially while OpenAI's older Evals platform is winding down toward read-only status on October 31, 2026.

openai, evals, quality, datasets, regression testing

OpenAI evals quickstart and datasets

Guide · OpenAI · Intermediate

You want OpenAI's current quickstart for turning examples into dataset-backed evals and improvement loops instead of relying on a deprecated docs path.

openai, datasets, evals, fine-tuning, quality

OpenAI Working with evals

Guide · OpenAI · Intermediate

You need API-level guidance for testing outputs, comparing models, and catching regressions during upgrades.

openai, evals, quality, regression testing, reliability