Open-source eval and red-team framework · OpenAI / Promptfoo · Intermediate to advanced
Declarative regression tests and side-by-side comparisons of prompts, models, RAG systems, and agent configurations, especially when red teaming is also required.
evals, llm evaluation, ai quality, open-source frameworks
Pytest-style LLM evaluation framework · Confident AI · Intermediate to advanced
Engineering teams that want LLM and agent evaluations to behave like software unit tests, with thresholds, assertions, and CI-friendly failures.
evals, llm evaluation, ai quality, open-source frameworks
RAG and AI-application evaluation library · Vibrant Labs · Intermediate to advanced
Evaluating retrieval quality, grounded generation, agent tool use, and production-aligned test data for RAG applications.
evals, llm evaluation, ai quality, open-source frameworks
Model and agent evaluation harness · UK AI Security Institute · Intermediate to advanced
Rigorous, reproducible model and agent capability or safety evaluations involving tools, multi-turn interaction, coding, or sandboxed environments.
evals, llm evaluation, ai quality, open-source frameworks
Reusable evaluator library · LangChain · Intermediate to advanced
Python or TypeScript developers who want composable evaluator functions without adopting a full evaluation platform.
evals, llm evaluation, ai quality, open-source frameworks
Agent testing and red-team framework · Giskard AI · Intermediate to advanced
Behavioral tests and adversarial scans of multi-turn agents, chatbots, and RAG systems from a pytest-compatible workflow.
evals, llm evaluation, ai quality, open-source frameworks
Framework docs · Qwen · Intermediate
You want the official Qwen framework for instruction following, tool use, planning, and memory in agent-style applications.
qwen, agents, planning, memory, tool use