W&B Weave

What it does

Weave adds tracing and repeatable evaluations to the W&B stack. Evaluation objects connect datasets, model or application functions, scorers, per-example results, multiple trials, latency, and cost.

• Repeatable evaluation objects and runs over versioned datasets
• Function-based, class-based, custom, and built-in scorers
• Single-turn, multi-turn, and trajectory evaluation linked to traces, latency, and cost

Where it fits

Teams already using Weights & Biases or wanting versioned datasets, scorers, traces, and evaluation comparisons in an experiment-oriented workflow.

Use it after defining the behavior that matters, collecting representative cases, and deciding how each case will be graded. A tool can execute and organize an eval, but it cannot decide which failures are costly for your users.

Limitations and cautions

• Weave traces and scores agents but does not run or sandbox them.
• TypeScript lacks some Python class abstractions, and production self-management requires Kubernetes operations.

How to evaluate the evaluator

Before using any score as a release gate, create a small evaluator-validation set that humans have labeled. Compare false passes, false failures, disagreement by failure type, run-to-run variance, latency, and cost. Keep deterministic checks for verifiable facts, calibrate model judges against domain experts, and review raw traces when aggregate scores move.