Evals & Quality

These packages give you a deterministic recording and replay system for AI agent interactions. You'd adopt them to debug agent behavior without burning LLM tokens on every iteration — record a trace once, then replay it in stubbed, partial, or diff modes for zero-cost debugging and regression testing. The system is built around a trace-based data model with hierarchical spans and events, with interceptors that monkey-patch OpenAI and Anthropic SDKs transparently, so recording happens without modifying your agent code.

These packages give you a linter, validator, and scaffolder for AGENTS.md and SKILL.md files — the markdown documents that define how AI agents behave and what skills they have. You would adopt them to enforce a consistent, machine-readable structure across agent definitions in a multi-agent system, catching formatting errors, missing sections, and broken skill references before they cause runtime issues. The toolkit is built as a pipeline of independent packages (parser → validator → linter → reporter) that share canonical Zod schemas, with an MCP server exposing the same tools directly to AI agents over Stdio or HTTP.

These packages give you a deterministic engine for deciding what content to include, summarize, or drop when packing prompts into an LLM's context window. You'd adopt them to solve the problem of overflowing a model's token budget—replacing ad-hoc truncation with a configurable planner that enforces budgets, reserves space for generation, and emits structured warnings about every decision. The most distinctive thing is the pluggable strategy system: you can swap between priority-greedy, sliding-window, summarize-and-replace, or RAG relevance selection strategies, or compose custom ones, all while using typed context item primitives and tokenizer adapters for different model families.

These packages give you a complete system for using LLMs to evaluate generated text, with built-in prompt templates for five criteria (faithfulness, relevance, coherence, safety, tool-use), multi-provider support (OpenAI, Anthropic, local endpoints), and a judgment engine that handles retries, caching, and rate limiting. You would adopt them to replace ad-hoc LLM evaluation scripts with a structured pipeline that includes statistical calibration against human labels, bias detection (position, length, style), and multi-judge consensus strategies. The packages are designed as independently installable modules that share a common type system and pluggable interfaces—the engine, providers, templates, cache backends, and bias detectors each implement a shared contract, so you can swap implementations or use only the pieces you need without pulling in the rest.

These packages give you Git-like version control for AI prompts — an API server, TypeScript SDK, CLI, and MCP server that let you track prompt changes, gate promotions on eval results, and serve A/B deployments. You'd adopt them to solve the problem of managing prompt iterations across development, staging, and production without manual copy-pasting or ad-hoc versioning. The most distinctive thing is that the entire lifecycle — from creating a draft to promoting it to production — is gated by evaluation harness results, with AI agents able to pull managed prompts at runtime via the MCP server.