Evals & Quality
Agent and RAG eval harnesses, LLM-as-judge, prompt versioning.
Filtering by
2 repos
reaatech/agent-replay
These packages give you a deterministic recording and replay system for AI agent interactions. You'd adopt them to debug agent behavior without burning LLM tokens on every iteration — record a trace once, then replay it in stubbed, partial, or diff modes for zero-cost debugging and regression testing. The system is built around a trace-based data model with hierarchical spans and events, with interceptors that monkey-patch OpenAI and Anthropic SDKs transparently, so recording happens without modifying your agent code.
- packages
- 7
- updated
- 2 months ago
reaatech/prompt-version-control
These packages give you Git-like version control for AI prompts — an API server, TypeScript SDK, CLI, and MCP server that let you track prompt changes, gate promotions on eval results, and serve A/B deployments. You'd adopt them to solve the problem of managing prompt iterations across development, staging, and production without manual copy-pasting or ad-hoc versioning. The most distinctive thing is that the entire lifecycle — from creating a draft to promoting it to production — is gated by evaluation harness results, with AI agents able to pull managed prompts at runtime via the MCP server.
- packages
- 5
- updated
- 4 months ago