Skip to content
reaatech

Evals & Quality

Agent and RAG eval harnesses, LLM-as-judge, prompt versioning.

Filtering by

2 repos

reaatech/context-window-planner

★ 0
These packages give you a deterministic engine for deciding what content to include, summarize, or drop when packing prompts into an LLM's context window. You'd adopt them to solve the problem of overflowing a model's token budget—replacing ad-hoc truncation with a configurable planner that enforces budgets, reserves space for generation, and emits structured warnings about every decision. The most distinctive thing is the pluggable strategy system: you can swap between priority-greedy, sliding-window, summarize-and-replace, or RAG relevance selection strategies, or compose custom ones, all while using typed context item primitives and tokenizer adapters for different model families.
packages
2
updated
5 months ago

reaatech/llm-judge-toolkit

★ 0
These packages give you a complete system for using LLMs to evaluate generated text, with built-in prompt templates for five criteria (faithfulness, relevance, coherence, safety, tool-use), multi-provider support (OpenAI, Anthropic, local endpoints), and a judgment engine that handles retries, caching, and rate limiting. You would adopt them to replace ad-hoc LLM evaluation scripts with a structured pipeline that includes statistical calibration against human labels, bias detection (position, length, style), and multi-judge consensus strategies. The packages are designed as independently installable modules that share a common type system and pluggable interfaces—the engine, providers, templates, cache backends, and bias detectors each implement a shared contract, so you can swap implementations or use only the pieces you need without pulling in the rest.
packages
10
updated
5 months ago