@reaatech/llm-judge-templates

Status: Pre-1.0 — APIs may change in minor versions. Pin to a specific version in production.

Evaluation prompt templates implementing the JudgmentTemplate interface. Each template (faithfulness, relevance, coherence, safety, tool-use) builds structured prompts with JSON-parsed responses and fallback parsing.

Installation

npm install @reaatech/llm-judge-templates
# or
pnpm add @reaatech/llm-judge-templates

Feature Overview

• Five evaluation templates — faithfulness, relevance, coherence, safety, and tool-use criteria
• JudgmentTemplate interface — build custom templates with buildPrompt and parseResponse
• Structured JSON output — every template returns score, reasoning, confidence, and metadata
• Robust parsing — cleanAndParse strips markdown fences; parseFallback handles malformed output
• Input validation — 500K character length limit and required field checks enforced per template
• Score normalization — safeScore clamps and validates scores to 0–1, defaults to 0.5

Quick Start

import { FaithfulnessTemplate } from "@reaatech/llm-judge-templates";
 
const template = new FaithfulnessTemplate();
 
const prompt = template.buildPrompt({
  query: "What is the capital of France?",
  response: "The capital of France is Paris.",
  context: "Paris is the capital and most populous city of France.",
});
// prompt = { system: "...", user: "..." }
 
const parsed = template.parseResponse(`{
  "score": 1.0,
  "reasoning": "All claims are supported by the source material.",
  "confidence": 0.95
}`);
 
console.log(parsed.score, parsed.confidence);
// 1.0, 0.95

API Reference

FaithfulnessTemplate

RelevanceTemplate

CoherenceTemplate

SafetyTemplate

ToolUseTemplate

JudgmentTemplate Interface

TemplateContext

Helpers

Related Packages

• @reaatech/llm-judge-types — Core types including EvaluationCriteria and JudgmentTemplate interface
• @reaatech/llm-judge-engine — Judgment engine that consumes templates

License

MIT