@reaatech/llm-judge-calibration

Status: Pre-1.0 — APIs may change in minor versions. Pin to a specific version in production.

Calibration suite for measuring LLM judge accuracy against human-labeled gold-standard datasets. Computes Cohen’s kappa, confusion matrices, F1 scores, and detects calibration drift over time.

Installation

npm install @reaatech/llm-judge-calibration
# or
pnpm add @reaatech/llm-judge-calibration

Feature Overview

• Cohen’s kappa inter-rater reliability coefficient
• Confusion matrix with precision/recall/F1 computation
• Bundled gold-standard datasets for 5 evaluation criteria
• CalibrationRunner for automated batch evaluation
• DriftDetector to detect performance regression
• Concurrency-controlled parallel execution

Quick Start

import { CalibrationRunner } from '@reaatech/llm-judge-calibration';
import { DatasetManager } from '@reaatech/llm-judge-calibration';
 
const runner = new CalibrationRunner({
  engine: judgmentEngine,
  criteria: 'faithfulness',
  concurrency: 5,
  onProgress: (done, total) => console.log(`${done}/${total}`),
});
 
const { report, judgments } = await runner.run();
console.log(report.cohensKappa, report.accuracy);

import { CalibrationMetrics } from '@reaatech/llm-judge-calibration';
 
const report = CalibrationMetrics.generateReport(judgments, humanLabels);
console.log(report);
// { cohensKappa: 0.78, accuracy: 0.91, precision: 0.88, recall: 0.85, f1Score: 0.86, ... }

API Reference

CalibrationMetrics

Static class for computing calibration statistics.

CalibrationRunner

DatasetManager

DriftDetector

Key Types

CalibrationReport

CalibrationDataset

DriftReport

Related Packages

• @reaatech/llm-judge-engine — JudgmentEngine consumed by CalibrationRunner
• @reaatech/llm-judge-types — CalibrationReport, ConfusionMatrix, EvaluationCriteria types
• @reaatech/llm-judge-infra — BatchProcessor infrastructure utilities

License

MIT