@reaatech/llm-judge-consensus

Status: Pre-1.0 — APIs may change in minor versions. Pin to a specific version in production.

Multi-judge consensus strategies for combining individual judgment scores into a final evaluation. Includes majority voting, weighted voting, and a cheap-first tiebreaker strategy to minimize API costs.

Installation

npm install @reaatech/llm-judge-consensus
# or
pnpm add @reaatech/llm-judge-consensus

Feature Overview

• Three consensus strategies implementing shared interface
• Confidence-weighted score aggregation
• Automatic agreement score computation using variance-based formula
• Cheap-first pattern uses N cheap model judgments with optional expensive tiebreakers
• Zero external dependencies beyond types

Quick Start

import { MajorityVoting, CheapFirstTiebreaker } from '@reaatech/llm-judge-consensus';
 
const strategy = new MajorityVoting();
const result = strategy.execute([judgment1, judgment2, judgment3]);
 
console.log(result.finalScore, result.agreementScore);
// 0.82, 0.94

const cheapFirst = new CheapFirstTiebreaker(2);
 
const result = cheapFirst.execute([
  gpt4oMiniJudgment,
  gpt4oMiniJudgment2,
  gpt4oJudgment,    // tiebreaker — only used if cheap judges disagree
]);
 
console.log(result.tiebreakerUsed);
// false (or true if cheap judges disagreed)

API Reference

MajorityVoting

CheapFirstTiebreaker

WeightedVoting

ConsensusStrategy Interface

ConsensusResult

Related Packages

• @reaatech/llm-judge-types — ConsensusStrategy, ConsensusResult, Judgment types

License

MIT