@reaatech/llm-judge-bias

Status: Pre-1.0 — APIs may change in minor versions. Pin to a specific version in production.

Bias detection suite for identifying systematic biases in LLM judgments. Detects position bias (order effects), length bias (verbosity preference), and style bias (formatting/register effects).

Installation

npm install @reaatech/llm-judge-bias
# or
pnpm add @reaatech/llm-judge-bias

Feature Overview

• Position bias detection with original/swapped order comparison
• Length bias detection via Pearson correlation between response length and score
• Style bias detection comparing formal/casual/bullet-point transformations
• Automatic debiasing by averaging both orders
• ComprehensiveBiasDetector orchestrates all three detectors in one pass
• Configurable thresholds for each bias dimension

Quick Start

import { PositionBiasDetector } from '@reaatech/llm-judge-bias';
 
const detector = new PositionBiasDetector(0.1);
 
const report = await detector.detect(engine, [
  { id: 'a', content: 'Response A...' },
  { id: 'b', content: 'Response B...' },
]);
 
console.log(report.hasBias, report.recommendation);

import { ComprehensiveBiasDetector } from '@reaatech/llm-judge-bias';
 
const detector = new ComprehensiveBiasDetector({
  positionThreshold: 0.1,
  lengthThreshold: 0.3,
  styleThreshold: 0.1,
});
 
const report = await detector.runAll(engine, {
  candidates: [{ id: 'a', content: '...' }, { id: 'b', content: '...' }],
  responses: [{ id: '1', content: '...' }],
  styleBaseResponse: 'Some response...',
  styleContext: { query: '...', response: '...', context: '...' },
});
 
console.log(report.hasBias, report.recommendation);

API Reference

PositionBiasDetector

LengthBiasDetector

StyleBiasDetector

ComprehensiveBiasDetector

Report Types

Default Style Transforms

Related Packages

• @reaatech/llm-judge-engine — JudgmentEngine consumed by detectors
• @reaatech/llm-judge-types — Core type definitions

License

MIT