@reaatech/rag-eval-cost
Status: Pre-1.0 — APIs may change in minor versions. Pin to a specific version in production.
Cost tracking, pricing, budgeting, and reporting infrastructure for RAG evaluations. Tracks per-sample token consumption, enforces budget limits with configurable alert thresholds, and generates cost reports in JSON and JUnit XML formats.
Installation
npm install @reaatech/rag-eval-cost
# or
pnpm add @reaatech/rag-eval-costFeature Overview
- Multi-provider pricing — built-in pricing for Anthropic (Claude), OpenAI (GPT), and Google (Gemini) models
- Per-sample cost tracking — record costs by sample ID, metric, and provider with token-level detail
- Budget enforcement — configurable per-run budget limits with hard/soft stop modes
- Alert thresholds — trigger actions at 50%, 75%, and 90% of budget with configurable callbacks
- Report generation — JSON cost reports and JUnit XML for CI integration
- Extensible pricing — add custom model pricing for any provider
Quick Start
import { CostTracker, Pricing, BudgetManager, CostReporter } from "@reaatech/rag-eval-cost";
// Track costs with budget limits
const tracker = new CostTracker({ budgetLimit: 10.00, hardLimit: false });
const result = tracker.recordCost(
"sample-1",
0.025,
{ input: 500, output: 75, total: 575 },
"faithfulness",
"anthropic"
);
console.log(result.withinBudget); // true
console.log(tracker.getTotalCost()); // 0.025
// Generate report for CI
const reporter = new CostReporter();
const report = reporter.generateReport(tracker.getBreakdown());
console.log(report.totalCost); // 0.025
console.log(report.costPerSample); // 0.025API Reference
CostTracker
Per-evaluation cost accounting with budget limits.
import { CostTracker, type CostTrackerOptions } from "@reaatech/rag-eval-cost";
const tracker = new CostTracker({
budgetLimit: 50.00,
hardLimit: true,
alertThresholds: [0.5, 0.75, 0.9],
});CostTrackerOptions
| Property | Type | Default | Description |
|---|---|---|---|
budgetLimit | number | Infinity | Maximum total cost before stopping |
hardLimit | boolean | false | If true, shouldStop returns true when budget exceeded |
alertThresholds | number[] | [0.5, 0.75, 0.9] | Budget usage percentages that trigger alerts |
Instance Methods
| Method | Returns | Description |
|---|---|---|
recordCost(id, cost, tokens?, metric?, provider?) | { cost, withinBudget, shouldStop } | Record a cost entry and check budget |
getTotalCost() | number | Total accumulated cost |
getCount() | number | Number of recorded samples |
getAverageCost() | number | Mean cost per sample |
getBreakdown() | CostBreakdown | Cost breakdown by metric, provider, and per-sample |
isWithinBudget() | boolean | Whether total cost is within budget |
reset() | void | Clear all recorded costs |
Pricing
Provider model pricing lookup and cost calculation.
import { Pricing, type ModelPricing } from "@reaatech/rag-eval-cost";
const pricing = new Pricing();
// Look up pricing for a model
const config = pricing.getPricing("claude-opus", "anthropic");
console.log(config.input); // 15.00 (per million tokens)
console.log(config.output); // 75.00 (per million tokens)
// Calculate cost from token counts
const cost = pricing.calculateCost("gpt-4", "openai", 1000000, 500000);
console.log(cost); // 25.00Built-in Pricing
| Model | Provider | Input $/1M | Output $/1M |
|---|---|---|---|
claude-opus | anthropic | 15.00 | 75.00 |
claude-sonnet-4-6 | anthropic | 3.00 | 15.00 |
claude-haiku-4-5 | anthropic | 0.80 | 4.00 |
gpt-4 | openai | 30.00 | 60.00 |
gpt-4-turbo | openai | 10.00 | 30.00 |
gpt-4o | openai | 5.00 | 15.00 |
gemini-pro | 2.50 | 7.50 | |
gemini-flash | 0.075 | 0.30 |
Methods
| Method | Returns | Description |
|---|---|---|
getPricing(model, provider) | ModelPricing | Get pricing config for a model |
calculateCost(model, provider, inputTokens, outputTokens) | number | Calculate cost from token counts |
addModelPricing(pricing) | void | Register custom model pricing |
BudgetManager
Budget enforcement with alert thresholds.
import { BudgetManager } from "@reaatech/rag-eval-cost";
const manager = new BudgetManager({
budgetLimit: 10.00,
hardLimit: true,
alertThresholds: [0.5, 0.75, 0.9],
});
const result = manager.recordSpend(7.00);
console.log(result.withinBudget); // true
console.log(manager.getBudgetUsage()); // 0.7
const alerts = manager.getActiveAlerts();
// → [{ threshold: 0.5, message: "Budget 50% used", level: "warn" }]BudgetManagerOptions
| Property | Type | Default | Description |
|---|---|---|---|
budgetLimit | number | (required) | Maximum budget |
hardLimit | boolean | false | Stop execution when budget exceeded |
alertThresholds | number[] | [0.5, 0.75, 0.9] | Usage thresholds for alerts |
CostReporter
Report generation in multiple formats.
import { CostReporter } from "@reaatech/rag-eval-cost";
const reporter = new CostReporter();
const report = reporter.generateReport(breakdown);
// → { totalCost: 1.50, costPerSample: 0.075, trend: "stable", ... }
const xml = reporter.generateJUnitXml(breakdown);
// → <testsuite name="CostReport" tests="1" ...>| Method | Returns | Description |
|---|---|---|
generateReport(breakdown) | CostReport | Generate JSON cost report with trends |
generateJUnitXml(breakdown) | string | Generate JUnit XML for CI reporting |
addHistoricalCost(date, cost) | void | Add historical data for trend calculation |
Usage Patterns
Budget Enforcement in Pipelines
const tracker = new CostTracker({ budgetLimit: 10.00, hardLimit: true });
for (const sample of samples) {
const { shouldStop } = tracker.recordCost(sample.id, sample.cost, sample.tokens, sample.metric);
if (shouldStop) {
console.warn("Budget exceeded — stopping evaluation");
break;
}
await evaluateSample(sample);
}CI Cost Gate
import { CostReporter, CostTracker } from "@reaatech/rag-eval-cost";
const tracker = new CostTracker({ budgetLimit: 50.00 });
// ... run evaluations ...
const reporter = new CostReporter();
const xml = reporter.generateJUnitXml(tracker.getBreakdown());
writeFileSync("reports/cost-report.xml", xml);
// Fail CI if over budget
if (!tracker.isWithinBudget()) {
console.error(`Budget exceeded: $${tracker.getTotalCost()}`);
process.exit(1);
}Related Packages
@reaatech/rag-eval-core— Types and schemas@reaatech/rag-eval-judge— LLM judge (uses pricing for cost estimation)@reaatech/rag-eval-suite— Central orchestrator (uses tracker for budget enforcement)
