@reaatech/agent-replay-core

Status: Pre-1.0 — APIs may change in minor versions. Pin to a specific version in production.

Deterministic recording, replay, and debugging engine for AI agent interactions. Capture traces once, replay infinitely — without consuming LLM tokens.

Installation

npm install @reaatech/agent-replay-core
# or
pnpm add @reaatech/agent-replay-core

Feature Overview

• RecordingEngine — capture agent interactions with span lifecycle management, event recording, and checkpoint creation
• ReplayEngine — stubbed, live, partial, and diff replay modes with progress callbacks
• Partial Replay — replay up to any checkpoint with stubbed responses, restore state, then go live
• ReplayDebugger — step-through debugging with conditional breakpoints, watch expressions, and state inspection
• DiffEngine — structural and semantic comparison of recorded vs replayed traces
• SemanticDiffEngine — text similarity-based comparison of LLM outputs, tool calls, and routing decisions
• RegressionDetector — automated regression detection for CI/CD pipelines
• DivergenceDetector — pinpoint exactly where live replay diverges from the recorded trace
• AnomalyDetector — detect duration spikes, error bursts, token spikes, and infinite loops
• CI/CD Helper — single-function entry point for running all checks in automation
• TraceSerializer — line-delimited JSON with gzip compression and streaming deserialization
• TraceComparator — multi-trace statistical comparison
• TraceSummarizer — automatic trace summarization with highlights and concerns
• AnnotationManager — collaborative annotations on traces for post-hoc analysis
• Streaming — tee-based stream recording and deterministic stream replay with optional timing preservation
• State Capture — structured clone, snapshotter registry, and determinism control (clock freezing, random seeding)

Quick Start

Recording

import { RecordingEngine, LocalFileStorage } from "@reaatech/agent-replay-core";
 
const engine = new RecordingEngine();
const session = engine.startRecording({
  name: "my-agent-run",
  tags: ["production", "v1.2.0"],
});
 
const spanId = engine.startSpan("gpt-4-chat", "llm_call");
engine.captureEvent(
  {
    timestamp: Date.now(),
    type: "request",
    name: "llm-request",
    attributes: { model: "gpt-4" },
    data: { messages: [{ role: "user", content: "Hello" }] },
  },
  { spanId }
);
// ... make your LLM call ...
engine.captureEvent(
  {
    timestamp: Date.now(),
    type: "response",
    name: "llm-response",
    attributes: {},
    data: { content: "Hello! How can I help?" },
  },
  { spanId }
);
engine.endSpan(spanId, "ok");
 
const trace = engine.stopRecording(session);
 
// Persist to disk
const storage = new LocalFileStorage("./traces");
await storage.save(trace);

Replaying

import { ReplayEngine, LocalFileStorage } from "@reaatech/agent-replay-core";
 
const storage = new LocalFileStorage("./traces");
const trace = await storage.load("trace-1714348800000-0");
 
const replay = new ReplayEngine();
const result = replay.replay(trace, {
  mode: "stubbed",
  onProgress: (p) => console.log(`${p.percent}% complete`),
});
 
console.log(result.outputs); // Replayed LLM responses — zero tokens consumed

API Reference

RecordingEngine

Primary API for capturing agent interactions.

RecordingSession

ReplayEngine

Executes replay of recorded traces in four modes.

const result = replay.replay(trace, { mode: "stubbed" });
// result: { trace: Trace, outputs: unknown[], duration: number, divergence?: DivergenceReport }

PartialReplayOrchestrator

Advanced replay with checkpoint-based state restoration and go-live transitions.

const orchestrator = new PartialReplayOrchestrator();
 
// Full workflow: find checkpoint → stub replay → restore state → go live → execute
const result = await orchestrator.partialReplay(
  trace,
  "cp-3",
  { mode: "partial", checkpointId: "cp-3" },
  async (liveSpans) => {
    // Your live executor making actual LLM calls
    return { trace, outputs, duration };
  }
);

ReplayDebugger

Interactive step-through debugging with breakpoints and watchpoints.

const debugger = new ReplayDebugger(trace);
 
debugger.addBreakpoint({ kind: "llm_call", name: /error/i });
debugger.setBreakpointHandler(async (hit, session) => {
  console.log("Breakpoint hit:", hit.span.name);
  return true; // pause execution
});
 
const session = debugger.start();
await debugger.runToCompletion();
 
// Inspect results
console.log(formatDebugSession(debugger.getSession()));

Diff & Comparison

DiffEngine

Structural and semantic comparison of recorded vs replayed traces.

SemanticDiffEngine

Text similarity-based semantic comparison of LLM outputs.

const engine = new SemanticDiffEngine({ textSimilarityThreshold: 0.95 });
const result = engine.compare(baselineTrace, currentTrace);
// result: { differences, overallSimilarity, maxSeverity }

DivergenceDetector

Pinpoints exactly where live replay diverges from the recorded trace.

RegressionDetector

Automated regression detection with configurable thresholds.

AnomalyDetector

Detects unusual patterns in traces.

TraceComparator

Multi-trace statistical comparison.

CI/CD Helper

Single-function entry point for running all checks in automation.

import { runCICDCheck } from "@reaatech/agent-replay-core";
 
const result = runCICDCheck(currentTrace, {
  baseline: baselineTrace,
  failOnRegression: true,
  minSimilarity: 0.95,
  failOnAnomaly: true,
  failOnDivergence: false,
});
 
if (!result.passed) {
  console.error(result.formattedReport);
  process.exit(1);
}

Storage & Serialization

LocalFileStorage

Filesystem-based TraceStorage implementation.

const storage = new LocalFileStorage("./traces");
 
await storage.save(trace);
const trace = await storage.load("trace-123");
const summaries = await storage.list({ tags: ["production"] });
const results = await storage.search({ text: "error", limit: 10 });
await storage.delete("trace-123");

TraceSerializer

Line-delimited JSON serialization with gzip support.

Trace Migration

Streaming

StreamingRecorder

Tee-based stream recording — passes chunks through to the consumer while recording them.

const recorder = new StreamingRecorder();
for await (const chunk of recorder.record(source, normalizeChunk)) {
  yield chunk; // Consumer receives chunks in real time
}
const recorded = recorder.finalize(aggregatedResponse);

StreamingStubEngine

Deterministic stream replay with optional timing preservation.

const stub = new StreamingStubEngine({ preserveTiming: true });
for await (const chunk of stub.replayStream(recordedStream, denormalizeChunk)) {
  yield chunk;
}
// Or aggregate into a single response:
const response = stub.toResponse(recordedStream);

State Capture

AnnotationManager

Collaborative annotations for post-hoc trace analysis.

TraceSummarizer

Automatic trace summarization into human-readable reports.

Replay Modes in Detail

Stubbed (Default)

Replays recorded LLM responses from the trace. Zero tokens, zero API calls. Fast and deterministic. Use for rapid iteration during development.

Live

Re-executes LLM calls against the actual provider. Requires interceptors from @reaatech/agent-replay-interceptors. Use for validating that code changes produce correct results.

Partial

Replays the first N steps with stubbed responses (zero cost), restores agent state from the checkpoint, then switches to live execution for the remaining steps. Ideal for debugging a specific portion of a long agent run.

Diff

Compares live LLM outputs against the recorded trace, detecting any divergence. Reports structural changes, semantic differences, and overall severity. Use in CI/CD to catch regressions before deployment.

File Format

Traces use the .artrace.json extension with line-delimited JSON:

Line 1:   TraceHeader {"version": "1.0.0", "format": "artrace-json-v1", "metadata": {...}, "schema": {...}}
Lines 2-N: {"_kind": "span", "id": "span-0", ...} or {"_kind": "checkpoint", "id": "cp-0", ...}
Last:     {"kind": "footer", "indexes": {...}, "summary": {...}}

Optional gzip compression (.artrace.json.gz) is supported by TraceSerializer.

Related Packages

• @reaatech/agent-replay-shared — Types, errors, and configuration
• @reaatech/agent-replay-interceptors — LLM provider interceptors (OpenAI, Anthropic)
• @reaatech/agent-replay-integrations — Framework integrations (LangChain, LangGraph)
• @reaatech/agent-replay-cli — Command-line interface
• @reaatech/agent-replay — Convenience entry point re-exporting all packages

License

MIT