A 8-person dev tools startup uses multiple LLM providers (OpenAI, Anthropic) and stores API keys in environment variables. Their security policy requires key rotation every 90 days, but manual rotation causes outages when keys are changed without updating all services. They need an automated rotation system that updates keys across all agents and services without downtime, with audit logging for compliance.
A complete, working implementation of this recipe — downloadable as a zip or browsable file by file. Generated by our build pipeline; tested with full coverage before publishing.
LLM API key rotation is a compliance standard (90-day rotation is common), but doing it manually causes outages — services pick up the new key at different times, and a single stale key in a long-running agent can bring down production. This recipe builds an automated secret rotation sidecar that rotates API keys across AWS Secrets Manager or Vercel Environment Variables without dropping a single request. You’ll get a CLI tool (srotation) and a Hono HTTP server that expose rotation, validation, status, and streaming event endpoints — all backed by the REAA rotation engine.
Prerequisites
Node.js >= 22 — the project uses the built-in fetch, TextEncoder, and modern async patterns
pnpm 10+ — the package manager, pinned in the project via packageManager
An OpenAI API key and/or Anthropic API key to validate during rotation (placeholders in .env.example)
An AWS account (for AWS Secrets Manager) or a Vercel project (for Vercel Environment Variables) — choose one or both
Familiarity with TypeScript, environment variables, and basic REST endpoints
Step 1: Examine the project scaffold
The project is a Next.js 16 shell (App Router) with all dependencies pre-installed. Look at the layout:
Expected output: You should see the directory tree above when you run ls -la from the project root.
Step 2: Configure environment variables
Copy the example env file and fill in your provider of choice:
terminal
cp .env.example .env
Open .env and configure at least one secret store provider. The full variable reference is in .env.example:
env
# Env vars used by agnostic-secret-rotation-sidecar.# Keep placeholders only — never commit real values.# Rotary provider type: aws, vercel, or both comma-separatedSRK_PROVIDER=# AWS provider config (required when SRK_PROVIDER includes aws)SRK_AWS_REGION=us-east-1SRK_AWS_ENDPOINT=# Vercel provider config (required when SRK_PROVIDER includes vercel)SRK_VERCEL_TOKEN=SRK_VERCEL_PROJECT_ID=SRK_VERCEL_TEAM_ID=SRK_VERCEL_TARGET=production# LLM provider API keys (for pre/post rotation key validation)OPENAI_API_KEY=ANTHROPIC_API_KEY=# HTTP API serverSRK_PORT=8080SRK_HOST=127.0.0.1SRK_AUTH_TOKEN=SRK_CORS_ORIGIN=http://localhost:*# LoggingSRK_LOG_LEVEL=infoSRK_LOG_STRUCTURED=true# Rotation schedulingSRK_ROTATION_INTERVAL_MS=SRK_SECRETS=
Set SRK_PROVIDER to aws, vercel, or aws,vercel for dual-provider setups. Then fill in the matching section (SRK_AWS_REGION or SRK_VERCEL_TOKEN + SRK_VERCEL_PROJECT_ID). Finally, set at least one LLM provider API key (OPENAI_API_KEY or ANTHROPIC_API_KEY).
Expected output: You should now have a .env file with your provider credentials and at least one LLM API key set.
Step 3: Define the TypeScript types
The foundation is a set of interfaces in src/types.ts. These describe the config shape, the rotation response, and the health/status endpoints:
Expected output:src/types.ts compiles cleanly — pnpm typecheck passes at this point since there are no imports.
Step 4: Build the configuration loader
src/config.ts reads environment variables and produces a typed AppConfig. It imports dotenv/config for the .env file and provides envString, envInt, and envBool helpers:
The loader supports a comma-separated SRK_PROVIDER=aws,vercel for dual-provider setups. Each secret is auto-classified as openai or anthropic based on its name.
Expected output: With SRK_PROVIDER=aws set, loadConfig() returns an object where aws.region is "us-east-1" and vercel is undefined. Without SRK_PROVIDER set, it throws ConfigurationValidationError.
Step 5: Create the error hierarchy
src/errors.ts defines four custom error classes that carry metadata about which secret failed and at which stage:
Expected output:new ProviderConnectionError("msg", "s1") has canRetry === true and stage === "provider-connection". ConfigurationValidationError is a plain Error subclass.
Step 6: Implement LLM provider key validators
The sidecar validates API keys before and after rotation by making real SDK calls. Two provider classes live under src/providers/.
validateKey() calls models.list() — a lightweight read-only endpoint. If the SDK returns 401 or 403, the key is invalid. Any other error (network, timeout) propagates so the caller can retry.
Anthropic doesn’t have a lightweight list endpoint, so validation sends a minimal 1-token message. If it succeeds, the key is valid.
Factory — src/providers/index.ts:
ts
import { OpenAIKeyProvider } from "./openai-provider.js";import { AnthropicKeyProvider } from "./anthropic-provider.js";export function createProvider( type: "openai" | "anthropic", config: { apiKey: string },): OpenAIKeyProvider | AnthropicKeyProvider { if (!config.apiKey || config.apiKey.length === 0) { throw new Error("apiKey must not be empty"); } switch (type) { case "openai": return new OpenAIKeyProvider(config); case "anthropic": return new AnthropicKeyProvider(config); default: { const exhaustiveCheck: never = type; throw new Error(`Unknown provider type: ${String(exhaustiveCheck)}`); } }}
Expected output:createProvider("openai", { apiKey: "sk-test" }) returns an OpenAIKeyProvider instance. An empty apiKey throws.
Step 7: Build the rotation service orchestrator
src/rotation/llm-rotation-service.ts is the brain of the sidecar. It wires the REAA packages together: RotationManager from @reaatech/secret-rotation-core drives key lifecycle, AWSProvider or VercelProvider handles secret storage, and SidecarServer from @reaatech/secret-rotation-sidecar provides the management API.
ts
import { RotationManager } from "@reaatech/secret-rotation-core";import { AWSProvider } from "@reaatech/secret-rotation-provider-aws";import { VercelProvider } from "@reaatech/secret-rotation-provider-vercel";import { LoggerService, MetricsService } from "@reaatech/secret-rotation-observability";import { SidecarServer } from "@reaatech/secret-rotation-sidecar";import type { Counter, Gauge } from "@reaatech/secret-rotation-observability";import type { RotationState } from "@reaatech/secret-rotation-types";import { createProvider } from "../providers/index.js";import type { AppConfig, RotateResponse, HealthStatus, SecretStatusResponse } from "../types.js";import { RotationServiceError }
The flow: initialize() creates the secret store provider (AWS or Vercel), the RotationManager, and the SidecarServer. rotate() delegates to the manager and tracks metrics. preValidateKey / postValidateKey hit the actual LLM SDKs to confirm the key works before and after the rotation.
Expected output:new LlmRotationService(config).initialize() logs “Rotation service initialized”. Without a provider in config, it throws “No provider configured”.
Step 8: Wire the Hono API server
src/api/server.ts exposes REST endpoints backed by the rotation service:
ts
import { Hono, type Context, type Next } from "hono";import { cors } from "hono/cors";import { serve } from "@hono/node-server";import type { LlmRotationService } from "../rotation/llm-rotation-service.js";import type { ApiConfig } from "../types.js";export function createServer(service: LlmRotationService, config: ApiConfig): Hono { const app = new Hono(); app.use("*", cors({ origin: config.corsOrigin ?? "http://localhost:*"
Endpoint
Method
Description
/health
GET
Health check with uptime and timestamp
/secrets
GET
List managed secrets
/secrets/:name
GET
Status for one secret
/rotate
POST
Rotate a secret (body: { "secretName": "..." })
/secrets/:name/rotate
POST
Rotate a secret by URL param
/metrics
GET
Prometheus-formatted metrics
/events
GET
SSE stream of key_activated and rotation_failed events
Expected output: After createServer(service, config), hitting /health returns 200 with { status: "healthy", uptime: …, timestamp: "…" }.
Step 9: Build the CLI
src/cli/index.ts provides four commands via Commander:
ts
import { Command } from "commander";import { loadConfig } from "../config.js";import { LlmRotationService } from "../rotation/llm-rotation-service.js";import { createServer, startServer } from "../api/server.js";export async function main(argv: string[] = process.argv): Promise<void> { const program = new Command(); program .name("srotation") .description("Automated API key rotation for LLM providers") .version("0.1.0"
When no subcommand is given, the CLI defaults to starting the server — so npx srotation and npx srotation server are equivalent.
Expected output: Running npx srotation --help prints the available commands: rotate, validate, status, server, help.
Step 10: Run the test suite
The project includes unit tests for every module plus an integration test that exercises the full rotation flow. Run them all with:
terminal
pnpm test
The tests are written with Vitest and mock all external dependencies (OpenAI, Anthropic, AWS, Vercel) so they run offline. Here’s a sample from the rotation service test showing the mock setup with vi.hoisted():
ts
import { describe, it, expect, vi, beforeEach, afterEach } from "vitest";const mockRotate = vi.hoisted(() => vi.fn());const mockGetState = vi.hoisted(() => vi.fn());const mockEvents = vi.hoisted(() => ({ on: vi.fn() }));vi.mock("@reaatech/secret-rotation-core", () => ({ RotationManager: class { rotate = mockRotate; getState = mockGetState; start = vi.fn(); stop = vi.fn(); events = mockEvents; },}));// ... other mocks ...import { LlmRotationService } from "../../src/rotation/llm-rotation-service.js";describe("LlmRotationService", () => { it("rotate() calls manager.rotate and returns success", async () => { mockRotate.mockResolvedValue({ rotationId: "rot-1", success: true, newKeyId: "k1", duration: 100, timestamp: new Date() }); const service = new LlmRotationService(awsConfig); service.initialize(); const result = await service.rotate("test-secret"); expect(mockRotate).toHaveBeenCalledWith("test-secret"); expect(result.success).toBe(true); expect(result.rotationId).toBe("rot-1"); }); it("rotate() returns failure when manager.rotate throws", async () => { mockRotate.mockRejectedValue(new Error("rotation failed")); const service = new LlmRotationService(awsConfig); service.initialize(); const result = await service.rotate("test-secret"); expect(result.success).toBe(false); expect(result.error).toContain("rotation failed"); });});
The server tests use Hono’s app.request() helper to hit routes directly without HTTP:
Expected output:pnpm test exits 0. The coverage report shows >= 90% for lines, branches, functions, and statements across runtime source files.
Next steps
Add a database-backed state store — the current in-memory state resets on restart. Wire the rotation service to PostgreSQL or Redis via the REAA core’s persistence hooks for production durability.
Support additional LLM providers — implement provider adapters for Google Gemini, Cohere, or local models by creating new provider classes that implement the same validateKey() contract.
Integrate with a secrets manager webhook — set up AWS EventBridge or Vercel webhooks to trigger automatic rotation when a key approaches expiry, replacing the polling-based schedule.
function envInt(key: string, fallback: number): number {
const raw = process.env[key];
if (raw === undefined || raw === "") return fallback;
const parsed = Number(raw);
if (Number.isNaN(parsed)) {
throw new ConfigurationValidationError(`Invalid integer for ${key}: "${raw}"`);
}
return parsed;
}
function envBool(key: string, fallback: boolean): boolean {
const raw = process.env[key];
if (raw === undefined || raw === "") return fallback;
return raw === "true" || raw === "1";
}
export function loadConfig(): AppConfig {
const providerRaw = envString("SRK_PROVIDER");
if (!providerRaw) {
throw new ConfigurationValidationError("SRK_PROVIDER is required (aws, vercel, or both comma-separated)");
