Skip to content
/ solutions / xero-reliability-suite-for-smb-accounting-operations Xero Reliability Suite for SMB Accounting Operations Auto-generate Xero incident runbooks, wrap API calls with circuit breakers, rotate OAuth secrets, and get Slack alerts — all without an SRE.
The problem SMBs that depend on automated Xero data flows grind to a halt when APIs error or rate-limit — with no on-call staff, each outage means hours of manual recovery.
Example artifact A complete, working implementation of this recipe — downloadable as a zip or browsable file by file. Generated by our build pipeline; tested with full coverage before publishing.
200 kB · 108 tests· 92.8% coverage· vitest passing
SHA-256 c772b54ebe8a86905b648ce358176c96b54498bc376255fd98b0633a5b51c1a4 Comments Sign in to commentSign in with GitHub to comment and vote.
Las Vegas, Nevada USA © 2026 REAA Technologies Inc. — Open-Source AI Solutions for Small Business.
On this page Intro
Small and medium businesses that depend on automated Xero data flows grind to a halt when APIs error or rate-limit. Without on-call staff, each outage means hours of manual recovery. This tutorial walks you through building a Xero Reliability Suite — a Next.js-powered operations hub that auto-generates incident runbooks, wraps every Xero API call with circuit breakers, rotates OAuth secrets on a cron schedule, and pushes real-time Slack alerts. By the end you’ll have a deployable dashboard that keeps your Xero integration running even when you’re not watching.
Prerequisites
Node.js 22+ and pnpm 10
A Xero app (free developer account at developer.xero.com) with OAuth 2.0 credentials
A Slack workspace with a bot token (create one at api.slack.com/apps)
An Anthropic API key for AI-powered runbook generation
Basic TypeScript familiarity — this tutorial is copy-paste-along
Step 1: Scaffold the Next.js project
Create the project and install all dependencies. The package.json uses exact version pins so your build is deterministic.
pnpm create next-app@latest xero-reliability-suite --typescript --app --src-dir --import-alias "@/*"
cd xero-reliability-suite
Open and replace the and with these exact pins:
package.json
dependencies
devDependencies
{
"dependencies" : {
"@reaatech/agent-runbook-agent" : "0.1.0" ,
"@reaatech/agent-runbook-alerts" : "0.1.0" ,
"@reaatech/agent-runbook-analyzer" : "0.1.0" ,
"@reaatech/agent-runbook-cli" : "0.1.0" ,
"@reaatech/circuit-breaker-agents" : "0.1.1" ,
"@reaatech/secret-rotation-core" : "0.1.0" ,
"@slack/web-api" : "7.17.0" ,
"@trigger.dev/sdk" : "4.4.6" ,
"dotenv" : "17.4.2" ,
"next" : "16.2.9" ,
"p-retry" : "8.0.0" ,
"react" : "19.2.4" ,
"react-dom" : "19.2.4" ,
"xero-node" : "18.0.0" ,
"zod" : "4.4.3"
},
"devDependencies" : {
"@types/node" : "22.19.21" ,
"@types/react" : "19.2.17" ,
"@types/react-dom" : "19.2.3" ,
"@vitest/coverage-v8" : "4.1.9" ,
"eslint" : "9.39.4" ,
"eslint-config-next" : "16.2.9" ,
"msw" : "2.14.6" ,
"typescript" : "5.9.3" ,
"typescript-eslint" : "8.61.1" ,
"vitest" : "4.1.9"
}
} Expected output: pnpm resolves and links all 25 packages, exits with status 0, and a pnpm-lock.yaml appears.
Add these scripts to package.json:
"scripts" : {
"dev" : "next dev" ,
"build" : "next build" ,
"start" : "next start" ,
"lint" : "eslint ." ,
"typecheck" : "tsc --noEmit" ,
"test" : "vitest run --coverage --reporter=json --outputFile=vitest-report.json"
}
Step 2: Configure environment variables Create .env.example as your team’s template and .env.local with real values for local development.
NODE_ENV=development
# Xero OAuth 2.0 credentials
XERO_CLIENT_ID=<your-xero-client-id>
XERO_CLIENT_SECRET=<your-xero-client-secret>
XERO_REDIRECT_URI=http://localhost:3000/callback
XERO_SCOPES=openid profile email accounting.settings accounting.transactions offline_access
XERO_HTTP_TIMEOUT=30000
# Slack alerting
SLACK_TOKEN=<your-slack-bot-token>
SLACK_CHANNEL=<your-slack-channel-id>
# Runbook AI provider
ANTHROPIC_API_KEY=<your-anthropic-key>
# Trigger.dev for scheduled rotation
TRIGGER_SECRET_KEY=<your-trigger-secret>
TRIGGER_API_URL=<your-trigger-api-url>
# Secret rotation schedule
SECRET_ROTATION_INTERVAL_MS=86400000
SECRET_PROVIDER_REGION=us-east-1 cp .env.example .env.local Expected output: Two files exist — .env.example (shared, placeholder values) and .env.local (gitignored, your real credentials).
Step 3: Create the TypeScript types All shared interfaces live in src/types.ts. This file defines the contracts every module depends on.
// src/types.ts
export interface XeroConfig {
clientId : string ;
clientSecret : string ;
redirectUris : string [];
scopes : string [];
httpTimeout : number ;
}
export interface CircuitBreakerConfig {
name : string ;
failureThreshold : number ;
recoveryTimeoutMs : number ;
}
export interface RotationConfig {
secretNames : string [];
rotationIntervalMs : number ;
}
export interface AlertConfig {
slackToken : string ;
slackChannel : string ;
sloTargets : {
availability : number ;
latencyP99 : number ;
};
}
export interface RunbookConfig {
repoPath : string ;
provider : string ;
model : string ;
sections : string [];
output ?: string ;
format ?: string ;
}
export interface BreakerStatus {
name : string ;
state : string ;
failureCount : number ;
lastFailure : number | null ;
lastSuccess : number | null ;
}
export interface RotationStatus {
secretName : string ;
lastRotation : number | null ;
nextRotation : number | null ;
state : string ;
}
export interface AnalysisContext {
repoPath : string ;
serviceType : string ;
language : string ;
framework : string ;
externalServices : string [];
}
export interface ApiEndpoint {
endpoint : string ;
filePath : string ;
lineNumber : number ;
} Expected output: src/types.ts compiles cleanly with no TypeScript errors.
Step 4: Build the config loader with Zod validation The src/config.ts module reads process.env and validates every value through Zod schemas, then transforms raw env strings into strongly typed configuration objects.
// src/config.ts
import dotenv from "dotenv" ;
import { z } from "zod" ;
import type { XeroConfig, AlertConfig, RotationConfig } from "./types.js" ;
dotenv. config ();
const xeroEnvFields = {
XERO_CLIENT_ID: z. string (). min ( 1 ),
XERO_CLIENT_SECRET: z. string (). min ( 1 ),
XERO_REDIRECT_URI: z. string (). min ( 1 ),
XERO_SCOPES: z
. string ()
. default (
"openid profile email accounting.settings accounting.transactions offline_access" ,
),
XERO_HTTP_TIMEOUT: z.coerce. number (). positive (). default ( 30000 ),
} as const ;
const slackEnvFields = {
SLACK_TOKEN: z. string (). default ( "" ),
SLACK_CHANNEL: z. string (). default ( "" ),
} as const ;
const rotationEnvFields = {
SECRET_ROTATION_INTERVAL_MS: z.coerce. number (). positive (). default ( 86400000 ),
SECRET_PROVIDER_REGION: z. string (). default ( "us-east-1" ),
} as const ;
export const XeroConfigSchema = z. object ({ ... xeroEnvFields });
export const SlackConfigSchema = z. object ({ ... slackEnvFields });
export const RotationConfigSchema = z. object ({ ... rotationEnvFields });
export const AppConfigSchema = z
. object ({
... xeroEnvFields,
... slackEnvFields,
... rotationEnvFields,
})
. transform ((raw) => ({
xero: {
clientId: raw.XERO_CLIENT_ID,
clientSecret: raw.XERO_CLIENT_SECRET,
redirectUris: [raw.XERO_REDIRECT_URI],
scopes: raw.XERO_SCOPES. split ( " " ). filter (Boolean),
httpTimeout: raw.XERO_HTTP_TIMEOUT,
} satisfies XeroConfig ,
slack: {
slackToken: raw.SLACK_TOKEN,
slackChannel: raw.SLACK_CHANNEL,
sloTargets: {
availability: 99.9 ,
latencyP99: 500 ,
},
} satisfies AlertConfig ,
rotation: {
secretNames: [ "xero-oauth-secret" ],
rotationIntervalMs: raw.SECRET_ROTATION_INTERVAL_MS,
} satisfies RotationConfig ,
}));
export type AppConfig = {
xero : XeroConfig ;
slack : AlertConfig ;
rotation : RotationConfig ;
};
export function loadConfig () : AppConfig {
return AppConfigSchema. parse (process.env);
}
export interface ValidateConfigResult {
success : boolean ;
config ?: AppConfig ;
error ?: string ;
}
export function validateConfig () : ValidateConfigResult {
const result = AppConfigSchema. safeParse (process.env);
if (result.success) {
return { success: true , config: result.data };
}
return { success: false , error: result.error.message };
} Expected output: Run pnpm typecheck — zero errors. The loadConfig() function either returns a typed AppConfig or throws a Zod validation error with a clear message.
Step 5: Create the Xero API client The src/xero-client.ts module wraps xero-node with a reusable XeroService class — it handles OAuth token initialization, refresh, and tenant discovery.
// src/xero-client.ts
import { XeroClient, TokenSet } from "xero-node" ;
import type { XeroConfig } from "./types.js" ;
export function createXeroClient (config : XeroConfig ) : XeroClient {
return new XeroClient ({
clientId: config.clientId,
clientSecret: config.clientSecret,
redirectUris: config.redirectUris,
scopes: config.scopes,
httpTimeout: config.httpTimeout,
});
}
export async function acquireTokenSet (
xero : XeroClient ,
tokenStore : {
get : () => TokenSet | null ;
set : (token : TokenSet ) => void ;
},
) : Promise < void > {
await xero. initialize ();
const stored = tokenStore. get ();
if (stored != null ) {
xero. setTokenSet (stored);
if (stored. expired ()) {
const newToken : TokenSet = await xero. refreshToken ();
tokenStore. set (newToken);
}
}
}
export async function getInvoices (
xero : XeroClient ,
tenantId : string ,
) : Promise < unknown > {
const result = await xero.accountingApi. getInvoices (tenantId);
return result;
}
export async function getAccounts (
xero : XeroClient ,
tenantId : string ,
) : Promise < unknown > {
const result = await xero.accountingApi. getAccounts (tenantId);
return result;
}
export async function getContacts (
xero : XeroClient ,
tenantId : string ,
) : Promise < unknown > {
const result = await xero.accountingApi. getContacts (tenantId);
return result;
}
export class XeroService {
private xero : XeroClient ;
private config : XeroConfig ;
private token : TokenSet | null = null ;
constructor (config : XeroConfig ) {
this.config = config;
this.xero = createXeroClient (config);
}
async initialize () : Promise < void > {
await this.xero. initialize ();
if (this.token != null ) {
this.xero. setTokenSet (this.token);
if (this.token. expired ()) {
this.token = await this.xero. refreshToken ();
}
}
await this.xero. updateTenants ();
}
setToken (token : TokenSet ) : void {
this.token = token;
}
getTenantId () : string {
if (Array. isArray (this.xero.tenants) && this.xero.tenants.length > 0 ) {
const first = this.xero.tenants[ 0 ] as { tenantId ?: string };
return first.tenantId ?? "" ;
}
return "" ;
}
async getInvoices (tenantId ?: string ) : Promise < unknown > {
const tid : string = tenantId ?? this. getTenantId ();
return await this.xero.accountingApi. getInvoices (tid);
}
async getAccounts (tenantId ?: string ) : Promise < unknown > {
const tid : string = tenantId ?? this. getTenantId ();
return await this.xero.accountingApi. getAccounts (tid);
}
async getContacts (tenantId ?: string ) : Promise < unknown > {
const tid : string = tenantId ?? this. getTenantId ();
return await this.xero.accountingApi. getContacts (tid);
}
} Expected output: TypeScript accepts the file. The XeroService class wraps xero-node with token refresh and tenant auto-discovery.
Step 6: Wire circuit breakers around every Xero API call The src/circuit-breakers.ts module uses @reaatech/circuit-breaker-agents to wrap each Xero endpoint (invoices, accounts, contacts) with its own circuit breaker. When a breaker trips open, the service returns a degraded response instead of crashing.
// src/circuit-breakers.ts
import { CircuitBreaker, InMemoryAdapter, DefaultMetricsCollector, CircuitOpenError } from "@reaatech/circuit-breaker-agents" ;
import pRetry from "p-retry" ;
import { CircuitBreakerConfig, BreakerStatus } from "./types.js" ;
const registry : CircuitBreaker [] = [];
export interface XeroService {
getInvoices (tenantId : string ) : Promise < unknown >;
getAccounts (tenantId : string ) : Promise < unknown >;
getContacts (tenantId : string ) : Promise < unknown
Expected output: Three circuit breakers (xero-invoices, xero-accounts, xero-contacts) register on construction. Each API call gets 3 retries via p-retry, then the breaker state transitions: closed → open → half-open → closed.
Step 7: Add AI-powered runbook scanning The src/runbook-scan.ts module uses @reaatech/agent-runbook-agent to analyze a Xero integration repository and produce a structured Markdown runbook covering failure modes, alerts, rollback procedures, and health checks.
// src/runbook-scan.ts
import { createAnalysisAgent } from "@reaatech/agent-runbook-agent" ;
import { generateRunbook } from "@reaatech/agent-runbook-cli" ;
import type { RunbookConfig, AnalysisContext } from "./types.js" ;
type AgentMethods = {
analyzeRepository (ctx : Record < string , string >) : Promise < Array < Record < string , unknown >>>;
identifyFailureModes (ctx : Record < string , string >) : Promise < string []>;
generateRunbookSection (section : string , ctx : Record < string , string >) : Promise < string >;
};
export async function scanXeroIntegration (config : RunbookConfig , analysisContext : AnalysisContext ) : Promise < string > {
const agent = createAnalysisAgent ({
provider: config.provider as "claude" | "openai" | "gemini" | "mock" ,
model: config.model,
apiKey: process.env.ANTHROPIC_API_KEY,
temperature: 0.3 ,
});
const ag : AgentMethods = agent as never ;
const ctx : Record < string , string > = {
repoPath: analysisContext.repoPath,
serviceType: analysisContext.serviceType,
language: analysisContext.language,
framework: analysisContext.framework,
};
const insights = await ag. analyzeRepository (ctx);
const failureModes = await ag. identifyFailureModes (ctx);
const alertsSection = await ag. generateRunbookSection ( "alerts" , ctx);
const rollbackSection = await ag. generateRunbookSection ( "rollback" , ctx);
const healthChecksSection = await ag. generateRunbookSection ( "health-checks" , ctx);
const runbookSections : string [] = [
"# Runbook" ,
"" ,
"## Insights" ,
];
if (Array. isArray (insights)) {
for ( const i of insights) {
const cat = typeof i.category === "string" ? i.category : "" ;
const finding = typeof i.finding === "string" ? i.finding : "" ;
const conf = typeof i.confidence === "number" ? i.confidence : 0 ;
runbookSections. push ( `- **${ cat }** (${ String ( conf ) }): ${ finding }` );
}
}
runbookSections. push ( "" , "## Failure Modes" );
if (Array. isArray (failureModes)) {
for ( const fm of failureModes) {
runbookSections. push ( `- ${ fm }` );
}
}
runbookSections. push (
"" ,
"## Alerts" ,
alertsSection,
"" ,
"## Rollback" ,
rollbackSection,
"" ,
"## Health Checks" ,
healthChecksSection,
);
return runbookSections. join ( "\n" );
}
export async function generateFullRunbook (config : RunbookConfig ) : Promise < string > {
const fn = (generateRunbook as never ) as (args : {
path : string ;
output ?: string ;
format ?: string ;
provider : string ;
model : string ;
sections : string [];
}) => Promise < Record < string , unknown >>;
const runbook = await fn ({
path: config.repoPath,
output: config.output ?? "runbook.md" ,
format: config.format ?? "markdown" ,
provider: config.provider,
model: config.model,
sections: config.sections,
});
return JSON. stringify (runbook);
} Also create the analyzer module that discovers Xero API call sites:
// src/analyzer.ts
import { scanRepository, mapDependencies, parseConfigs, analyzeCode } from "@reaatech/agent-runbook-analyzer" ;
import type { ApiEndpoint } from "./types.js" ;
interface EndpointLike {
path ?: string ;
endpoint ?: string ;
file ?: string ;
filePath ?: string ;
line ?: number ;
lineNumber ?: number ;
}
export async function analyzeXeroApiCalls (repoPath : string ) : Promise < ApiEndpoint []> {
const analysis = await scanRepository (repoPath, { depth: 5 });
mapDependencies (repoPath);
parseConfigs (repoPath);
const code = analyzeCode (repoPath, analysis.language, analysis.framework);
const endpoints : EndpointLike [] = code.apiEndpoints;
return endpoints
. filter ((ep) => {
const path = ep.path ?? ep.endpoint ?? "" ;
return path. toLowerCase (). includes ( "/api.xro/2.0/" ) && /\/invoices | \/accounts | \/contacts/ i . test (path);
})
. map ((ep) => ({
endpoint: ep.path ?? ep.endpoint ?? "" ,
filePath: ep.file ?? ep.filePath ?? "" ,
lineNumber: ep.line ?? ep.lineNumber ?? 0 ,
}));
} Expected output: pnpm typecheck passes. scanXeroIntegration() returns a multi-section Markdown document. analyzeXeroApiCalls() filters raw API endpoints to only Xero accounting API paths.
Step 8: Build the secret rotation system The src/secrets-rotation.ts module uses @reaatech/secret-rotation-core to manage OAuth key rotation with a mock provider suitable for local development.
// src/secrets-rotation.ts
import { RotationManager } from "@reaatech/secret-rotation-core" ;
import type { KeyActivatedEvent } from "@reaatech/secret-rotation-types" ;
import type { RotationConfig } from "./types.js" ;
export function createRotationManager (config : RotationConfig ) : RotationManager {
const p = createMockProvider ();
const manager = new RotationManager ({ providerInstance: p, rotationIntervalMs: config.rotationIntervalMs });
manager.events. on ( "key_activated" , (event) => {
const { secretName, keyId } = event as KeyActivatedEvent ;
console. log ( `Key activated for ${ secretName }: ${ keyId }` );
});
return manager;
}
export function createMockProvider () {
return {
name: "mock" ,
priority: 0 ,
supportsRotation : () => true ,
capabilities : () => ({ supportsRotation: true , supportsVersioning: false , supportsLabels: false }),
health : () => Promise . resolve ({ status: "healthy" as const , latency: 0 , lastChecked: new Date () }),
rotate : (n : string ) => Promise . resolve ({ version: n }),
getSecret : (n : string ) => Promise . resolve ({ value: "mock" , versionId: n, createdAt: new Date (), versionStages: [ "current" ], metadata: {} }),
beginRotation : (n : string ) => Promise . resolve ({ sessionId: "s1" , secretName: n, provider: "mock" , state: {}, startedAt: new Date () }),
completeRotation : (s : unknown ) => { void s; return Promise . resolve (); },
createSecret : (n : string , v : string ) => { void n; void v; return Promise . resolve (); },
storeSecretValue : (n : string , v : string ) => Promise . resolve ({ value: v, versionId: n, createdAt: new Date (), versionStages: [], metadata: {} }),
deleteSecret : (n : string ) => { void n; return Promise . resolve (); },
listVersions : (n : string ) => Promise . resolve ([{ versionId: n, createdAt: new Date () }]),
getVersion : (n : string , vid : string ) => Promise . resolve ({ value: n, versionId: vid, createdAt: new Date (), versionStages: [], metadata: {} }),
deleteVersion : (n : string , vid : string ) => { void n; void vid; return Promise . resolve (); },
cancelRotation : (s : unknown ) => { void s; return Promise . resolve (); },
};
}
export interface MinimalRotationManager {
rotate (secretName : string ) : Promise < unknown >;
getState (secretName : string ) : Promise < unknown >;
start (secretNames : string []) : Promise < void >;
stop () : Promise < void >;
}
export async function rotateXeroSecrets (
manager : MinimalRotationManager ,
) : Promise < unknown > {
return manager. rotate ( "xero-oauth-secret" );
}
export async function getRotationStatus (
manager : MinimalRotationManager ,
secretName : string ,
) : Promise < unknown > {
return manager. getState (secretName);
}
export async function startAutoRotation (
manager : MinimalRotationManager ,
secretNames : string [],
) : Promise < void > {
await manager. start (secretNames);
}
export async function stopAutoRotation (
manager : MinimalRotationManager ,
) : Promise < void > {
await manager. stop ();
} Then wire the cron scheduler in src/trigger-cron.ts:
// src/trigger-cron.ts
import { schedules } from "@trigger.dev/sdk" ;
import { rotateXeroSecrets, type MinimalRotationManager } from "./secrets-rotation.js" ;
import { sendRotationAlert, sendAlert } from "./slack-alerts.js" ;
export interface SlackService {
chat : { postMessage : (opts : { text : string ; channel : string }) => Promise <{ ts ?: string }> };
}
export function getTriggerSchedules () : unknown {
return schedules;
}
export function defineRotationJob (
rotateFn : () => Promise < Record < string , unknown >>,
alertFn : () => Promise < void >,
) : { start : () => void ; stop : () => void } {
return {
start () : void {
const run = async () : Promise < void > => {
try {
await rotateFn ();
await alertFn ();
} catch (err) {
console. error ( "Rotation job failed:" , err);
}
};
void run ();
},
stop () : void {
},
};
}
export function createRotationCron (
manager : MinimalRotationManager ,
web : SlackService ,
channel : string ,
intervalMs : number ,
) : { start : () => void ; stop : () => void } {
let timeout : ReturnType < typeof setTimeout> | null = null ;
let running = false ;
const tick = async () : Promise < void > => {
try {
await rotateXeroSecrets (manager);
await sendRotationAlert (web, channel, "xero-oauth-secret" , "key_activated" );
} catch (err) {
const message = err instanceof Error ? err.message : String (err);
console. error ( "Rotation cron failed:" , message);
await sendAlert (web, channel, `Rotation failed: ${ message }` );
}
};
return {
start () : void {
if (running) return ;
running = true ;
const schedule = () : void => {
timeout = setTimeout (() => {
void tick (). then (() => {
if (running) schedule ();
});
}, intervalMs);
};
schedule ();
},
stop () : void {
running = false ;
if (timeout !== null ) {
clearTimeout (timeout);
timeout = null ;
}
},
};
} Expected output: createRotationManager() returns a RotationManager wired to a mock provider. createRotationCron() returns a start/stop controller that rotates secrets on a configurable interval and posts Slack alerts on success or failure.
Step 9: Implement Slack alerting The src/slack-alerts.ts module provides typed alert functions for every event the suite generates — circuit breaker transitions, secret rotation events, and runbook generation results.
// src/slack-alerts.ts
import { WebClient } from "@slack/web-api" ;
import { extractAlerts, generateAlerts, formatAlertsForPlatform, calculateSloThresholds } from "@reaatech/agent-runbook-alerts" ;
export function createSlackClient (token : string ) : WebClient {
return new WebClient (token);
}
export async function sendAlert (
web : { chat : { postMessage : (opts : { text : string ; channel : string }) => Promise <{ ts ?: string }> } },
channel : string ,
text : string ,
) : Promise <{ ts : string }> {
if (text.length === 0 ) {
throw new Error ( "Message text must not be empty" );
}
const result = await web.chat. postMessage ({ text, channel });
return { ts: result.ts ?? "" };
}
export async function sendCircuitBreakerAlert (
web : { chat : { postMessage : (opts : { text : string ; channel : string }) => Promise < unknown > } },
channel : string ,
breakerName : string ,
state : string ,
) : Promise < void > {
const text = `⚠️ Breaker '${ breakerName }' is now ${ state }` ;
await web.chat. postMessage ({ text, channel });
}
export async function sendRotationAlert (
web : { chat : { postMessage : (opts : { text : string ; channel : string }) => Promise < unknown > } },
channel : string ,
secretName : string ,
eventType : string ,
) : Promise < void > {
const text = `🔄 Secret '${ secretName }' rotation event: ${ eventType }` ;
await web.chat. postMessage ({ text, channel });
}
export async function sendRunbookGeneratedAlert (
web : { chat : { postMessage : (opts : { text : string ; channel : string }) => Promise < unknown > } },
channel : string ,
runbookTitle : string ,
) : Promise < void > {
const text = `📗 Runbook generated: '${ runbookTitle }'` ;
await web.chat. postMessage ({ text, channel });
}
function invoke < TResult >( fn : ( ... args : never []) => TResult , ... args : never []) : TResult {
return fn ( ... args);
}
export function generateAlertDefinitions (context : Record < string , unknown >) : string {
void calculateSloThresholds ({ availability: 99.9 , latencyP99: 500 });
void extractAlerts ( "" );
const config = { sloTargets: { availability: 99.9 , latencyP99: 500 } as const , platform: "prometheus" as const };
const alerts = invoke < never >(generateAlerts as never , context as never , config as never );
return invoke < string >(formatAlertsForPlatform, alerts, "prometheus" as never );
} Expected output: You can call sendAlert(web, "C123456", "Hello") and it posts to Slack. generateAlertDefinitions() returns Prometheus-compatible alert rule YAML derived from SLO targets.
Step 10: Assemble the XeroReliabilitySuite glue The src/glue.ts module is the central orchestrator — it wires XeroService, ProtectedXeroClient, Slack, secret rotation, and runbook scanning into a single XeroReliabilitySuite class.
// src/glue.ts
import { WebClient } from "@slack/web-api" ;
import type {
AlertConfig,
BreakerStatus,
} from "./types.js" ;
import { createRotationManager, rotateXeroSecrets, startAutoRotation, stopAutoRotation } from "./secrets-rotation.js" ;
import { createSlackClient } from "./slack-alerts.js" ;
import { createRotationCron } from "./trigger-cron.js" ;
import { ProtectedXeroClient, getBreakerStatuses, resetBreaker as resetSingleBreaker } from "./circuit-breakers.js" ;
import { analyzeXeroApiCalls } from "./analyzer.js" ;
import { scanXeroIntegration } from "./runbook-scan.js" ;
import { type AppConfig }
Create the singleton factory in app/api/_lib/suite.ts:
// app/api/_lib/suite.ts
import { XeroReliabilitySuite } from "@/src/glue.js" ;
import { loadConfig } from "@/src/config.js" ;
let suite : XeroReliabilitySuite | null = null ;
export function getSuite () : XeroReliabilitySuite {
if ( ! suite) {
suite = new XeroReliabilitySuite ( loadConfig ());
}
return suite;
} Expected output: XeroReliabilitySuite compiles cleanly. The getSuite() singleton lazily initializes once, loading config through Zod validation.
Step 11: Build the API routes Create the Next.js App Router route handlers. Each route delegates to getSuite() and returns typed JSON responses via NextResponse.json().
// app/api/dashboard/route.ts
import { NextRequest, NextResponse } from "next/server" ;
import { getSuite } from "../_lib/suite.js" ;
export function GET (_req : NextRequest ) {
void _req;
try {
const data = getSuite (). getDashboardData ();
return NextResponse. json (data);
} catch (error) {
return NextResponse. json ({ error: (error as Error ).message }, { status: 500 });
}
} // app/api/circuit-breaker/status/route.ts
import { NextRequest, NextResponse } from "next/server" ;
import { getSuite } from "../../_lib/suite.js" ;
export function GET (_req : NextRequest ) {
void _req;
try {
const data = getSuite (). getDashboardData ();
return NextResponse. json ({ breakers: data.breakers });
} catch (error) {
return NextResponse. json ({ error: (error as Error ).message }, { status: 500 });
}
} // app/api/circuit-breaker/reset/route.ts
import { NextRequest, NextResponse } from "next/server" ;
import { getSuite } from "../../_lib/suite.js" ;
export async function POST (req : NextRequest ) {
try {
const { name } = await req. json () as { name : string };
if ( ! name) {
return NextResponse. json ({ error: "Missing required field: name" }, { status: 400 });
}
getSuite (). resetBreaker (name);
return NextResponse. json ({ success: true });
} catch (error) {
return NextResponse. json ({ error: (error as Error ).message }, { status: 500 });
}
} // app/api/runbook/generate/route.ts
import { NextRequest, NextResponse } from "next/server" ;
import { getSuite } from "../../_lib/suite.js" ;
export async function POST (req : NextRequest ) {
try {
const { repoPath } = await req. json () as { repoPath : string };
if ( ! repoPath) {
return NextResponse. json ({ error: "Missing required field: repoPath" }, { status: 400 });
}
await getSuite (). triggerRunbookScan (repoPath);
const data = getSuite (). getDashboardData ();
return NextResponse. json ({ runbook: data.lastRunbook });
} catch (error) {
return NextResponse. json ({ error: (error as Error ).message }, { status: 500 });
}
} // app/api/runbook/latest/route.ts
import { NextRequest, NextResponse } from "next/server" ;
import { getSuite } from "../../_lib/suite.js" ;
export function GET (_req : NextRequest ) {
void _req;
try {
const data = getSuite (). getDashboardData ();
return NextResponse. json ({ runbook: data.lastRunbook });
} catch (error) {
return NextResponse. json ({ error: (error as Error ).message }, { status: 500 });
}
} // app/api/secrets/rotate/route.ts
import { NextRequest, NextResponse } from "next/server" ;
import { getSuite } from "../../_lib/suite.js" ;
export async function POST (_req : NextRequest ) {
void _req;
try {
const result = await getSuite (). triggerManualRotation ();
return NextResponse. json ({ success: true , result });
} catch (error) {
return NextResponse. json ({ error: (error as Error ).message }, { status: 500 });
}
} // app/api/secrets/status/route.ts
import { NextRequest, NextResponse } from "next/server" ;
import { getSuite } from "../../_lib/suite.js" ;
export function GET (_req : NextRequest ) {
void _req;
try {
const data = getSuite (). getDashboardData ();
return NextResponse. json ({ status: data.rotations });
} catch (error) {
return NextResponse. json ({ error: (error as Error ).message }, { status: 500 });
}
} // app/api/secrets/start/route.ts
import { NextRequest, NextResponse } from "next/server" ;
import { getSuite } from "../../_lib/suite.js" ;
export function POST (_req : NextRequest ) {
void _req;
try {
void getSuite (). startAll ();
return NextResponse. json ({ success: true });
} catch (error) {
return NextResponse. json ({ error: (error as Error ).message }, { status: 500 });
}
} // app/api/secrets/stop/route.ts
import { NextRequest, NextResponse } from "next/server" ;
import { getSuite } from "../../_lib/suite.js" ;
export function POST (_req : NextRequest ) {
void _req;
try {
getSuite (). stopAll ();
return NextResponse. json ({ success: true });
} catch (error) {
return NextResponse. json ({ error: (error as Error ).message }, { status: 500 });
}
} Expected output: 9 route handlers covering dashboard, circuit breaker status/reset, runbook generate/latest, secrets rotate/status/start/stop. Run pnpm typecheck — zero errors. All handlers use NextRequest/NextResponse from next/server.
Step 12: Create the dashboard page The dashboard page at app/dashboard/page.tsx is a server component that fetches from GET /api/dashboard and renders circuit breaker states, rotation status, and action buttons.
// app/dashboard/page.tsx
async function getDashboardData () {
const baseUrl = process.env.NEXT_PUBLIC_API_URL ?? "http://localhost:3000" ;
const res = await fetch ( `${ baseUrl }/api/dashboard` , { cache: "no-store" });
if ( ! res.ok) {
throw new Error ( `Dashboard API returned ${ String ( res . status ) }` );
}
return res. json () as Promise <{
breakers : { name : string ; state : string ; failureCount : number
Update the landing page at app/page.tsx:
// app/page.tsx
import Link from "next/link" ;
export default function Home () {
return (
<main style ={{ padding: "2rem" , maxWidth: "48rem" , margin: "0 auto" }}>
<h1>Xero Reliability Suite</h1>
<p style ={{ marginTop: "0.5rem" , fontSize: "1.125rem" }}>
Monitoring, circuit breaking, secret rotation, and runbook automation for SMB Xero accounting operations.
</p>
<div style ={{ marginTop: "2rem" }}>
< Link
href = "/dashboard"
style ={{
display: "inline-block" ,
padding: "0.75rem 1.5rem" ,
borderRadius: "0.375rem" ,
background: "#0070f3" ,
color: "#fff" ,
fontWeight: 600 ,
}}
>
View Dashboard
</ Link >
</div>
</main>
);
} Expected output: Visit http://localhost:3000/dashboard — the page renders breaker cards with color-coded states (green for closed, red for open, amber for half-open), a rotation status section, and action buttons for runbook generation and manual secret rotation.
Step 13: Run the tests The test suite covers every module with happy-path, error, and boundary cases. Here’s a sample of the circuit-breaker tests to show the pattern:
// tests/circuit-breakers.test.ts
import { describe, it, expect, vi, beforeEach } from "vitest" ;
import {
createBreaker,
ProtectedXeroClient,
getBreakerStatuses,
} from "../src/circuit-breakers.js" ;
const { mockExecute, mockGetState, mockGetStats, mockReset, MockCircuitBreaker, MockCircuitOpenError } = vi. hoisted (() => {
const mockExecute = vi. fn ();
const mockGetState = vi. fn ();
const mockGetStats = vi. fn ();
const mockReset = vi. fn ();
class MockCircuitOpenError extends Error {
Expected output: All 33 test suites pass with 108 tests. Coverage exceeds 90% across lines, branches, and functions.
✓ tests/analyzer.test.ts
✓ tests/config.test.ts
✓ tests/circuit-breakers.test.ts
✓ tests/xero-client.test.ts
✓ tests/slack-alerts.test.ts
✓ tests/runbook-scan.test.ts
✓ tests/secrets-rotation.test.ts
✓ tests/trigger-cron.test.ts
✓ tests/glue.test.ts
✓ tests/app/api/dashboard.test.ts
✓ tests/app/api/runbook/generate.test.ts
✓ tests/app/api/circuit-breaker/status.test.ts
✓ tests/app/api/circuit-breaker/reset.test.ts
✓ tests/app/api/secrets/rotate.test.ts
✓ tests/integration/reliability.test.ts
Test Files 15 passed (15)
Tests 108 passed (108)
Coverage 90%+ lines, 90%+ branches, 90%+ functions
Next steps
Swap the mock secret provider for a real AWS Secrets Manager or HashiCorp Vault provider by implementing the SecretProvider interface from @reaatech/secret-rotation-core
Add a Prometheus metrics endpoint at /api/metrics that exposes breaker state and rotation latency as Prometheus gauge metrics for Grafana dashboards
Extend the runbook scanning to cover additional Xero API resources like BankTransactions, CreditNotes, and PurchaseOrders by adding them to the analyzeXeroApiCalls() filter
>;
}
export function createBreaker (name : string , failureThreshold : number , recoveryTimeoutMs : number ) : CircuitBreaker {
const breaker = new CircuitBreaker ({
name,
failureThreshold,
recoveryTimeoutMs,
persistence: new InMemoryAdapter (),
metricsCollector: new DefaultMetricsCollector (),
});
registry. push (breaker);
return breaker;
}
export class ProtectedXeroClient {
private readonly xeroService : XeroService ;
private readonly invoicesBreaker : CircuitBreaker ;
private readonly accountsBreaker : CircuitBreaker ;
private readonly contactsBreaker : CircuitBreaker ;
private readonly breakerMap : Record < string , CircuitBreaker >;
constructor (xeroService : XeroService , breakerConfig ?: Partial < CircuitBreakerConfig >) {
this.xeroService = xeroService;
const config = {
name: "xero" ,
failureThreshold: breakerConfig?.failureThreshold ?? 5 ,
recoveryTimeoutMs: breakerConfig?.recoveryTimeoutMs ?? 30000 ,
} satisfies CircuitBreakerConfig ;
this.invoicesBreaker = createBreaker ( "xero-invoices" , config.failureThreshold, config.recoveryTimeoutMs);
this.accountsBreaker = createBreaker ( "xero-accounts" , config.failureThreshold, config.recoveryTimeoutMs);
this.contactsBreaker = createBreaker ( "xero-contacts" , config.failureThreshold, config.recoveryTimeoutMs);
this.breakerMap = {
"xero-invoices" : this.invoicesBreaker,
"xero-accounts" : this.accountsBreaker,
"xero-contacts" : this.contactsBreaker,
};
}
getBreakers () : CircuitBreaker [] {
return [this.invoicesBreaker, this.accountsBreaker, this.contactsBreaker];
}
getBreaker (name : string ) : CircuitBreaker | undefined {
return this.breakerMap[name];
}
async getInvoices (tenantId : string ) : Promise < unknown > {
try {
return await this.invoicesBreaker. execute (() =>
pRetry (() => this.xeroService. getInvoices (tenantId), { retries: 3 })
);
} catch (error) {
if (error instanceof CircuitOpenError ) {
return { degraded: true , breaker: "xero-invoices" , message: "Circuit open — Xero API temporarily unavailable" };
}
throw error;
}
}
async getAccounts (tenantId : string ) : Promise < unknown > {
try {
return await this.accountsBreaker. execute (() =>
pRetry (() => this.xeroService. getAccounts (tenantId), { retries: 3 })
);
} catch (error) {
if (error instanceof CircuitOpenError ) {
return { degraded: true , breaker: "xero-accounts" , message: "Circuit open — Xero API temporarily unavailable" };
}
throw error;
}
}
async getContacts (tenantId : string ) : Promise < unknown > {
try {
return await this.contactsBreaker. execute (() =>
pRetry (() => this.xeroService. getContacts (tenantId), { retries: 3 })
);
} catch (error) {
if (error instanceof CircuitOpenError ) {
return { degraded: true , breaker: "xero-contacts" , message: "Circuit open — Xero API temporarily unavailable" };
}
throw error;
}
}
}
export function getBreakerStatuses () : BreakerStatus [] {
return registry. map ((breaker) => {
const stats = breaker. getStats ();
return {
name: stats.circuit_id,
state: stats.state,
failureCount: stats.failure_count,
lastFailure: stats.last_failure_time ?? null ,
lastSuccess: null ,
};
});
}
export function resetBreaker (breaker : CircuitBreaker ) : void {
breaker. reset ();
}
from
"./config.js"
;
import { XeroService } from "./xero-client.js" ;
export class XeroReliabilitySuite {
private readonly xeroService : XeroService ;
private readonly protectedClient : ProtectedXeroClient ;
private readonly slackClient : WebClient ;
private readonly rotationManager : ReturnType < typeof createRotationManager>;
private readonly cron : ReturnType < typeof createRotationCron>;
private readonly alertConfig : AlertConfig ;
private lastRunbook : string | null = null ;
constructor (config : AppConfig ) {
this.xeroService = new XeroService (config.xero);
this.protectedClient = new ProtectedXeroClient (this.xeroService);
this.slackClient = createSlackClient (config.slack.slackToken);
this.rotationManager = createRotationManager (config.rotation);
this.alertConfig = config.slack;
this.cron = createRotationCron (
this.rotationManager,
this.slackClient,
config.slack.slackChannel,
config.rotation.rotationIntervalMs,
);
}
getDashboardData () : {
breakers : BreakerStatus [];
rotations : Record < string , unknown >;
lastRunbook : string | null ;
} {
let breakers : BreakerStatus [] = [];
try {
breakers = getBreakerStatuses ();
} catch {
breakers = [];
}
return { breakers, rotations: {}, lastRunbook: this.lastRunbook };
}
async triggerRunbookScan (repoPath : string ) : Promise < string > {
try {
await analyzeXeroApiCalls (repoPath);
const analysisContext = {
repoPath,
serviceType: "web-api" ,
language: "typescript" ,
framework: "nextjs" ,
externalServices: [ "xero" ],
};
const runbook = await scanXeroIntegration (
{ repoPath, provider: "claude" , model: "claude-sonnet-4-6" , sections: [ "alerts" , "rollback" , "health-checks" ] },
analysisContext,
);
this.lastRunbook = runbook;
return runbook;
} catch (err) {
const message = err instanceof Error ? err.message : String (err);
this.lastRunbook = `# Runbook generation failed\\n\\n${ message }` ;
return this.lastRunbook;
}
}
async triggerManualRotation () : Promise < unknown > {
return rotateXeroSecrets (this.rotationManager);
}
resetBreaker (name : string ) : void {
const breaker = this.protectedClient. getBreaker (name);
if (breaker) resetSingleBreaker (breaker);
}
stopAll () : void {
void stopAutoRotation (this.rotationManager);
this.cron. stop ();
}
async startAll () : Promise < void > {
await startAutoRotation (this.rotationManager, [ "xero-oauth-secret" ]);
this.cron. start ();
}
}
; lastFailure
:
number
|
null
; lastSuccess
:
number
|
null
}[];
rotations : { secretName : string ; lastRotation : number | null ; nextRotation : number | null ; state : string };
lastRunbook : string | null ;
}>;
}
function formatTimestamp (ts : number | null ) : string {
if (ts === null ) return "N/A" ;
return new Date (ts). toLocaleString ();
}
export default async function Dashboard () {
let data;
let error : string | null = null ;
try {
data = await getDashboardData ();
} catch (e) {
error = (e as Error ).message;
}
return (
<main style ={{ padding: "2rem" , maxWidth: "64rem" , margin: "0 auto" }}>
<h1>Reliability Dashboard</h1>
{ error && (
<div style ={{ padding: "1rem" , background: "#fee2e2" , borderRadius: "0.375rem" , marginTop: "1rem" }}>
<strong>Error:</strong> { error }
</div>
)}
{ data && (
<>
<section style ={{ marginTop: "2rem" }}>
<h2>Circuit Breakers</h2>
<div style ={{ display: "grid" , gridTemplateColumns: "repeat(auto-fill, minmax(18rem, 1fr))" , gap: "1rem" , marginTop: "1rem" }}>
{ data . breakers .length === 0 && <p>No breakers configured.</p>}
{ data . breakers . map (( breaker ) => (
<div
key ={ breaker . name }
style ={{
padding: "1rem" ,
borderRadius: "0.5rem" ,
border: "1px solid #e5e7eb" ,
background: breaker . state === "open" ? "#fef2f2" : breaker . state === "half-open" ? "#fffbeb" : "#f0fdf4" ,
}}
>
<h3 style ={{ margin: 0 }}>{ breaker . name }</h3>
<dl style ={{ marginTop: "0.5rem" , fontSize: "0.875rem" }}>
<dt>State</dt>
<dd><strong>{ breaker . state }</strong></dd>
<dt>Failure Count</dt>
<dd>{ breaker . failureCount }</dd>
<dt>Last Failure</dt>
<dd>{ formatTimestamp ( breaker . lastFailure )}</dd>
<dt>Last Success</dt>
<dd>{ formatTimestamp ( breaker . lastSuccess )}</dd>
</dl>
<form action = "/api/circuit-breaker/reset" method = "POST" style ={{ marginTop: "0.5rem" }}>
<input type = "hidden" name = "name" value ={ breaker . name } />
<button type = "submit" style ={{ padding: "0.25rem 0.75rem" , borderRadius: "0.25rem" , border: "1px solid #d1d5db" , background: "#fff" , cursor: "pointer" , fontSize: "0.75rem" }}>Reset</button>
</form>
</div>
))}
</div>
</section>
<section style ={{ marginTop: "2rem" }}>
<h2>Rotation Status</h2>
<div style ={{ padding: "1rem" , borderRadius: "0.5rem" , border: "1px solid #e5e7eb" , marginTop: "1rem" }}>
<dl>
<dt>Secret</dt>
<dd><strong>{ data . rotations . secretName }</strong></dd>
<dt>State</dt>
<dd>{ data . rotations . state }</dd>
<dt>Last Rotation</dt>
<dd>{ formatTimestamp ( data . rotations . lastRotation )}</dd>
<dt>Next Rotation</dt>
<dd>{ formatTimestamp ( data . rotations . nextRotation )}</dd>
</dl>
</div>
</section>
<section style ={{ marginTop: "2rem" }}>
<h2>Actions</h2>
<div style ={{ display: "flex" , gap: "1rem" , marginTop: "1rem" , flexWrap: "wrap" }}>
<form action = "/api/runbook/generate" method = "POST" >
<input type = "hidden" name = "repoPath" value = "/home/repo" />
<button
type = "submit"
style ={{
padding: "0.5rem 1rem" ,
borderRadius: "0.375rem" ,
border: "none" ,
background: "#0070f3" ,
color: "#fff" ,
fontWeight: 600 ,
cursor: "pointer" ,
}}
>
Generate Runbook
</button>
</form>
<form action = "/api/secrets/rotate" method = "POST" >
<button
type = "submit"
style ={{
padding: "0.5rem 1rem" ,
borderRadius: "0.375rem" ,
border: "none" ,
background: "#059669" ,
color: "#fff" ,
fontWeight: 600 ,
cursor: "pointer" ,
}}
>
Rotate Secrets Now
</button>
</form>
</div>
</section>
</>
)}
</main>
);
}
constructor
() {
super( "Circuit open" );
}
}
class MockCircuitBreaker {
readonly name : string ;
readonly failureThreshold : number ;
readonly recoveryTimeoutMs : number ;
execute = mockExecute;
getState = mockGetState;
getStats = mockGetStats;
reset = mockReset;
constructor (opts : { name : string ; failureThreshold : number ; recoveryTimeoutMs : number }) {
this.name = opts.name;
this.failureThreshold = opts.failureThreshold;
this.recoveryTimeoutMs = opts.recoveryTimeoutMs;
}
}
return { mockExecute, mockGetState, mockGetStats, mockReset, MockCircuitBreaker, MockCircuitOpenError };
});
vi. mock ( "@reaatech/circuit-breaker-agents" , () => ({
CircuitBreaker: MockCircuitBreaker,
InMemoryAdapter: vi. fn (),
DefaultMetricsCollector: vi. fn (),
CircuitOpenError: MockCircuitOpenError,
}));
vi. mock ( "p-retry" , () => ({
default: vi. fn ( async ( fn : () => unknown ) => await fn ()),
}));
describe ( "circuit-breakers" , () => {
beforeEach (() => {
vi. clearAllMocks ();
});
it ( "happy: ProtectedXeroClient.getInvoices() returns data normally when breaker is closed" , async () => {
const invoiceData = { body: { invoices: [{ invoiceID: "inv-1" }] } };
mockExecute. mockImplementation ( async ( fn : () => unknown ) => await fn ());
mockGetState. mockReturnValue ( "closed" );
const xeroService = {
getInvoices: vi. fn (). mockResolvedValue (invoiceData),
getAccounts: vi. fn (),
getContacts: vi. fn (),
};
const client = new ProtectedXeroClient (xeroService);
const result = ( await client. getInvoices ( "tenant-1" )) as { body : { invoices : Array <{ invoiceID : string }> } };
expect (result). toBeDefined ();
expect (result.body.invoices[ 0 ].invoiceID). toBe ( "inv-1" );
});
it ( "error: circuit breaker open returns degraded response" , async () => {
mockExecute. mockRejectedValue ( new MockCircuitOpenError ());
const xeroService = {
getInvoices: vi. fn (),
getAccounts: vi. fn (),
getContacts: vi. fn (),
};
const client = new ProtectedXeroClient (xeroService);
const result = ( await client. getInvoices ( "tenant-1" )) as { degraded : boolean ; breaker : string };
expect (result). toBeDefined ();
expect (result.degraded). toBe ( true );
});
it ( "boundary: error from Xero propagates when circuit is closed" , async () => {
const apiError = new Error ( "Xero API error" );
mockExecute. mockRejectedValue (apiError);
const xeroService = {
getInvoices: vi. fn (),
getAccounts: vi. fn (),
getContacts: vi. fn (),
};
const client = new ProtectedXeroClient (xeroService);
await expect (client. getInvoices ( "tenant-1" )).rejects. toThrow ( "Xero API error" );
});
it ( "happy: getBreakerStatuses() returns state for all breakers" , () => {
mockGetState. mockReturnValue ( "closed" );
mockGetStats. mockReturnValue ({
circuit_id: "xero-invoices" ,
state: "closed" ,
failure_count: 0 ,
last_failure_time: null ,
});
createBreaker ( "xero-invoices" , 5 , 30000 );
const statuses = getBreakerStatuses ();
expect (Array. isArray (statuses)). toBe ( true );
statuses. forEach ((s) => {
expect (s.name). toBeDefined ();
expect (s.state). toBeDefined ();
});
});
it ( "happy: resetBreaker() calls breaker.reset()" , async () => {
const { resetBreaker } = await import ( "../src/circuit-breakers.js" );
const breaker = createBreaker ( "test" , 5 , 30000 );
resetBreaker (breaker);
expect (mockReset). toHaveBeenCalled ();
});
});