Files · Google Gemini Runbook Automation for PagerDuty SMB Incidents

# Google Gemini Runbook Automation for PagerDuty SMB Incidents
 
Generate up-to-date incident runbooks for every PagerDuty-monitored service so your small team always knows how to respond during an outage.
 
## Problem
 
Small DevOps teams using PagerDuty often have few written runbooks because writing and maintaining them is time-consuming. When a critical incident hits, responders waste precious minutes guessing recovery steps instead of following a documented plan.
 
## Architecture
 
This CLI-powered automation pulls PagerDuty incident and service metadata, then uses `@reaatech/agent-runbook` modules (analyzer, alerts, health-checks) to produce a complete runbook. The `@reaatech/confidence-router-core` determines if an existing runbook is similar enough to skip generation (avoiding duplicate noise). `@reaatech/agents-markdown` utilities format the output into clean markdown, and an Inngest workflow triggers the CLI on a schedule and on PagerDuty webhook events.
 
### Components
 
- **src/lib/pagerduty-client.ts** — REST client for PagerDuty API (incidents, services, escalation policies)
- **src/lib/llm.ts** — Gemini LLM service with Langfuse tracing
- **src/cli/runbook-gen.ts** — CLI entry point that orchestrates the full runbook generation pipeline
- **src/inngest/trigger-runbook-gen.ts** — Inngest functions for scheduled (every 6h) and webhook-triggered generation
- **app/api/runbooks/** — Next.js route handlers listing/serving generated runbooks
 
## Environment Variables
 
| Variable | Description |
|----------|-------------|
| `GEMINI_API_KEY` | Google Gemini API key |
| `PAGERDUTY_API_KEY` | PagerDuty REST API token |
| `INNGEST_EVENT_KEY` | Inngest event key |
| `INNGEST_SIGNING_KEY` | Inngest signing key |
| `LANGFUSE_PUBLIC_KEY` | Langfuse public key (tracing) |
| `LANGFUSE_SECRET_KEY` | Langfuse secret key |
| `LANGFUSE_HOST` | Langfuse host URL |
| `RUNBOOKS_OUTPUT_DIR` | Directory for generated runbooks (default: `./src/runbooks`) |
| `CONFIDENCE_ROUTE_THRESHOLD` | Threshold for "definitely new" runbook (default: `0.8`) |
| `CONFIDENCE_FALLBACK_THRESHOLD` | Threshold for "definitely duplicate" (default: `0.3`) |
 
## Inngest Integration
 
Inngest triggers the runbook generator on a scheduled basis (every 6 hours via cron `0 */6 * * *`) and via PagerDuty webhook events.
 
### Scheduled Generation
 
The `scheduledRunbookGen` function fetches all PagerDuty services and generates a runbook for each, skipping any that are duplicates (based on `@reaatech/confidence-router-core`).
 
### Webhook Generation
 
The `pagerdutyWebhook` function listens for `pagerduty/incident.triggered` events. The expected webhook payload format:
 
```json
{
  "name": "pagerduty/incident.triggered",
  "data": {
    "serviceId": "SERVICE_ID"
  }
}
```
 
Webhook events are sent via Inngest when PagerDuty fires an incident trigger.
 
## Runbook Output
 
Generated runbooks are saved as markdown (`.md`) files in the configured output directory (default: `./src/runbooks/`). The output can be viewed via the API at `/api/runbooks/`.
 
## CLI Usage
 
```bash
pnpm tsx src/cli/runbook-gen.ts --service-id <id> [--repo-path <path>] [--force] [--output-dir <dir>]
```
 
- `--service-id` — PagerDuty service ID (required)
- `--repo-path` — optional path to a repository for scanning
- `--force` — skip duplicate detection
- `--output-dir` — override the output directory
 
## Getting Started
 
```bash
cp .env.example .env
# Fill in your API keys in .env
pnpm install
pnpm tsx src/cli/runbook-gen.ts --service-id <your-service-id>
pnpm dev  # start Next.js dev server for API
```
 
## Running tests
 
```bash
pnpm test
```
 
## Project layout
 
```
app/api/               Next.js App Router API routes
app/api/runbooks/      Generated runbook listing + detail endpoints
app/api/health/        Health check endpoint
app/api/inngest/       Inngest serve handler
src/lib/               Service classes (PagerDuty, Gemini LLM)
src/cli/               CLI entry point (runbook-gen.ts)
src/inngest/           Inngest function definitions
src/runbooks/          Generated runbook output directory (gitignored)
tests/                 Vitest test suite mirroring src/
packages/              API references for every dependency
DEV_PLAN.md            Build plan for this recipe
```
 
## License
 
MIT — see [LICENSE](./LICENSE).