@reaatech/pi-bench-corpus
Status: Pre-1.0 — APIs may change in minor versions. Pin to a specific version in production.
Attack corpus builder, validator, and variant generation engine for prompt-injection-bench. Generates balanced, validated injection samples from YAML templates with synonym and obfuscation strategies.
Installation
terminal
npm install @reaatech/pi-bench-corpus
# or
pnpm add @reaatech/pi-bench-corpusFeature Overview
- Template-based generation — Build corpora from YAML attack templates with variable substitution
- Variant strategies — Synonym replacement, Unicode homoglyphs, whitespace tricks, case manipulation, comment injection
- Automatic balancing — Ensure even distribution across all 8 attack categories
- Validation — Detect duplicate samples, harmful content patterns, severity distribution gaps
- Manifest generation — Versioned snapshots with hash proofs for reproducibility
- Dual ESM/CJS output — works with
importandrequire
Quick Start
typescript
import {
generateDefaultCorpus,
createCorpusBuilder,
createCorpusValidator,
} from "@reaatech/pi-bench-corpus";
// Generate a balanced 40-sample-per-category default corpus
const corpus = generateDefaultCorpus();
console.log(`Generated ${corpus.length} injection samples`);
// Build a custom corpus
const builder = createCorpusBuilder({ samplesPerCategory: 100, minSeverity: "medium" });
builder.generateFromTemplates(customTemplates);
builder.balance();
const samples = builder.getSamples();
const manifest = builder.getManifest();
// Validate
const validator = createCorpusValidator();
const result = validator.validate(samples);
if (!result.valid) {
for (const err of result.errors) console.error(err);
}API Reference
CorpusBuilder
| Method | Description |
|---|---|
addSamples(samples) | Add pre-built samples to the corpus |
generateFromTemplates(templates) | Generate samples from attack template definitions |
balance() | Trim categories to equal size for even distribution |
getSamples() | Return all corpus samples |
getSamplesByCategory(category) | Filter samples by attack category |
getManifest() | Generate a CorpusManifest with counts and distribution |
CorpusConfig
| Property | Type | Default | Description |
|---|---|---|---|
version | string | 2026.04 | Corpus version identifier |
samplesPerCategory | number | 40 | Target samples per category |
minSeverity | SeverityLevel | low | Minimum severity to include |
createCorpusBuilder(config?)
Factory function. Returns a CorpusBuilder instance.
CorpusValidator
| Method | Description |
|---|---|
validate(samples) | Run full validation suite, returns ValidationResult |
ValidationResult
| Property | Type | Description |
|---|---|---|
valid | boolean | Whether the corpus passes all checks |
errors | string[] | Blocking validation errors |
warnings | string[] | Non-blocking warnings |
stats | object | Sample counts, severity and category breakdowns |
createCorpusValidator(config?)
Factory function.
TemplateEngine
| Export | Description |
|---|---|
generateVariants(template) | Generate variant strings from an AttackTemplate |
templateToSample(template, index) | Convert template + variant index into an InjectionSample |
generateSamplesFromTemplate(template) | Generate all samples for a single template |
validateTemplate(unknown) | Validate a raw template object |
createTemplate(definition) | Build a template from a definition object |
applyVariantStrategies(prompt, strategies) | Apply obfuscation strategies to a prompt |
Usage Patterns
Loading from YAML Templates
typescript
import { parse } from "yaml";
import { readFileSync } from "node:fs";
import { createCorpusBuilder } from "@reaatech/pi-bench-corpus";
const raw = readFileSync("corpus/examples/role-playing.yaml", "utf8");
const templates = parse(raw).templates;
const builder = createCorpusBuilder({ samplesPerCategory: 20 });
builder.generateFromTemplates(templates);Validation in CI
typescript
import { generateDefaultCorpus, createCorpusValidator } from "@reaatech/pi-bench-corpus";
const corpus = generateDefaultCorpus();
const result = createCorpusValidator().validate(corpus);
if (!result.valid) {
console.error("Corpus validation failed");
process.exit(1);
}Related Packages
@reaatech/pi-bench-core— Core types and taxonomyprompt-injection-bench— CLI and umbrella package