Files · Agent Feedback Loop for Automated Fine-Tune Dataset Generation

# Agent Feedback Loop for Automated Fine-Tune Dataset Generation
 
> Collect agent decisions and user corrections to build a dataset for model fine-tuning.
 
A boutique marketing agency uses an AI agent to generate ad copy. The agent often misses the brand voice, requiring manual edits. This system logs agent outputs, captures user feedback (accept/reject/edit), and periodically exports a clean dataset for fine-tuning. This reduces reliance on expensive API calls and improves quality over time.
 
## Tech Stack
 
- **`ai` + `@ai-sdk/openai`** — Provider-agnostic LLM interface (swap provider by changing `AGENT_MODEL`)
- **`@reaatech/agent-replay-core`** — Trace recording and replay engine for agent interactions
- **`@reaatech/agent-replay-interceptors`** — Transparent SDK interception to record LLM calls
- **`@reaatech/agent-replay-integrations`** — State adapters for LangChain/LangGraph framework state capture
- **`@reaatech/agent-replay-cli`** — Programmatic trace replay in stubbed/live/diff modes
- **`@reaatech/agent-eval-harness-golden`** — Golden trajectory management and comparison
- **`@reaatech/agent-eval-harness-suite`** — Batch evaluation suite runner with results aggregation
- **`drizzle-orm` + `postgres`** — Type-safe SQL database layer
- **`langfuse`** — LLM observability tracing
- **`zod`** — Request/response schema validation
 
## Getting Started
 
```bash
cp .env.example .env
# Fill in DATABASE_URL, OPENAI_API_KEY, LANGFUSE_* keys
pnpm install
pnpm dev
```
 
## API Reference
 
### `POST /api/agent/generate`
 
Generate ad copy from an LLM agent.
 
**Body:** `{ "prompt": "string", "brandVoice?": "string", "tone?": "string", "sessionId?": "string" }`
**Response 200:** `{ "data": { "id": "uuid", "prompt": "string", "output": "string", "modelUsed": "string", "inputTokens": "number", "outputTokens": "number", "durationMs": "number", "tracePath?": "string", "createdAt": "date" } }`
**Errors:** 400 validation, 500 service error
 
### `POST /api/agent/feedback`
 
Record user feedback on an agent run.
 
**Body:** `{ "agentRunId": "uuid", "decision": "accept|reject|edit", "correctedOutput?": "string", "userNotes?": "string" }`
**Response 201:** `{ "data": { "id": "uuid", "agentRunId": "uuid", "decision": "string", "correctedOutput?": "string", "userNotes?": "string", "createdAt": "date" } }`
**Errors:** 400 validation, 404 agent run not found
 
### `GET /api/datasets`
 
List all fine-tune datasets.
 
**Response 200:** `{ "data": [ { "id": "uuid", "name": "string", "status": "string", "recordCount": "number", "exportFormat": "string", "createdAt": "date", "updatedAt": "date" } ] }`
 
### `POST /api/datasets`
 
Create a new dataset.
 
**Body:** `{ "name": "string", "description?": "string", "maxExamples?": "number", "exportFormat?": "jsonl" }`
**Response 201:** `{ "data": { DatasetRecord } }`
 
### `POST /api/datasets/:id/generate`
 
Build the dataset from accepted/corrected feedback records.
 
**Response 200:** `{ "data": { "recordCount": "number", "format": "string" } }`
**Errors:** 404 dataset not found
 
### `GET /api/datasets/:id/export`
 
Export the dataset as JSONL fine-tuning examples.
 
**Response 200:** `{ "data": [ { "messages": [ { "role": "user", "content": "string" }, { "role": "assistant", "content": "string" } ] } ], "recordCount": "number" }`
**Errors:** 404 dataset not found
 
## Project Layout
 
```
app/api/              Next.js App Router API routes
src/lib/              Core utilities (LLM provider, observability)
src/services/         Business logic services
tests/                Vitest unit + integration tests
packages/             API references for every dependency (read these first)
DEV_PLAN.md           Build plan
```
 
## License
 
MIT — see [LICENSE](./LICENSE).