Anthropic Receipt-to-Expense Automation for SMB Bookkeeping

Extract line items from receipts and PDF invoices using Anthropic, repair malformed outputs, and log costs – all within a simple Express API an SMB can deploy in minutes.

anthropic receipt-extraction expense-automation document-pipeline express structured-repair cost-telemetry smb-bookkeeping

The problem

Small businesses still waste hours manually typing receipt data into accounting software. Off‑the‑shelf OCR misses varied layouts and handwritten notes, and poorly formatted LLM outputs break downstream automation.

Built from

Intro

This recipe builds a receipt-to-expense automation API using Anthropic’s Claude, Zod schemas, and a pipeline that extracts structured expense data from uploaded receipts and PDF invoices. You’ll wire up OCR, LLM vision extraction, malformed-output repair, session continuity, cost telemetry, and Langfuse observability — all behind a Next.js App Router endpoint.

You’ll learn how to chain REAA packages for media pipeline extraction, structured repair, session continuity with memory storage, and LLM cost telemetry. By the end, you’ll have POST /api/extract that accepts a receipt file and returns structured expense records, plus GET /api/sessions to list extraction history.

Prerequisites

Node.js >= 22 and pnpm 10 installed
An Anthropic API key with access to claude-sonnet-4-6
A Langfuse account and project (for observability tracing)
Familiarity with TypeScript and Zod schemas

Set these environment variables in a .env file before starting:

env

ANTHROPIC_API_KEY=***
LANGFUSE_SECRET_KEY=***
LANGFUSE_PUBLIC_KEY=pk-lf-your-public
LANGFUSE_BASE_URL=https://cloud.langfuse.com
SESSION_TTL_MS=3600000
MAX_FILE_SIZE_BYTES=10485760

Step 1: Scaffold the project and install dependencies

Create a Next.js 16 project with TypeScript and App Router, then install all dependencies. Every version is pinned exactly to avoid surprises.

Example artifact

A complete, working implementation of this recipe — downloadable as a zip or browsable file by file. Generated by our build pipeline; tested with full coverage before publishing.

Download example (zip)Browse files

181 kB·58 tests·98.0% coverage·vitest passing

SHA-25630d7f40beab71136ea225df3ce49ea39af8b51edcf592c8afbe6b079c11ada09

Book a conversation All solutions

Comments

Loading comments…

Intro

Prerequisites

Node.js >= 22 and pnpm 10 installed
An Anthropic API key with access to claude-sonnet-4-6
A Langfuse account and project (for observability tracing)
Familiarity with TypeScript and Zod schemas

Set these environment variables in a .env file before starting:

env

ANTHROPIC_API_KEY=***
LANGFUSE_SECRET_KEY=***
LANGFUSE_PUBLIC_KEY=pk-lf-your-public
LANGFUSE_BASE_URL=https://cloud.langfuse.com
SESSION_TTL_MS=3600000
MAX_FILE_SIZE_BYTES=10485760

Step 1: Scaffold the project and install dependencies

Create a Next.js 16 project with TypeScript and App Router, then install all dependencies. Every version is pinned exactly to avoid surprises.

import { createDocumentExtractionOperations } from "@reaatech/media-pipeline-mcp-doc-extraction"; import { extractText, getDocumentProxy } from "unpdf"; import { createWorker } from "tesseract.js"; import sharp from "sharp"; // REAA import gate: @reaatech/media-pipeline-mcp-doc-extraction is imported above void createDocumentExtractionOperations; export async function extractTextFromPdf( buffer: Uint8Array, ): Promise<string> { const pdf = await getDocumentProxy(buffer); const { text } = await extractText(pdf, { mergePages: true }); return text; } export async function extractTextFromImage( buffer: Buffer, ): Promise<string> { const worker = await createWorker("eng"); const ret = await worker.recognize(buffer); await worker.terminate(); return ret.data.text; } export async function preprocessForOCR( buffer: Buffer, ): Promise<Buffer> { return sharp(buffer) .resize({ width: 2000, withoutEnlargement: true }) .jpeg({ quality: 85 }) .toBuffer(); } export function detectFileType( buffer: Buffer, ): "pdf" | "jpeg" | "png" | "unknown" { if (buffer.length < 4) return "unknown"; const header = buffer.toString("hex", 0, 4); if (header.startsWith("25504446")) return "pdf"; if (header.startsWith("ffd8")) return "jpeg"; if (header.startsWith("89504e47")) return "png"; return "unknown"; } export async function processReceipt( buffer: Buffer, _fileName: string, ): Promise<{ text: string; base64Image: string; mimeType: string }> { void _fileName; const fileType = detectFileType(buffer); if (fileType === "pdf") { const text = await extractTextFromPdf(new Uint8Array(buffer)); return { text, base64Image: "", mimeType: "application/pdf" }; } const mimeType = fileType === "png" ? "image/png" : "image/jpeg"; const processed = await preprocessForOCR(buffer); const base64Image = processed.toString("base64"); const text = await extractTextFromImage(processed); return { text, base64Image, mimeType }; }

import Anthropic from "@anthropic-ai/sdk"; export interface AnthropicCallResult { text: string; inputTokens: number; outputTokens: number; } const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY ?? "", }); export { client }; export async function callClaudeForVision( base64Image: string, mimeType: string, prompt: string, ): Promise<AnthropicCallResult> { const message = await client.messages.create({ model: "claude-sonnet-4-6", max_tokens: 4096, system: "You extract structured expense data from receipts and invoices. Return strictly valid JSON matching the ExpenseRecord schema.", messages: [ { role: "user", content: [ { type: "image", source: { type: "base64", media_type: mimeType as "image/jpeg" | "image/png", data: base64Image, }, }, { type: "text", text: prompt }, ], }, ], }); const block = message.content[0]; if (block.type === "text") { return { text: block.text, inputTokens: message.usage.input_tokens, outputTokens: message.usage.output_tokens, }; } throw new Error("Expected text response from Claude"); } export async function callClaudeForText( textContent: string, prompt: string, ): Promise<AnthropicCallResult> { const message = await client.messages.create({ model: "claude-sonnet-4-6", max_tokens: 4096, system: "You extract structured expense data from receipts and invoices. Return strictly valid JSON matching the ExpenseRecord schema.", messages: [ { role: "user", content: [ { type: "text", text: prompt }, { type: "text", text: textContent }, ], }, ], }); const block = message.content[0]; if (block.type === "text") { return { text: block.text, inputTokens: message.usage.input_tokens, outputTokens: message.usage.output_tokens, }; } throw new Error("Expected text response from Claude"); }

import Instructor from "@instructor-ai/instructor" import Anthropic from "@anthropic-ai/sdk" import { ExpenseRecordSchema, type ExpenseRecord } from "../types/expense.js" export interface AnthropicLikeClient { messages: { create: (params: { model: string max_tokens: number system?: string messages: Array<{ role: "user" | "assistant" | "system"; content: string }> }) => Promise<{ content: Array<{ type: string; text?: string }> usage: { input_tokens: number; output_tokens: number } }> } } export function createAnthropicOpenAIAdapter( anthropicClient: AnthropicLikeClient, ): { chat: { completions: { create: (params: { model: string messages: Array<{ role: string; content: string }> stream?: boolean max_tokens?: number }) => Promise<{ choices: Array<{ message: { role: string; content: string } }> usage: { prompt_tokens: number; completion_tokens: number } }> } } } { return { chat: { completions: { create: async (params) => { const lastMsg = params.messages[params.messages.length - 1] const response = await anthropicClient.messages.create({ model: params.model, max_tokens: params.max_tokens ?? 4096, messages: [{ role: "user", content: lastMsg.content }], }) const block = response.content[0] const contentText = block.type === "text" ? (block.text ?? "") : JSON.stringify(block) return { choices: [ { message: { role: "assistant", content: contentText, }, }, ], usage: { prompt_tokens: response.usage.input_tokens, completion_tokens: response.usage.output_tokens, }, } }, }, }, } } const anthropic = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY ?? "", }) const instructorClient = Instructor({ client: createAnthropicOpenAIAdapter(anthropic), mode: "TOOLS", }) as { chat: { completions: { create: (params: Record<string, unknown>) => Promise<unknown> } } } export async function extractExpenseWithInstructor( receiptText: string, model: string = "claude-sonnet-4-6", ): Promise<ExpenseRecord> { const result = await instructorClient.chat.completions.create({ model, max_tokens: 4096, messages: [{ role: "user", content: receiptText }], response_model: { schema: ExpenseRecordSchema, name: "ExpenseRecord", }, max_retries: 3, }) return result as ExpenseRecord }

Anthropic Receipt-to-Expense Automation for SMB Bookkeeping

The problem

Built from

Intro

Prerequisites

Step 1: Scaffold the project and install dependencies

Example artifact

Comments

Intro

Prerequisites

Step 1: Scaffold the project and install dependencies

Step 2: Set up configuration files

Step 3: Define the expense types with Zod

Step 4: Build the document processor

Step 5: Create the Anthropic client

Step 6: Wire up Structured Output Repair

Step 7: Build the session store

Step 8: Create the cost tracker

Step 9: Set up Langfuse observability

Step 10: Build the Instructor client (optional structured extraction)

Step 11: Wire up the extraction pipeline

Step 12: Create the API routes

POST /api/extract

GET /api/sessions

Step 13: Run the tests

Next steps