A 3-person consulting firm sends a weekly industry newsletter to 2,000 subscribers. They manually scan 20+ RSS feeds, select articles, and write summaries. This takes 5 hours per week. They also want to include internal company updates but often forget. They need an automated pipeline that fetches articles, scores them by relevance, generates summaries, and compiles a draft newsletter.
Example artifact
A complete, working implementation of this recipe — downloadable as a zip or browsable file by file. Generated by our build pipeline; tested with full coverage before publishing.
You’ll build an Automated Industry Newsletter Curator — a pipeline that fetches articles from RSS feeds, extracts and enriches content, scores articles by relevance to your topic, generates AI summaries, and compiles a polished newsletter draft to disk. This recipe is built with the @reaatech/* hybrid RAG ecosystem, Hono as the HTTP framework, Mozilla Readability for content extraction, and any OpenAI-compatible LLM provider. A small consulting team that spends hours a week manually curating a client newsletter could swap this in and cut that time to near zero.
Prerequisites
Node.js >= 22 and pnpm 10
An OpenAI API key (or any OpenAI-compatible endpoint — set LLM_BASE_URL to point elsewhere)
A Qdrant instance (local Docker or cloud) for vector storage and hybrid retrieval
Working knowledge of TypeScript and Hono
Step 1: Set up the project
The scaffold agent has already created the project shell with exact-pinned dependencies. Verify the structure:
terminal
ls
You should see package.json, tsconfig.json, next.config.ts, vitest.config.ts, eslint.config.mjs, .env.example, src/, and tests/. The dependencies already include all six REAA packages and every third-party library you’ll need.
Copy the environment file and fill in the required credentials:
Expected output: You now have a valid .env file with your credentials. All remaining vars have sensible defaults — LLM_BASE_URL points to OpenAI, LLM_MODEL is gpt-5.2, and the output directory is ./output.
Step 2: Create the core types
All the domain types live in src/types.ts. These define the shape of every object that flows through the pipeline — RSS items, articles, scores, newsletter drafts, internal company updates, and LLM config.
Expected output: The file compiles clean — run pnpm typecheck to confirm. The byline field matches Mozilla Readability’s output field name, and the Zod schema validates internal update inputs at the API boundary.
Step 3: Build the RSS feed fetcher
The RSS fetcher parses XML feeds using fast-xml-parser, handles both RSS 2.0 and Atom formats, deduplicates articles by normalized URL, and gracefully handles per-feed failures so one broken feed doesn’t take down the whole pipeline.
ts
import { XMLParser } from "fast-xml-parser";import { nanoid } from "nanoid";import type { RssFeedSource, RssItem, Article } from "../types.js";import { getLogger } from "@reaatech/hybrid-rag-observability";interface ParsedFeed { rss?: { channel?: Record<string, unknown>; }; feed?: Record<string, unknown>;}const parser = new XMLParser();function parseRssItem(raw:
Expected output:pnpm typecheck passes. URL normalization strips www., lowercases the hostname, and removes trailing slashes — so https://www.Example.com/Article/ and https://example.com/article are treated as duplicates. The function also extracts the atom:link href as a fallback when channel.link is absent.
Step 4: Build the article content extractor
The article extractor uses Mozilla Readability (the same library behind Firefox’s Reader View) to turn raw HTML into clean article content. It also converts HTML to Markdown using node-html-markdown for the LLM-friendly version.
Expected output: Typecheck passes. The JSDOM(html, { url }) parameter is critical for Readability to resolve relative URLs in images and links. When Readability returns null (e.g., a page with no main article), the function falls back to returning the raw HTML. When extraction throws, it still returns the raw content rather than an empty article.
Step 5: Build the internal updates service
Your team can post internal updates through a CRUD API — these get included in the newsletter alongside external articles. The service uses an in-memory store with Zod validation at the boundary.
ts
import { nanoid } from "nanoid";import type { InternalUpdate } from "../types.js";import { InternalUpdateInputSchema, type InternalUpdateInput } from "../types.js";const updates: InternalUpdate[] = [];export function addUpdate(input: InternalUpdateInput): InternalUpdate { const validated = InternalUpdateInputSchema.parse(input); const update: InternalUpdate = { id: nanoid(), content: validated.content, author: validated.author, tags: validated.tags ?? [], date: new Date().toISOString(), }; updates.push(update); return update;}export function getRecentUpdates(days: number): InternalUpdate[] { const cutoff = Date.now() - days * 86_400_000; return updates.filter((u) => new Date(u.date).getTime() > cutoff);}export function listUpdates(opts?: { tags?: string[]; since?: string;}): InternalUpdate[] { let result = [...updates]; if (opts?.tags && opts.tags.length > 0) { const filterTags = opts.tags; result = result.filter((u) => filterTags.some((t) => u.tags.includes(t)), ); } if (opts?.since) { const sinceMs = new Date(opts.since).getTime(); result = result.filter((u) => new Date(u.date).getTime() >= sinceMs); } return result;}export function deleteUpdate(id: string): boolean { const idx = updates.findIndex((u) => u.id === id); if (idx === -1) return false; updates.splice(idx, 1); return true;}
Expected output: Typecheck passes. The addUpdate function throws a ZodError if content or author are empty strings — that validation is also enforced at the API route level via safeParse.
Step 6: Build the LLM service with retry logic
The LLM service wraps OpenAI’s chat completions API with a unified retry helper (exponential backoff up to 3 attempts). It provides five operations: generating article summaries, scoring articles for relevance, classifying articles into sections, composing section intros, and generating newsletter titles.
ts
import OpenAI from "openai";import type { Article, ArticleScore, LLMConfig, NewsletterSection } from "../types.js";export function createLLMClient(config: LLMConfig): OpenAI { return new OpenAI({ apiKey: config.apiKey, baseURL: config.baseUrl });}async function withRetry<T>( fn: () => Promise<T>, retries = 3,): Promise<T> { let lastErr: unknown; for (
Expected output: Typecheck passes. The scoreArticle function requests response_format: { type: "json_object" } so the LLM returns structured JSON, then computes finalScore = relevanceScore * 0.6 + noveltyScore * 0.4 — weighting relevance higher than novelty. Malformed JSON falls back to all zeros gracefully.
Step 7: Build the RAG pipeline service with REAA packages
This is the core integration — you wire together five @reaatech/* packages into a single RagPipelineService class. The pipeline handles document ingestion (preprocessing, validation, chunking) and hybrid retrieval (vector + BM25 search with RRF fusion).
ts
import { RAGPipeline } from "@reaatech/hybrid-rag-pipeline";import { DocumentSchema, ChunkingStrategy, type Document, type Chunk, type ChunkingConfig, type RetrievalResult, type HybridResult, ChunkingConfigSchema } from "@reaatech/hybrid-rag";import { TextPreprocessor, DocumentValidator, chunkDocument } from "@reaatech/hybrid-rag-ingestion";import { HybridRetriever, RerankerEngine } from "@reaatech/hybrid-rag-retrieval";import { createLogger, getMetricsCollector, getTracingManager } from "@reaatech/hybrid-rag-observability";import type { Article } from "../types.js";export class RagPipelineService { private pipeline: RAGPipeline; private preprocessor
Expected output: Typecheck passes. The RAGPipeline is configured in cost-conscious mode — rerankerProvider: null skips the expensive reranking step. The searchArticles method builds a HybridResult object that tracks per-source scores (vector vs BM25) for debugging retrieval quality.
Step 8: Build the newsletter compiler
The compiler ties everything together: it scores and ranks articles, classifies them into sections, generates section intros and a newsletter title, then produces a Markdown file on disk.
ts
import { nanoid } from "nanoid";import * as fs from "node:fs/promises";import * as path from "node:path";import type { Article, ArticleScore, NewsletterDraft, NewsletterSection, InternalUpdate } from "../types.js";import { scoreArticle, assignSection, composeSectionIntro, generateNewsletterTitle } from "./llm-service.js";import type OpenAI from "openai";const MAX_ARTICLES = parseInt(process.env.NEWSLETTER_MAX_ARTICLES ?? "15", 10);export async function scoreAndRankArticles( articles: Article[],
Expected output: Typecheck passes. The output directory is created with mkdir -p semantics, so it works even on first run. The Markdown output uses # for the title, ## for sections, ### for articles, and [Read more](url) links.
Step 9: Wire the Hono API and MCP server
The application entry point (src/app.ts) wires all the services together into a Hono HTTP server with 10 routes. It also starts an MCP server for AI agent integration using createMCPServer from @reaatech/hybrid-rag-mcp-server.
ts
import { Hono } from "hono";import { cors } from "hono/cors";import { serve } from "@hono/node-server";import { RagPipelineService } from "./services/rag-pipeline-service.js";import { createLLMClient } from "./services/llm-service.js";import { fetchAllFeeds } from "./services/rss-fetcher.js";import { extractArticle } from "./services/article-extractor.js";import { addUpdate, listUpdates, deleteUpdate } from "./services/internal-updates.js";import { scoreAndRankArticles, compileNewsletter, saveNewsletterToDisk } from "./services/newsletter-compiler.js";import { InternalUpdateInputSchema, type RssFeedSource } from "./types.js";
Expected output: Typecheck passes. The server skips the serve() call during Vitest runs by checking process.env.VITEST, so your tests can import and exercise routes directly. The MCP server starts asynchronously alongside the HTTP server — if it fails, it logs the error but doesn’t crash the app.
Step 10: Run the tests
The test suite covers every service and route — 92 test cases including happy paths, error boundaries, and edge cases like malformed RSS, network failures, and empty arrays.
Run the full verification pipeline:
terminal
pnpm typecheck
terminal
pnpm lint
terminal
pnpm test
Expected output: All three commands exit 0. Typecheck produces no errors. Lint produces no warnings. The test report shows numFailedTests: 0, numTotalTests: 92, and line/branch/function/statement coverage ≥ 90% on runtime code in src/.
Step 11: Generate your first newsletter
Start the server with your .env configured with real API keys:
terminal
pnpm dev
In another terminal, send a POST to the generate endpoint:
terminal
curl -X POST http://localhost:3000/api/newsletter/generate
Expected output: A 201 response with { draftId, file, articleCount, chunkCount }. The newsletter Markdown file is written to ./output/newsletter-YYYY-MM-DD.md. Open it to see sections, AI-generated intros, summaries, and links — a full weekly draft ready for review.
You can also manage RSS sources and internal updates through the REST API:
terminal
# Add an RSS sourcecurl -X POST http://localhost:3000/api/sources \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com/feed.xml", "name": "Example Blog"}'# Post an internal updatecurl -X POST http://localhost:3000/api/updates \ -H "Content-Type: application/json" \ -d '{"content": "New client onboarding complete", "author": "Alice", "tags": ["sales"]}'# Search indexed articlescurl -X POST http://localhost:3000/api/pipeline/search \ -H "Content-Type: application/json" \ -d '{"query": "AI safety", "topK": 5}'
Next steps
Add a reranker — set rerankerProvider to "cohere" and provide an API key in RAGPipelineConfig to improve result quality with cross-encoder scoring.
Customize the prompt templates — tune the scoreArticle, composeSectionIntro, and other prompts in llm-service.ts to match your industry’s vocabulary and newsletter style.
Persist internal updates — swap the in-memory store in internal-updates.ts for a SQLite or Postgres backend so updates survive server restarts.
Switch the LLM provider — set LLM_BASE_URL to any OpenAI-compatible endpoint (Anthropic via proxy, local Ollama, Groq, etc.) and the pipeline uses it transparently.
