Vertex AI Voice Agent for Field Service Dispatch

Intro

You’ll build a real-time voice agent that answers field service phone calls, classifies the caller’s needs with Vertex AI Gemini, and routes the call to the right team — emergency repair, maintenance, or estimates. By the end, you’ll have a working Next.js application that receives Twilio webhooks, creates LiveKit rooms for voice sessions, runs speech-to-text through Deepgram and text-to-speech through ElevenLabs, tracks your LLM spending with a budget engine, and serves a dashboard that shows live call logs and cost data.

Prerequisites

Node.js >= 22 and pnpm 10.14.0 — the exact version pinned in package.json
A Twilio account with a voice-enabled phone number — you’ll need your Account SID and auth token
A LiveKit server (self-hosted or LiveKit Cloud) — you’ll need an API key, secret, and WebSocket URL
A Google Cloud project with Vertex AI enabled — you’ll need the project ID and a location like us-central1
A Deepgram API key for real-time speech-to-text
An ElevenLabs API key for text-to-speech
A Firebase project with Firestore enabled — you’ll need a service account key JSON file
Familiarity with TypeScript and Next.js App Router routing

Step 1: Scaffold the project

Create an empty directory and initialize a Next.js TypeScript project with all the dependencies the voice agent needs.

terminal

mkdir voice-agent && cd voice-agent
pnpm init

Set the package type and engine requirements. Create package.json with the exact versions from the artifact:

json

{
  "name": "vertex-ai-voice-agent-for-field-service-dispatch",
  "version": "1.0.0",
  "private": true,
  "type": "module",
  "engines"

Install everything and add the TypeScript config:

terminal

pnpm install

Create tsconfig.json:

json

{
  "compilerOptions": {
    "strict": true,
    "esModuleInterop": true,
    "moduleResolution": "bundler",
    "target": "ES2022",
    "module": "ESNext",
    "jsx": "react-jsx",
    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true,
    "resolveJsonModule": true,
    "isolatedModules": true,
    "noUncheckedIndexedAccess"

Create vitest.config.ts for the test runner:

import { defineConfig } from 'vitest/config';
import path from 'path';
 
// Custom reporter that merges coverage-summary into vitest-report.json
const coverageMergingReporter = {
  onFinished: async (_files: unknown[], _errors: unknown[], _coverage: unknown)

Expected output: pnpm install downloads all packages. The src directory doesn’t exist yet — you’ll create it as you add files in the following steps.

Step 2: Set environment variables

Create .env.local at the project root. Every variable here is read by the application at runtime. Fill in your own values for the placeholders:

terminal

# Twilio
TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_AUTH_TOKEN=your_twilio_auth_token
TWILIO_PHONE_NUMBER=+15551234567
 
# LiveKit
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
LIVEKIT_URL=wss://your-livekit-server.com
 
# Google Cloud Vertex AI
GOOGLE_CLOUD_PROJECT=your-gcp-project-id
GOOGLE_CLOUD_LOCATION=us-central1
GOOGLE_GENAI_USE_VERTEXAI=true
 
# Deepgram (speech-to-text)

Step 3: Handle incoming calls with the Twilio webhook

When a customer calls your Twilio phone number, Twilio sends a POST request to your /api/calls endpoint. This route validates the request, creates a LiveKit room for the voice session, calls the dispatcher to classify and route, and returns TwiML that connects the caller.

Create the directories and the route file:

terminal

mkdir -p src/api/calls

Create src/api/calls/route.ts:

This route does several things: validates Twilio’s cryptographic signature (preventing spoofed webhooks), creates a LiveKit room named after the call SID, generates an access token for the room, dispatches the call for classification, and returns a <Connect><Sip> TwiML response that bridges the PSTN call into the LiveKit room.

Step 4: Classify and route with the agent dispatcher

The dispatcher is the brain of the voice agent. It uses Vertex AI Gemini to classify the caller’s transcript into one of three categories (emergency, maintenance, estimate), retrieves customer history from Agent Memory, selects the best-fit agent using CapabilityBasedRouter, and persists the session via Agent Mesh Session.

Create the agents directory and the dispatcher file:

terminal

mkdir -p src/agents

Create src/agents/dispatcher.ts. This file is ~300 lines; the excerpt below shows the classification, routing, and session management logic, plus the public dispatchCall and endCall exports. The full file (including type definitions, helper functions, and the complete error handling) is in the downloadable artifact.

The dispatcher registers three agents — Emergency Repair, Maintenance, and Estimate — each with skills mapped to the classification categories. It uses the CapabilityBasedRouter to score and select the best agent match. If classification or routing fails, the system falls back gracefully to the maintenance agent so no call goes unanswered.

Step 5: Build the voice pipeline

The voice pipeline wires together the three services that process live audio: Deepgram for speech-to-text (STT), Vertex AI Gemini for response generation, and ElevenLabs for text-to-speech (TTS). It also checks the budget engine before each LLM call and records usage afterward.

Create src/agents/pipeline.ts:

The pipeline connects to Deepgram’s WebSocket for real-time streaming STT using the Nova-3 model, builds a conversation context from past turns, and generates responses with Gemini 1.5 Pro. When the budget is tight, it automatically downgrades to a cheaper model. Every successful LLM call records cost data through the budget engine.

Step 6: Track spending with the budget engine

The budget engine enforces spending limits so your voice agent doesn’t surprise you with a massive bill. It defines Vertex AI pricing for Gemini models, persists spend records to Firestore with atomic transactions, and checks each LLM request against your monthly budget — downgrading models or blocking requests when thresholds are hit.

Create the library directory and two files:

terminal

mkdir -p src/lib

Create src/lib/firestore-admin.ts — a small helper that initializes the Firebase Admin SDK once and provides a Firestore client:

import { initializeApp, getApps } from 'firebase-admin/app';
import { getFirestore, Firestore } from 'firebase-admin/firestore';
 
let db: Firestore | null = null;
 
export function getFirestoreDb(): Firestore {
  if (db) return db;
 
  if (!getApps().length) {
    initializeApp({ projectId: process.env.GOOGLE_CLOUD_PROJECT ?? 'default' });
  }
 
  db = getFirestore();

Create src/lib/budget.ts. This file is ~220 lines; the excerpt below shows the pricing, controller setup, checkRequest, and recordUsage functions:

The budget policy defines a soft cap at 80% (warnings) and a hard cap at 100% (blocking). The auto-downgrade rule swaps gemini-1.5-pro for the cheaper gemini-1.5-flash when spend nears the limit. All spend records are persisted to Firestore with atomic transactions so concurrent requests can’t double-count.

Step 7: Add health monitoring

Your voice agent depends on five external services: Twilio, LiveKit, Vertex AI, Deepgram, and ElevenLabs. The health check system probes each one with timeouts and reports an aggregate status (healthy, degraded, or unhealthy). When more than two services fail, it logs a structured alert.

Create src/health.ts:

Now expose the checks at a route. Create src/api/health/route.ts:

terminal

mkdir -p src/api/health

import { NextRequest, NextResponse } from 'next/server';
import { runHealthChecks } from '../../health';
import type { HealthCheckResult } from '../../health';
 
export async function GET(_request: NextRequest): Promise<NextResponse> {
  const result: HealthCheckResult = await runHealthChecks();
 
  let httpStatus: number;
  if (result.status ===

You can verify the health endpoint by visiting http://localhost:3000/api/health. A fully configured deployment returns HTTP 200 with JSON like:

json

{
  "status": "healthy",
  "timestamp": "2026-05-12T17:00:00.000Z",
  "checks": [
    { "service": "Twilio", "status": "pass", "latencyMs": 120 },
    { "service": "LiveKit", "status": "pass", "latencyMs": 85 },
    { "service": "VertexAI", "status": "pass", "latencyMs": 340 },
    {

Step 8: Build the operator dashboard

The dashboard at /dashboard shows a real-time budget overview and a table of recent call logs. It’s protected by an API key that must be passed as the x-api-key header.

Create the dashboard directory structure:

terminal

mkdir -p src/app/dashboard

Add middleware to protect dashboard routes. Create src/middleware.ts:

import { NextRequest, NextResponse } from 'next/server';
 
export function middleware(request: NextRequest): NextResponse {
  const pathname = request.nextUrl.pathname;
 
  if (!pathname.startsWith('/dashboard')) {
    return NextResponse.next();
  }
 
  const apiKey = request.headers.get('x-api-key');
  const

Create the dashboard layout at src/app/dashboard/layout.tsx:

tsx

interface DashboardLayoutProps {
  children: React.ReactNode;
}
 
export default function DashboardLayout({ children }: DashboardLayoutProps) {
  return (
    <div className="dashboard-layout">
      <header className="dashboard-header">
        <h1>Field Service Dispatch</h1>
      </header>
      <main className="dashboard-main">{children}</main>

The dashboard page at src/app/dashboard/page.tsx composes two client components — a budget card and a call log table — each wrapped in an error boundary:

tsx

import CallLogTable from '@/components/CallLogTable';
import BudgetCard from '@/components/BudgetCard';
import ErrorBoundary from '@/components/ErrorBoundary';
 
export default function DashboardPage() {
  return (
    <div className="dashboard-content">
      <section className="dashboard-section budget-section">
        <h2>Budget Overview</h2>
        <ErrorBoundary>

Create src/components/ErrorBoundary.tsx — a class component that catches render errors and shows a retry button:

tsx

'use client';
 
import { Component, ErrorInfo, ReactNode } from 'react';
 
interface ErrorBoundaryProps {
  children: ReactNode;
  fallback?: ReactNode

Finally, create the two dashboard components. The BudgetCard fetches /api/budget and renders a progress bar with cost breakdown by spend tier color. The CallLogTable fetches /api/calls/logs and renders a table with status badges. Both handle loading, error, and empty states.

Create src/components/BudgetCard.tsx:

terminal

mkdir -p src/components

Create src/components/CallLogTable.tsx:

With all dashboard files in place, start the dev server and visit http://localhost:3000/dashboard with the x-api-key header set to your DASHBOARD_API_KEY value. You’ll see the budget overview card and the call log table.

Step 9: Run the tests

The artifact includes a full test suite using Vitest with React Testing Library and MSW for mocking HTTP requests. All external services are mocked — no live API calls happen during tests.

Create the test setup file at src/test-setup.ts to configure environment stubs and the MSW server:

import '@testing-library/jest-dom/vitest';
import { setupServer } from 'msw/node';
import { beforeAll, afterAll, afterEach, vi } from 'vitest';
import { handlers } from './mocks/handlers';
 
vi.stubEnv('TWILIO_ACCOUNT_SID', 'ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx');
vi.stubEnv('TWILIO_AUTH_TOKEN', '');
vi.stubEnv('LIVEKIT_URL'

The MSW handlers and test data factories are in src/mocks/. The full implementations are in the downloadable artifact.

Run the test suite:

terminal

pnpm test

Expected output:

code

✓ src/api/calls/route.test.ts (9 tests)
✓ src/api/health/route.test.ts (8 tests)
✓ src/agents/dispatcher.test.ts (9 tests)
✓ src/agents/pipeline.test.ts (12 tests)
✓ src/components/__tests__/BudgetCard.test.tsx (9 tests)
✓ src/components/__tests__/CallLogTable.test.tsx (8 tests)
✓ src/integration.test.ts (9 tests)
✓ src/lib/budget.test.ts (9 tests)

Test Files  8 passed (8)
     Tests  73 passed (73)

To run with coverage (which enforces 90% thresholds on lines, branches, functions, and statements):

terminal

pnpm test:coverage

When all tests pass, start the dev server to see the application running:

terminal

pnpm dev

Then configure your Twilio phone number’s voice webhook URL to https://your-domain.com/api/calls. Incoming calls will now trigger the full pipeline: Twilio webhook → LiveKit room creation → Vertex AI Gemini classification → agent routing → Deepgram STT → Gemini response generation → ElevenLabs TTS.

Next steps

Connect the dashboard API routes (/api/budget and /api/calls/logs) by wiring the budget controller’s getRemainingBudget and the Firestore getCallLogs function into Next.js route handlers — the budget engine and Firestore client are already written, they just need route files to expose them.
Deploy to a platform like Vercel or Cloud Run and configure your Twilio phone number’s voice webhook to point at the production URL.
Extend the dispatcher agent registry with additional agent types — for example, a Spanish-speaking agent or an after-hours on-call agent with different routing rules.

Intro

Prerequisites

Node.js >= 22 and pnpm 10.14.0 — the exact version pinned in package.json
A Twilio account with a voice-enabled phone number — you’ll need your Account SID and auth token
A LiveKit server (self-hosted or LiveKit Cloud) — you’ll need an API key, secret, and WebSocket URL
A Google Cloud project with Vertex AI enabled — you’ll need the project ID and a location like us-central1
A Deepgram API key for real-time speech-to-text
An ElevenLabs API key for text-to-speech
A Firebase project with Firestore enabled — you’ll need a service account key JSON file
Familiarity with TypeScript and Next.js App Router routing

Step 1: Scaffold the project

Create an empty directory and initialize a Next.js TypeScript project with all the dependencies the voice agent needs.

terminal

mkdir voice-agent && cd voice-agent
pnpm init

Set the package type and engine requirements. Create package.json with the exact versions from the artifact:

json

{
  "name": "vertex-ai-voice-agent-for-field-service-dispatch",
  "version": "1.0.0",
  "private": true,
  "type": "module",
  "engines"

Install everything and add the TypeScript config:

terminal

pnpm install

Create tsconfig.json:

json

{
  "compilerOptions": {
    "strict": true,
    "esModuleInterop": true,
    "moduleResolution": "bundler",
    "target": "ES2022",
    "module": "ESNext",
    "jsx": "react-jsx",
    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true,
    "resolveJsonModule": true,
    "isolatedModules": true,
    "noUncheckedIndexedAccess"

Create vitest.config.ts for the test runner:

import { defineConfig } from 'vitest/config';
import path from 'path';
 
// Custom reporter that merges coverage-summary into vitest-report.json
const coverageMergingReporter = {
  onFinished: async (_files: unknown[], _errors: unknown[], _coverage: unknown)

Expected output: pnpm install downloads all packages. The src directory doesn’t exist yet — you’ll create it as you add files in the following steps.

Step 2: Set environment variables

Create .env.local at the project root. Every variable here is read by the application at runtime. Fill in your own values for the placeholders:

terminal

# Twilio
TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_AUTH_TOKEN=your_twilio_auth_token
TWILIO_PHONE_NUMBER=+15551234567
 
# LiveKit
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
LIVEKIT_URL=wss://your-livekit-server.com
 
# Google Cloud Vertex AI
GOOGLE_CLOUD_PROJECT=your-gcp-project-id
GOOGLE_CLOUD_LOCATION=us-central1
GOOGLE_GENAI_USE_VERTEXAI=true
 
# Deepgram (speech-to-text)

Step 3: Handle incoming calls with the Twilio webhook

Create the directories and the route file:

terminal

mkdir -p src/api/calls

Create src/api/calls/route.ts:

Step 4: Classify and route with the agent dispatcher

Create the agents directory and the dispatcher file:

terminal

mkdir -p src/agents

Step 5: Build the voice pipeline

Create src/agents/pipeline.ts:

Step 6: Track spending with the budget engine

Create the library directory and two files:

terminal

mkdir -p src/lib

Create src/lib/firestore-admin.ts — a small helper that initializes the Firebase Admin SDK once and provides a Firestore client:

import { initializeApp, getApps } from 'firebase-admin/app';
import { getFirestore, Firestore } from 'firebase-admin/firestore';
 
let db: Firestore | null = null;
 
export function getFirestoreDb(): Firestore {
  if (db) return db;
 
  if (!getApps().length) {
    initializeApp({ projectId: process.env.GOOGLE_CLOUD_PROJECT ?? 'default' });
  }
 
  db = getFirestore();

Create src/lib/budget.ts. This file is ~220 lines; the excerpt below shows the pricing, controller setup, checkRequest, and recordUsage functions:

Step 7: Add health monitoring

Create src/health.ts:

Now expose the checks at a route. Create src/api/health/route.ts:

terminal

mkdir -p src/api/health

import { NextRequest, NextResponse } from 'next/server';
import { runHealthChecks } from '../../health';
import type { HealthCheckResult } from '../../health';
 
export async function GET(_request: NextRequest): Promise<NextResponse> {
  const result: HealthCheckResult = await runHealthChecks();
 
  let httpStatus: number;
  if (result.status ===

You can verify the health endpoint by visiting http://localhost:3000/api/health. A fully configured deployment returns HTTP 200 with JSON like:

json

{
  "status": "healthy",
  "timestamp": "2026-05-12T17:00:00.000Z",
  "checks": [
    { "service": "Twilio", "status": "pass", "latencyMs": 120 },
    { "service": "LiveKit", "status": "pass", "latencyMs": 85 },
    { "service": "VertexAI", "status": "pass", "latencyMs": 340 },
    {

Step 8: Build the operator dashboard

The dashboard at /dashboard shows a real-time budget overview and a table of recent call logs. It’s protected by an API key that must be passed as the x-api-key header.

Create the dashboard directory structure:

terminal

mkdir -p src/app/dashboard

Add middleware to protect dashboard routes. Create src/middleware.ts:

import { NextRequest, NextResponse } from 'next/server';
 
export function middleware(request: NextRequest): NextResponse {
  const pathname = request.nextUrl.pathname;
 
  if (!pathname.startsWith('/dashboard')) {
    return NextResponse.next();
  }
 
  const apiKey = request.headers.get('x-api-key');
  const

Create the dashboard layout at src/app/dashboard/layout.tsx:

tsx

interface DashboardLayoutProps {
  children: React.ReactNode;
}
 
export default function DashboardLayout({ children }: DashboardLayoutProps) {
  return (
    <div className="dashboard-layout">
      <header className="dashboard-header">
        <h1>Field Service Dispatch</h1>
      </header>
      <main className="dashboard-main">{children}</main>

The dashboard page at src/app/dashboard/page.tsx composes two client components — a budget card and a call log table — each wrapped in an error boundary:

tsx

import CallLogTable from '@/components/CallLogTable';
import BudgetCard from '@/components/BudgetCard';
import ErrorBoundary from '@/components/ErrorBoundary';
 
export default function DashboardPage() {
  return (
    <div className="dashboard-content">
      <section className="dashboard-section budget-section">
        <h2>Budget Overview</h2>
        <ErrorBoundary>

Create src/components/ErrorBoundary.tsx — a class component that catches render errors and shows a retry button:

tsx

'use client';
 
import { Component, ErrorInfo, ReactNode } from 'react';
 
interface ErrorBoundaryProps {
  children: ReactNode;
  fallback?: ReactNode

Create src/components/BudgetCard.tsx:

terminal

mkdir -p src/components

Create src/components/CallLogTable.tsx:

Step 9: Run the tests

The artifact includes a full test suite using Vitest with React Testing Library and MSW for mocking HTTP requests. All external services are mocked — no live API calls happen during tests.

Create the test setup file at src/test-setup.ts to configure environment stubs and the MSW server:

import '@testing-library/jest-dom/vitest';
import { setupServer } from 'msw/node';
import { beforeAll, afterAll, afterEach, vi } from 'vitest';
import { handlers } from './mocks/handlers';
 
vi.stubEnv('TWILIO_ACCOUNT_SID', 'ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx');
vi.stubEnv('TWILIO_AUTH_TOKEN', '');
vi.stubEnv('LIVEKIT_URL'

The MSW handlers and test data factories are in src/mocks/. The full implementations are in the downloadable artifact.

Run the test suite:

terminal

pnpm test

Expected output:

code

✓ src/api/calls/route.test.ts (9 tests)
✓ src/api/health/route.test.ts (8 tests)
✓ src/agents/dispatcher.test.ts (9 tests)
✓ src/agents/pipeline.test.ts (12 tests)
✓ src/components/__tests__/BudgetCard.test.tsx (9 tests)
✓ src/components/__tests__/CallLogTable.test.tsx (8 tests)
✓ src/integration.test.ts (9 tests)
✓ src/lib/budget.test.ts (9 tests)

Test Files  8 passed (8)
     Tests  73 passed (73)

To run with coverage (which enforces 90% thresholds on lines, branches, functions, and statements):

terminal

pnpm test:coverage

When all tests pass, start the dev server to see the application running:

terminal

pnpm dev

Next steps

Connect the dashboard API routes (/api/budget and /api/calls/logs) by wiring the budget controller’s getRemainingBudget and the Firestore getCallLogs function into Next.js route handlers — the budget engine and Firestore client are already written, they just need route files to expose them.
Deploy to a platform like Vercel or Cloud Run and configure your Twilio phone number’s voice webhook to point at the production URL.
Extend the dispatcher agent registry with additional agent types — for example, a Spanish-speaking agent or an after-hours on-call agent with different routing rules.

Vertex AI Voice Agent for Field Service Dispatch

The problem

Built from

Intro

Prerequisites

Step 1: Scaffold the project

Step 2: Set environment variables

Step 3: Handle incoming calls with the Twilio webhook

Step 4: Classify and route with the agent dispatcher

Step 5: Build the voice pipeline

Step 6: Track spending with the budget engine

Step 7: Add health monitoring

Step 8: Build the operator dashboard

Step 9: Run the tests

Next steps

Example artifact

Comments

Intro

Prerequisites

Step 1: Scaffold the project

Step 2: Set environment variables

Step 3: Handle incoming calls with the Twilio webhook

Step 4: Classify and route with the agent dispatcher

Step 5: Build the voice pipeline

Step 6: Track spending with the budget engine

Step 7: Add health monitoring

Step 8: Build the operator dashboard

Step 9: Run the tests

Next steps