Phone Voice Agent

View as Markdown

This example builds a voice agent you can call on the phone, with Twilio and the Grok API's realtime voice. It's a starting point for phone lines that talk with callers in real time, like support or bookings.

Note: These are example implementations for learning and development, and they aren't production-ready without additional hardening.

Overview

These examples demonstrate how to build voice agents that can be accessed via phone calls. Perfect for IVR systems, call centers, and voice-based customer service.

Available examples

XAI Native

Native XAI implementation with Twilio Media Streams.

Features:

Direct WebSocket integration with XAI
Real-time voice processing
Session management

Tech Stack:

Node.js + TypeScript
Twilio Media Streams
WebSockets
Express server

Quick start

XAI Native (recommended)

cd xai
npm install

# Configure environment
cp .env.example .env
# Edit .env with your keys

# Start server
npm run dev

# Expose to internet (for Twilio)
ngrok http 3000

# Configure Twilio webhook with ngrok URL
# Call your Twilio number!

Architecture

Call flow

┌─────────┐    1. SIP    ┌─────────────┐   2. WebSocket   ┌──────────────┐
│  Phone  │ ←──────────→ │   Twilio    │ ←──────────────→ │  Your Server │
│  Call   │   Audio      │Media Streams│  μ-law (native)  │  (Node.js)   │
└─────────┘              └─────────────┘                  └──────────────┘
                                                                 ↓
                                                           3. WebSocket
                                                                 ↓
                                                          ┌──────────────┐
                                                          │   XAI API    │
                                                          │  (Realtime)  │
                                                          └──────────────┘

Data flow

Phone → Twilio: Caller dials your Twilio number
Twilio → Server: Twilio streams μ-law PCM audio via WebSocket
Server → XAI: Server forwards μ-law audio directly to XAI Realtime API (no conversion)
XAI → Server: AI responds with μ-law audio and text
Server → Twilio: Server forwards μ-law audio directly to Twilio (no conversion)
Twilio → Phone: Caller hears AI response

Prerequisites

Required accounts

XAI Account
- Get API key: console.x.ai
- Realtime API access
Twilio Account
- Sign up: console.twilio.com
- Phone number with voice capabilities
- Media Streams enabled

Technical requirements

Node.js: 18+
Public Endpoint: ngrok or deployed server
Port: 3000 (configurable)

Setup guide

1. Get Twilio credentials

# From Twilio Console
TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_AUTH_TOKEN=your_auth_token
TWILIO_PHONE_NUMBER=+1234567890

2. Configure the environment

cd xai
cp .env.example .env

# Edit .env with:
XAI_API_KEY=your_xai_api_key_here
HOSTNAME=your-ngrok-domain.ngrok.app

3. Expose to the internet

Using ngrok:

# Install ngrok
brew install ngrok  # macOS
# or download from ngrok.com

# Start your server first
npm run dev

# In another terminal, expose it
ngrok http 3000

# Copy the HTTPS URL (e.g., https://abc123.ngrok.io)

4. Configure the Twilio webhook

Go to Twilio Console
Navigate to Phone Numbers → Manage → Active Numbers
Click your phone number
Under "Voice & Fax", set:
- A call comes in: Webhook
- URL: https://your-ngrok-url.ngrok.io/twiml
- Method: POST
Save

5. Test the call

# Call your Twilio number
# You should hear the AI agent!

Features

XAI Native features

Real-time voice processing
Low latency responses
Session management
Call logging
Error handling
Graceful disconnection

Configuration options

Environment variables

# Required
XAI_API_KEY=your_key_here
HOSTNAME=your-ngrok-domain.ngrok.app

# Optional (with defaults)
API_URL=wss://api.x.ai/v1/realtime
PORT=3000

Note: Twilio credentials are configured in the Twilio Console, not as environment variables.

Voice configuration

// In your code
const sessionConfig = {
  type: "session.update",
  session: {
    voice: "Ara", // Choose from: Ara, Rex, Sal, Eve, Leo
    instructions: "You are a helpful assistant.",
    // Audio format settings (these are the defaults if not specified)
    audio: {
      input: { format: { type: "audio/pcm", rate: 24000 } },
      output: { format: { type: "audio/pcm", rate: 24000 } }
    }
  }
};

Audio processing

Audio format (no conversion needed!)

End-to-End Format:

Format: μ-law PCM (audio/pcmu)
Sample Rate: 8kHz
Encoding: Base64

Twilio → Server → XAI:

μ-law audio passes through without conversion
XAI API natively supports PCMU (μ-law) format

XAI → Server → Twilio:

μ-law audio passes through without conversion
Direct passthrough improves latency and audio quality

Audio pipeline

// Incoming audio from Twilio (μ-law @ 8kHz)
const twilioAudio = μ-law PCM @ 8kHz

// Send directly to XAI (configured for native μ-law input)
await xaiWebSocket.send(twilioAudio)

// Receive from XAI (μ-law @ 8kHz - native telephony format)
const xaiResponse = μ-law PCM @ 8kHz

// Send directly to Twilio
await twilioWebSocket.send(xaiResponse)

Key Improvement: XAI API supports native μ-law (PCMU) and A-law (PCMA) formats, eliminating the need for PCM16 conversion and improving audio quality and latency.

Monitoring & debugging

Enable logging

# Set log level
LOG_LEVEL=debug npm run dev

Check the WebSocket connection

// Connection events
ws.on('open', () => console.log('Connected to XAI'));
ws.on('close', () => console.log('Disconnected from XAI'));
ws.on('error', (err) => console.error('WebSocket error:', err));

Testing

Local testing

# Start server
npm run dev

# Use ngrok for external access
ngrok http 3000

# Call your Twilio number

Test call script

# Make a test call via Twilio API
curl -X POST https://api.twilio.com/2010-04-01/Accounts/$TWILIO_ACCOUNT_SID/Calls.json \
  --data-urlencode "Url=http://your-server.com/twiml" \
  --data-urlencode "To=+1234567890" \
  --data-urlencode "From=$TWILIO_PHONE_NUMBER" \
  -u $TWILIO_ACCOUNT_SID:$TWILIO_AUTH_TOKEN

Documentation

XAI Implementation README - XAI implementation details
Twilio Docs - Twilio Media Streams
XAI API Docs - XAI Realtime API

Examples

Use cases

IVR System:

Menu navigation
Call routing
Information gathering

Customer Support:

First-line support
FAQ handling
Appointment scheduling

Surveys:

Voice surveys
Feedback collection
Market research

Notifications:

Appointment reminders
Payment alerts
Status updates