Phone Voice Agent
This example builds a voice agent you can call on the phone, with Twilio and the Grok API's realtime voice. It's a starting point for phone lines that talk with callers in real time, like support or bookings.
Note: These are example implementations for learning and development, and they aren't production-ready without additional hardening.
Overview
These examples demonstrate how to build voice agents that can be accessed via phone calls. Perfect for IVR systems, call centers, and voice-based customer service.
Available examples
XAI Native
Native XAI implementation with Twilio Media Streams.
Features:
- Direct WebSocket integration with XAI
- Real-time voice processing
- Session management
Tech Stack:
- Node.js + TypeScript
- Twilio Media Streams
- WebSockets
- Express server
Quick start
XAI Native (recommended)
Bash
cd xai
npm install
# Configure environment
cp .env.example .env
# Edit .env with your keys
# Start server
npm run dev
# Expose to internet (for Twilio)
ngrok http 3000
# Configure Twilio webhook with ngrok URL
# Call your Twilio number!
Architecture
Call flow
Text
┌─────────┐ 1. SIP ┌─────────────┐ 2. WebSocket ┌──────────────┐
│ Phone │ ←──────────→ │ Twilio │ ←──────────────→ │ Your Server │
│ Call │ Audio │Media Streams│ μ-law (native) │ (Node.js) │
└─────────┘ └─────────────┘ └──────────────┘
↓
3. WebSocket
↓
┌──────────────┐
│ XAI API │
│ (Realtime) │
└──────────────┘
Data flow
- Phone → Twilio: Caller dials your Twilio number
- Twilio → Server: Twilio streams μ-law PCM audio via WebSocket
- Server → XAI: Server forwards μ-law audio directly to XAI Realtime API (no conversion)
- XAI → Server: AI responds with μ-law audio and text
- Server → Twilio: Server forwards μ-law audio directly to Twilio (no conversion)
- Twilio → Phone: Caller hears AI response
Prerequisites
Required accounts
-
XAI Account
- Get API key: console.x.ai
- Realtime API access
-
Twilio Account
- Sign up: console.twilio.com
- Phone number with voice capabilities
- Media Streams enabled
Technical requirements
- Node.js: 18+
- Public Endpoint: ngrok or deployed server
- Port: 3000 (configurable)
Setup guide
1. Get Twilio credentials
Bash
# From Twilio Console
TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_AUTH_TOKEN=your_auth_token
TWILIO_PHONE_NUMBER=+1234567890
2. Configure the environment
Bash
cd xai
cp .env.example .env
# Edit .env with:
XAI_API_KEY=your_xai_api_key_here
HOSTNAME=your-ngrok-domain.ngrok.app
3. Expose to the internet
Using ngrok:
Bash
# Install ngrok
brew install ngrok # macOS
# or download from ngrok.com
# Start your server first
npm run dev
# In another terminal, expose it
ngrok http 3000
# Copy the HTTPS URL (e.g., https://abc123.ngrok.io)
4. Configure the Twilio webhook
- Go to Twilio Console
- Navigate to Phone Numbers → Manage → Active Numbers
- Click your phone number
- Under "Voice & Fax", set:
- A call comes in: Webhook
- URL:
https://your-ngrok-url.ngrok.io/twiml - Method: POST
- Save
5. Test the call
Bash
# Call your Twilio number
# You should hear the AI agent!
Features
XAI Native features
- Real-time voice processing
- Low latency responses
- Session management
- Call logging
- Error handling
- Graceful disconnection
Configuration options
Environment variables
Bash
# Required
XAI_API_KEY=your_key_here
HOSTNAME=your-ngrok-domain.ngrok.app
# Optional (with defaults)
API_URL=wss://api.x.ai/v1/realtime
PORT=3000
Note: Twilio credentials are configured in the Twilio Console, not as environment variables.
Voice configuration
JavaScript
// In your code
const sessionConfig = {
type: "session.update",
session: {
voice: "Ara", // Choose from: Ara, Rex, Sal, Eve, Leo
instructions: "You are a helpful assistant.",
// Audio format settings (these are the defaults if not specified)
audio: {
input: { format: { type: "audio/pcm", rate: 24000 } },
output: { format: { type: "audio/pcm", rate: 24000 } }
}
}
};
Audio processing
Audio format (no conversion needed!)
End-to-End Format:
- Format: μ-law PCM (audio/pcmu)
- Sample Rate: 8kHz
- Encoding: Base64
Twilio → Server → XAI:
- μ-law audio passes through without conversion
- XAI API natively supports PCMU (μ-law) format
XAI → Server → Twilio:
- μ-law audio passes through without conversion
- Direct passthrough improves latency and audio quality
Audio pipeline
JavaScript
// Incoming audio from Twilio (μ-law @ 8kHz)
const twilioAudio = μ-law PCM @ 8kHz
// Send directly to XAI (configured for native μ-law input)
await xaiWebSocket.send(twilioAudio)
// Receive from XAI (μ-law @ 8kHz - native telephony format)
const xaiResponse = μ-law PCM @ 8kHz
// Send directly to Twilio
await twilioWebSocket.send(xaiResponse)
Key Improvement: XAI API supports native μ-law (PCMU) and A-law (PCMA) formats, eliminating the need for PCM16 conversion and improving audio quality and latency.
Monitoring & debugging
Enable logging
Bash
# Set log level
LOG_LEVEL=debug npm run dev
Check the WebSocket connection
JavaScript
// Connection events
ws.on('open', () => console.log('Connected to XAI'));
ws.on('close', () => console.log('Disconnected from XAI'));
ws.on('error', (err) => console.error('WebSocket error:', err));
Testing
Local testing
Bash
# Start server
npm run dev
# Use ngrok for external access
ngrok http 3000
# Call your Twilio number
Test call script
Bash
# Make a test call via Twilio API
curl -X POST https://api.twilio.com/2010-04-01/Accounts/$TWILIO_ACCOUNT_SID/Calls.json \
--data-urlencode "Url=http://your-server.com/twiml" \
--data-urlencode "To=+1234567890" \
--data-urlencode "From=$TWILIO_PHONE_NUMBER" \
-u $TWILIO_ACCOUNT_SID:$TWILIO_AUTH_TOKEN
Documentation
- XAI Implementation README - XAI implementation details
- Twilio Docs - Twilio Media Streams
- XAI API Docs - XAI Realtime API
Examples
Use cases
IVR System:
- Menu navigation
- Call routing
- Information gathering
Customer Support:
- First-line support
- FAQ handling
- Appointment scheduling
Surveys:
- Voice surveys
- Feedback collection
- Market research
Notifications:
- Appointment reminders
- Payment alerts
- Status updates