Decoupling AI Telephony: A BYOM Architecture with Asterisk PJSIP and VAPI

Current Situation Analysis

Modern AI voice platforms have abstracted telephony into a managed SaaS layer, but this abstraction introduces a linear cost curve that becomes unsustainable at scale. When you route AI agents through hosted telephony providers, you pay for signaling, media relay, transcription, and inference on a per-minute basis. For a single pilot project, this pricing model is acceptable. For multi-tenant deployments or businesses running hundreds of concurrent AI interactions, the cumulative overhead fractures unit economics.

The core misunderstanding lies in treating real-time voice as a standard REST API. Telephony is a stateful, latency-sensitive streaming protocol. SaaS platforms charge premiums to manage NAT traversal, codec negotiation, jitter buffering, and global SIP trunking. While convenient, this model locks you into vendor-specific rate limits, obscures actual media routing paths, and prevents protocol-level optimizations.

Data from production deployments shows a clear inflection point. Hosted AI voice solutions typically bill $0.10–$0.15 per minute. At 500 minutes monthly per business unit, that translates to $50–$75 in telephony overhead alone. By decoupling the AI inference layer from the telephony transport layer using a Bring-Your-Own-Media (BYOM) architecture, you shift from variable per-minute pricing to fixed infrastructure costs. A 4GB RAM / 2 vCPU instance running Asterisk 18 with PJSIP, paired with a standard SIP trunk and VAPI for AI orchestration, stabilizes baseline expenditure at approximately $47 monthly. Scaling is then bounded only by CPU cycles and network I/O, not by platform rate limits.

WOW Moment: Key Findings

The architectural shift from managed telephony to self-hosted SIP routing fundamentally changes how you model AI voice costs and performance. The following comparison isolates the operational and economic deltas between the two approaches.

Approach	Monthly Baseline Cost	Per-Minute Overhead	Max Concurrent Streams	Latency Overhead	Infrastructure Control
Hosted AI Voice SaaS	$29–$99 (platform fee)	$0.10–$0.15	Platform-capped (often 5–10)	200–350ms (media relay hops)	None (black-box routing)
Self-Hosted Asterisk + VAPI BYOM	~$47 (VPS + DID + trunk)	$0.008–$0.012 (SIP trunk rate)	12–20 (hardware-bound)	150–200ms (direct PJSIP path)	Full (dialplan, codecs, NAT, TLS)

This finding matters because it decouples AI capability from telephony economics. You retain the ability to swap LLM providers, adjust TTS parameters, or modify agent behavior through VAPI's API, while maintaining complete sovereignty over SIP signaling, RTP media paths, and codec negotiation. The performance delta is equally significant: eliminating third-party media relays reduces hop count, stabilizes jitter buffers, and brings end-to-end latency into the conversational threshold (<200ms).

Core Solution

Building a BYOM telephony stack requires three coordinated layers: a SIP signaling engine, an AI orchestration API, and a transport bridge that routes RTP streams without transcoding bottlenecks. Asterisk 18 with PJSIP serves as the signaling and media controller. VAPI acts as the AI agent runtime. A standard SIP trunk provider handles PSTN ingress/egress.

Architecture Rationale

1. PJSIP over chan_sip: PJSIP is the actively maintained SIP stack in Asterisk. It supports WebRTC, modern NAT traversal, and granular codec negotiation. chan_sip is deprecated and lacks the performance characteristics required for AI media streams.
2. BYOM Decoupling: VAPI's BYOM model allows you to register a SIP endpoint directly to your Asterisk instance. This eliminates VAPI's media relay, routing audio directly between the caller's RTP stream and the AI agent's WebSocket pipeline.
3. Codec Locking: AI voice pipelines are optimized for 8kHz PCM. Enabling HD codecs (G.722, Opus) forces transcoding at the SIP trunk or VAPI boundary, introducing 50–100ms of latency and CPU overhead. Explicitly whitelisting G.711 (ulaw/alaw) ensures deterministic media handling.

Implementation Steps

Step 1: Provision the SIP Endpoint in Asterisk

Configure PJSIP to accept registrations from VAPI's cloud infrastructure. The endpoint must enforce strict codec ordering, disable direct media (to ensure Asterisk handles NAT/routing), and enable basic media encryption for signaling security.

Step 2: Define Dialplan Routing

The dialplan acts as the traffic controller. Inbound calls from the SIP trunk are routed to the VAPI endpoint. Outbound calls initiated by VAPI are forwarded to the PSTN trunk. Hangup handlers ensure clean channel teardown and resource release.

Step 3: Orchestrate VAPI via TypeScript

Use a typed Node.js module to provision virtual DIDs, configure agent profiles, and manage API authentication. The module should abstract VAPI's REST endpoints into reusable methods with proper error boundaries and retry logic.

TypeScript Integration Module

import { z } from 'zod';

const VapiConfigSchema = z.object({
  apiKey: z.string().min(1),
  baseUrl: z.string().url(),
  assistantId: z.string().uuid(),
  sipServer: z.string().ip(),
  sipUsername: z.string().min(1),
  sipPassword: z.string().min(1),
});

type VapiConfig = z.infer<typeof VapiConfigSchema>;

export class TelephonyOrchestrator {
  private config: VapiConfig;
  private headers: Record<string, string>;

  constructor(config: VapiConfig) {
    this.config = VapiConfigSchema.parse(config);
    this.headers = {
      Authorization: `Bearer ${this.config.apiKey}`,
      'Content-Type': 'application/json',
    };
  }

  async provisionVirtualDID(): Promise<Record<string, unknown>> {
    const payload = {
      provider: 'byom',
      byomPhoneNumber: {
        server: this.config.sipServer,
        username: this.config.sipUsername,
        password: this.config.sipPassword,
      },
      assistantId: this.config.assistantId,
    };

    const response = await fetch(`${this.config.baseUrl}/phone-number`, {
      method: 'POST',
      headers: this.headers,
      body: JSON.stringify(payload),
    });

    if (!response.ok) {
      throw new Error(`DID provisioning failed: ${response.statusText}`);
    }

    return response.json();
  }

  async configureAgentProfile(): Promise<Record<string, unknown>> {
    const payload = {
      model: {
        provider: 'openai',
        model: 'gpt-4',
        temperature: 0.1,
      },
      voice: {
        provider: 'eleven-labs',
        voiceId: 'process_voice_id_here',
      },
      firstMessage: 'System initialized. Awaiting caller input.',
    };

    const response = await fetch(`${this.config.baseUrl}/assistant`, {
      method: 'POST',
      headers: this.headers,
      body: JSON.stringify(payload),
    });

    if (!response.ok) {
      throw new Error(`Agent configuration failed: ${response.statusText}`;
    }

    return response.json();
  }
}

This implementation replaces ad-hoc scripting with a schema-validated, type-safe orchestrator. The zod validation prevents misconfigured payloads from reaching VAPI's API. Error handling is explicit, and the class structure allows easy extension for webhook listeners, call analytics, or multi-tenant routing.

Pitfall Guide

Self-hosting telephony infrastructure introduces protocol-level complexities that managed platforms hide. The following pitfalls represent the most frequent production failures and their resolutions.

1. Codec Negotiation Mismatch

Explanation: Allowing Asterisk to advertise multiple codecs causes VAPI's media engine to fallback to transcoding when the SIP trunk prefers a different format. This adds 50–100ms latency and increases CPU load. Fix: Explicitly disallow all codecs and whitelist only ulaw and alaw in the PJSIP endpoint. Set disallow=all followed by allow=ulaw and allow=alaw.

2. NAT/RTP Asymmetry

Explanation: When Asterisk sits behind a firewall or cloud VPS NAT, RTP packets may route to private IPs, causing one-way audio or complete media drop. Fix: Configure external_media_address and external_signaling_address in the [global] section. Enable rtp_symmetric=yes to force media back to the source port.

3. SIP Registration Storms

Explanation: VAPI's cloud infrastructure may attempt multiple concurrent registrations during failover or scaling events. Without contact limits, Asterisk's AOR table fills, causing registration rejections. Fix: Set max_contacts=5 and remove_existing=yes on the AOR. Add qualify=yes to monitor endpoint health and prune stale registrations.

4. Unbounded Dialplan Execution

Explanation: Missing timeouts or hangup handlers leave channels in a DIAL state indefinitely, consuming file descriptors and memory. Fix: Always specify a timeout in Dial(). Attach a hangup_handler_wipe to clean up variables, release RTP resources, and log termination events.

5. VAPI Webhook Rate Limits

Explanation: High call volume triggers VAPI's webhook endpoints (call start, end, transcription). Without backoff logic, your server may hit 429 responses, dropping call state. Fix: Implement exponential backoff with jitter. Queue webhook payloads in Redis or a message broker, and process them asynchronously.

6. Missing SRTP/TLS for Signaling

Explanation: Transmitting SIP credentials and SDP offers over plaintext UDP exposes authentication tokens and media negotiation data to network sniffing. Fix: Enable media_encryption=sdes on the endpoint. For production, migrate signaling to TLS (port 5061) and enforce transport=tls in the PJSIP transport definition.

7. Ignoring CPU Throttling on Concurrent Streams

Explanation: Each concurrent AI call consumes CPU for RTP jitter buffering, DTMF detection, and hangup handler execution. A 2 vCPU instance will saturate at ~12–15 streams, causing audio artifacts. Fix: Monitor astctl metrics. Implement rtp_keepalive=yes to detect dead channels early. Scale vertically or shard dialplans across multiple Asterisk instances when approaching 70% CPU utilization.

Production Bundle

Action Checklist

• Provision VPS with 4GB+ RAM and 2+ vCPUs; configure UFW/iptables to allow UDP 5060 and UDP 10000-20000
• Install Asterisk 18 from official repositories; enable PJSIP modules and disable deprecated chan_sip
• Configure pjsip.conf with strict codec whitelisting, NAT parameters, and AOR contact limits
• Define extensions.conf contexts for inbound VAPI routing, outbound trunk dialing, and hangup cleanup
• Deploy TypeScript orchestrator; validate VAPI BYOM registration and assistant provisioning
• Run concurrent load test (10–12 calls); monitor CPU, RTP jitter, and setup latency via asterisk -rvvv
• Implement webhook queue with exponential backoff; configure Prometheus/Grafana for SIP metrics

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Startup / Pilot (<100 min/mo)	Hosted AI Voice SaaS	Zero infrastructure overhead; fast deployment	$10–$15/mo
Multi-tenant Agency (500+ min/mo)	Self-Hosted Asterisk + VAPI BYOM	Linear cost decoupling; full protocol control	~$47/mo flat
Compliance-Heavy (HIPAA/PCI)	Self-Hosted + On-Prem SIP Trunk	Media never leaves controlled network; auditability	Higher CapEx, lower OpEx
Global Scale (Multi-Region)	Asterisk Cluster + VAPI Regional Endpoints	Latency optimization; failover routing	$80–$120/mo (multi-node)

Configuration Template

/etc/asterisk/pjsip.conf

[transport-udp]
type=transport
protocol=udp
bind=0.0.0.0:5060

[global]
external_media_address=YOUR_PUBLIC_IP
external_signaling_address=YOUR_PUBLIC_IP
rtp_symmetric=yes

[vapi-sip-endpoint]
type=endpoint
context=ai-voice-context
disallow=all
allow=ulaw
allow=alaw
auth=vapi-credentials
aors=vapi-contact-list
direct_media=no
ice_support=yes
media_encryption=sdes
qualify=yes
max_audio_streams=1

[vapi-credentials]
type=auth
auth_type=userpass
username=ai_agent_sip
password=STRONG_RANDOM_PASSWORD_HERE

[vapi-contact-list]
type=aor
max_contacts=5
remove_existing=yes

/etc/asterisk/extensions.conf

[ai-voice-context]
exten => _X.,1,NoOp(Routing inbound to AI agent: ${CALLERID(num)})
same => n,Set(CHANNEL(hangup_handler_wipe)=cleanup-ai-call,s,1)
same => n,Dial(PJSIP/vapi-sip-endpoint,30)
same => n,Hangup()

[cleanup-ai-call]
exten => s,1,NoOp(AI call terminated. Releasing resources.)
same => n,Return()

[outbound-pstn]
exten => _1NXXNXXXXXX,1,NoOp(Outbound PSTN: ${EXTEN})
same => n,Dial(PJSIP/${EXTEN}@sip-trunk-provider,60)
same => n,Hangup()

Quick Start Guide

1. Initialize the VPS: Deploy a 4GB/2 vCPU instance. Install Asterisk 18 via package manager. Enable pjsip, res_pjsip, and app_dial modules.
2. Apply Configuration: Copy the pjsip.conf and extensions.conf templates. Replace YOUR_PUBLIC_IP, credentials, and trunk references. Run asterisk -rx "module reload" to apply.
3. Provision VAPI: Execute the TypeScript orchestrator with your API key and SIP server IP. Verify the BYOM registration appears in asterisk -rx "pjsip show endpoints".
4. Test Call Flow: Dial the provisioned DID. Confirm RTP streams establish within 2.3 seconds. Monitor asterisk -rvvv for codec negotiation and hangup handler execution.
5. Validate Metrics: Run a 10-call concurrent test. Verify CPU stays below 70%, latency remains <200ms, and no RTP drops occur. Deploy monitoring hooks for production traffic.