rver-Side Credential Minting

Never expose provider API keys to the client. Instead, mint a short-lived session token that grants temporary access to the realtime stream. The gateway handles provider routing, so the client only needs the token and endpoint URL.

// app/api/voice/session/route.ts
import { gateway } from '@ai-sdk/gateway';
import { NextRequest } from 'next/server';

export async function POST(req: NextRequest) {
  const { searchParams } = new URL(req.url);
  const modelId = searchParams.get('model') || 'openai/gpt-realtime-2';

  try {
    const session = await gateway.experimental_realtime.getToken({
      model: modelId,
      metadata: {
        source: 'web-voice-agent',
        environment: process.env.NODE_ENV,
      },
    });

    return Response.json({
      endpoint: session.url,
      credential: session.token,
      capabilities: ['audio_in', 'audio_out', 'tool_execution'],
    });
  } catch (error) {
    console.error('Voice session mint failed:', error);
    return Response.json({ error: 'Session initialization failed' }, { status: 500 });
  }
}

Why this structure: Separating credential minting from client logic ensures API keys remain server-bound. Attaching metadata enables gateway-level observability tagging, which is critical for cost allocation and debugging across multiple voice features.

Step 2: Client-Side Stream Management

The browser requires a WebSocket wrapper that handles microphone capture, audio playback, and session state. AI SDK 7 provides a hook that abstracts the protocol details.

// components/VoiceInterface.tsx
'use client';

import { experimental_useRealtime as useVoiceStream } from '@ai-sdk/react';
import { gateway } from '@ai-sdk/gateway';
import { useCallback, useEffect, useRef } from 'react';

interface VoiceProps {
  modelId: string;
  onToolCall: (toolName: string, args: Record<string, unknown>) => Promise<unknown>;
}

export function VoiceInterface({ modelId, onToolCall }: VoiceProps) {
  const modelRef = useRef(
    gateway.experimental_realtime(modelId)
  );

  const {
    connectionState,
    establishConnection,
    terminateConnection,
    captureMicrophone,
    releaseMicrophone,
    handleServerEvent,
  } = useVoiceStream({
    model: modelRef.current,
    api: { credential: '/api/voice/session' },
    sessionConfig: {
      voice: 'alloy',
      turnDetection: { type: 'server-vad' },
      temperature: 0.7,
    },
  });

  useEffect(() => {
    if (connectionState === 'open') {
      captureMicrophone();
    }
    return () => releaseMicrophone();
  }, [connectionState]);

  const processToolCall = useCallback(async (event: MessageEvent) => {
    const payload = JSON.parse(event.data);
    if (payload.type === 'tool_call') {
      const result = await onToolCall(payload.name, payload.arguments);
      handleServerEvent({
        type: 'tool_result',
        callId: payload.callId,
        output: JSON.stringify(result),
      });
    }
  }, [onToolCall, handleServerEvent]);

  return (
    <div>
      <span>Status: {connectionState}</span>
      <button onClick={establishConnection}>Start Session</button>
      <button onClick={terminateConnection}>End Session</button>
    </div>
  );
}

Why this structure: The hook manages WebSocket lifecycle, audio encoding, and playback buffering. Server-side VAD eliminates client-side silence detection, reducing CPU usage and improving interruption accuracy. Tool execution is decoupled from the audio stream, allowing async operations without blocking playback.

Step 3: Async Audio Generation & Transcription

For non-realtime workloads, the gateway exposes single-request endpoints for synthesis and transcription. These functions compose cleanly with realtime sessions for hybrid workflows.

// lib/audio-pipeline.ts
import { generateSpeech, transcribe } from 'ai';
import { gateway } from '@ai-sdk/gateway';

export async function synthesizeSpokenResponse(
  text: string,
  voiceId: string = 'eve'
): Promise<Uint8Array> {
  const output = await generateSpeech({
    model: gateway('xai/grok-tts'),
    text,
    voice: voiceId,
    outputFormat: 'mp3',
  });
  return output.audio.uint8Array;
}

export async function extractTranscript(
  audioBuffer: ArrayBuffer | string | URL
): Promise<string> {
  const result = await transcribe({
    model: gateway('openai/whisper-1'),
    audio: audioBuffer,
  });
  return result.text;
}

Why this structure: Using the gateway wrapper ensures consistent routing, observability, and BYOK support across all audio operations. MP3 output balances quality and bandwidth for web delivery. The transcription function accepts multiple input types, enabling flexible integration with file uploads, base64 payloads, or direct URLs.

Pitfall Guide

1. Client-Side Key Exposure

Explanation: Hardcoding provider API keys in frontend bundles allows malicious actors to extract credentials and incur unauthorized costs. Fix: Always mint short-lived tokens server-side. Use HTTP-only cookies or secure headers for token delivery. Rotate credentials automatically via gateway budget controls.

2. Client-Side Silence Detection Overhead

Explanation: Implementing local VAD or manual silence timers consumes CPU, introduces false positives, and degrades interruption accuracy. Fix: Configure turnDetection: { type: 'server-vad' } in the session config. Let the gateway handle audio endpointing and barge-in detection.

3. Tool Call Deadlocks During Streaming

Explanation: Blocking the audio thread while waiting for tool execution causes playback stalls and session timeouts. Fix: Decouple tool execution from the stream loop. Queue tool requests, execute them asynchronously, and push results back via tool_result events without interrupting audio playback.

4. Unbounded Audio Session Costs

Explanation: Continuous realtime sessions can accumulate significant token usage if left open or if users engage in long conversations without timeout logic. Fix: Implement session expiration policies. Use gateway spend limits and budget alerts. Add client-side idle detection to terminate inactive sessions after 3–5 minutes.

5. Format Mismatch in Transcription Pipelines

Explanation: Feeding uncompressed WAV or high-sample-rate audio to transcription models increases latency and costs without improving accuracy. Fix: Normalize audio to 16kHz mono PCM or MP3 before transcription. Use gateway routing to automatically select the optimal model variant based on input format.

6. Ignoring Gateway Observability Tags

Explanation: Failing to attach metadata to requests makes it impossible to attribute costs, debug failures, or analyze usage patterns across features. Fix: Include metadata fields in token minting and model calls. Tag requests with feature, user_tier, and environment for granular dashboard filtering.

7. Mixing Modalities Without Unified Routing

Explanation: Calling text, image, and audio models through separate SDKs or endpoints fragments logs, complicates budget tracking, and increases maintenance overhead. Fix: Route all modalities through AI Gateway. Use a single API key, unified observability dashboard, and consistent error handling patterns across the stack.

Production Bundle

Action Checklist

• Configure server-side token minting with short-lived credentials and metadata tagging
• Enable server-side VAD in session configuration to handle interruptions natively
• Implement async tool execution queue to prevent audio stream blocking
• Set gateway budget limits and spend alerts for continuous audio sessions
• Normalize audio formats to 16kHz PCM or MP3 before transcription
• Attach consistent observability metadata across all modalities
• Add client-side idle detection to terminate inactive sessions automatically
• Test barge-in behavior and tool-call concurrency under network degradation

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Interactive voice assistant with natural conversation	Realtime Gateway (gpt-realtime-2)	Server-side VAD enables barge-in and sub-500ms latency	Higher per-minute, but reduces infrastructure overhead
Batch voice note transcription	Async STT (whisper-1) via Gateway	Single-request, optimized for accuracy over latency	Predictable per-minute pricing, easy budget control
Dynamic voiceovers for generated content	Async TTS (grok-tts) via Gateway	On-demand synthesis with format flexibility	Low cost, scales linearly with content volume
Multi-provider fallback requirements	Gateway routing with BYOK	Automatic failover and unified observability	Minimal overhead, eliminates provider lock-in
Mobile/low-bandwidth environments	Realtime with Opus encoding + server VAD	Reduces payload size and client CPU usage	Slightly higher gateway routing cost, lower bandwidth

Configuration Template

// gateway.config.ts
import { gateway } from '@ai-sdk/gateway';

export const voiceGateway = gateway.configure({
  apiKey: process.env.AI_GATEWAY_KEY,
  defaultModel: 'openai/gpt-realtime-2',
  observability: {
    enabled: true,
    tags: {
      team: 'voice-platform',
      environment: process.env.NODE_ENV || 'development',
    },
  },
  budgets: {
    maxMonthlySpend: 500,
    alertThreshold: 0.8,
    sessionTimeout: 300000, // 5 minutes
  },
  routing: {
    fallbackModels: ['xai/grok-tts', 'openai/whisper-1'],
    preferLowLatency: true,
  },
});

Quick Start Guide

1. Install dependencies: npm install ai @ai-sdk/react @ai-sdk/gateway
2. Create token route: Implement a server endpoint that calls gateway.experimental_realtime.getToken() and returns the credential and endpoint URL.
3. Initialize client hook: Import experimental_useRealtime, pass the gateway model instance, and configure turnDetection: { type: 'server-vad' }.
4. Handle tool execution: Intercept tool_call events, run your business logic, and push results back via tool_result without blocking audio playback.
5. Deploy with observability: Attach metadata tags to all requests, configure gateway spend limits, and monitor latency/usage through the unified dashboard.