voice-pipeline-config.yaml

for handling user interruptions interface OverlapHandler { handleInterruption(turnId: string): Promise<void>; }

// Filler injection strategy interface FillerStrategy { shouldInject(metrics: LatencyMetrics): boolean; getFillerAudio(): Buffer; }

class AdaptiveVoiceOrchestrator { private metrics: LatencyMetrics; private config: VoiceTurnConfig; private overlapHandler: OverlapHandler; private fillerStrategy: FillerStrategy;

constructor(config: VoiceTurnConfig, overlapHandler: OverlapHandler, fillerStrategy: FillerStrategy) { this.config = config; this.overlapHandler = overlapHandler; this.fillerStrategy = fillerStrategy; this.metrics = { p95: 0, p99: 0, currentTurnMs: 0 }; }

/**

• Processes a voice turn with adaptive filler injection and overlap handling. */ async processVoiceTurn(audioStream: AsyncIterable<Buffer>): Promise<void> { const turnStart = performance.now(); const turnId = crypto.randomUUID();

try {

  // 1. Stream STT and LLM inference
  // In production, this would involve chunked processing
  const llmResponse = await this.runInference(audioStream, turnId);
  
  // 2. Check latency budget and inject filler if needed
  const elapsed = performance.now() - turnStart;
  if (this.fillerStrategy.shouldInject({ ...this.metrics, currentTurnMs: elapsed })) {
    await this.injectFiller(turnId);
  }

  // 3. Stream TTS output
  await this.streamAudioOutput(llmResponse, turnId);

  // 4. Update metrics
  this.updateMetrics(performance.now() - turnStart);

} catch (error) {
  if (this.isOverlapError(error)) {
    await this.overlapHandler.handleInterruption(turnId);
  } else {
    throw error;
  }
}

}

private async injectFiller(turnId: string): Promise<void> { // Low-latency filler audio injection const fillerAudio = this.fillerStrategy.getFillerAudio(); // Send filler to output stream immediately await this.writeToOutput(fillerAudio, turnId); }

private isOverlapError(error: unknown): boolean { // Detect interruption signals from STT pipeline return error instanceof Error && error.message.includes('INTERRUPTION_DETECTED'); }

private updateMetrics(durationMs: number): void { // Update rolling p95/p99 calculations // Implementation depends on metrics library (e.g., Prometheus, Datadog) } }

**Architecture Decisions:**
*   **Predictive Filler Injection:** The system checks elapsed time against the `fillerTriggerMs`. If the LLM is taking longer than expected, a filler is injected before the user crosses the 500ms cliff. This prevents the overlap trap proactively.
*   **Safe Overlap Handling:** The `OverlapHandler` ensures that when a user interrupts, the pipeline aborts the current TTS stream, clears the LLM context if necessary, and resets the STT buffer. This prevents the "overlap death spiral" where pipeline state becomes corrupted.
*   **p95-Centric Metrics:** The orchestrator tracks p95 and p99 latency, not averages. Alerts and auto-scaling decisions should be based on these tail metrics to ensure cliff thresholds are respected.

#### 3. Optimization Techniques

*   **Speculative Decoding:** Use a draft model to propose tokens verified by the larger model, reducing effective inference time by 10-15%.
*   **Prompt Caching:** Cache embeddings for system prompts and common context to skip redundant processing.
*   **Edge Deployment:** Deploy inference closer to the user to reduce network RTT. Edge computing has reduced average voice agent latency from 2,500ms to ~600ms in production environments.
*   **Audio Tokenization:** Leverage models that accept audio tokens directly, bypassing the STT step for specific use cases, reducing pipeline stages.

### Pitfall Guide

1.  **The Linear Latency Illusion:** Treating latency as a continuous metric. Crossing a cliff (e.g., 500ms) changes the interaction type entirely. Optimization must prioritize staying below cliffs rather than incremental improvements within a failed zone. A 490ms response is fundamentally different from a 510ms response.
2.  **Downstream Optimization Bias:** Spending weeks optimizing TTS latency while the LLM inference stage dominates the critical path. Always profile the full pipeline; if LLM takes 400ms, shaving 50ms off TTS will not save you from the 500ms cliff.
3.  **Interruption Cascade:** Failing to handle user interruptions gracefully. When a user talks over the agent, the pipeline must abort the current stream and reset state. Without this, latency compounds, leading to rapid abandonment and pipeline corruption.
4.  **The Average Trap:** Reporting average latency hides the p95 tail where cliff crossings occur. Users form their worst impressions based on the slowest interactions. Always monitor and optimize p95/p99 metrics.
5.  **Delayed Auditory Feedback Blindspot:** Assuming latency is the only issue. Delayed auditory feedback causes physiological speech disruption. High-quality Acoustic Echo Cancellation (AEC) is mandatory for any system that may exceed 800ms or operate in noisy environments.
6.  **First-Turn Anchor Effect:** The first response sets the user's mental model. A slow first turn causes high drop-off even if subsequent turns are fast. Front-load optimization on the initial exchange; cache common openings.
7.  **The Waiter Demographic Fallacy:** Assuming users will wait. In testing, 8 out of 12 users overlapped at 500ms. You cannot design for the demographic that waits 3 seconds after a light turns green. Design for the majority who interrupt.

### Production Bundle

#### Action Checklist

- [ ] **Instrument p95 Metrics:** Configure monitoring to track p95 and p99 latency, not averages. Set alerts for p95 > 450ms.
- [ ] **Configure Fillers:** Enable filler injection for turns expected to exceed 400ms. Validate filler audio quality and latency.
- [ ] **Validate AEC:** Test Acoustic Echo Cancellation in production environments. Ensure no delayed feedback reaches the user.
- [ ] **Cache First Turn:** Implement caching for common opening exchanges to guarantee sub-300ms first responses.
- [ ] **Implement Overlap Handling:** Ensure the pipeline can abort streams and reset state on user interruption without data loss.
- [ ] **Route Simple Queries:** Configure model routing to send simple queries to fast-path edge models. Reserve heavy reasoning for complex turns.
- [ ] **Load Test with Interruptions:** Simulate user interruptions during load testing to verify pipeline stability and recovery.

#### Decision Matrix

| Scenario | Recommended Approach | Why | Cost Impact |
| :--- | :--- | :--- | :--- |
| **Simple FAQ / Transactional** | Edge LLM + Prompt Caching | Keeps latency < 300ms; avoids cloud RTT. | Low; edge compute is cheaper. |
| **Complex Reasoning / Analysis** | Cloud LLM + Aggressive Fillers | Cloud models are slower; fillers prevent overlap. | Medium; filler audio adds minimal cost. |
| **Noisy Environment** | High-Quality AEC + Robust STT | Echo and noise exacerbate latency perception. | Medium; AEC requires DSP resources. |
| **High Concurrency** | Speculative Decoding + Quantization | Reduces inference time per request. | Low; improves throughput efficiency. |
| **Budget-Constrained** | Kokoro TTS + Deepgram STT | Open-weight models reduce API costs. | Low; self-hosted models lower OpEx. |

#### Configuration Template

```yaml
# voice-pipeline-config.yaml
latency_budget:
  p95_target_ms: 450
  p99_target_ms: 800
  first_turn_target_ms: 300

fillers:
  enabled: true
  trigger_threshold_ms: 400
  audio_assets:
    - "mmhmm.wav"
    - "let_me_check.wav"
    - "one_second.wav"

routing:
  simple_queries:
    model: "edge-llm-7b-quantized"
    max_tokens: 100
  complex_queries:
    model: "cloud-llm-70b"
    max_tokens: 500

overlap_handling:
  strategy: "abort_and_reset"
  max_retries: 2
  silence_detection_ms: 500

metrics:
  provider: "prometheus"
  labels: ["turn_type", "model_name", "region"]

Quick Start Guide

1. Define Thresholds: Set your p95 target to 450ms and configure filler injection at 400ms. This provides a 50ms buffer before the 500ms overlap cliff.
2. Integrate Filler Service: Add a low-latency filler audio service to your pipeline. Ensure fillers can be injected within 50ms of the trigger.
3. Deploy Edge Inference: Route simple queries to an edge-deployed model. Measure the reduction in RTT and verify p95 improvements.
4. Instrument p95 Monitoring: Configure your metrics pipeline to track p95 and p99 latency. Set up dashboards to visualize cliff crossings in real-time.
5. Test Interruptions: Run load tests with simulated user interruptions. Verify that the pipeline aborts cleanly and recovers without state corruption.