Benchmarking five live translation systems with an open-source eval harness (including OpenAI's GPT-Realtime-Translate)

Engineering Reliable Real-Time Translation: A Framework for Evaluating Speech-to-Speech Latency and Accuracy

Current Situation Analysis

The shift from text-based translation to live speech-to-speech (S2S) translation introduces a dual constraint that traditional NLP benchmarks fail to capture: conversational latency and semantic fidelity under streaming conditions. As platforms like OpenAI's GPT-Realtime-Translate and other proprietary engines enter the market, development teams face a critical evaluation gap. Most organizations rely on static metrics like Word Error Rate (WER) and Time-to-First-Byte (TTFB), which provide misleading signals for real-time audio interactions.

This problem is often overlooked because S2S systems are treated as black boxes where audio in equals audio out. However, the user experience is defined by the gap between the speaker finishing an utterance and the listener hearing the translation, not just the initial response time. Furthermore, accuracy in live translation requires preserving intent and nuance across languages, which character-level metrics like BLEU or WER cannot adequately assess.

Data from recent evaluations of live S2S platforms demonstrates that systems optimized for low TTFB often sacrifice semantic completeness, while high-accuracy models may introduce latency that breaks conversational flow. The industry requires a standardized evaluation harness that measures Ear-Voice Span (EVS) for latency and GEMBA-MQM v2 for accuracy. These metrics align technical performance with human perception, enabling teams to make data-driven decisions when selecting or tuning translation engines.

WOW Moment: Key Findings

The divergence between traditional benchmarking and live-optimized evaluation reveals why many production deployments fail to meet UX expectations. The table below contrasts the limitations of legacy metrics against the insights provided by EVS and GEMBA-MQM v2.

Metric Category	Traditional Approach	Live-Optimized Approach	Impact on Production
Latency	TTFB (Time-to-First-Byte)	EVS (Ear-Voice Span)	TTFB ignores streaming overhead and audio synthesis time. EVS captures the actual delay perceived by the listener, including ASR, translation, and TTS pipeline duration.
Accuracy	WER / BLEU	GEMBA-MQM v2	WER penalizes harmless variations and misses semantic errors. GEMBA-MQM v2 categorizes errors by severity (Minor/Major/Critical) based on multilingual quality standards, providing actionable quality signals.
Streaming	Static Batch Evaluation	Real-Time Stream Analysis	Batch evaluation misses partial result stability and mid-stream corrections. Live analysis tracks how the system handles interruptions and evolving context.
Decision Value	Low	High	Traditional metrics may lead to selecting a fast but inaccurate engine. Live-optimized metrics ensure the selected engine supports natural, reliable conversation.

Why this matters: By adopting EVS and GEMBA-MQM v2, engineering teams can quantify the trade-off between speed and quality with precision. This enables the configuration of thresholds that prevent "hallucinated" translations in critical scenarios while maintaining latency below the 800ms threshold required for natural dialogue.

Core Solution

Building a robust evaluation harness for live S2S translation requires a pipeline that captures audio streams, synchronizes timestamps, and computes metrics without interfering with the engine's real-time processing. The following architecture implements this using TypeScript, focusing on modularity and extensibility.

Architecture Decisions

1. Timestamp Synchronization: EVS calculation depends on precise timing. The harness uses a unified clock reference to align speaker end-times with listener start-times.
2. Metric Decoupling: Latency and accuracy metrics are computed independently. This allows teams to swap accuracy scorers (e.g., from GEMBA-MQM v2 to a custom model) without affecting latency tracking.
3. Stream-Aware Processing: The harness processes audio in chunks to support streaming engines. It handles partial results and final segments separately to measure stability.

Implementation

The following code defines the core evaluation harness. It includes interfaces for results, a latency tracker for EVS, and an accuracy scorer integrating GEMBA-MQM v2.

1. Core Interfaces and Harness Structure

import { EventEmitter } from 'events';

export interface S2SEngineConfig {
  provider: string;
  model: string;
  apiKey: string;
  targetLanguage: string;
}

export interface EvaluationMetrics {
  evsMs: number;
  gembaScore: number;
  errorCategories: string[];
}

export interface EvaluationResult {
  segmentId: string;
  sourceAudio: Buffer;
  translatedAudio: Buffer;
  metrics: EvaluationMetrics;
  timestamp: number;
}

export class LiveS2SEvaluator extends EventEmitter {
  private latencyTracker: LatencyTracker;
  private accuracyScorer: AccuracyScorer;

  constructor() {
    super();
    this.latencyTracker = new LatencyTracker();
    this.accuracyScorer = new AccuracyScorer();
  }

  async evaluateStream(
    engine: S2SEngine,
    inputAudioStream: AsyncIterable<Buffer>
  ): Promise<EvaluationResult[]> {
    const results: EvaluationResult[] = [];
    let segmentId = 0;

    for await (const audioChunk of inputAudioStream) {
      const startTime = performance.now();
      
      // Capture speaker end time from audio activity detection
      const speakerEnd = await this.latencyTracker.detectSpeechEnd(audioChunk);
      
      // Send to engine and receive translation
      const translationStream = engine.translate(audioChunk);
      const translatedAudio = await this.collectAudio(translationStream);
      
      const listenerStart = performance.now();
      
      // Calculate EVS
      const evs = this.latencyTracker.computeEVS(speakerEnd, listenerStart);
      
      // Score accuracy using GEMBA-MQM v2
      const accuracy = await this.accuracyScorer.score(
        audioChunk, 
        translatedAudio, 
        engine.config.targetLanguage
      );

      const result: EvaluationResult = {
        segmentId: `seg_${segmentId++}`,
        sourceAudio: audioChunk,
        translatedAudio,
        metrics: {
          evsMs: evs,
          gembaScore: accuracy.score,
          errorCategories: accuracy.errors
        },
        timestamp: Date.now()
      };

      results.push(result);
      this.emit('segmentEvaluated', result);
    }

    return results;
  }

  private async collectAudio(stream: AsyncIterable<Buffer>): Promise<Buffer> {
    const chunks: Buffer[] = [];
    for await (const chunk of stream) {
      chunks.push(chunk);
    }
    return Buffer.concat(chunks);
  }
}

2. Latency Tracker with EVS Calculation

Ear-Voice Span is calculated as the difference between the time the speaker finishes and the time the listener begins hearing the translation. This includes all pipeline stages.

export class LatencyTracker {
  private clockOffset: number = 0;

  async detectSpeechEnd(audioChunk: Buffer): Promise<number> {
    // Implementation would use VAD (Voice Activity Detection)
    // to find the precise timestamp where speech ends.
    // Returns timestamp in ms relative to unified clock.
    return performance.now() + this.clockOffset;
  }

  computeEVS(speakerEndMs: number, listenerStartMs: number): number {
    // EVS = Listener Start - Speaker End
    // Negative values indicate overlap (interruption handling)
    const evs = listenerStartMs - speakerEndMs;
    return Math.max(0, evs);
  }

  calibrateClock(offset: number): void {
    this.clockOffset = offset;
  }
}

3. Accuracy Scorer with GEMBA-MQM v2

GEMBA-MQM v2 provides a score based on Multilingual Quality Metric standards. It requires transcribing the target audio to text for comparison, then scoring semantic equivalence.

export class AccuracyScorer {
  async score(
    sourceAudio: Buffer, 
    targetAudio: Buffer, 
    targetLang: string
  ): Promise<{ score: number; errors: string[] }> {
    // Step 1: Transcribe target audio to text for evaluation
    // In production, use a high-quality ASR model for the target language
    const targetText = await this.transcribeTarget(targetAudio, targetLang);
    
    // Step 2: Extract source text (assuming source is transcribed or provided)
    const sourceText = await this.extractSourceText(sourceAudio);
    
    // Step 3: Invoke GEMBA-MQM v2
    // This would call the GEMBA API or local model
    const gembaResult = await this.invokeGembaMqmV2(sourceText, targetText, targetLang);
    
    return {
      score: gembaResult.overallScore,
      errors: gembaResult.errorCategories
    };
  }

  private async invokeGembaMqmV2(
    source: string, 
    target: string, 
    lang: string
  ): Promise<GembaResponse> {
    // Mock implementation of GEMBA-MQM v2 integration
    // Returns score between 0 and 1, and list of error types
    return {
      overallScore: 0.85,
      errorCategories: ['Minor: Terminology', 'No Error']
    };
  }
  
  // Helper methods for transcription would be implemented here
  private async transcribeTarget(audio: Buffer, lang: string): Promise<string> { return ''; }
  private async extractSourceText(audio: Buffer): Promise<string> { return ''; }
}

interface GembaResponse {
  overallScore: number;
  errorCategories: string[];
}

Rationale:

• EVS over TTFB: TTFB measures network latency to the first byte of response, which is irrelevant for audio synthesis. EVS measures the actual user-perceived delay.
• GEMBA-MQM v2: Unlike BLEU, which compares n-gram overlap, GEMBA-MQM v2 evaluates semantic meaning and categorizes errors. This is crucial for S2S where a translation might be grammatically correct but semantically wrong.
• Modular Design: Separating LatencyTracker and AccuracyScorer allows independent optimization and testing.

Pitfall Guide

1. Measuring TTFB Instead of EVS

   • Explanation: Teams often optimize for low TTFB, assuming it correlates with responsiveness. However, TTFB ignores the time required for text-to-speech synthesis and streaming audio delivery.
   • Fix: Implement EVS calculation that captures the full pipeline from speaker end to listener start. Ensure timestamps are synchronized across all components.

2. Relying on WER for Semantic Accuracy

   • Explanation: WER penalizes word substitutions that may not affect meaning, while missing semantic hallucinations. A low WER score can mask critical translation errors.
   • Fix: Adopt GEMBA-MQM v2 or similar semantic metrics. Use high-quality ASR for ground truth generation to avoid transcription errors skewing results.

3. Ignoring Network Jitter and Real-World Conditions

   • Explanation: Benchmarks run on localhost or high-bandwidth connections do not reflect production environments where packet loss and latency spikes occur.
   • Fix: Integrate network simulation tools (e.g., tc on Linux or network throttling libraries) to test under realistic conditions. Report metrics with confidence intervals.

4. Clock Drift in Distributed Systems

   • Explanation: If the evaluation harness and the S2S engine run on different machines, clock skew can invalidate EVS calculations.
   • Fix: Use NTP synchronization or embed timestamps in the audio stream metadata. Calibrate the clock offset at the start of each evaluation session.

5. Evaluating Only Final Results

   • Explanation: Live S2S engines often stream partial results. Evaluating only the final output misses issues with mid-stream corrections or instability.
   • Fix: Capture and analyze partial results. Measure how often the engine revises translations and the impact on user experience.

6. Single-Pass Evaluation

   • Explanation: Running a single test pass can yield skewed results due to transient network conditions or engine warm-up effects.
   • Fix: Run multiple iterations and aggregate results. Use statistical methods to determine significance. Discard warm-up segments.

7. Language Pair Bias

   • Explanation: Performance varies significantly across language pairs. Optimizing for English-to-Spanish may not generalize to English-to-Japanese.
   • Fix: Test a diverse set of language pairs relevant to your user base. Report metrics per language pair to identify specific weaknesses.

Production Bundle

Action Checklist

• Define EVS Thresholds: Establish maximum acceptable EVS based on UX research (e.g., <800ms for natural conversation).
• Integrate GEMBA-MQM v2: Set up the accuracy scorer to use GEMBA-MQM v2 for semantic evaluation. Ensure access to the model or API.
• Implement Audio Capture Pipeline: Build the harness to capture input and output audio streams with precise timestamping.
• Calibrate Clocks: Synchronize clocks across all evaluation components and validate offset calculations.
• Run Baseline Tests: Evaluate OpenAI GPT-Realtime-Translate and other target engines to establish baseline metrics.
• Simulate Network Conditions: Configure network throttling to test robustness under realistic constraints.
• Automate Regression: Integrate the harness into CI/CD pipelines to detect performance regressions on engine updates.
• Review Error Categories: Analyze GEMBA-MQM v2 error categories to identify systematic issues in translation quality.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Real-Time Customer Support	Prioritize EVS < 600ms; Accept moderate GEMBA score.	Low latency is critical for agent efficiency. Minor errors can be corrected verbally.	Moderate. May require streaming-optimized engines.
Medical Translation	Prioritize GEMBA-MQM v2 > 0.90; EVS < 1000ms.	Semantic accuracy is paramount. Safety risks outweigh latency concerns.	High. May require premium models and human-in-the-loop verification.
Gaming / Live Events	Prioritize EVS < 400ms; Use lightweight models.	Ultra-low latency required for synchronization. Accuracy can be lower.	Low to Moderate. Focus on edge deployment and optimization.
General Purpose App	Balance EVS < 800ms and GEMBA > 0.85.	Provides a good trade-off for most use cases.	Moderate. Standard cloud APIs usually suffice.

Configuration Template

Use this YAML configuration to define evaluation parameters for the harness.

evaluation:
  engine:
    provider: "openai"
    model: "gpt-realtime-translate"
    api_key: "${OPENAI_API_KEY}"
    target_language: "es"
  
  metrics:
    latency:
      type: "evs"
      max_threshold_ms: 800
    accuracy:
      type: "gemba_mqm_v2"
      min_score: 0.85
  
  network:
    simulation:
      enabled: true
      latency_ms: 50
      jitter_ms: 10
      packet_loss_percent: 0.5
  
  execution:
    iterations: 10
    warmup_segments: 2
    output_format: "json"
    report_path: "./reports/eval_report.json"

Quick Start Guide

1. Install Dependencies:

   npm install @codcompass/s2s-eval-harness
   

2. Configure Environment: Set your API keys and configuration in .env or the YAML template.

   export OPENAI_API_KEY="your-key-here"
   

3. Run Evaluation: Execute the harness against your audio dataset.

   npx s2s-eval run --config eval_config.yaml --input ./audio_samples/
   

4. Review Results: Analyze the generated report for EVS distribution and GEMBA-MQM v2 scores.

   cat reports/eval_report.json | jq '.metrics'
   

5. Iterate: Adjust engine parameters or switch providers based on the metrics. Re-run to validate improvements.

By implementing this framework, teams can move beyond superficial benchmarks and ensure their live translation systems meet the rigorous demands of production environments. The combination of EVS and GEMBA-MQM v2 provides a comprehensive view of performance, enabling data-driven optimization for both latency and accuracy.