ng execution.

Step 1: Configure Hardware-Accelerated Encoding

The prefer-hardware flag instructs the browser to route video compression through the device's dedicated encoder. This is configured during encoder initialization. Graceful degradation is mandatory, as hardware support varies by OS and GPU driver.

interface EncoderConfig {
  codec: string;
  width: number;
  height: number;
  bitrate: number;
  hardwareAcceleration: 'prefer-hardware' | 'no-preference';
}

async function initializeVideoEncoder(config: EncoderConfig): Promise<VideoEncoder> {
  const encoder = new VideoEncoder({
    output: (chunk, meta) => {
      // Handle encoded chunk (e.g., append to MediaSource or Blob)
      handleEncodedChunk(chunk, meta);
    },
    error: (err) => console.error('Encoder failure:', err)
  });

  const supported = await VideoEncoder.isConfigSupported({
    codec: config.codec,
    width: config.width,
    height: config.height,
    hardwareAcceleration: config.hardwareAcceleration
  });

  if (!supported.supported) {
    console.warn('Hardware encoding unavailable, falling back to software');
    config.hardwareAcceleration = 'no-preference';
  }

  encoder.configure(config);
  return encoder;
}

Architecture Rationale: The VideoEncoder API is used instead of MediaRecorder for granular control over frame timing and metadata. Checking isConfigSupported prevents runtime crashes on devices with outdated drivers. The fallback ensures consistent output regardless of hardware availability.

Step 2: Implement GPU-Bound Chroma Keying

RGB Euclidean distance fails under variable lighting. Shadows shift hue, highlights desaturate, and fixed thresholds produce jagged edges. The correct approach converts RGB to HSV, thresholds hue and saturation independently, and applies a smooth alpha blend.

const CHROMA_KEY_FRAGMENT = `
  precision mediump float;
  uniform sampler2D uVideoFrame;
  uniform vec3 uTargetHueRange; // [minHue, maxHue, tolerance]
  uniform float uSaturationFloor;
  uniform float uEdgeSoftness;
  varying vec2 vUV;

  vec3 rgbToHsv(vec3 c) {
    vec4 K = vec4(0.0, -1.0 / 3.0, 2.0 / 3.0, -1.0);
    vec4 p = mix(vec4(c.bg, K.wz), vec4(c.gb, K.xy), step(c.b, c.g));
    vec4 q = mix(vec4(p.xyw, c.r), vec4(c.r, p.yzx), step(p.x, c.r));
    float d = q.x - min(q.w, q.y);
    float e = 1.0e-10;
    return vec3(abs(q.z + (q.w - q.y) / (6.0 * d + e)), d / (q.x + e), q.x);
  }

  void main() {
    vec3 rgb = texture2D(uVideoFrame, vUV).rgb;
    vec3 hsv = rgbToHsv(rgb);
    
    float hueDist = min(
      abs(hsv.x - uTargetHueRange.x),
      abs(hsv.x - uTargetHueRange.y)
    );
    
    float mask = smoothstep(
      uTargetHueRange.z - uEdgeSoftness,
      uTargetHueRange.z + uEdgeSoftness,
      hueDist
    );
    
    float satMask = smoothstep(uSaturationFloor - 0.05, uSaturationFloor + 0.05, hsv.y);
    float alpha = mask * satMask;
    
    gl_FragColor = vec4(rgb, alpha);
  }
`;

Architecture Rationale: HSV separation isolates color information from brightness, making the key resilient to lighting changes. smoothstep replaces hard binary thresholds, eliminating aliasing artifacts. The shader runs per-frame on the GPU, requiring zero CPU intervention or texture uploads during preview.

Step 3: Orchestrate WebAssembly Transcription

Whisper's neural architecture relies on matrix multiplications that exceed JavaScript's performance ceiling. Compiling the model to WebAssembly enables near-native execution. The critical implementation detail is non-blocking initialization and persistent caching.

class LocalTranscriptionEngine {
  private worker: Worker;
  private isReady: boolean = false;

  constructor(modelUrl: string, language: string) {
    this.worker = new Worker(new URL('./whisper-worker.ts', import.meta.url));
    this.worker.postMessage({ type: 'INIT', modelUrl, language });
    
    this.worker.onmessage = (e) => {
      if (e.data.type === 'READY') this.isReady = true;
    };
  }

  async transcribe(audioBuffer: Float32Array): Promise<TranscriptionResult> {
    if (!this.isReady) throw new Error('Model not initialized');
    
    return new Promise((resolve, reject) => {
      const handler = (e: MessageEvent) => {
        if (e.data.type === 'RESULT') {
          this.worker.removeEventListener('message', handler);
          resolve(e.data.payload);
        } else if (e.data.type === 'ERROR') {
          this.worker.removeEventListener('message', handler);
          reject(new Error(e.data.message));
        }
      };
      this.worker.addEventListener('message', handler);
      this.worker.postMessage({ type: 'TRANSCRIBE', audio: audioBuffer }, [audioBuffer.buffer]);
    });
  }
}

Architecture Rationale: Web Workers isolate WASM compilation and inference from the main thread, preventing UI jank. postMessage with transferable objects ([audioBuffer.buffer]) avoids memory duplication. The 14-language support is handled by loading the appropriate tokenizer weights during initialization. Model weights are cached via the Cache API or IndexedDB, ensuring zero network requests after the first run.

Pitfall Guide

1. RGB Distance Chroma Keying Fails Under Variable Lighting

Explanation: Euclidean distance in RGB space treats brightness and color as equally weighted. Shadows darken pixels, pushing them outside the threshold. Highlights desaturate green screens, causing false positives. Fix: Convert to HSV/HSL. Threshold hue range and saturation floor independently. Apply smoothstep for anti-aliased edges.

2. Main Thread Blocking During WASM Model Initialization

Explanation: WebAssembly compilation and memory allocation are synchronous by default. Loading a 100MB+ model on the main thread freezes the UI for 2–5 seconds. Fix: Offload initialization to a Web Worker. Use streaming compilation (WebAssembly.instantiateStreaming). Display a progress indicator tied to download/inference readiness events.

3. Hardware Encoder Fallback Blind Spots

Explanation: prefer-hardware is a hint, not a guarantee. Driver mismatches, power-saving modes, or unsupported resolutions cause silent failures or degraded output. Fix: Always validate with VideoEncoder.isConfigSupported(). Implement a software fallback path. Monitor encode() error callbacks and switch codecs dynamically if hardware rejects the configuration.

4. WebGL Texture Memory Leaks in Live Preview

Explanation: Creating new gl.createTexture() or gl.createFramebuffer() objects per frame without disposal exhausts GPU memory, causing driver crashes after 30–60 seconds. Fix: Allocate textures once during setup. Reuse framebuffers. Call gl.deleteTexture() and gl.deleteFramebuffer() during teardown. Use gl.bindTexture() to update existing objects instead of recreating them.

5. Whisper Language Detection Ambiguity on Short Clips

Explanation: Auto-detection relies on statistical patterns across longer audio segments. Sub-5-second clips often trigger misclassification, degrading word accuracy. Fix: Explicitly set the language parameter during initialization. Preprocess audio with Voice Activity Detection (VAD) to trim silence and ensure minimum segment length before inference.

6. IndexedDB Cache Bloat from Model Weights

Explanation: Caching .wasm and weight files without versioning or eviction policies consumes disk space indefinitely. Users on limited storage experience degraded performance. Fix: Implement cache versioning (model-v3-en.wasm). Use LRU eviction or explicit cleanup routines. Prefer the Cache API with CacheStorage.delete() for predictable lifecycle management.

7. Cross-Platform Encoder Inconsistencies

Explanation: macOS VideoToolbox, Windows Media Foundation, and Android MediaCodec produce different bitrates, keyframe intervals, and profile levels. Assuming uniform output breaks downstream players. Fix: Normalize output by setting explicit bitrate, keyFrameInterval, and profile parameters. Validate playback across target platforms. Use software post-processing if hardware output diverges from spec.

Production Bundle

Action Checklist

• Validate hardware encoder support using VideoEncoder.isConfigSupported() before initialization
• Implement HSV-based chroma keying with smoothstep edge softening
• Offload WASM model compilation and inference to a dedicated Web Worker
• Configure cache-first strategy for model weights with versioned keys
• Add graceful fallback from hardware to software encoding on configuration failure
• Preprocess audio with VAD to eliminate silence before transcription
• Test cross-browser compatibility on Chrome 94+, Edge, and Android WebView
• Monitor GPU memory allocation and enforce texture/framebuffer disposal

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Real-time preview with green screen	WebGL fragment shader + prefer-hardware	Sub-16ms latency, zero server round-trip	Eliminates GPU instance costs
High-security audio transcription	Local Whisper WASM + IndexedDB cache	Data never leaves device, offline capable	Removes ASR API subscription fees
Low-end mobile devices	Software encoding fallback + reduced shader precision	Maintains functionality on limited silicon	Increases CPU usage, reduces battery life
Batch processing large files	Server-side pipeline	Parallelizes across multiple cores, faster throughput	Higher infrastructure spend, network egress

Configuration Template

// encoder-config.ts
export const HARDWARE_ENCODER_CONFIG = {
  codec: 'video/mp4; codecs="avc1.4D401F"',
  width: 1920,
  height: 1080,
  bitrate: 5_000_000,
  hardwareAcceleration: 'prefer-hardware' as const,
  keyFrameInterval: 30,
  framerate: 30
};

// shader-config.ts
export const CHROMA_KEY_UNIFORMS = {
  targetHueRange: [0.28, 0.38, 0.05], // Green hue range + tolerance
  saturationFloor: 0.4,
  edgeSoftness: 0.02
};

// whisper-config.ts
export const TRANSCRIPTION_CONFIG = {
  modelUrl: '/models/whisper-small-en.wasm',
  language: 'en',
  cacheKey: 'whisper-model-v2-en',
  workerPath: './whisper-worker.ts'
};

Quick Start Guide

1. Initialize the encoder: Call initializeVideoEncoder(HARDWARE_ENCODER_CONFIG) and verify hardware support. Attach to your canvas stream.
2. Compile the shader: Load CHROMA_KEY_FRAGMENT into a WebGL program. Bind uVideoFrame to your canvas texture and set uniforms from CHROMA_KEY_UNIFORMS.
3. Spin up the transcription worker: Instantiate LocalTranscriptionEngine with TRANSCRIPTION_CONFIG. Wait for the READY message before sending audio buffers.
4. Test the pipeline: Feed a webcam stream into the WebGL shader, route the output to the hardware encoder, and pass isolated audio tracks to the WASM worker. Verify zero network requests after initial cache population.