JS video player with ffmpeg HTTP streaming in PHP: state machine, watchdog, subtitles

Adaptive Video Streaming Pipeline: FFmpeg Orchestration, State Machines, and Silent Failure Recovery

Current Situation Analysis

Modern web browsers support a narrow subset of media formats. The industry standard remains H.264 video with AAC audio inside an MP4 container. However, user-generated content and archival media rarely conform to this standard. Files frequently arrive as MKV containers with HEVC video, DTS or AC3 audio, and bitmap subtitle tracks.

When a browser encounters an unsupported codec, the native <video> element often fails silently. No error event fires. No console warnings appear. The player simply spins indefinitely. This "silent failure" pattern creates a poor user experience and makes debugging difficult.

Developers often attempt to solve this with client-side detection using HTMLMediaElement.canPlayType(). This approach is unreliable. Browsers may report support for a codec based on software capabilities, yet fail to decode the stream due to hardware limitations or specific bitstream quirks. Furthermore, trial-and-error streaming on the client side introduces significant latency. Waiting seconds to detect a failure before switching strategies wastes bandwidth and frustrates users.

The core challenge is bridging the gap between the browser's rigid requirements and the chaotic reality of media files. This requires a server-side orchestration layer that can inspect, adapt, and stream media dynamically, paired with a client-side state machine capable of recovering from silent failures without user intervention.

WOW Moment: Key Findings

The most efficient streaming strategy depends entirely on the relationship between the source codec and browser capabilities. A one-size-fits-all transcoding approach wastes CPU resources and increases latency. The optimal pipeline dynamically selects between three modes based on real-time probe data.

Strategy	CPU Load	Startup Latency	Browser Compatibility	Seek Precision	Best Use Case
Direct Serve	None	Instant	Low	Perfect	MP4/H.264/AAC files
Remux	Low (Audio only)	Fast	High	High	MKV with H.264/DTS
Transcode	High	Slow	Universal	Variable	HEVC/VP9/AV1/Dolby

Why this matters: Remuxing allows you to serve MKV files containing H.264 video with near-zero CPU cost by repackaging the stream into fragmented MP4. Transcoding should be reserved as a fallback for truly incompatible codecs. This tiered approach reduces server load by up to 90% for compatible files while maintaining universal playback support.

Core Solution

The solution consists of a PHP backend that manages FFmpeg processes and a JavaScript frontend that orchestrates playback modes.

Server-Side Architecture

The backend exposes a streaming endpoint that accepts a mode parameter. Before streaming, the system probes the file to determine the appropriate mode.

1. Codec Probing with SQLite Cache

Running ffprobe on large files is expensive, taking 2–12 seconds depending on disk I/O and file size. We cache the results in SQLite, keyed by file path and modification time. This ensures instant mode selection for subsequent requests.

<?php
class CodecCache {
    private PDO $db;

    public function __construct(string $dbPath) {
        $this->db = new PDO("sqlite:$dbPath");
        $this->db->exec("CREATE TABLE IF NOT EXISTS probe_cache (
            key TEXT PRIMARY KEY, 
            data TEXT, 
            created_at INTEGER
        )");
    }

    public function getMetadata(string $filePath): array {
        $key = hash('sha256', $filePath . filemtime($filePath));
        
        $stmt = $this->db->prepare("SELECT data FROM probe_cache WHERE key = ?");
        $stmt->execute([$key]);
        $row = $stmt->fetch(PDO::FETCH_ASSOC);

        if ($row) {
            return json_decode($row['data'], true);
        }

        return $this->probeAndCache($filePath, $key);
    }

    private function probeAndCache(string $filePath, string $key): array {
        $cmd = sprintf(
            'ffprobe -v quiet -print_format json -show_streams %s',
            escapeshellarg($filePath)
        );
        
        $output = shell_exec($cmd);
        $metadata = json_decode($output, true);

        $stmt = $this->db->prepare("INSERT OR REPLACE INTO probe_cache (key, data, created_at) VALUES (?, ?, ?)");
        $stmt->execute([$key, json_encode($metadata), time()]);

        return $metadata;
    }
}

2. Streaming Modes and FFmpeg Flags

The streaming engine supports three modes. Each mode uses specific FFmpeg flags to ensure browser compatibility.

<?php
class MediaStreamer {
    private const MAX_CONCURRENT = 4;
    private const SEM_KEY = 'video_stream_sem';

    public function stream(string $filePath, string $mode): void {
        $semId = sem_get(ftok(__FILE__, self::SEM_KEY), self::MAX_CONCURRENT);
        
        if (!sem_acquire($semId)) {
            http_response_code(503);
            echo "Server busy";
            return;
        }

        register_shutdown_function(function() use ($semId, &$process) {
            if (is_resource($process)) {
                proc_terminate($process);
            }
            sem_release($semId);
        });

        $cmd = $this->buildCommand($filePath, $mode);
        $process = proc_open($cmd, [
            0 => ['pipe', 'r'],
            1 => ['pipe', 'w'],
            2 => ['pipe', 'w']
        ], $pipes);

        if (!is_resource($process)) {
            sem_release($semId);
            return;
        }

        $stdout = $pipes[1];
        
        // Stream output to client
        while (!feof($stdout) && !connection_aborted()) {
            echo fread($stdout, 65536);
            flush();
        }

        proc_close($process);
    }

    private function buildCommand(string $file, string $mode): string {
        $baseFlags = '-movflags frag_keyframe+empty_moov+default_base_moof -min_frag_duration 300000 -f mp4 pipe:1';

        return match ($mode) {
            'remux' => sprintf(
                'ffmpeg -i %s -c:v copy -c:a aac %s',
                escapeshellarg($file),
                $baseFlags
            ),
            'transcode' => sprintf(
                'ffmpeg -i %s -c:v libx264 -preset ultrafast -crf 23 -c:a aac %s',
                escapeshellarg($file),
                $baseFlags
            ),
            default => ''
        };
    }
}

Critical Flag Explanation:

• empty_moov: Essential for fragmented MP4 over HTTP. Without this, the browser waits for the moov atom at the end of the file, which never arrives on a pipe.
• frag_keyframe: Ensures fragments start on keyframes, enabling accurate seeking.
• default_base_moof: Required for compatibility with certain browser implementations of MSE.

3. Concurrency Management

FFmpeg processes consume CPU for the duration of the stream. A semaphore limits concurrent processes to prevent server overload. The connection_aborted() check ensures that if a client disconnects, the PHP script terminates the FFmpeg process immediately, freeing the semaphore slot.

Client-Side State Machine

The frontend uses a state machine to manage mode selection, recovery, and subtitle rendering.

1. Mode Selection Logic

class StreamController {
    private state = {
        mode: 'native',
        validatedMode: null as string | null,
        stallRetries: 0,
        lastUpdate: Date.now()
    };

    public selectMode(probeData: any): string {
        const videoStream = probeData.streams.find((s: any) => s.codec_type === 'video');
        const audioStream = probeData.streams.find((s: any) => s.codec_type === 'audio');
        
        const vCodec = videoStream?.codec_name;
        const aCodec = audioStream?.codec_name;
        const container = this.getFileExtension().toLowerCase();

        // H.264 in MKV requires remuxing
        if (vCodec === 'h264' && container === 'mkv') return 'remux';
        
        // H.264/AAC in MP4 can be served natively
        if (vCodec === 'h264' && aCodec === 'aac' && container === 'mp4') return 'native';

        // Check browser support for other codecs
        const mime = this.getMimeType(vCodec);
        const support = document.createElement('video').canPlayType(mime);
        
        return support ? 'native' : 'transcode';
    }
}

2. Silent Failure Recovery

Browsers may report support for HEVC via canPlayType() but fail to decode silently due to hardware limitations. The only reliable signal is videoWidth remaining zero after playback attempts.

private checkSilentFailure(): void {
    setTimeout(() => {
        if (this.video.videoWidth === 0 && this.state.mode === 'native') {
            console.warn('Silent decode failure detected. Switching to transcode.');
            this.state.mode = 'transcode';
            this.reloadStream();
        }
    }, 1500);
}

3. Watchdog with Exponential Backoff

A fixed timeout causes issues with transcoding, which has a longer startup time. Exponential backoff prevents the watchdog from triggering retries while FFmpeg is still initializing.

private startWatchdog(): void {
    clearTimeout(this.watchdogTimer);
    
    const baseTimeouts: Record<string, number> = {
        native: 5,
        remux: 10,
        transcode: 20,
        burnSub: 30
    };

    const base = baseTimeouts[this.state.validatedMode || this.state.mode] || 15;
    const timeout = Math.min(base * Math.pow(2, this.state.stallRetries), 120) * 1000;

    this.watchdogTimer = setTimeout(() => {
        this.state.stallRetries++;
        this.reloadStream(this.video.currentTime);
    }, timeout);
}

// Reset watchdog on active playback
this.video.addEventListener('timeupdate', () => this.startWatchdog());

4. Subtitle Rendering

Text subtitles are extracted as WebVTT and rendered using a custom overlay. To handle files with thousands of cues efficiently, we use binary search for the initial position and a linear pointer for subsequent updates.

class SubtitleRenderer {
    private cues: any[] = [];
    private pointer: number = 0;

    public locateCue(time: number): number {
        // Linear advance from pointer
        while (this.pointer < this.cues.length - 1 && this.cues[this.pointer].end < time) {
            this.pointer++;
        }
        
        // Binary search if pointer is too far behind
        if (this.cues[this.pointer].start > time + 1) {
            this.pointer = this.binarySearch(time);
        }
        
        return this.pointer;
    }

    private binarySearch(time: number): number {
        let lo = 0, hi = this.cues.length - 1;
        while (lo < hi) {
            const mid = (lo + hi) >> 1;
            if (this.cues[mid].end < time) lo = mid + 1;
            else hi = mid;
        }
        return lo;
    }
}

Font sizing is dynamically adjusted using a ResizeObserver to maintain readability across different viewport sizes.

Pitfall Guide

1. PTS Normalization Desync

   • Issue: Using -fflags +genpts or first_pts=0 on fragmented streams causes audio/video desync after seeking.
   • Fix: Remove timestamp normalization flags. Browsers handle PTS correctly on fragmented MP4; modifying them introduces errors.

2. Orphaned FFmpeg Processes

   • Issue: Client disconnects leave FFmpeg processes running, consuming CPU and semaphore slots.
   • Fix: Implement connection_aborted() checks in the streaming loop and use register_shutdown_function to terminate processes.

3. Silent HEVC Failures

   • Issue: canPlayType() returns support, but hardware decode fails silently with no events.
   • Fix: Monitor videoWidth after a delay. If zero, assume failure and fallback to transcode.

4. Watchdog Saturation

   • Issue: Fixed timeouts cause rapid retries during slow transcoding startups, creating zombie processes.
   • Fix: Use exponential backoff with a high cap (e.g., 120s) for transcode mode.

5. FPM Worker Exhaustion

   • Issue: Streaming blocks PHP-FPM workers, causing 503 errors for other requests.
   • Fix: Increase pm.max_children in PHP-FPM configuration to accommodate concurrent streams.

6. Subtitle Performance Degradation

   • Issue: Linear scanning of cues on every timeupdate causes UI jank in files with many subtitles.
   • Fix: Use binary search for initial positioning and maintain a pointer for O(1) subsequent lookups.

7. Missing Fragment Flags

   • Issue: Omitting empty_moov or frag_keyframe causes playback to hang or seeking to break.
   • Fix: Always include frag_keyframe+empty_moov+default_base_moof in FFmpeg flags for HTTP streaming.

Production Bundle

Action Checklist

• Verify FFmpeg installation supports required codecs (libx264, AAC).
• Configure SQLite cache database with proper permissions.
• Set PHP-FPM pm.max_children to handle expected concurrent streams.
• Implement semaphore limits to prevent CPU overload.
• Add connection_aborted() checks to all streaming loops.
• Test silent failure detection with HEVC files on Safari/Chrome.
• Validate subtitle binary search performance with large VTT files.
• Monitor semaphore usage and process cleanup in production.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
MP4/H.264/AAC	Direct Serve	Zero CPU, instant playback	None
MKV/H.264/DTS	Remux	Low CPU, high compatibility	Minimal
HEVC/VP9/AV1	Transcode	Universal support	High CPU
PGS/VOBSUB	Burn-in	Browser compatibility	Very High CPU

Configuration Template

Nginx Configuration for X-Accel-Redirect:

location /internal/video/ {
    internal;
    alias /var/media/storage/;
}

PHP-FPM Tuning:

[pool]
pm = dynamic
pm.max_children = 25
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 10

FFmpeg Flags Reference:

# Remux Command
ffmpeg -i input.mkv \
  -c:v copy \
  -c:a aac \
  -movflags frag_keyframe+empty_moov+default_base_moof \
  -min_frag_duration 300000 \
  -f mp4 pipe:1

# Transcode Command
ffmpeg -i input.mkv \
  -c:v libx264 \
  -preset ultrafast \
  -crf 23 \
  -c:a aac \
  -movflags frag_keyframe+empty_moov+default_base_moof \
  -min_frag_duration 300000 \
  -f mp4 pipe:1

Quick Start Guide

1. Install Dependencies: Ensure PHP, FFmpeg, and SQLite are installed.
2. Deploy Backend: Place MediaStreamer and CodecCache classes in your project.
3. Configure Nginx: Add the internal location block for direct file serving.
4. Initialize Frontend: Instantiate StreamController with your video element.
5. Test Playback: Load an MKV file with H.264 video to verify remuxing works. Check logs for mode selection and watchdog behavior.