JS video player with ffmpeg HTTP streaming in PHP: state machine, watchdog, subtitles
Adaptive Video Streaming Pipeline: FFmpeg Orchestration, State Machines, and Silent Failure Recovery
Current Situation Analysis
Modern web browsers support a narrow subset of media formats. The industry standard remains H.264 video with AAC audio inside an MP4 container. However, user-generated content and archival media rarely conform to this standard. Files frequently arrive as MKV containers with HEVC video, DTS or AC3 audio, and bitmap subtitle tracks.
When a browser encounters an unsupported codec, the native <video> element often fails silently. No error event fires. No console warnings appear. The player simply spins indefinitely. This "silent failure" pattern creates a poor user experience and makes debugging difficult.
Developers often attempt to solve this with client-side detection using HTMLMediaElement.canPlayType(). This approach is unreliable. Browsers may report support for a codec based on software capabilities, yet fail to decode the stream due to hardware limitations or specific bitstream quirks. Furthermore, trial-and-error streaming on the client side introduces significant latency. Waiting seconds to detect a failure before switching strategies wastes bandwidth and frustrates users.
The core challenge is bridging the gap between the browser's rigid requirements and the chaotic reality of media files. This requires a server-side orchestration layer that can inspect, adapt, and stream media dynamically, paired with a client-side state machine capable of recovering from silent failures without user intervention.
WOW Moment: Key Findings
The most efficient streaming strategy depends entirely on the relationship between the source codec and browser capabilities. A one-size-fits-all transcoding approach wastes CPU resources and increases latency. The optimal pipeline dynamically selects between three modes based on real-time probe data.
| Strategy | CPU Load | Startup Latency | Browser Compatibility | Seek Precision | Best Use Case |
|---|---|---|---|---|---|
| Direct Serve | None | Instant | Low | Perfect | MP4/H.264/AAC files |
| Remux | Low (Audio only) | Fast | High | High | MKV with H.264/DTS |
| Transcode | High | Slow | Universal | Variable | HEVC/VP9/AV1/Dolby |
Why this matters: Remuxing allows you to serve MKV files containing H.264 video with near-zero CPU cost by repackaging the stream into fragmented MP4. Transcoding should be reserved as a fallback for truly incompatible codecs. This tiered approach reduces server load by up to 90% for compatible files while maintaining universal playback support.
Core Solution
The solution consists of a PHP backend that manages FFmpeg processes and a JavaScript frontend that orchestrates playback modes.
Server-Side Architecture
The backend exposes a streaming endpoint that accepts a mode parameter. Before streaming, the system probes the file to determine the appropriate mode.
1. Codec Probing with SQLite Cache
Running ffprobe on large files is expensive, taking 2β12 seconds depending on disk I/O and file size. We cache the results in SQLite, keyed by file path and modification time. This ensures instant mode selection for subsequent requests.
<?php
class CodecCache {
private PDO $db;
public function __construct(string $dbPath) {
$this->db = new PDO("sqlite:$dbPath");
$this->db->exec("CREATE TABLE IF NOT EXISTS probe_cache (
key TEXT PRIMARY KEY,
data TEXT,
created_at INTEGER
)");
}
public function getMetadata(string $filePath): array {
$key = hash('sha256', $filePath . filemtime($filePath));
$stmt = $this->db->prepare("SELECT data FROM probe_cache WHERE key = ?");
$stmt->execute([$key]);
$row = $stmt->fetch(PDO::FETCH_ASSOC);
if ($row) {
return json_decode($row['data'], true);
}
return $this->probeAndCache($filePath, $key);
}
private function probeAndCache(string $filePath, string $key): array {
$cmd = sprintf(
'ffprobe -v quiet -print_format json -show_streams %s',
escapeshellarg($filePath)
);
$output = shell_exec($cmd);
$metadata = json_decode($output, true);
$stmt = $this->db->prepare("INSERT OR REPLACE INTO probe_cache (key, data, created_at) VALUES (?, ?, ?)");
$stmt->execute([$key, json_encode($metadata), time()]);
return $metadata;
}
}
2. Streaming Modes and FFmpeg Flags
The streaming engine supports three modes. Each mode uses specific FFmpeg flags to ensure browser compatibility.
<?php
class MediaStreamer {
private const MAX_CONCURRENT = 4;
private const SEM_KEY = 'video_stream_sem';
public function stream(string $filePath, string $mode): void {
$semId = sem_get(ftok(__FILE__, self::SEM_KEY), self::MAX_CONCURRENT);
if (!sem_acquire($semId)) {
http_response_code(503);
echo "Server busy";
return;
}
register_shutdown_function(function() use ($semId, &$process) {
if (is_resource($process)) {
proc_terminate($process);
}
sem_release($semId);
});
$cmd = $this->buildCommand($filePath, $mode);
$process = proc_open($cmd, [
0 => ['pipe', 'r'],
1 => ['pipe', 'w'],
2 => ['pipe', 'w']
], $pipes);
if (!is_resource($process)) {
sem_release($semId);
return;
}
$stdout = $pipes[1];
// Stream output to client
while (!feof($stdout) && !connection_aborted()) {
echo fread($stdout, 65536);
flush();
}
proc_close($process);
}
private function buildCommand(string $file, string $mode): string {
$baseFlags = '-movflags frag_keyframe+empty_moov+default_base_moof -min_frag_duration 300000 -f mp4 pipe:1';
return match ($mode) {
'remux' => sprintf(
'ffmpeg -i %s -c:v copy -c:a aac %s',
escapeshellarg($file),
$baseFlags
),
'transcode' => sprintf(
'ffmpeg -i %s -c:v libx264 -preset ultrafast -crf 23 -c:a aac %s',
escapeshellarg($file),
$baseFlags
),
default => ''
};
}
}
Critical Flag Explanation:
empty_moov: Essential for fragmented MP4 over HTTP. Without this, the browser waits for themoovatom at the end of the file, which never arrives on a pipe.frag_keyframe: Ensures fragments start on keyframes, enabling accurate seeking.default_base_moof: Required for compatibility with certain browser implementations of MSE.
3. Concurrency Management
FFmpeg processes consume CPU for the duration of the stream. A semaphore limits concurrent processes to prevent server overload. The connection_aborted() check ensures that if a client disconnects, the PHP script terminates the FFmpeg process immediately, freeing the semaphore slot.
Client-Side State Machine
The frontend uses a state machine to manage mode selection, recovery, and subtitle rendering.
1. Mode Selection Logic
class StreamController {
private state = {
mode: 'native',
validatedMode: null as string | null,
stallRetries: 0,
lastUpdate: Date.now()
};
public selectMode(probeData: any): string {
const videoStream = probeData.streams.find((s: any) => s.codec_type === 'video');
const audioStream = probeData.streams.find((s: any) => s.codec_type === 'audio');
const vCodec = videoStream?.codec_name;
const aCodec = audioStream?.codec_name;
const container = this.getFileExtension().toLowerCase();
// H.264 in MKV requires remuxing
if (vCodec === 'h264' && container === 'mkv') return 'remux';
// H.264/AAC in MP4 can be served natively
if (vCodec === 'h264' && aCodec === 'aac' && container === 'mp4') return 'native';
// Check browser support for other codecs
const mime = this.getMimeType(vCodec);
const support = document.createElement('video').canPlayType(mime);
return support ? 'native' : 'transcode';
}
}
2. Silent Failure Recovery
Browsers may report support for HEVC via canPlayType() but fail to decode silently due to hardware limitations. The only reliable signal is videoWidth remaining zero after playback attempts.
private checkSilentFailure(): void {
setTimeout(() => {
if (this.video.videoWidth === 0 && this.state.mode === 'native') {
console.warn('Silent decode failure detected. Switching to transcode.');
this.state.mode = 'transcode';
this.reloadStream();
}
}, 1500);
}
3. Watchdog with Exponential Backoff
A fixed timeout causes issues with transcoding, which has a longer startup time. Exponential backoff prevents the watchdog from triggering retries while FFmpeg is still initializing.
private startWatchdog(): void {
clearTimeout(this.watchdogTimer);
const baseTimeouts: Record<string, number> = {
native: 5,
remux: 10,
transcode: 20,
burnSub: 30
};
const base = baseTimeouts[this.state.validatedMode || this.state.mode] || 15;
const timeout = Math.min(base * Math.pow(2, this.state.stallRetries), 120) * 1000;
this.watchdogTimer = setTimeout(() => {
this.state.stallRetries++;
this.reloadStream(this.video.currentTime);
}, timeout);
}
// Reset watchdog on active playback
this.video.addEventListener('timeupdate', () => this.startWatchdog());
4. Subtitle Rendering
Text subtitles are extracted as WebVTT and rendered using a custom overlay. To handle files with thousands of cues efficiently, we use binary search for the initial position and a linear pointer for subsequent updates.
class SubtitleRenderer {
private cues: any[] = [];
private pointer: number = 0;
public locateCue(time: number): number {
// Linear advance from pointer
while (this.pointer < this.cues.length - 1 && this.cues[this.pointer].end < time) {
this.pointer++;
}
// Binary search if pointer is too far behind
if (this.cues[this.pointer].start > time + 1) {
this.pointer = this.binarySearch(time);
}
return this.pointer;
}
private binarySearch(time: number): number {
let lo = 0, hi = this.cues.length - 1;
while (lo < hi) {
const mid = (lo + hi) >> 1;
if (this.cues[mid].end < time) lo = mid + 1;
else hi = mid;
}
return lo;
}
}
Font sizing is dynamically adjusted using a ResizeObserver to maintain readability across different viewport sizes.
Pitfall Guide
PTS Normalization Desync
- Issue: Using
-fflags +genptsorfirst_pts=0on fragmented streams causes audio/video desync after seeking. - Fix: Remove timestamp normalization flags. Browsers handle PTS correctly on fragmented MP4; modifying them introduces errors.
- Issue: Using
Orphaned FFmpeg Processes
- Issue: Client disconnects leave FFmpeg processes running, consuming CPU and semaphore slots.
- Fix: Implement
connection_aborted()checks in the streaming loop and useregister_shutdown_functionto terminate processes.
Silent HEVC Failures
- Issue:
canPlayType()returns support, but hardware decode fails silently with no events. - Fix: Monitor
videoWidthafter a delay. If zero, assume failure and fallback to transcode.
- Issue:
Watchdog Saturation
- Issue: Fixed timeouts cause rapid retries during slow transcoding startups, creating zombie processes.
- Fix: Use exponential backoff with a high cap (e.g., 120s) for transcode mode.
FPM Worker Exhaustion
- Issue: Streaming blocks PHP-FPM workers, causing 503 errors for other requests.
- Fix: Increase
pm.max_childrenin PHP-FPM configuration to accommodate concurrent streams.
Subtitle Performance Degradation
- Issue: Linear scanning of cues on every
timeupdatecauses UI jank in files with many subtitles. - Fix: Use binary search for initial positioning and maintain a pointer for O(1) subsequent lookups.
- Issue: Linear scanning of cues on every
Missing Fragment Flags
- Issue: Omitting
empty_moovorfrag_keyframecauses playback to hang or seeking to break. - Fix: Always include
frag_keyframe+empty_moov+default_base_moofin FFmpeg flags for HTTP streaming.
- Issue: Omitting
Production Bundle
Action Checklist
- Verify FFmpeg installation supports required codecs (libx264, AAC).
- Configure SQLite cache database with proper permissions.
- Set PHP-FPM
pm.max_childrento handle expected concurrent streams. - Implement semaphore limits to prevent CPU overload.
- Add
connection_aborted()checks to all streaming loops. - Test silent failure detection with HEVC files on Safari/Chrome.
- Validate subtitle binary search performance with large VTT files.
- Monitor semaphore usage and process cleanup in production.
Decision Matrix
| Scenario | Recommended Approach | Why | Cost Impact |
|---|---|---|---|
| MP4/H.264/AAC | Direct Serve | Zero CPU, instant playback | None |
| MKV/H.264/DTS | Remux | Low CPU, high compatibility | Minimal |
| HEVC/VP9/AV1 | Transcode | Universal support | High CPU |
| PGS/VOBSUB | Burn-in | Browser compatibility | Very High CPU |
Configuration Template
Nginx Configuration for X-Accel-Redirect:
location /internal/video/ {
internal;
alias /var/media/storage/;
}
PHP-FPM Tuning:
[pool]
pm = dynamic
pm.max_children = 25
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 10
FFmpeg Flags Reference:
# Remux Command
ffmpeg -i input.mkv \
-c:v copy \
-c:a aac \
-movflags frag_keyframe+empty_moov+default_base_moof \
-min_frag_duration 300000 \
-f mp4 pipe:1
# Transcode Command
ffmpeg -i input.mkv \
-c:v libx264 \
-preset ultrafast \
-crf 23 \
-c:a aac \
-movflags frag_keyframe+empty_moov+default_base_moof \
-min_frag_duration 300000 \
-f mp4 pipe:1
Quick Start Guide
- Install Dependencies: Ensure PHP, FFmpeg, and SQLite are installed.
- Deploy Backend: Place
MediaStreamerandCodecCacheclasses in your project. - Configure Nginx: Add the internal location block for direct file serving.
- Initialize Frontend: Instantiate
StreamControllerwith your video element. - Test Playback: Load an MKV file with H.264 video to verify remuxing works. Check logs for mode selection and watchdog behavior.
Mid-Year Sale β Unlock Full Article
Base plan from just $4.99/mo or $49/yr
Sign in to read the full article and unlock all tutorials.
Sign In / Register β Start Free Trial7-day free trial Β· Cancel anytime Β· 30-day money-back
