Data-Driven Audio Stem Separation: Benchmarking HTDemucs Variants Without Running Inference

Current Situation Analysis

Shipping AI-powered audio stem separation in production requires navigating a fragmented model landscape. Marketing documentation frequently claims "state-of-the-art" performance, but engineering teams must translate those claims into concrete trade-offs between separation fidelity, inference latency, and compute cost. The four primary HTDemucs variants (htdemucs, htdemucs_ft, htdemucs_6s, mdx_extra_q) each optimize for different acoustic characteristics, making blind selection a liability.

This decision is routinely misunderstood or deferred. Teams often rely on subjective listening tests across a handful of reference tracks, which introduces confirmation bias and fails to capture edge cases. Alternatively, engineering groups attempt to replicate academic benchmarks like MUSDB18-HQ locally. Doing so requires provisioning ~22 GB of storage, configuring GPU environments, and dedicating multiple days to run BSS Eval v4 metrics across dozens of tracks. The operational overhead delays releases and creates inconsistent evaluation standards across sprints.

The industry gap is clear: developers need a deterministic, queryable evaluation layer that decouples model selection from inference execution. Pre-computed benchmark datasets now solve this by publishing standardized metrics across 800 evaluation rows (50 reference tracks × 4 model variants × 4 stem categories). These datasets expose median Source-to-Distortion Ratio (SDR), Real-Time Factor (RTF), and auxiliary metrics (ISR, SIR, SAR) without requiring local audio processing. By treating the benchmark as a configuration source rather than a research artifact, teams can route model selection through data-driven constraints instead of guesswork.

WOW Moment: Key Findings

The benchmark dataset reveals a clear performance-latency trade-off matrix that directly maps to product requirements. Instead of treating all variants as interchangeable, the data shows distinct specialization patterns:

Model Variant	Median Vocal SDR (dB)	Median Drum SDR (dB)	Median RTF	Primary Optimization
htdemucs	8.53	10.01	0.06	Baseline speed & compatibility
htdemucs_ft	9.19	10.11	0.07	Vocal isolation & balanced fidelity
htdemucs_6s	8.66	9.54	0.09	Multi-stem granularity (piano/guitar split)
mdx_extra_q	9.04	11.49	0.08	Percussive/bass-heavy material & maximum SDR

Why this matters: The table transforms model selection from an artistic debate into a constraint-satisfaction problem. If your product prioritizes vocal clarity for karaoke or lyric extraction, htdemucs_ft provides a measurable +0.66 dB SDR advantage over the baseline. If you're building rhythm analysis or DJ tools, mdx_extra_q delivers +1.48 dB drum isolation at a marginal RTF increase. The RTF column enables hard latency budgeting: a 0.07 RTF means the model processes audio 14× faster than real-time, which is critical for preview generation or streaming pipelines. This data layer allows you to embed model routing directly into configuration files, feature flags, or API gateways with auditable justification.

Core Solution

The implementation strategy treats the benchmark as a deterministic configuration source. Rather than running inference during selection, you load the pre-computed metrics, apply product constraints (target stem, maximum RTF, quality floor), and resolve the optimal variant programmatically. This approach decouples evaluation from deployment, enables CI regression testing, and supports dynamic routing based on user intent.

Architecture Decisions & Rationale

1. Median over Mean: SDR distributions in audio separation are heavily skewed by difficult tracks (dense mixes, overlapping frequencies). The median provides a robust central tendency that reflects typical user experience better than arithmetic averages.
2. Percentile Latency Validation: Average RTF masks tail latency. Production systems must validate against p95 or p99 RTF to guarantee SLA compliance during peak load or complex audio material.
3. Constraint-First Filtering: Model selection should fail fast when constraints cannot be met. Raising explicit errors prevents silent fallbacks to suboptimal variants.
4. Immutable Benchmark Snapshot: Cache the dataset locally or pin a specific revision in CI to prevent silent drift when upstream metrics are updated.

Implementation

The following implementation uses a modular class structure with explicit type contracts. It separates data loading, constraint filtering, candidate resolution, and validation into distinct responsibilities.

import pandas as pd
from datasets import load_dataset
from typing import Optional, List, Tuple
import logging

logger = logging.getLogger(__name__)

class AudioBenchmarkLoader:
    """Handles dataset retrieval and metric normalization."""
    
    DATASET_REPO = "StemSplitio/stem-separation-benchmark-2026"
    CONFIG_NAME = "metrics_only"
    SPLIT_NAME = "results"
    
    def fetch_metrics(self) -> pd.DataFrame:
        """Load benchmark data into a DataFrame with standardized columns."""
        raw_dataset = load_dataset(
            self.DATASET_REPO,
            self.CONFIG_NAME,
            split=self.SPLIT_NAME,
            trust_remote_code=True
        )
        metrics_df = raw_dataset.to_pandas()
        
        # Standardize column names and types for downstream filtering
        metrics_df.rename(columns={
            "sdr_median": "quality_score_db",
            "rtf": "latency_factor",
            "track_id": "reference_track",
            "stem": "target_stem",
            "model_id": "variant_id"
        }, inplace=True)
        
        return metrics_df

class ModelRouter:
    """Resolves optimal HTDemucs variant based on product constraints."""
    
    def __init__(self, metrics: pd.DataFrame):
        self.metrics = metrics
        self._validate_schema()
        
    def _validate_schema(self) -> None:
        required_cols = {"variant_id", "target_stem", "quality_score_db", "latency_factor"}
        if not required_cols.issubset(self.metrics.columns):
            raise ValueError(f"Missing required columns: {required_cols - set(self.metrics.columns)}")
            
    def resolve_variant(
        self,
        target_stem: str,
        max_latency: Optional[float] = None,
        allowed_variants: Optional[List[str]] = None
    ) -> str:
        """Return the variant with highest median SDR under latency constraints."""
        filtered = self.metrics[self.metrics["target_stem"] == target_stem].copy()
        
        if allowed_variants:
            filtered = filtered[filtered["variant_id"].isin(allowed_variants)]
            
        if max_latency is not None:
            filtered = filtered[filtered["latency_factor"] <= max_latency]
            
        if filtered.empty:
            raise RuntimeError(
                f"No variant satisfies stem={target_stem}, max_latency={max_latency}"
            )
            
        # Group by variant and compute median quality
        variant_scores = filtered.groupby("variant_id")["quality_score_db"].median()
        best_variant = variant_scores.idxmax()
        
        logger.info(f"Resolved variant: {best_variant} for stem={target_stem}")
        return best_variant
        
    def audit_failure_modes(
        self, 
        variant: str, 
        stem: str, 
        top_n: int = 5
    ) -> pd.DataFrame:
        """Identify tracks where separation quality degrades significantly."""
        subset = self.metrics[
            (self.metrics["variant_id"] == variant) & 
            (self.metrics["target_stem"] == stem)
        ]
        
        return subset.nsmallest(top_n, "quality_score_db")[
            ["reference_track", "quality_score_db", "latency_factor"]
        ].reset_index(drop=True)

class CIPolicyValidator:
    """Enforces quality floors without running inference in CI pipelines."""
    
    def __init__(self, metrics: pd.DataFrame):
        self.metrics = metrics
        
    def assert_quality_floor(self, variant: str, stem: str, min_sdr_db: float) -> bool:
        """Verify median SDR meets product specification."""
        subset = self.metrics[
            (self.metrics["variant_id"] == variant) & 
            (self.metrics["target_stem"] == stem)
        ]
        median_score = subset["quality_score_db"].median()
        
        if median_score < min_sdr_db:
            raise AssertionError(
                f"Variant {variant} median SDR ({median_score:.2f} dB) "
                f"below floor ({min_sdr_db} dB) for stem={stem}"
            )
        return True
        
    def validate_latency_budget(self, variant: str, stem: str, max_p95_rtf: float) -> bool:
        """Ensure p95 latency stays within SLA thresholds."""
        subset = self.metrics[
            (self.metrics["variant_id"] == variant) & 
            (self.metrics["target_stem"] == stem)
        ]
        p95_latency = subset["latency_factor"].quantile(0.95)
        
        if p95_latency > max_p95_rtf:
            raise AssertionError(
                f"Variant {variant} p95 RTF ({p95_latency:.3f}) exceeds budget ({max_p95_rtf})"
            )
        return True

Usage Pattern

Wire the router into your application configuration layer. The selector runs once at startup or during configuration reload, caching the resolved variant for the request lifecycle.

# Initialize loader and router
loader = AudioBenchmarkLoader()
benchmark_data = loader.fetch_metrics()
router = ModelRouter(benchmark_data)

# Resolve variants per product requirement
vocal_variant = router.resolve_variant("vocals", max_latency=0.08)
drum_variant = router.resolve_variant("drums", max_latency=None)

# Audit known failure tracks before deployment
failure_audit = router.audit_failure_modes(vocal_variant, "vocals", top_n=5)
print("Tracks requiring manual QA:", failure_audit["reference_track"].tolist())

# CI validation (runs in <2 seconds)
validator = CIPolicyValidator(benchmark_data)
validator.assert_quality_floor(vocal_variant, "vocals", min_sdr_db=9.0)
validator.validate_latency_budget(vocal_variant, "vocals", max_p95_rtf=0.10)

This architecture ensures model selection is deterministic, auditable, and decoupled from inference infrastructure. The router can be exposed as a configuration endpoint, integrated into feature flags, or used to generate deployment manifests.

Pitfall Guide

1. Ignoring the htdemucs_6s Stem Artifact

Explanation: The 6-stem variant splits piano and guitar out of the traditional other category. This causes the other stem SDR to drop artificially (~0.2 dB) because the evaluation metric treats the split as separation failure rather than intentional granularity. Fix: Exclude other from cross-variant comparisons, or aggregate piano + guitar + other before evaluation. Only compare htdemucs_6s against others on vocals, drums, and bass.

2. Optimizing for Mean Instead of Median

Explanation: Audio separation scores follow a heavy-tailed distribution. A few highly complex tracks can drag the mean down, masking strong performance on typical material. Fix: Always use .median() for SDR aggregation. Reserve .mean() for diagnostic analysis of outlier behavior, not model selection.

3. Hardcoding RTF Thresholds Without Percentile Validation

Explanation: Average RTF hides tail latency. A model might average 0.07 RTF but spike to 0.15 on dense mixes, violating SLA guarantees during peak usage. Fix: Validate against .quantile(0.95) or .quantile(0.99) in CI and load testing. Set production timeouts based on p99, not averages.

4. Assuming Benchmark Tracks Represent Production Uploads

Explanation: MUSDB18-HQ contains studio-quality, multi-track stems. Real-world uploads include low-bitrate MP3s, phone recordings, and heavily compressed streams, which degrade separation quality unpredictably. Fix: Use the benchmark for variant selection, but layer a golden-file test suite with your actual user uploads. Run periodic inference audits on production samples to detect distribution shift.

5. Running Inference in CI for Model Validation

Explanation: Executing Demucs in CI pipelines introduces GPU dependencies, long runtimes, and non-deterministic results due to hardware variance. Fix: Assert against the published benchmark dataset. Pin the dataset revision in your pipeline cache. Only run inference in staging or dedicated QA environments.

6. Overlooking Stem-Specific Routing

Explanation: Treating all stems equally leads to suboptimal routing. A model optimized for vocals may perform poorly on percussive material, and vice versa. Fix: Implement dynamic routing based on the requested stem type. Maintain a configuration map that ties product features to their optimal variant.

7. Failing to Cache Dataset Downloads

Explanation: Repeated load_dataset calls in CI or development environments trigger network requests and rate limits, slowing down feedback loops. Fix: Set HF_HOME or HF_DATASETS_CACHE to a persistent volume. Pin dataset revisions using revision="v1.0" to ensure reproducible builds.

Production Bundle

Action Checklist

• Load benchmark dataset once during application startup or configuration reload
• Define product constraints: target stem, maximum RTF, minimum SDR floor
• Implement median-based scoring to avoid outlier skew
• Validate latency budgets against p95/p99 distributions, not averages
• Exclude other stem when comparing htdemucs_6s against other variants
• Cache dataset locally and pin revision in CI pipelines
• Layer golden-file testing with production uploads for distribution shift detection
• Document model routing decisions in architecture runbooks with benchmark links

Decision Matrix

Scenario	Recommended Variant	Why	Cost Impact
Vocal isolation (karaoke, lyrics, podcast cleanup)	htdemucs_ft	Highest vocal SDR (+0.66 dB over baseline) with moderate latency	Low compute overhead, optimal for streaming
Drum/bass-heavy material (DJ tools, rhythm analysis)	mdx_extra_q	Superior percussive separation (+1.48 dB drum SDR)	Slightly higher RTF, acceptable for batch processing
Low-latency preview generation	htdemucs	Fastest RTF (0.06) with acceptable baseline quality	Minimal compute cost, ideal for real-time UI feedback
Multi-instrument granularity (piano/guitar extraction)	htdemucs_6s	Splits other into distinct instrumental stems	Higher RTF (0.09), requires stem-aware post-processing
Budget-constrained edge deployment	htdemucs	Smallest footprint, fastest inference, proven stability	Lowest memory/CPU usage, suitable for mobile/IoT

Configuration Template

# audio_separation_config.yaml
benchmark:
  dataset_repo: "StemSplitio/stem-separation-benchmark-2026"
  config: "metrics_only"
  split: "results"
  revision: "v1.0"
  cache_dir: "/var/cache/hf_datasets"

routing:
  vocals:
    max_rtf: 0.08
    min_sdr_db: 9.0
    fallback_variant: "htdemucs"
  drums:
    max_rtf: null
    min_sdr_db: 10.0
    fallback_variant: "htdemucs"
  bass:
    max_rtf: 0.10
    min_sdr_db: 9.5
    fallback_variant: "htdemucs"
  other:
    max_rtf: 0.10
    min_sdr_db: 6.0
    fallback_variant: "htdemucs"

ci_policy:
  assert_quality_floor: true
  validate_p95_latency: true
  max_p95_rtf: 0.10

Quick Start Guide

1. Install dependencies: pip install datasets pandas pyyaml
2. Initialize the loader: Create an instance of AudioBenchmarkLoader and call fetch_metrics() to retrieve the cached dataset.
3. Configure constraints: Define your product requirements in a YAML file or environment variables (target stem, max RTF, SDR floor).
4. Resolve variant: Pass constraints to ModelRouter.resolve_variant() to get the optimal model_id string.
5. Validate in CI: Run CIPolicyValidator assertions during pull request checks to prevent regression before deployment.

This pattern transforms model selection from a subjective exercise into a deterministic, auditable configuration step. By treating the benchmark as a source of truth, you eliminate inference overhead during selection, enforce quality SLAs in CI, and maintain a clear audit trail for architectural decisions.