Honest Perf Benchmarks for a Paid-API Compiler

Engineering Deterministic Benchmarks for Hybrid AI Compilation Pipelines

Current Situation Analysis

Modern compilation and transformation pipelines no longer operate in purely deterministic environments. It is increasingly common to see build steps that combine static analysis, AST manipulation, and structured output generation with stochastic, paid LLM calls for reasoning, summarization, or semantic validation. This hybrid architecture introduces a benchmarking paradox: how do you measure performance when half your pipeline is non-deterministic and costs real money per execution?

Most teams handle this by either benchmarking only the deterministic stages (leaving the actual bottleneck unmeasured) or running the full pipeline blindly in CI (draining API budgets and generating noisy, non-reproducible metrics). The problem is frequently misunderstood because performance regression detection requires three things that hybrid pipelines actively resist: byte-identical inputs across machines, explicit cost consent, and honest handling of skipped stages. When any of these are compromised, benchmark suites produce false confidence. A 15% "improvement" might just be a different random seed. A "stable" metric might be masking a skipped stage that quietly recorded zero milliseconds.

The engineering reality is that paid API steps cannot be treated like standard function calls. They introduce latency variance, rate limits, and budget constraints. Without a deterministic corpus, cross-machine comparisons become statistically meaningless. Without explicit gating, CI pipelines either fail silently when budgets exhaust or burn tokens on every commit. The industry standard approach of "just run it and average the results" collapses under the weight of stochastic variance and financial overhead.

WOW Moment: Key Findings

The breakthrough in hybrid pipeline benchmarking comes from treating determinism, cost control, and result integrity as first-class architectural constraints rather than afterthoughts. The following comparison demonstrates why conventional approaches fail and why the deterministic-gated-skip pattern succeeds:

Approach	Input Reproducibility	Cost Control	Trend Accuracy	CI Stability
Committed Fixtures	High	None	Low (stale data)	High
Random Generation	None	None	None (noise)	Low
Single-Gate API Calls	High	Weak (fails open)	Medium	Medium
Seeded Generator + Double Gate + Skip Recording	Byte-identical	Explicit consent	High (no zero-skew)	High

This finding matters because it decouples performance measurement from execution cost. Teams can now run deterministic stages on every pull request, gate paid stages behind explicit opt-in, and maintain a continuous JSON timeline that accurately reflects what ran, what didn't, and why. The benchmark suite stops being a liability and becomes a regression detection system that respects budget constraints while preserving statistical validity.

Core Solution

Building a reliable benchmark suite for a hybrid pipeline requires four coordinated components: a deterministic corpus factory, a capability-intent gating system, a skip-aware result recorder, and a runner that captures execution metadata without framework lock-in.

Step 1: Deterministic Corpus Generation

Random generation is necessary for scalable, representative test data, but it must be cryptographically reproducible. The solution uses a seeded pseudo-random number generator paired with a deterministic UUID v4 constructor. This guarantees that identical seeds produce byte-identical outputs across any machine, Node version, or CI runner.

class SeededRNG {
  private state: number;

  constructor(seed: number) {
    this.state = seed;
  }

  next(): number {
    let t = (this.state += 0x6d2b79f5);
    t = Math.imul(t ^ (t >>> 15), t | 1);
    t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  }
}

class DeterministicUUIDFactory {
  static generate(rng: SeededRNG): string {
    const buffer = new Uint8Array(16);
    for (let i = 0; i < 16; i++) {
      buffer[i] = Math.floor(rng.next() * 256);
    }
    buffer[6] = (buffer[6] & 0x0f) | 0x40; // Version 4
    buffer[8] = (buffer[8] & 0x3f) | 0x80; // Variant 1
    return this.formatHex(buffer);
  }

  private static formatHex(bytes: Uint8Array): string {
    const hex = Array.from(bytes).map(b => b.toString(16).padStart(2, '0')).join('');
    return `${hex.slice(0,8)}-${hex.slice(8,12)}-${hex.slice(12,16)}-${hex.slice(16,20)}-${hex.slice(20)}`;
  }
}

Architecture Rationale: crypto.randomUUID() is explicitly avoided because it draws from OS entropy and breaks reproducibility. The seeded approach guarantees that front-matter identifiers, content hashes, and cache keys remain stable across runs. This is non-negotiable for pipelines that rely on content-addressable caching or incremental compilation.

Step 2: Capability-Intent Gating

Paid API stages require a two-layer guard. The first layer verifies that credentials exist. The second layer verifies that the operator explicitly intends to spend tokens. This prevents accidental budget drain when developers have API keys configured for local tooling but run benchmarks in shared environments.

class BenchmarkGate {
  static isPaidStageAllowed(): boolean {
    const hasKey = !!process.env.ANTHROPIC_API_KEY;
    const hasConsent = process.env.BENCH_ALLOW_PAID === '1';
    return hasKey && hasConsent;
  }

  static getSkipReason(): string {
    if (!process.env.ANTHROPIC_API_KEY) {
      return 'Missing ANTHROPIC_API_KEY';
    }
    return 'BENCH_ALLOW_PAID not set to 1';
  }
}

Architecture Rationale: Single-flag designs fail open. Developers routinely have API keys in their shell profiles for CLI tools, IDE extensions, or local debugging. Requiring an explicit consent flag creates a deliberate friction point that aligns with operational security principles. The pattern mirrors production deployment guards: capability proves you can, consent proves you should.

Step 3: Skip-Aware Result Recording

When a stage is gated, recording zero milliseconds corrupts trend analysis. Omitting the record entirely breaks timeline continuity. The correct approach records a structured skip state that preserves scenario visibility without polluting performance histograms.

interface BenchmarkRecord {
  scenario: string;
  gitSha: string;
  nodeVersion: string;
  platform: string;
  timestamp: string;
  medianMs?: number;
  rssDeltaMb?: number;
  iterations: number;
  skipped: boolean;
  skipReason?: string;
}

function recordScenario(result: Partial<BenchmarkRecord>): BenchmarkRecord {
  return {
    scenario: result.scenario!,
    gitSha: result.gitSha!,
    nodeVersion: result.nodeVersion!,
    platform: result.platform!,
    timestamp: new Date().toISOString(),
    iterations: result.iterations ?? 0,
    skipped: result.skipped ?? false,
    skipReason: result.skipReason,
    medianMs: result.medianMs,
    rssDeltaMb: result.rssDeltaMb,
  };
}

Architecture Rationale: The Partial typing allows timing fields to legitimately absent when skipped. Baseline comparison scripts can now distinguish between regression, improvement, and non-execution. Skipped records remain in the JSON timeline, proving the scenario was evaluated, but are excluded from statistical aggregations.

Step 4: Runner Architecture

The benchmark runner must capture execution metadata, enforce warmup cycles, calculate medians across iterations, and measure memory footprint deltas. Custom implementation avoids framework constraints that don't align with cross-run comparison requirements.

class PipelineBenchmark {
  private readonly iterations: number;
  private readonly warmupCount: number;

  constructor(iterations = 5, warmupCount = 2) {
    this.iterations = iterations;
    this.warmupCount = warmupCount;
  }

  async measure<T>(fn: () => Promise<T>): Promise<{ median: number; rssDelta: number }> {
    // Warmup phase
    for (let i = 0; i < this.warmupCount; i++) await fn();

    const timings: number[] = [];
    const initialRss = process.memoryUsage().rss;

    for (let i = 0; i < this.iterations; i++) {
      const start = performance.now();
      await fn();
      timings.push(performance.now() - start);
    }

    const finalRss = process.memoryUsage().rss;
    timings.sort((a, b) => a - b);
    const median = timings[Math.floor(timings.length / 2)];

    return {
      median,
      rssDelta: (finalRss - initialRss) / (1024 * 1024),
    };
  }
}

Architecture Rationale: Vitest's built-in bench was evaluated but rejected because it lacks native support for custom JSON timeline schemas and cross-run baseline comparison. Building a lightweight runner provides full control over output shape, metadata injection, and memory tracking. The median is preferred over mean to resist outlier latency spikes common in I/O and network-bound stages.

Pitfall Guide

1. The crypto.randomUUID Mirage

Explanation: Using Node's native UUID generator seems correct for production code but breaks benchmark reproducibility. Each run generates different identifiers, causing content-hash caches to invalidate differently across machines. Fix: Implement a seeded UUID v4 constructor that derives bytes from a deterministic PRNG stream. Reserve crypto.randomUUID() for production runtime only.

2. Single-Flag API Gating

Explanation: Checking only for ANTHROPIC_API_KEY fails when developers have credentials configured for local tooling. Benchmarks run unintentionally, draining budgets or hitting rate limits. Fix: Require explicit consent via a secondary environment variable. Capability proves access; consent proves intent. Document this pattern in CI configuration files.

3. The Zero-Value Skew

Explanation: Recording 0ms for skipped stages tricks trend analysis into believing performance improved infinitely. Histograms become meaningless, and regression alerts stop firing. Fix: Record a structured skip state with skipped: true and a skipReason. Exclude skipped records from statistical calculations while preserving timeline continuity.

4. Regex State Leakage Across Iterations

Explanation: Module-level regular expressions with the global (/g) flag retain lastIndex state between calls. Subsequent benchmark iterations operate on truncated or empty strings, producing artificially fast timings. Fix: Construct regular expressions inside the function scope or reset lastIndex = 0 before each iteration. Prefer String.prototype.matchAll() or non-global patterns for stateless operations.

5. Parser Boundary Mismatches

Explanation: Different YAML/front-matter parsers handle quoting inconsistently. gray-matter may quote string values while a custom validator strips them, causing datetime or enum checks to fail with literal quote characters. Fix: Document parser behavior at module boundaries. Normalize front-matter output before validation, or align the validator's parsing rules with the generator's serialization format.

6. Framework Mismatch for Timeline Storage

Explanation: Standard testing frameworks optimize for unit test assertions, not cross-run performance comparison. Bolting JSON timeline storage onto existing benchmark runners creates maintenance debt and schema drift. Fix: Implement a lightweight custom runner when cross-machine comparison, metadata injection, and skip-aware recording are required. The trade-off favors control over convenience.

7. Ignoring Memory Footprint

Explanation: Focusing solely on execution time misses memory leaks, cache bloat, and garbage collection pressure. A pipeline can appear fast while consuming increasing heap space across iterations. Fix: Measure process.memoryUsage().rss before and after benchmark cycles. Track RSS delta alongside timing metrics. Alert when memory growth exceeds acceptable thresholds.

Production Bundle

Action Checklist

• Seed your corpus generator with a fixed PRNG and verify byte-identical output across local and CI environments
• Implement a double-gate system requiring both API credentials and explicit consent flags for paid stages
• Record skipped scenarios with structured metadata instead of zero values or omission
• Replace module-level global regexes with scoped instances to prevent lastIndex state leakage
• Align front-matter serialization and validation parsers to prevent quote-handling mismatches
• Measure RSS delta alongside execution time to detect memory pressure and cache bloat
• Store benchmark results in a gitignored directory with explicit baseline tracking
• Validate benchmark output schema against a strict TypeScript interface before timeline ingestion

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Deterministic stages (lint, parse, transform)	Run on every PR with seeded corpus	Zero marginal cost, high regression detection value	None
Paid API stages (summarization, reasoning)	Gate with explicit consent flag	Prevents budget drain, aligns with operational security	Controlled, opt-in only
Cross-machine comparison	Seeded PRNG + deterministic UUIDs	Guarantees byte-identical inputs regardless of environment	None
Trend analysis & baseline tracking	Skip-aware JSON timeline with median/RSS	Preserves statistical validity, prevents zero-skew	None
Quick local validation	Reduced iteration count (3) + warmup only	Faster feedback loop during development	None
Production regression monitoring	Full iteration count (5+) + RSS tracking	Higher confidence, detects memory drift	None

Configuration Template

// bench.config.ts
export const BENCHMARK_CONFIG = {
  corpus: {
    seed: 42,
    fileCount: 50,
    wordsPerFile: 500,
    outputDir: './.bench-corpus',
  },
  runner: {
    iterations: 5,
    warmupIterations: 2,
    timeoutMs: 30000,
  },
  gates: {
    paidApi: {
      credentialEnv: 'ANTHROPIC_API_KEY',
      consentEnv: 'BENCH_ALLOW_PAID',
      consentValue: '1',
    },
  },
  output: {
    resultsDir: './.bench-results',
    baselineFile: './benchmarks/baseline.json',
    timelineFile: './benchmarks/timeline.json',
  },
  thresholds: {
    ingest: { maxMs: 2000, maxRssDeltaMb: 50 },
    lint: { maxMs: 30000, maxRssDeltaMb: 100 },
    render: { maxMs: 5000, maxRssDeltaMb: 75 },
  },
};

Quick Start Guide

1. Initialize the benchmark workspace: Create a dedicated benchmarks/ directory with a package.json that isolates dependencies. Add .gitkeep to the results directory to ensure git tracking without committing transient data.
2. Generate the deterministic corpus: Run the seeded generator with a fixed seed value. Verify output consistency by running twice and comparing SHA-256 hashes of the generated files.
3. Configure gating and thresholds: Set BENCH_ALLOW_PAID=0 by default in CI. Define per-scenario thresholds in the configuration file. Ensure baseline comparison scripts exclude skipped records from statistical calculations.
4. Execute and validate: Run the benchmark suite locally. Verify that deterministic stages produce stable medians, paid stages record skip states when gated, and RSS deltas remain within thresholds. Commit the baseline JSON to version control.
5. Integrate with CI: Add a workflow step that runs deterministic stages on every push. Gate paid stages behind manual triggers or scheduled runs. Configure baseline comparison to fail PRs when median execution time exceeds thresholds or RSS growth indicates memory regression.