Local LLM benchmarking

xt combinations before deployment avoid costly hardware upgrades, reduce OOM incidents, and align model selection with actual workload profiles rather than marketing benchmarks.

Core Solution

Building a reproducible local LLM benchmarking pipeline requires decoupling measurement from execution. The inference engine should run as a standalone service, while a benchmark orchestrator handles request generation, metric collection, statistical aggregation, and result export. This architecture ensures cross-platform compatibility, eliminates framework lock-in, and enables headless CI/CD integration.

Step 1: Isolate the Inference Environment

Run the LLM backend as a stateless HTTP service. llama.cpp with llama-server is the industry standard for local deployment due to its CPU/GPU hybrid execution, quantization support, and streaming SSE compatibility. Configure it with fixed context windows, disabled caching, and explicit tensor splitting if using multiple GPUs.

llama-server \
  -m model.gguf \
  -c 32768 \
  --gpu-layers 35 \
  --threads 8 \
  --host 127.0.0.1 \
  --port 8080 \
  --no-mmap

Step 2: Implement the Benchmark Orchestrator

The orchestrator handles warm-up, request dispatch, streaming metric extraction, and statistical aggregation. TypeScript is ideal for this layer due to its strong async primitives, HTTP client capabilities, and JSON-native data handling.

import { spawn, ChildProcess } from 'child_process';
import { performance } from 'perf_hooks';
import { createReadStream, writeFileSync } from 'fs';
import { join } from 'path';

interface BenchmarkConfig {
  endpoint: string;
  prompt: string;
  maxTokens: number;
  runs: number;
  contextWindow: number;
}

interface MetricSample {
  ttft: number;
  throughput: number;
  totalTokens: number;
  duration: number;
}

export class LLMBenchmark {
  private config: BenchmarkConfig;
  private samples: MetricSample[] = [];

  constructor(config: BenchmarkConfig) {
    this.config = config;
  }

  async run(): Promise<void> {
    console.log(`Starting benchmark: ${this.config.runs} runs, context=${this.config.contextWindow}`);
    
    // Warm-up phase to stabilize GPU clocks and KV cache allocation
    await this.executeRun(true);
    
    for (let i = 0; i < this.config.runs; i++) {
      const sample = await this.executeRun(false);
      this.samples.push(sample);
      console.log(`Run ${i + 1}/${this.config.runs} | TTFT: ${sample.ttft.toFixed(1)}ms | Throughput: ${sample.throughput.toFixed(1)} tok/s`);
    }

    this.exportResults();
  }

  private async executeRun(warmup: boolean): Promise<MetricSample> {
    const payload = {
      prompt: this.config.prompt,
      n_predict: this.config.maxTokens,
      temperature: 0.0,
      stream: true
    };

    const start = performance.now();
    let firstTokenTime: number | null = null;
    let tokenCount = 0;

    const response = await fetch(`${this.config.endpoint}/completion`, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify(payload)
    });

    if (!response.ok) throw new Error(`Server error: ${response.statusText}`);

    const reader = response.body?.getReader();
    if (!reader) throw new Error('No response stream');

    while (true) {
      const { done, value } = await reader.read();
      if (done) break;

      const chunk = new TextDecoder().decode(value);
      const lines = chunk.split('\n').filter(line => line.startsWith('data:'));
      
      for (const line of lines) {
        try {
          const json = JSON.parse(line.replace('data: ', ''));
          if (!firstTokenTime && json.content) {
            firstTokenTime = performance.now();
          }
          if (json.content) tokenCount++;
          if (json.stop) break;
        } catch {}
      }
    }

    const end = performance.now();
    const ttft = firstTokenTime ? firstTokenTime - start : 0;
    const duration = end - start;
    const throughput = (tokenCount / duration) * 1000;

    return {
      ttft,
      throughput,
      totalTokens: tokenCount,
      duration
    };
  }

  private exportResults(): void {
    const avgTTFT = this.samples.reduce((a, b) => a + b.ttft, 0) / this.samples.length;
    const avgThroughput = this.samples.reduce((a, b) => a + b.throughput, 0) / this.samples.length;
    const p95TTFT = this.samples.map(s => s.ttft).sort((a, b) => a - b)[Math.floor(this.samples.length * 0.95)];
    const p95Throughput = this.samples.map(s => s.throughput).sort((a, b) => a - b)[Math.floor(this.samples.length * 0.05)];

    const report = {
      config: this.config,
      aggregates: { avgTTFT, avgThroughput, p95TTFT, p95Throughput },
      samples: this.samples
    };

    writeFileSync(join(process.cwd(), 'benchmark-report.json'), JSON.stringify(report, null, 2));
    console.log('Benchmark complete. Report saved to benchmark-report.json');
  }
}

Step 3: Instrument Hardware Telemetry

Inference metrics alone are insufficient. Production benchmarking must capture GPU utilization, memory bandwidth saturation, thermal state, and power draw. Integrate nvml-node or parse nvidia-smi output asynchronously during benchmark runs. Correlate throughput drops with temperature thresholds to identify thermal throttling boundaries.

Step 4: Aggregate with Statistical Rigor

Single-run variance in local LLM inference routinely exceeds 15% due to OS scheduler jitter, PCIe bus contention, and dynamic memory allocation. Always collect p50 and p95 latencies, discard warm-up runs, and enforce fixed random seeds for deterministic prompt encoding. Export results in structured JSON/CSV for CI/CD pipeline consumption.

Architecture Rationale

The decoupled design separates concerns: the inference engine handles tensor computation and KV cache management, while the orchestrator manages measurement, statistical validation, and reporting. This prevents benchmarking logic from contaminating production inference paths. Streaming SSE parsing captures true TTFT rather than relying on server-side estimates. Statistical aggregation with percentiles filters out scheduler noise, producing reproducible capacity metrics. The TypeScript harness runs on any platform with Node.js 18+, enabling cross-environment validation (local workstation, CI runner, edge device).

Pitfall Guide

1. Benchmarking Cold KV Cache States

Local LLMs allocate KV cache dynamically during the first inference pass. Cold runs show artificially high TTFT and memory allocation overhead. Production systems operate on warmed caches. Always execute 3–5 warm-up runs before measurement, and reset cache state only when testing context window scaling.

2. Conflating Prompt Encoding and Decode Speeds

Prefill (prompt processing) is compute-bound and scales quadratically with context length. Decode (token generation) is memory-bandwidth-bound and scales linearly. Reporting a single "tokens/sec" metric masks which phase is bottlenecked. Separate TTFT and generation throughput in all reports.

3. Ignoring Thermal Throttling Boundaries

Consumer GPUs reduce clock speeds when junction temperature exceeds 83–87°C. Throughput can drop 30–45% after 10 minutes of sustained inference. Benchmark runs shorter than 5 minutes produce unrealistic peak numbers. Enforce minimum 8-minute continuous runs and log temperature curves alongside throughput.

4. Single-Run Variance Acceptance

OS process scheduling, PCIe bus arbitration, and dynamic memory defragmentation introduce 10–20% latency variance per run. Reporting mean values without standard deviation or percentiles leads to flawed capacity planning. Collect minimum 15 runs, discard outliers beyond 2σ, and report p95 latency for SLA alignment.

5. Overlooking Quantization Degradation on Long Context

INT4 quantization compresses weights aggressively, but KV cache remains FP16/FP32. At 32K+ context, the model spends more time reading uncompressed cache than compressed weights, diminishing quantization benefits. Accuracy degradation also compounds with context length due to attention head precision loss. Validate quantization impact at target context windows, not just 4K.

6. Using Accuracy Leaderboards for Inference Selection

MMLU, GSM8K, and HellaSwag measure reasoning capability, not production viability. A model scoring 85% on MMLU may have 3x higher TTFT than an 82% model due to architecture differences (MoE vs dense, layer count, attention mechanism). Separate accuracy evaluation from inference benchmarking. Use accuracy gates for model selection, but use throughput/latency/VRAM metrics for deployment sizing.

7. Neglecting Batch Size and Concurrency Effects

Local LLM servers handle concurrent requests by queueing or dynamic batching. Throughput scales non-linearly with concurrency due to KV cache fragmentation and context switching. Benchmark single-request latency first, then test 2, 4, and 8 concurrent streams. Measure queue depth, timeout rates, and OOM triggers under load.

Production Best Practices:

• Lock context window size during benchmarking; do not allow dynamic expansion.
• Disable speculative decoding and caching during measurement to isolate baseline performance.
• Use fixed temperature (0.0) and deterministic sampling to eliminate generation variance.
• Correlate throughput drops with nvidia-smi clock state and thermal readings.
• Export raw samples, not just aggregates, to enable post-hoc statistical analysis.

Production Bundle

Action Checklist

• Isolate inference environment: Deploy model via stateless HTTP server with fixed context window and explicit GPU layer allocation.
• Implement benchmark orchestrator: Build TS harness that streams responses, extracts TTFT and throughput, and enforces warm-up phases.
• Instrument hardware telemetry: Log GPU utilization, memory bandwidth, temperature, and power draw during each run.
• Enforce statistical rigor: Execute minimum 15 runs, calculate p50/p95 percentiles, discard warm-up data, and export raw samples.
• Validate quantization at target context: Test INT4/INT8/FP16 at production context windows, not just 4K baselines.
• Test concurrency boundaries: Measure throughput degradation under 2, 4, and 8 concurrent streams; identify OOM and queue depth limits.
• Correlate thermal behavior: Run continuous 8–10 minute inference sessions; map throughput drops to temperature thresholds.
• Integrate with CI/CD: Automate benchmark execution on hardware matrix; fail deployments when p95 latency exceeds SLA thresholds.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Latency-critical chat interface	INT8 / 8K context / FP16 KV cache	Balances TTFT <150ms with acceptable accuracy; avoids INT4 degradation on reasoning	Moderate VRAM overhead; eliminates need for GPU upgrade
High-throughput batch processing	INT4 / 32K context / Dynamic batching	Maximizes tokens/sec; compresses weights to free memory bandwidth for concurrent streams	Higher accuracy loss; requires thermal monitoring to prevent throttling
Memory-constrained edge device	INT4 / 4K context / CPU offload fallback	Fits within 8–12 GB VRAM; maintains deployability on consumer hardware	Increased TTFT; requires graceful degradation logic for OOM conditions

Configuration Template

{
  "benchmark": {
    "endpoint": "http://127.0.0.1:8080",
    "model_path": "./models/llama-3-7b-instruct-Q8_0.gguf",
    "context_window": 8192,
    "max_tokens": 256,
    "temperature": 0.0,
    "runs": 20,
    "warmup_runs": 3,
    "concurrency": 1,
    "metrics": {
      "collect_ttft": true,
      "collect_throughput": true,
      "collect_gpu_telemetry": true,
      "gpu_tool": "nvidia-smi",
      "poll_interval_ms": 500
    },
    "export": {
      "format": "json",
      "path": "./reports/benchmark-report.json",
      "include_percentiles": [50, 95],
      "include_raw_samples": true
    },
    "thermal_thresholds": {
      "max_temp_c": 85,
      "throttle_alert": true,
      "log_curve": true
    }
  }
}

Quick Start Guide

1. Install dependencies: npm init -y && npm install typescript @types/node
2. Deploy inference server: Download quantized GGUF model, start llama-server with fixed context and GPU layers.
3. Run benchmark harness: Execute the TS orchestrator against the local endpoint; monitor console output for TTFT and throughput per run.
4. Analyze results: Open benchmark-report.json, verify p95 latency aligns with SLA, cross-reference thermal logs with throughput drops, and adjust quantization/context parameters accordingly.