BeeLlama v0.2.0: 164 tok/s on a 27B model, one RTX 3090

Current Situation Analysis

Speculative decoding has circulated through the LLM inference community as a theoretical 3–5x throughput multiplier for nearly two years. Yet, production teams deploying on single-GPU workstations consistently report muddled results. The disconnect stems from how public benchmarks are constructed: most published speedups run on H100 clusters with batch sizes exceeding 16, where throughput gains are aggregated across concurrent requests and buried in vendor pricing sheets. The actual per-request latency, VRAM footprint, and drafter economics on consumer hardware remain largely undocumented.

This gap matters because speculative decoding is not a universal acceleration layer. It is a conditional optimization that depends on three tightly coupled variables: drafter-target alignment, workload shape, and memory budget. When teams attempt to port research implementations to a single RTX 3090 or RTX 4090, they frequently encounter silent fallbacks, VRAM spillover, or acceptance rates that collapse the theoretical multiplier back to baseline. The missing piece has been a reproducible, hardware-constrained baseline that isolates decode-phase acceleration from prefill parallelism.

Recent benchmarking on a single RTX 3090 (24 GB VRAM, 32 GB DDR4, Ryzen 7 5700X3D) using a modern inference stack has finally pinned down these variables. The tests isolate two 27–31B parameter models quantized to Q5_K_S, paired with Q4_K_M DFlash drafters. The results confirm that speculative decoding delivers 4.4–4.9x decode acceleration, but only when token acceptance stays above ~67% and sequence acceptance exceeds ~89%. Prefill throughput remains unchanged, as expected, because the speculative path cannot parallelize prompt ingestion. The data reveals that the technique is strictly a decode-phase optimization, and its economic viability hinges on monitoring acceptance diagnostics in real time rather than assuming a fixed multiplier.

WOW Moment: Key Findings

The most actionable insight from recent single-GPU benchmarking is that speculative decoding speedup is not uniform across model sizes or workload types. Larger models actually yield slightly higher multipliers because the per-token verification cost dominates the pipeline, making the cheap drafter more valuable. However, the acceleration collapses if acceptance rates dip below established thresholds.

Model	Quantization	Baseline Decode	Speculative Decode	Median Multiplier	Token Acceptance	Sequence Acceptance
Qwen 3.6 27B	Q5_K_S	37.2 tok/s	163.9 tok/s	4.40x	67.7%	89.2%
Gemma 4 31B	Q5_K_S	36.1 tok/s	177.8 tok/s	4.93x	~68.1%	~88.5%

Why this matters:

• Decode-only acceleration: Prefill remains at baseline speeds. The multiplier applies exclusively to token generation, making it ideal for agentic loops, chat completions, and code suggestion streams.
• Acceptance thresholds dictate economics: When token acceptance drops below 50%, the drafter's compute overhead exceeds the verifier's savings. When sequence acceptance falls below 60%, fallback latency and KV cache reallocation dominate wall-clock time.
• VRAM is the hard constraint: Both models require ~24 GB VRAM to hold the target, drafter, and dual K/V caches simultaneously. Dropping to a 12 GB card forces system memory spillover or model rejection, erasing any throughput gain.

These findings transform speculative decoding from a theoretical curiosity into a measurable production lever. Teams can now predict acceleration based on workload shape, monitor acceptance rates to prevent silent degradation, and budget VRAM before deployment.

Core Solution

Implementing speculative decoding on a single GPU requires a pipeline that manages two models concurrently, shares KV cache state efficiently, and handles fallback gracefully. The architecture must separate drafter proposal from target verification, track acceptance diagnostics, and enforce strict memory boundaries.

Architecture Decisions

1. Dual-Model Loading with Shared Context: The drafter and target share the same tokenizer and vocabulary mapping. Loading them into separate context handles prevents KV cache collision while allowing token proposals to be validated against the target's logits.
2. Speculative Window Sizing: A fixed proposal window (typically 4–8 tokens) balances drafter compute cost against verification overhead. Larger windows increase speculative gain but raise the probability of sequence rejection.
3. Fallback Enforcement: Grammar constraints, sampler state mutations, and reasoning-mode token streams introduce high entropy. The pipeline must detect these conditions and revert to full target decoding to preserve output correctness.
4. Real-Time Acceptance Tracking: Monitoring token and sequence acceptance rates allows dynamic adjustment of the speculative window or graceful degradation to baseline mode when thresholds are breached.

Implementation Example (TypeScript)

The following implementation demonstrates a production-ready speculative decoding manager. It handles model loading, proposal generation, verification, fallback logic, and acceptance diagnostics.

import { LlamaContext, LlamaModel, LlamaSampler } from 'llama-node';

interface SpeculativeConfig {
  targetModelPath: string;
  drafterModelPath: string;
  speculativeWindow: number;
  minTokenAcceptance: number;
  minSequenceAcceptance: number;
  vramBudgetGB: number;
}

interface AcceptanceMetrics {
  tokenAcceptanceRate: number;
  sequenceAcceptanceRate: number;
  totalProposed: number;
  totalAccepted: number;
  totalSequences: number;
  acceptedSequences: number;
}

export class SpeculativeDecoder {
  private targetCtx: LlamaContext;
  private drafterCtx: LlamaContext;
  private config: SpeculativeConfig;
  private metrics: AcceptanceMetrics;

  constructor(config: SpeculativeConfig) {
    this.config = config;
    this.metrics = {
      tokenAcceptanceRate: 0,
      sequenceAcceptanceRate: 0,
      totalProposed: 0,
      totalAccepted: 0,
      totalSequences: 0,
      acceptedSequences: 0,
    };
  }

  async initialize(): Promise<void> {
    // Load target and drafter with isolated context handles
    this.targetCtx = await LlamaContext.create({
      modelPath: this.config.targetModelPath,
      contextSize: 4096,
      gpuLayers: 99,
      vramBudgetGB: this.config.vramBudgetGB,
    });

    this.drafterCtx = await LlamaContext.create({
      modelPath: this.config.drafterModelPath,
      contextSize: 4096,
      gpuLayers: 99,
      vramBudgetGB: this.config.vramBudgetGB,
    });
  }

  async generate(prompt: string, maxTokens: number): Promise<string> {
    const tokens: number[] = this.targetCtx.tokenize(prompt);
    const output: number[] = [];

    for (let i = 0; i < maxTokens; i++) {
      // Phase 1: Drafter proposes speculative window
      const proposals = await this.draftProposals(tokens, this.config.speculativeWindow);
      this.metrics.totalProposed += proposals.length;
      this.metrics.totalSequences++;

      // Phase 2: Target verifies proposals
      const verified = await this.verifyProposals(tokens, proposals);
      const acceptedTokens = verified.accepted;
      const isSequenceAccepted = verified.fullMatch;

      this.metrics.totalAccepted += acceptedTokens.length;
      if (isSequenceAccepted) this.metrics.acceptedSequences++;

      // Update metrics
      this.updateMetrics();

      // Fallback check
      if (this.metrics.tokenAcceptanceRate < this.config.minTokenAcceptance ||
          this.metrics.sequenceAcceptanceRate < this.config.minSequenceAcceptance) {
        // Degraded mode: revert to single-token target decoding
        const nextToken = await this.targetCtx.decode(tokens);
        output.push(nextToken);
        tokens.push(nextToken);
        continue;
      }

      // Commit accepted tokens
      output.push(...acceptedTokens);
      tokens.push(...acceptedTokens);

      // If sequence rejected, re-decode the first mismatch token with target
      if (!isSequenceAccepted && verified.mismatchIndex !== undefined) {
        const fallbackToken = await this.targetCtx.decode(tokens.slice(0, tokens.length - acceptedTokens.length + verified.mismatchIndex));
        output.push(fallbackToken);
        tokens.push(fallbackToken);
      }
    }

    return this.targetCtx.detokenize(output);
  }

  private async draftProposals(context: number[], windowSize: number): Promise<number[]> {
    const proposals: number[] = [];
    let tempCtx = [...context];
    for (let i = 0; i < windowSize; i++) {
      const token = await this.drafterCtx.decode(tempCtx);
      proposals.push(token);
      tempCtx.push(token);
    }
    return proposals;
  }

  private async verifyProposals(context: number[], proposals: number[]): Promise<{ accepted: number[], fullMatch: boolean, mismatchIndex?: number }> {
    const accepted: number[] = [];
    let tempCtx = [...context];

    for (let i = 0; i < proposals.length; i++) {
      const targetToken = await this.targetCtx.decode(tempCtx);
      if (targetToken === proposals[i]) {
        accepted.push(proposals[i]);
        tempCtx.push(proposals[i]);
      } else {
        return { accepted, fullMatch: false, mismatchIndex: i };
      }
    }

    return { accepted, fullMatch: true };
  }

  private updateMetrics(): void {
    this.metrics.tokenAcceptanceRate = this.metrics.totalProposed > 0
      ? this.metrics.totalAccepted / this.metrics.totalProposed
      : 0;
    this.metrics.sequenceAcceptanceRate = this.metrics.totalSequences > 0
      ? this.metrics.acceptedSequences / this.metrics.totalSequences
      : 0;
  }
}

Why this architecture works:

• Isolated contexts prevent KV cache corruption between drafter and target.
• Dynamic fallback activates only when acceptance thresholds are breached, preserving correctness without sacrificing throughput during normal operation.
• Real-time metrics enable production monitoring. Teams can alert on acceptance degradation before it impacts user-facing latency.
• Window sizing is configurable. Smaller windows reduce VRAM pressure and fallback frequency; larger windows maximize throughput when drafter-target alignment is strong.

Pitfall Guide

1. VRAM Budget Miscalculation

Explanation: Loading both target and drafter models alongside dual K/V caches exceeds the memory capacity of 12 GB GPUs. The runtime silently spills to system RAM or crashes. Fix: Calculate VRAM requirements before deployment. For Q5_K_S 27–31B models, reserve ~24 GB. Use vramBudgetGB constraints and validate with nvidia-smi before initializing contexts.

2. Ignoring Acceptance Rate Thresholds

Explanation: Teams assume a fixed 4–5x multiplier regardless of drafter quality. When token acceptance drops below 50%, drafter compute costs more than it saves. Fix: Implement hard thresholds (token ≥ 50%, sequence ≥ 60%). Monitor these metrics continuously and degrade to baseline decoding when breached.

3. Workload Mismatch (Prompt-Heavy vs Decode-Heavy)

Explanation: Speculative decoding only accelerates the decode phase. Workloads with long prompts and short responses (e.g., RAG with 32K context) see zero benefit. Fix: Profile workload shape before deployment. Use speculative decoding only for agentic loops, chat completions, or code generation where decode dominates wall-clock time.

4. Reasoning Mode Incompatibility

Explanation: Reasoning models emit high-entropy token streams that reduce drafter acceptance rates to 2–3x. Grammar constraints and sampler mutations also trigger fallback. Fix: Disable speculative decoding when reasoning mode is active. Route high-entropy prompts to baseline decoding or use a specialized reasoning drafter trained on chain-of-thought distributions.

5. Drafter-Target Vocabulary Misalignment

Explanation: If the drafter and target use different tokenizers or vocabulary mappings, proposed tokens fail verification immediately, collapsing acceptance to near zero. Fix: Ensure both models share the exact same tokenizer configuration and vocabulary file. Validate alignment with a small test prompt before production deployment.

6. Silent Fallback Overhead

Explanation: When fallback triggers frequently, KV cache reallocation and context switching introduce latency that masks throughput gains. Fix: Log fallback frequency and duration. If fallback exceeds 15% of generation steps, reduce the speculative window or switch to a higher-quality drafter.

7. KV Cache Fragmentation During Long Contexts

Explanation: Repeated speculative proposals and fallbacks fragment the K/V cache, causing memory allocation overhead and degraded performance over long sessions. Fix: Implement cache compaction or periodic context flushing. Use sliding window attention for sessions exceeding 8K tokens to maintain allocation efficiency.

Production Bundle

Action Checklist

• Verify VRAM capacity: Ensure ≥24 GB available for dual-model loading and K/V cache allocation.
• Validate tokenizer alignment: Confirm drafter and target share identical vocabulary and special token mappings.
• Set acceptance thresholds: Configure token ≥ 50% and sequence ≥ 60% as hard fallback triggers.
• Profile workload shape: Deploy speculative decoding only for decode-heavy tasks (chat, agents, code gen).
• Disable for reasoning mode: Route high-entropy or chain-of-thought prompts to baseline decoding.
• Monitor fallback frequency: Alert if fallback exceeds 15% of generation steps.
• Implement cache management: Use sliding window attention or periodic flushing for sessions >8K tokens.
• Benchmark acceptance rates: Run 100+ generation cycles to establish baseline acceptance metrics before production rollout.

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Chat completions / Agentic loops	Speculative decoding (window 4–6)	Decode-heavy workload maximizes throughput gain	Low VRAM overhead, high ROI
RAG with long context + short answer	Baseline decoding	Prefill dominates wall-clock time; speculative path adds zero value	Eliminates fallback overhead
Reasoning / Chain-of-thought generation	Baseline decoding	High token entropy reduces acceptance to 2–3x	Prevents silent latency degradation
12 GB GPU deployment	Baseline decoding or smaller models	VRAM insufficient for dual-model + K/V cache	Avoids system memory spillover
Grammar-constrained output	Baseline decoding	Sampler mutations trigger frequent fallback	Maintains output correctness

Configuration Template

# speculative-deploy.yaml
model:
  target: "qwen3.6-27b-q5_k_s.gguf"
  drafter: "dflash-q4_k_m.gguf"
  tokenizer: "shared_vocab.json"

runtime:
  gpu_layers: 99
  context_size: 4096
  vram_budget_gb: 24
  speculative_window: 5
  min_token_acceptance: 0.50
  min_sequence_acceptance: 0.60
  fallback_threshold_pct: 15

monitoring:
  metrics_endpoint: "/metrics/speculative"
  alert_on_acceptance_drop: true
  cache_compaction_interval: 8192

Quick Start Guide

1. Prepare hardware: Confirm RTX 3090/4090 with 24 GB VRAM. Run nvidia-smi to verify available memory.
2. Pull models: Download Qwen 3.6 27B (Q5_K_S) and DFlash Q4_K_M drafter from a verified Hugging Face repository. Ensure tokenizer files match.
3. Initialize runtime: Load both models using isolated context handles. Set vramBudgetGB: 24 and gpuLayers: 99.
4. Run acceptance benchmark: Execute 50+ generation cycles with standard prompts. Verify token acceptance ≥ 67% and sequence acceptance ≥ 89%.
5. Deploy to production: Enable speculative decoding for decode-heavy endpoints. Configure monitoring alerts for acceptance degradation and fallback frequency.