Convert HF checkpoint to GGUF

# Convert HF checkpoint to GGUF
python convert-hf-to-gguf.py model_dir --outfile model.gguf

# Quantize to 4-bit mixed precision
./quantize model.gguf model-q4_k_m.gguf Q4_K_M

import { LlamaInstance, LlamaContext, LlamaChatSession } from 'llama-cpp-node';
import path from 'path';

const MODEL_PATH = path.resolve('./models/mistral-7b-instruct-q4_k_m.gguf');

export async function createInferenceEngine() {
  const instance = await LlamaInstance.init({
    model: MODEL_PATH,
    gpuLayers: -1, // Offload all layers to Metal/CUDA
    contextSize: 4096,
    threads: 8,
    mmap: true, // Memory-map model file instead of loading entirely into RAM
    verbose: false,
  });

  const context = await instance.createContext();
  return new LlamaChatSession({ context });
}

erence Pipeline Batching and streaming are non-negotiable for production. Streaming prevents memory spikes from long completions and enables progressive UI updates.

export async function generateResponse(
  session: LlamaChatSession,
  prompt: string,
  onToken: (token: string) => void
): Promise<string> {
  const response = await session.prompt(prompt, {
    maxTokens: 1024,
    temperature: 0.7,
    topP: 0.9,
    repeatPenalty: 1.1,
    onToken: (token) => {
      const decoded = instance?.tokenize(token) ? token : '';
      onToken(decoded);
    },
  });

  return response;
}

Step 4: Context Window Management

Transformer KV caches grow linearly with context length. Without management, memory exhaustion occurs after ~3000 tokens. Implement a sliding window or importance-based pruning strategy.

class ContextManager {
  private history: Array<{ role: 'user' | 'assistant'; content: string }> = [];
  private readonly maxTokens = 3500; // Reserve headroom for generation

  addEntry(role: 'user' | 'assistant', content: string) {
    this.history.push({ role, content });
    this.prune();
  }

  private prune() {
    while (this.estimateTokenCount() > this.maxTokens) {
      this.history.shift(); // Fallback: drop oldest. Production should use semantic importance scoring.
    }
  }

  private estimateTokenCount(): number {
    return this.history.reduce((acc, msg) => acc + msg.content.length / 4, 0);
  }

  formatPrompt(): string {
    return this.history.map(m => `<${m.role}>${m.content}</${m.role}>`).join('\n');
  }
}

Architecture Decisions and Rationale

• GGUF over ONNX/Safetensors: GGUF embeds quantization metadata, tokenizer alignment, and architecture hints in a single file. Runtimes parse it without external config files, reducing deployment surface area.
• Memory Mapping (mmap: true): Loads only accessed pages into RAM. Critical for systems with 16GB unified memory where the OS and application compete for space.
• GPU Layer Offload (gpuLayers: -1): Maximizes parallel matrix multiplication. Falls back gracefully to CPU if VRAM is insufficient, preventing hard crashes.
• Streaming with onToken: Decouples generation from completion. Enables cancellation, progress indicators, and memory-safe token accumulation.
• Sliding Context: Prevents OOM errors. Production systems should replace naive FIFO pruning with attention-weighted retention or RAG-augmented context compression.

Pitfall Guide

1. Ignoring Memory Mapping and Loading the Entire Model into RAM Loading a 4.2GB GGUF file directly into process memory leaves zero headroom for KV cache, tokenizer buffers, and application state. The process will OOM during the first generation. Always enable mmap and monitor RSS vs. VSZ metrics.

2. Tokenizer-Model Mismatch Using a tokenizer trained on a different vocabulary than the target model causes silent corruption. Tokens decode to garbage, and generation diverges. Verify tokenizer.json matches the GGUF header, or rely on the runtime’s bundled tokenizer.

3. Thermal Throttling on Mobile and Thin Laptops Sustained inference pushes silicon to 95°C+. Modern chips downclock aggressively, dropping throughput from 35 tok/s to 8 tok/s within 45 seconds. Implement duty cycling, fan control hooks, or batched inference windows to allow thermal recovery.

4. Context Window Overflow Without Pruning KV cache allocation is linear. A 4096-context window with 7B parameters consumes ~2.1GB. Exceeding it triggers allocation failure. Always cap active context and implement sliding windows or compression before production release.

5. Assuming Uniform Hardware Performance A 7B model runs at 32 tok/s on an M3 Pro but 14 tok/s on an Intel i7-13700H with integrated graphics. Performance varies by memory bandwidth, cache hierarchy, and backend optimization. Profile target hardware before setting SLAs.

6. Skipping Quantization Validation 4-bit quantization is not lossless. Code generation and mathematical reasoning degrade faster than creative text. Run automated benchmarks on task-specific datasets. If perplexity drifts >5%, switch to Q5_K_M or Q8_0 for critical paths.

7. No Fallback Strategy for Heavy Tasks On-device models excel at routing, summarization, and structured extraction. They struggle with multi-step reasoning, long-horizon planning, and domain-specific knowledge. Route complex tasks to cloud APIs using a circuit breaker pattern. Never force edge hardware to handle workloads outside its capability envelope.

Production Best Practices

• Profile with llama-bench across target hardware matrices before deployment.
• Implement token budgeting: cap maxTokens per request to prevent runaway generation.
• Cache frequent prompts using semantic hashing to skip redundant KV computation.
• Monitor temperature and throttling events; log them alongside generation metrics.
• Use structured output (JSON schema enforcement) to reduce token waste and improve parse reliability.

Production Bundle

Action Checklist

• Quantize target model to Q4_K_M and validate perplexity delta <5% on domain-specific samples
• Enable memory mapping (mmap: true) and verify RSS stays under 60% of available RAM
• Configure hardware offload (gpuLayers: -1) with CPU fallback path tested
• Implement sliding context window with semantic pruning or importance scoring
• Add streaming token handler with cancellation support and UI progress hooks
• Set strict maxTokens and repeatPenalty to prevent runaway generation
• Deploy circuit breaker: route >3-step reasoning or compliance-sensitive prompts to cloud API
• Profile thermal behavior on target hardware and implement duty cycling if throttling exceeds 20%

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Real-time chat UX (<50ms TTFB)	Optimized Q4_K_M on-device	Eliminates network round-trip, guarantees responsive streaming	Near-zero marginal cost
Multi-step reasoning or code generation	Cloud API with structured prompting	Edge quantization degrades logical consistency; cloud maintains FP16/FP8 quality	$3–$8 per 1M tokens
Offline-first mobile app	Q4_K_M + Metal backend	Guarantees functionality without connectivity, respects app size limits	One-time model shipping cost
Compliance-heavy data (HIPAA/GDPR)	On-device with encrypted KV cache	Data never leaves device, simplifies audit trails	Infrastructure cost shifts to hardware procurement
High-throughput batch processing	Cloud GPU cluster with batching	Parallelizes thousands of requests; edge hardware lacks queue management	Economies of scale at volume

Configuration Template

// config/inference.ts
export const INFERENCE_CONFIG = {
  modelPath: process.env.MODEL_PATH || './models/model-q4_k_m.gguf',
  contextSize: parseInt(process.env.CONTEXT_SIZE || '4096', 10),
  gpuLayers: process.env.USE_GPU === 'true' ? -1 : 0,
  threads: parseInt(process.env.WORKER_THREADS || '8', 10),
  mmap: true,
  maxTokens: parseInt(process.env.MAX_TOKENS || '1024', 10),
  temperature: parseFloat(process.env.TEMPERATURE || '0.7'),
  topP: parseFloat(process.env.TOP_P || '0.9'),
  repeatPenalty: parseFloat(process.env.REPEAT_PENALTY || '1.1'),
  pruningStrategy: 'sliding_window' | 'importance_score' | 'none',
  fallbackToCloud: process.env.ENABLE_CLOUD_FALLBACK === 'true',
  cloudFallbackEndpoint: process.env.CLOUD_API_URL || '',
  cloudFallbackApiKey: process.env.CLOUD_API_KEY || '',
};

Quick Start Guide

1. Download and quantize: Pull a 7B GGUF model from Hugging Face or quantize your own using llama.cpp tools. Place it in ./models/.
2. Install runtime: Run npm install llama-cpp-node. Ensure your OS has the required Metal/CUDA drivers installed.
3. Initialize engine: Import the configuration template, call createInferenceEngine(), and verify model load completes without OOM.
4. Test streaming: Pass a prompt to generateResponse() with a console logger. Confirm TTFB <50ms and stable token throughput.
5. Add context management: Wrap the session in ContextManager, enforce token budgets, and deploy a circuit breaker for fallback routing.