ve pipelines.

• IQ3_M and below: Aggressive compression. Necessary for fitting 70B models on 16GB GPUs, but introduces noticeable degradation in reasoning and instruction following.

Implementation Examples

Custom Inference Router (Python) This wrapper abstracts backend differences and routes requests based on workload type.

import asyncio
import httpx
from typing import Optional

class LocalInferenceRouter:
    def __init__(self, cpu_endpoint: str, gpu_endpoint: str):
        self.cpu_client = httpx.AsyncClient(base_url=cpu_endpoint)
        self.gpu_client = httpx.AsyncClient(base_url=gpu_endpoint)

    async def generate(self, prompt: str, model: str, priority: str = "interactive") -> str:
        client = self.gpu_client if priority == "interactive" else self.cpu_client
        payload = {
            "model": model,
            "prompt": prompt,
            "stream": False,
            "temperature": 0.7,
            "max_tokens": 1024
        }
        response = await client.post("/v1/completions", json=payload)
        response.raise_for_status()
        return response.json()["choices"][0]["text"].strip()

    async def close(self):
        await self.cpu_client.aclose()
        await self.gpu_client.aclose()

vLLM Async Server Deployment (Python) Configures continuous batching and memory optimization for concurrent serving.

from vllm import AsyncLLMEngine, AsyncEngineArgs, SamplingParams

def initialize_vllm_server(model_path: str, gpu_memory_utilization: float = 0.85):
    engine_args = AsyncEngineArgs(
        model=model_path,
        gpu_memory_utilization=gpu_memory_utilization,
        max_num_batched_tokens=4096,
        max_num_seqs=256,
        disable_log_requests=True
    )
    engine = AsyncLLMEngine.from_engine_args(engine_args)
    return engine

async def run_inference(engine, prompt: str, max_tokens: int = 512):
    sampling_params = SamplingParams(
        temperature=0.7,
        top_p=0.9,
        max_tokens=max_tokens,
        stop=["<|end|>", "\n\n"]
    )
    outputs = engine.generate(prompt, sampling_params, request_id="req_001")
    final_output = None
    async for output in outputs:
        final_output = output
    return final_output.outputs[0].text if final_output else ""

MLX-LM Generation Script (Python) Optimized for Apple Silicon unified memory architecture.

import mlx.core as mx
from mlx_lm import load, generate

def run_mlx_inference(model_name: str, system_prompt: str, user_input: str):
    model, tokenizer = load(model_name)
    messages = [
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": user_input}
    ]
    prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    output = generate(
        model=model,
        tokenizer=tokenizer,
        prompt=prompt,
        max_tokens=1024,
        temperature=0.7,
        top_p=0.9,
        verbose=False
    )
    return output.strip()

Architectural Rationale

• Routing by priority prevents interactive requests from queuing behind batch jobs.
• vLLM's max_num_batched_tokens and max_num_seqs parameters are tuned to prevent OOM crashes while maximizing GPU utilization.
• MLX-LM leverages Apple's memory hierarchy by avoiding explicit tensor transfers between CPU and GPU domains.
• Quantization defaults are enforced at the model loading stage to prevent accidental FP16 allocation.

Pitfall Guide

1. Ignoring NUMA Topology on Multi-Socket CPUs

   • Explanation: Modern workstations often span multiple NUMA nodes. If the inference process allocates memory on a different node than the CPU cores executing it, latency spikes and throughput drops by 30–50%.
   • Fix: Bind the process to a specific NUMA node using numactl --cpunodebind=0 --membind=0 ollama serve or equivalent systemd CPU affinity settings.

2. Over-Quantizing Reasoning Workloads

   • Explanation: Aggressive quantization (IQ3_M and below) degrades the model's ability to maintain coherent chain-of-thought. The weight distribution critical for logical steps gets flattened.
   • Fix: Reserve Q4_K_M or Q5_K_M for reasoning pipelines. Use IQ4_XS if memory is constrained but quality cannot be compromised.

3. Mismanaging KV Cache Memory

   • Explanation: The key-value cache scales linearly with context length. Long conversations or document ingestion can silently exhaust VRAM/RAM, causing silent failures or fallback to CPU swapping.
   • Fix: Implement context window limits at the application layer. Use sliding window attention or cache eviction strategies for long-running sessions.

4. Treating MoE Active Parameters as Total Parameters

   • Explanation: Models like Qwen 3 235B-A22B have 235B total parameters but only 22B active per token. Engineers often miscalculate memory requirements by assuming full model loading.
   • Fix: Size hardware based on active parameters plus routing overhead. A 22B active MoE fits comfortably on a 24GB GPU when quantized, despite the 235B label.

5. Deploying Single-Threaded Servers for Concurrent Users

   • Explanation: Ollama and basic llama.cpp servers handle one request at a time. Under concurrent load, requests queue linearly, destroying perceived performance.
   • Fix: Migrate to vLLM or LocalAI with continuous batching. Configure max_num_seqs to match expected concurrency, and monitor queue depth.

6. Chasing Context Length Over Throughput

   • Explanation: Extending context windows beyond 32k tokens increases KV cache size quadratically in some implementations, reducing tokens/sec by 40–60%.
   • Fix: Use retrieval-augmented generation (RAG) or chunking strategies instead of raw context extension. Keep local inference context windows between 8k–32k for optimal throughput.

7. Neglecting Sampler Configuration for Deterministic Outputs

   • Explanation: Default sampling parameters introduce variance in code generation and data extraction tasks. Temperature > 0.5 causes inconsistent formatting.
   • Fix: Set temperature=0.0 or 0.1 for deterministic pipelines. Use top_p=0.9 and repetition_penalty=1.1 to maintain coherence without sacrificing output stability.

Production Bundle

Action Checklist

• Audit hardware NUMA topology and bind inference processes to matching memory nodes
• Standardize on Q4_K_M quantization unless specific quality or memory constraints dictate otherwise
• Implement application-level context window limits to prevent KV cache exhaustion
• Route interactive requests to GPU nodes and batch jobs to CPU nodes via an inference router
• Configure vLLM continuous batching parameters (max_num_batched_tokens, max_num_seqs) to match concurrency targets
• Set deterministic sampling parameters (temperature ≤ 0.1) for code extraction and data transformation pipelines
• Monitor VRAM/RAM utilization during peak load and implement graceful degradation or request queuing
• Validate model tool-calling capabilities against actual API schemas before production deployment

Decision Matrix

Configuration Template

# docker-compose.yml - Local Inference Stack
version: "3.8"

services:
  vllm-gpu:
    image: vllm/vllm-openai:latest
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - VLLM_USE_V1=1
    command: >
      --model qwen/Qwen3-14B-Instruct
      --quantization awq
      --gpu-memory-utilization 0.85
      --max-num-batched-tokens 4096
      --max-num-seqs 128
      --disable-log-requests
    ports:
      - "8000:8000"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

  ollama-cpu:
    image: ollama/ollama:latest
    environment:
      - OLLAMA_NUM_GPU=0
      - OLLAMA_HOST=0.0.0.0
    command: serve
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama

  inference-router:
    build: ./router
    environment:
      - GPU_ENDPOINT=http://vllm-gpu:8000
      - CPU_ENDPOINT=http://ollama-cpu:11434
    ports:
      - "3000:3000"
    depends_on:
      - vllm-gpu
      - ollama-cpu

volumes:
  ollama_data:

Quick Start Guide

1. Install the serving runtime: Deploy Ollama for CPU fallback and vLLM for GPU acceleration using the provided Docker Compose template. Ensure NVIDIA container toolkit is installed on GPU nodes.
2. Pull and verify the model: Execute ollama pull qwen3:14b on the CPU node. On the GPU node, vLLM will automatically download and quantize the model on first request. Verify throughput using curl http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{"model":"qwen/Qwen3-14B-Instruct","prompt":"Test","max_tokens":10}'.
3. Configure the inference router: Build and start the routing service. Set environment variables to point to your GPU and CPU endpoints. Test priority routing by sending concurrent requests with priority: interactive and priority: batch.
4. Integrate with your application: Replace cloud API calls with the router endpoint (http://localhost:3000/v1/completions). Implement context window limits and deterministic sampling parameters in your request payload. Monitor latency and adjust max_num_seqs based on observed concurrency.