: 10 metrics:

• type: Pods pods: metric: name: gpu_utilization target: type: AverageValue averageValue: "75"

### Step 4: TypeScript Streaming Client
Implement backpressure-aware streaming with retry logic and token budget enforcement.

```typescript
import { createParser } from 'eventsource-parser';

export async function streamLLMRequest(
  prompt: string,
  maxTokens: number = 512,
  baseUrl: string = 'http://vllm-inference:8000/v1'
): Promise<AsyncIterable<string>> {
  const response = await fetch(`${baseUrl}/chat/completions`, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      model: 'meta-llama/Llama-3.1-8B',
      messages: [{ role: 'user', content: prompt }],
      max_tokens: maxTokens,
      stream: true,
      temperature: 0.7,
    }),
  });

  if (!response.ok) {
    throw new Error(`LLM request failed: ${response.status} ${response.statusText}`);
  }

  if (!response.body) {
    throw new Error('No response body for streaming');
  }

  const parser = createParser((event) => {
    if (event.data === '[DONE]') return;
    try {
      const json = JSON.parse(event.data);
      const content = json.choices[0]?.delta?.content;
      if (content) queue.push(content);
    } catch { /* malformed chunk, skip */ }
  });

  const queue: string[] = [];
  let resolved = false;

  const reader = response.body.getReader();
  const decoder = new TextDecoder();

  async function* generator(): AsyncIterable<string> {
    while (!resolved) {
      if (queue.length > 0) {
        yield queue.shift()!;
        continue;
      }
      const { done, value } = await reader.read();
      if (done) {
        resolved = true;
        break;
      }
      parser.feed(decoder.decode(value, { stream: true }));
    }
    while (queue.length > 0) {
      yield queue.shift()!;
    }
  }

  return generator();
}

// Usage
(async () => {
  const stream = await streamLLMRequest('Explain PagedAttention in 3 sentences.');
  for await (const chunk of stream) {
    process.stdout.write(chunk);
  }
})();

Architecture Decisions and Rationale

• vLLM over TGI/Triton: vLLM’s PagedAttention reduces KV cache fragmentation by 30–50% and provides native continuous batching. TGI requires manual batching configuration; Triton adds complexity without proportional gains for single-model serving.
• Kubernetes HPA with GPU metrics: CPU-based scaling fails for LLMs because GPU utilization dictates throughput. Custom metrics (via DCGM exporter) align scaling with actual inference capacity.
• Streaming-first client design: Token-by-token delivery reduces perceived latency and enables early cancellation. Backpressure handling prevents client memory exhaustion during long generations.
• Request routing layer: Place a lightweight proxy (Envoy or custom Go/TS service) in front of inference pods to handle fallback routing, rate limiting, and token budget enforcement without modifying the inference runtime.

Pitfall Guide

1. Ignoring KV Cache Memory Limits Every token generated expands the KV cache. Without --gpu-memory-utilization caps or sequence limits, VRAM fills linearly and triggers OOM kills. Best practice: enforce max_num_seqs, monitor gpu_cache_usage_pct, and implement request rejection when cache exceeds 85%.

2. Naive Request Batching Grouping requests by arrival time rather than sequence length causes head-of-line blocking. Long prompts delay short completions. Best practice: enable continuous batching (--enable-chunked-prefill) and prioritize requests by estimated token budget.

3. Over-Quantizing Reasoning Models FP8/INT8 quantization compresses weights but degrades multi-step reasoning and code generation accuracy. Best practice: benchmark quantized vs. full precision on domain-specific eval sets before production rollout. Reserve quantization for classification, summarization, and chat.

4. Static GPU Allocation Fixed replica counts waste budget during low traffic and bottleneck during spikes. Best practice: use HPA with GPU utilization and queue depth metrics. Set scale-up thresholds at 70–75% and scale-down at 30% with a 300s stabilization window.

5. Missing Streaming Backpressure Unbounded streaming clients accumulate tokens in memory, causing heap bloat and crashes. Best practice: implement chunk limits, pause/resume logic, and explicit cancellation on client disconnect.

6. No Request Routing or Fallback Single-model deployments fail silently when a provider endpoint degrades or a custom model drifts. Best practice: deploy a routing layer with health checks, token budget validation, and automatic fallback to secondary models or cached responses.

7. Inadequate Observability Tracking only request count and latency misses the root causes of inference failures. Best practice: export gpu_utilization, kv_cache_usage, queue_depth, time_to_first_token, and tokens_per_second. Alert on cache fragmentation >20% and TTFT >800ms.

Production Bundle

Action Checklist

• Benchmark model precision: validate FP16 vs FP8/INT8 on domain-specific tasks before quantization.
• Configure KV cache limits: set --gpu-memory-utilization and --max-num-seqs to prevent OOM.
• Enable continuous batching: activate chunked prefill and dynamic interleaving in the inference server.
• Implement GPU-aware autoscaling: deploy DCGM metrics exporter and configure HPA on utilization thresholds.
• Build streaming client with backpressure: enforce chunk limits, handle disconnects, and track token budgets.
• Deploy request routing layer: add health checks, fallback routing, and rate limiting before inference pods.
• Instrument end-to-end metrics: track TTFT, tokens/sec, cache usage, and queue depth; alert on degradation.
• Run chaos tests: simulate VRAM exhaustion, network partition, and batch starvation to validate resilience.

Decision Matrix

Configuration Template

# vllm-production-values.yaml (Helm-style overrides)
replicaCount: 2
image:
  repository: vllm/vllm-openai
  tag: latest
  pullPolicy: IfNotPresent

args:
  - serve
  - meta-llama/Llama-3.1-8B
  - --gpu-memory-utilization
  - "0.9"
  - --max-num-seqs
  - "256"
  - --enable-chunked-prefill
  - --disable-log-requests

resources:
  limits:
    nvidia.com/gpu: 1
  requests:
    nvidia.com/gpu: 1
    cpu: "4"
    memory: "16Gi"

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetGPUUtilization: 75
  targetQueueDepth: 120

monitoring:
  enabled: true
  exporter: dcgm
  metrics:
    - gpu_utilization
    - kv_cache_usage_pct
    - queue_depth
    - time_to_first_token
    - tokens_per_second

Quick Start Guide

1. Pull and run vLLM locally:

   docker run --runtime=nvidia --gpus all -p 8000:8000 \
     vllm/vllm-openai:latest \
     serve meta-llama/Llama-3.1-8B --gpu-memory-utilization 0.9
   

2. Verify inference endpoint:

   curl http://localhost:8000/v1/models
   

3. Test streaming with TypeScript: Use the streamLLMRequest function from Core Solution. Run with node --experimental-fetch stream-client.ts.

4. Deploy to Kubernetes: Apply the vllm-deployment.yaml and HPA manifest. Ensure nvidia-device-plugin is installed and DCGM exporter is running for custom metrics.

5. Validate scaling: Generate load with k6 or wrk. Monitor gpu_utilization and queue_depth. Confirm HPA scales pods within 60–90 seconds of threshold breach.

Deployment strategy is not a static choice; it is a runtime contract between workload characteristics and infrastructure capacity. Match precision to task, batch continuously, scale on GPU metrics, and instrument everything. The models will scale themselves; your infrastructure must keep pace.