.numKVHeads * config.headDimension * batchSize * maxSequenceLength * bytesPerParam; const kvCacheGB = kvCacheBytes / (1024 ** 3);

const activationOverheadGB = weightsGB * 0.25;

const totalRequiredGB = (weightsGB + kvCacheGB + activationOverheadGB) * safetyMargin;

const recommendedHBM = Math.ceil(totalRequiredGB / 80) * 80;

return { weightsGB: Math.round(weightsGB * 100) / 100, kvCacheGB: Math.round(kvCacheGB * 100) / 100, activationOverheadGB: Math.round(activationOverheadGB * 100) / 100, totalRequiredGB: Math.round(totalRequiredGB * 100) / 100, recommendedHBM }; }

This calculator accounts for weight storage, KV-cache expansion, activation overhead, and a configurable safety margin. Production deployments should never provision at 100% memory utilization. The 15% margin prevents OOM crashes during context window spikes and allows the runtime scheduler to maintain contiguous memory blocks.

### Step 3: Hardware Selection and Interconnect Architecture

Once the memory budget is established, match it to physical hardware constraints. If `totalRequiredGB` exceeds a single accelerator's HBM, you must implement tensor parallelism or pipeline parallelism. This introduces interconnect latency and synchronization overhead.

- **Single-node deployment**: Ideal when memory fits within 80-192 GB per device. Use NVLink or PCIe 5.0 x16 for intra-node communication. Latency remains predictable.
- **Multi-node scale-out**: Required for models exceeding 200B parameters or long-context workloads (>32K tokens). Prioritize hardware with high-bandwidth chip-to-chip links (NVLink 5.0, Infinity Fabric, or Ethernet RoCEv2). Google TPUs and Intel Gaudi 3 excel here due to native mesh topologies.

### Step 4: Runtime Configuration and Batching Strategy

Hardware selection is meaningless without a matching inference runtime. Modern serving engines (vLLM, TensorRT-LLM, TGI) implement continuous batching, PagedAttention, and speculative decoding. Configure these based on your workload:

- Interactive: Enable continuous batching with a max batch size of 32-64. Use speculative decoding to reduce TTFT.
- Batch: Disable speculative decoding. Increase max batch size to 128-256. Enable prefix caching for repeated prompts.

The architecture decision hinges on one principle: minimize memory fragmentation and maximize compute utilization. Choose hardware that aligns with your runtime's native optimization paths.

## Pitfall Guide

### 1. Optimizing for Average Latency Instead of Tail Latency
**Explanation**: Teams monitor mean TTFT and assume system health. In production, p99 and p999 latency dictate user retention. A single slow request can block scheduler queues, causing cascading delays.
**Fix**: Implement p99/p999 alerting. Configure runtime schedulers with strict timeout thresholds. Use request prioritization to isolate interactive traffic from batch jobs.

### 2. Ignoring KV-Cache Growth Under Variable Context Lengths
**Explanation**: Memory estimators often assume fixed sequence lengths. Real traffic exhibits heavy-tailed distributions. A 10% spike in average context length can double KV-cache pressure, triggering OOM kills.
**Fix**: Implement dynamic context window limits. Use sliding window attention or KV-cache eviction policies. Monitor memory utilization with rolling averages, not point-in-time snapshots.

### 3. Over-Sharding for Marginal Performance Gains
**Explanation**: Engineers split models across 4-8 GPUs to chase higher throughput. The interconnect synchronization overhead often negates compute gains, especially for models under 70B parameters.
**Fix**: Benchmark single-node vs. multi-node with your actual prompt distribution. Only shard when memory budget exceeds single-device capacity or when throughput targets cannot be met locally.

### 4. Neglecting Quantization Calibration Overhead
**Explanation**: Switching from FP16 to INT8/FP4 reduces memory footprint but introduces calibration steps and potential accuracy degradation. Unvalidated quantization causes silent quality drops in production.
**Fix**: Run automated evaluation suites (MMLU, GSM8K, custom domain benchmarks) post-quantization. Use per-token or per-channel quantization instead of per-tensor for better accuracy retention. Validate with shadow traffic before full rollout.

### 5. Chasing Peak TFLOPS Without Scheduler Alignment
**Explanation**: High compute density is useless if the inference runtime cannot keep pipelines full. Poor batch scheduling, inefficient memory allocation, or framework bottlenecks leave silicon idle.
**Fix**: Profile runtime utilization with tools like `nsys`, `rocm-smi`, or Habana Profiler. Tune batch size, prefill chunking, and token generation limits to maintain >70% compute utilization.

### 6. Underestimating Tooling and Migration Costs
**Explanation**: Selecting hardware based on raw specs while ignoring SDK maturity leads to weeks of debugging, custom kernel writing, and framework porting. Engineer time often outweighs hardware savings.
**Fix**: Factor in onboarding cost. If your team knows CUDA/vLLM, NVIDIA or AMD ROCm will deploy faster. If you're building a greenfield batch pipeline, TPUs or Inferentia2 may offer better ROI despite steeper initial learning curves.

### 7. Treating All Inference Workloads as Homogeneous
**Explanation**: Routing chat, code generation, and document summarization through the same hardware pool causes resource contention. Interactive requests starve when batch jobs consume memory and scheduler slots.
**Fix**: Implement workload-aware routing. Use separate node pools or GPU partitions for interactive vs. batch traffic. Apply quality-of-service (QoS) policies at the load balancer and runtime level.

## Production Bundle

### Action Checklist
- [ ] Classify workloads: Separate interactive, batch, and hybrid traffic patterns before hardware evaluation.
- [ ] Calculate memory budgets: Run the TypeScript estimator with max sequence length, batch size, and quantization target.
- [ ] Validate p99 latency: Benchmark tail latency under concurrent load, not just average throughput.
- [ ] Test quantization impact: Run domain-specific evaluation suites after weight compression.
- [ ] Profile scheduler utilization: Ensure runtime keeps compute pipelines >70% saturated.
- [ ] Implement workload routing: Isolate interactive and batch traffic at the infrastructure layer.
- [ ] Monitor KV-cache fragmentation: Set alerts for memory utilization spikes and OOM events.
- [ ] Document migration paths: Maintain fallback configurations if target hardware faces supply constraints.

### Decision Matrix

| Scenario | Recommended Approach | Why | Cost Impact |
|----------|---------------------|-----|-------------|
| Real-time chat assistant (<100ms TTFT) | NVIDIA H200/B200 or AMD MI300X | Mature schedulers, low p99 latency, extensive framework support | Higher upfront cost, lower engineering overhead |
| Large-context document processing (32K-128K tokens) | AMD MI300X or Intel Gaudi 3 | Superior HBM capacity and bandwidth, cost-effective scaling | Moderate cost, requires KV-cache tuning |
| High-volume batch summarization | AWS Inferentia2 or Google TPUs | Predictable cost-per-token, optimized for throughput over latency | Lowest operational cost, higher migration effort |
| Multi-node MoE/Reasoning scale-out | Google TPUs or NVIDIA B200 | High-bandwidth chip-to-chip interconnects, native MoE support | High infrastructure cost, requires distributed training/inference expertise |
| Budget-constrained startup MVP | AMD MI300X or NVIDIA L40S | Balanced performance/memory, accessible cloud availability | Moderate cost, faster time-to-production |

### Configuration Template

Below is a production-ready vLLM deployment configuration optimized for mixed interactive/batch workloads. Adjust parameters based on your memory budget and hardware tier.

```yaml
# vLLM Production Deployment Config
model: "meta-llama/Llama-3.1-70B-Instruct"
tensor_parallel_size: 2
max_model_len: 32768
max_num_seqs: 64
max_num_batched_tokens: 16384
gpu_memory_utilization: 0.85
quantization: "fp8"
enforce_eager: false
disable_log_stats: false
enable_prefix_caching: true
speculative_config:
  model: "nvidia/Llama-3.1-8B-Instruct"
  num_speculative_tokens: 4
  speculative_draft_tensor_parallel_size: 1

# Scheduler Tuning
chunked_prefill_enabled: true
max_num_batched_tokens_prefill: 8192
preemption_mode: "swap"

# Monitoring & Telemetry
enable_metrics: true
metrics_port: 8000
log_level: "INFO"

Quick Start Guide

1. Profile your traffic: Export request logs for 7 days. Calculate average/max sequence length, concurrency peaks, and latency tolerance.
2. Run the memory estimator: Input your model configuration and traffic profile into the TypeScript calculator. Note the recommendedHBM value.
3. Select hardware tier: Match your memory budget and latency requirements to the Decision Matrix. Provision a single-node test instance.
4. Deploy with baseline config: Use the YAML template above. Adjust tensor_parallel_size, max_model_len, and quantization to match your hardware.
5. Benchmark and iterate: Run load tests with locust or k6. Monitor p99 latency, GPU utilization, and memory fragmentation. Tune batch sizes and scheduler parameters until targets are met.