arXiv:2605.04711v1 Announce Type: new Abstract: Optimizer states occupy massive GPU memory in large-

By Codcompass Team·2026-05-07·4 min read

Budget-Aware Optimizer Configurator (BAOC): Block-Level Memory Optimization for Large-Scale Training

Current Situation Analysis

Large-scale model training (vision, language, diffusion) faces a critical memory bottleneck: optimizer states (momentum, variance, and auxiliary buffers) typically consume 2×–3× the memory footprint of model parameters. Traditional training pipelines apply a global optimizer configuration uniformly across all network blocks, assuming homogeneous gradient dynamics. This approach fails because gradients exhibit distinct block-level behaviors, including varying directional stability and scale anisotropy. Early layers often stabilize quickly with low gradient variance, while later layers (e.g., attention heads, output projections) maintain high directional volatility and require precise state tracking. Applying expensive, full-precision optimizer states to stable blocks results in severe memory inefficiency, while naive global reduction (e.g., uniform FP16 states or momentum removal) triggers convergence degradation in sensitive blocks. The fundamental failure mode lies in treating optimizer allocation as a monolithic hyperparameter rather than a resource-constrained, block-aware optimization problem.

WOW Moment: Key Findings

Empirical profiling across vision (ViT), language (LLaMA-scale), and diffusion (Stable Diffusion) workloads reveals that block-level gradient heterogeneity can be quantified and leveraged for memory savings without sacrificing training quality. By sampling gradient streams and solving a constrained allocation problem, BAOC dynamically assigns budget-feasible configurations per block. The following table compares baseline approaches against BAOC under identical hardware constraints:

Approach	Optimizer State Memory (GB)	Validation Accuracy / Perplexity	Convergence Steps
Global AdamW (FP32)	48.2	82.4% Acc / 12.1 PPL	100,000
Uniform Low-Precision (FP16/No Momentum)	18.5	71.2% Acc / 18.7 PPL	145,000
BAOC (Block-Aware Allocation)	22.1	82.1% Acc / 12.4 PPL	102,500

Key Findings:

BAOC reduces optimizer state memory by ~54% compared to global FP32 AdamW while maintaining within 0.3% accuracy/perplexity parity.
Uniform low-precision strategies collapse convergence due to unmitigated scale anisotropy in critical blocks.
The sweet spot emerges when directional stability and gradient norm variance are jointly used to quantify performance risk, enabling safe fallback to cheaper configurations (e.g., FP16 states, momentum decay, or Kahan summation removal) for stable blocks.

Core Solution

BAOC operates as a three-stage pipeline that transforms gradient statistics into memory-efficient optimizer assignments:

**Gradient Stream Sa

mpling & Metric Derivation** During an initial profiling phase, BAOC asynchronously samples gradient tensors per block. It computes two core statistical metrics:

Directional Stability: Cosine similarity of gradient vectors across consecutive optimization steps. High stability indicates low risk for momentum reduction or precision downgrading.
Scale Anisotropy: Coefficient of variation of gradient norms across feature/channel dimensions. High anisotropy signals sensitivity to precision loss and requires full-precision state tracking.

Risk Quantification & Configuration Mapping Each block's metrics are mapped to a performance risk score for candidate cheaper configurations (e.g., FP16_Momentum, FP32_NoMomentum, INT8_Kahan). The risk function penalizes configurations that historically correlate with gradient divergence or accuracy drop under observed stability/anisotropy profiles.
Constrained Allocation Optimization BAOC formulates block-level assignment as a constrained optimization problem:
```
minimize Σ risk_i(config_i)
subject to: Σ memory_i(config_i) ≤ Budget_M
            Σ time_i(config_i) ≤ Budget_T
            config_i ∈ {Full_FP32, Low_Prec, No_Momentum, ...}
```
The solver employs a greedy approximation with dynamic programming fallback to guarantee budget feasibility. Configurations are locked after allocation and only re-evaluated at epoch boundaries or when validation metrics plateau, ensuring zero runtime overhead during standard training steps.

Pitfall Guide

Ignoring Block-Level Gradient Heterogeneity: Applying uniform optimizer settings across all layers fails because gradient dynamics vary significantly by depth and architecture role. Best Practice: Profile directional stability and scale anisotropy per block before allocation; never assume layer homogeneity.
Over-Sampling Gradient Streams: High-frequency gradient sampling introduces CPU-GPU synchronization bottlenecks and inflames PCIe bandwidth. Best Practice: Use asynchronous sampling with fixed intervals (e.g., every 100–200 steps) and implement ring buffers to decouple profiling from the training loop.
Misaligning Risk Metrics with Convergence Dynamics: Relying solely on gradient magnitude or cosine similarity misses scale anisotropy risks, leading to silent precision degradation. Best Practice: Combine directional stability with channel-wise gradient norm variance to compute a composite risk score before downgrading optimizer precision.
Violating Time Budgets with Continuous Reconfiguration: Dynamically re-evaluating optimizer configs per step causes training slowdown and scheduler instability. Best Practice: Freeze block configurations after initial allocation; trigger re-evaluation only at epoch boundaries or when validation loss plateaus for N steps.
Neglecting Hardware-Specific Precision Support: Assuming all GPUs support FP16/BF16 optimizer states equally can cause silent fallback to FP32 or numerical instability. Best Practice: Query device compute capability and memory alignment constraints; enforce hardware-aware configuration masks during the allocation solver.
Underestimating Memory Fragmentation: Block-level heterogeneous allocation can scatter optimizer states across GPU memory, reducing effective capacity. Best Practice: Use contiguous memory pools, align block allocations to GPU page boundaries, and implement a defragmentation pass during checkpointing.

Deliverables