Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention

Abstract: Recent NVFP4 pretraining methods mainly target transformer linear layers, leaving optimizer states, optimizer arithmetic and attention underexplored in 4-bit pipelines. This critical gap blocks stable full-stack 4-bit pretraining, as the three core modules exhibit unique numerical failure patterns: linear layers hit hard quantization noise limits with dimension-propagated error amplification; AdamW second moments are heavy-tailed non-negative values fragile to low-precision denominators; attention carries error-prone computation paths demanding strict forward-backward quantization consistency. We propose Full-Stack FP4, the first complete NVFP4 pretraining framework resolving all three stability bottlenecks via module-wise precision strategies. For linear projections, LoRA-SVD lightweight decomposition suppresses quantization noise and breaks the direct-quantization error ceiling, shrinking the linear-only loss gap from 1.40% to 0.61%. For optimizers, we design AdamW second-moment transformation for robust NVFP4 storage and fully support native NVFP4 Newton-Schulz iterations for the Root (Muon) optimizer. For attention, a mixed-precision scheme quantizes Q/K/V and backward dS while guarding vulnerable paths in BF16, paired with unified tensor reuse to sustain forward-backward alignment. We further analyze fast error accumulation in naive low-bit matrix multiplication and the extreme sensitivity of PV / dOV^T attention branches. All modules are plug-and-play with cumulative stability and efficiency improvements. Our 3B/64B-token pretraining validates near-BF16 performance with merely 1.47% loss gap, verifying feasible stable end-to-end NVFP4 LLM pretraining.

• The paper introduces a full-stack FP4 pretraining pipeline that reduces the convergence gap to 1.47% compared to BF16 by addressing module-specific quantization challenges.
• The paper employs a novel LoRA-SVD method for linear layers and a tailored optimizer-state quantization pipeline that collectively reduce quantization error by up to 60%.
• The paper demonstrates a robust mixed-precision attention mechanism that preserves critical computational paths in higher precision, ensuring stable training and smooth loss landscapes.

Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention

Introduction and Context

The drive to reduce computational and memory overhead in LLM pretraining has led to aggressive quantization strategies, with NVFP4 enabling native 4-bit precision on modern Blackwell GPUs. While linear layers have seen substantial progress in FP4 quantization, prior work has left optimizer states, optimizer computations, and attention mechanisms only partially addressed due to distinct numerical pathologies in each module. The paper, "Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention" (2607.04422), presents a modular, end-to-end FP4 training pipeline. This system deploys module-specific precision policies and lightweight structural augmentations that systematically neutralize the pathological failure modes of naive quantization, yielding stable and efficient training at scale.

Figure 1: The Full-Stack FP4 system incorporates module-specific approaches for quantized linear projection (LoRA-SVD), quantized optimizers, and mixed-precision attention within standard transformer architecture.

LoRA-SVD for Linear Projection Quantization

Linear layers in the NVFP4 regime encounter a quantization noise ceiling, exacerbated by error growth linear in hidden dimension, which increasingly overwhelms the optimization signal. Existing quantization improvements (block scaling, stochastic rounding, Hadamard rotations) approach but do not fundamentally resolve this limit. The paper addresses this with LoRA-SVD: the full projection is decomposed into a low-rank principal subspace (retained in BF16 via LoRA-like SVD) and a residual projection handled in NVFP4.

This dual-branch design acts as a structural low-pass filter, systematically mitigating quantization noise in the dominant eigenspace while keeping overhead minimal. Explicit subspace constraints (via fast Cholesky-QR) maintain branch alignment over time, with periodic global SVD-based realignment.

Figure 2: LoRA-SVD consistently reduces quantization error by approximately 10% for outputs, input gradients, and weight gradients relative to naive NVFP4, and this effect is maintained throughout training.

This approach narrows the linear-only loss gap (relative to BF16) from 1.40% (vanilla NVFP4 4/6) to 0.61%. Notably, the quantization MSE increases superlinearly with network width in direct NVFP4, confirming the necessity of a structural, not merely numerical, approach.

Optimizer-State Quantization: AdamW and Root (Muon)

The optimizer pathway introduces additional instability: AdamW's second moment ($v_t$) is non-negative, heavy-tailed, and acts as a denominator, amplifying error variance after direct quantization. The paper introduces a 4-step pipeline for quantizing $v_t$: square root transformation, mean subtraction (tile-wise), Hadamard energy spreading, and NVFP4 quantization, with exactly invertible reconstruction.

The result is a 60% reduction in quantization error for the critical second moment, as shown empirically. All momentum vectors are handled with stochastic rounding and block scaling, matching the LoRA-SVD pathway. For the Root optimizer (generalized Muon), the system enables natively-quantized Newton–Schulz iterations with shape-optimized coefficients and full NVFP4 persistence, obviating the need for BF16 auxiliary operations.

Figure 3: The transformed-state pipeline for AdamW significantly curbs second-moment compression error, driving both stable convergence and robust training objectives.

This modular approach is essential for robust, quantization-sustainable optimizer pathways, especially as model scale increases.

Mixed-Precision Quantized Attention

Attention layers require distinct treatment due to the catastrophic instability induced by quantizing certain paths. Specifically, the $PV$ (forward) and $dOV^\top$ (backward) matmuls exhibit extreme numerical sensitivity: quantizing either breaks probability simplex constraints in $P$, and induces block-wise truncation, irreversibly shifting the output and exploding gradient noise.

The proposed method maintains $Q$, $K$, $V$ quantized in both passes, reusing quantized versions throughout for IO savings and forward-backward consistency, while $P$, $dO$, and $v_t$0 are computed in BF16. $v_t$1 is quantized in 1x16 and 16x1 tiles. All matmuls with insensitive operands are aggressively quantized, maximizing IO and memory reduction.

Figure 4: Quantizing $v_t$2 or $v_t$3 yields dramatic accuracy regressions, validating that these paths must be maintained in higher precision for stable optimization.

This scheme preserves the minimum loss gap and enables stable, efficient attention computation across all pretraining steps.

Main Experimental Results

The full-stack system was validated at the 3B parameter scale, pretraining on a 64B-token subset. The full-stack FP4 pipeline—including LoRA-SVD, optimizer-state quantization, and mixed-precision attention—achieves a convergence gap of only 1.47% with respect to the BF16 Root+AdamW baseline. The training-loss trajectories remain tightly aligned over the entire course of pretraining, without divergence, plateau, or collapse.

Figure 5: Loss curves over 64B tokens show the full-stack FP4 system aligns closely with the BF16 baseline (convergence gap 1.47%), validating stable large-scale 4-bit pretraining.

These results provide strong evidence that, contrary to prior expectations, practical, stable FP4 pretraining is tenable across the full stack without need for fallback BF16 blocks.

Ablation and Loss Landscape Analyses

Ablation studies at the 1B scale confirm the strict additivity and necessity of each component. Removal of any single module (e.g., LoRA-SVD or the optimizer quantization pipeline) produces a sharp upward inflection in loss. Additionally, visualization of loss landscapes throughout toy pretraining shows that the full-stack framework retains smooth topology and avoids the rugged surface and collapse observed in naive NVFP4 baselines.

Figure 6: Loss landscapes at key training steps reveal that only the full-stack framework maintains smooth, navigable topography across training, avoiding the collapse characteristic of direct NVFP4.

Theoretical Implications and Future Work

This work demonstrates that targeted, module-aware quantization policies, combined with lightweight structural augmentations, can push NVFP4 quantization beyond the previous limits imposed by noise amplification and numerical sensitivity. The approach is modular, admitting straightforward integration into existing transformer stacks. Noteworthy is the flexibility of LoRA-SVD for dynamic precision switching—early training in pure FP4, late-stage incremental BF16 restoration, and final quantized deployment.

Theoretical analysis confirms the necessity for structural SVD-based denoising in the linear pathway and high-precision preservation in key attention matmuls. These architectures are expected to be further enhanced by hardware-aware kernel fusion, enabling additional efficiency improvements on future Blackwell-class accelerators.

Conclusion

The Full-Stack FP4 framework solves the previously insurmountable stability barriers in 4-bit transformer pretraining. By addressing module-specific numerical pathologies—namely the linear quantization error ceiling, optimizer denominator fragility, and attention path sensitivity—the system closes the convergence gap with full-precision pretraining to less than 1.5% at scale. This modular architecture provides an actionable blueprint for future ultra-low-precision, resource-efficient LLM pretraining and transfer. Extension to larger scales, longer training regimes, and hardware-optimized implementations remains a theoretically important and practically impactful direction for ongoing research.