Normalized Architectures are Natively 4-Bit

Abstract: Training LLMs at 4-bit precision is critical for efficiency. We show that nGPT, an architecture that constrains weights and hidden representations to the unit hypersphere, is inherently more robust to low-precision arithmetic. This removes the need for interventions-such as applying random Hadamard transforms and performing per-tensor scaling calculations-to preserve model quality, and it enables stable end-to-end NVFP4 training. We validate this approach on both a 1.2B dense model and hybrid (Mamba-Transformer) MoE models of up to 3B/30B parameters. We trace this robustness to the dot product: while quantization noise remains largely uncorrelated in both standard and normalized architectures, the signal behaves differently. In nGPT, the hypersphere constraint enhances weak positive correlations among the element-wise products, leading to a constructive accumulation of the signal across the hidden dimension while the noise continues to average out. This yields a higher effective signal-to-noise ratio and a flatter loss landscape, with the effect strengthening as the hidden dimension grows, suggesting increasing advantages at scale. A reference implementation is available at https://github.com/anonymous452026/ngpt-nvfp4

• The paper demonstrates that enforcing hyperspherical normalization allows nGPT to achieve native 4-bit quantization without algorithmic interventions.
• The study finds that nGPT exhibits a consistent ~7 dB SNR advantage through constructive signal accumulation compared to standard GPT.
• Empirical evaluations reveal that nGPT scales favorably with model width, yielding robust performance and up to 3.6× speedup on NVFP4 GPUs.

Structural Quantization Robustness of Normalized Transformers

Architectural Basis for 4-Bit Robustness

The paper "Normalized Architectures are Natively 4-Bit" (2605.06067) presents a rigorous investigation into the quantization robustness of transformer architectures constrained to the unit hypersphere—specifically nGPT. The work confronts the persistent challenge of unstable 4-bit training for LLMs, which in standard transformers requires heavy algorithmic interventions such as randomized Hadamard transforms (RHT), dynamic per-tensor scaling, and stochastic rounding to mitigate divergence and quality loss under the NVFP4 format. The central finding is that nGPT, by enforcing normalization on weights and activations, achieves robust 4-bit precision natively, fundamentally bypassing these burdensome procedures.

The hyperspherical constraint in nGPT eliminates the architectural reliance on high-magnitude outlier coordinates—making constructive alignment across thousands of dimensions an intrinsic property. This normalization compels the optimizer to learn distributed representations which facilitate coherent signal accumulation during matrix multiplications; the outcome is a high effective dot product SNR even under severe quantization.

Mechanistic Analysis: Signal Accumulation vs. Noise Suppression

The authors distinguish between two mechanisms: noise suppression and signal accumulation. Empirical measurements in a 3.6B-parameter nGPT (12-layer, $D=4096$) show the SNR before summation (on weights, activations, and element-wise products) is identical for GPT and nGPT, but after summation, nGPT attains a consistent ~7 dB advantage versus GPT. This improvement emerges not from reduced quantization noise, but from constructive signal summation enabled by weak but pervasive positive pairwise correlations among coordinate-wise products—a phenomenon only present in nGPT.

Figure 1: Normalized signal $z_s$ strongly relates to SNR, distinguishing nGPT from GPT; normalized noise $\tilde{z}_n$ is similar and non-predictive across architectures.

By decomposing the dot product into normalized signal and noise, the SNR advantage is attributed solely to the signal term $z_s$; the noise term remains statistically indistinguishable. This coherence of signal accumulation—a product of hypersphere-induced correlation—enables nGPT to maintain high SNR throughout all transformer layers.

Figure 2: nGPT induces greater mean-centered signal correlation ($\bar{\rho}_s$) in quantized dot-product elements compared to GPT, driving higher SNR.

Theoretical analysis formalizes the SNR scaling: the positive pairwise correlation $\bar{\rho}_s$ is a multiplicative factor for SNR, and this architectural bias compounds with depth and width.

Scaling Laws: SNR Advantage with Model Width

The structural SNR advantage is not static; theoretical and empirical evidence indicates that it scales favorably with hidden dimension $D$. The ratio of SNR between nGPT and GPT grows linearly with $D$ until saturating at a plateau determined by the correlation ratio. For current models ($D=4096$), this sits in the linear regime; for massive LLMs ($D \gg 10^4$), nGPT's SNR advantage becomes increasingly pronounced.

Figure 3: nGPT's SNR tracks theory curves with increasing width, outperforming GPT with scaling regimes marked.

This scaling law implies that hypersphere normalization will be even more advantageous as model sizes continue their exponential growth trajectory.

Loss Landscape: Quantization and Flatness

Higher per-layer SNR translates to a flatter loss landscape under quantization perturbations. nGPT's architectural structure tolerates simultaneous weight perturbations significantly better than GPT—GPT's loss degrades 3.5$z_s$0 faster in direct noise injection experiments. This flatness persists across layers and is measurable in both empirical loss and SNR.

Figure 4: nGPT achieves higher per-layer gain (dB) in SNR, directly suppressing quantization impact; GPT loss is more sensitive to perturbations.

Empirical Evaluations: Dense and Hybrid Architectures

Extensive experiments validate the architectural robustness across model classes:

• 1.2B Dense Model: Trained over 1T tokens, nGPT in NVFP4 achieves lower relative error and superior downstream task performance—even as it omits RHT and per-tensor scaling.
• Hybrid Mamba-Transformer MoE (400M/600M, 3B/30B): nGPT maintains stability and achieves $z_s$10% relative error without quantization interventions, outperforming standard GPT under NVFP4. The architectural benefits transfer to massive MoE configurations.

  Figure 5: nGPT-NVFP4 eliminates loss gap and overhead in 1.2B dense; achieves sim0% relative error in 3B/30B hybrid MoE without quantization interventions.

Additionally, learning rate sensitivity experiments demonstrate nGPT's remarkable hyperparameter insensitivity under quantization: nGPT's optima in BF16 and NVFP4 are identical and broadly flat.

Practical Acceleration

The architectural elimination of per-tensor scaling and RHT not only enhances model robustness but simplifies the runtime path. Benchmarks on NVIDIA GB200 GPUs show that the nGPT NVFP4 configuration achieves up to $z_s$2 speedup relative to BF16 GPT in large hidden dimensions, as the fixed-scale, normalized structure unlocks highly-optimized NVFP4 tensor cores.

Figure 6: Layer-wise speedup on GB200 grows with model width; nGPT NVFP4 leverages normalized activations for maximal efficiency.

Implications and Future Directions

The findings imply a shift in the approach to 4-bit and lower quantization: architectural normalization is a primary determinant of robustness, superseding algorithmic post-hoc interventions. As LLMs scale both in parameter count and computational deployment, nGPT's hypersphere structural bias may enable aggressive reduction in precision without sacrificing model quality or requiring complex runtime mitigations.

Theoretical implications include a link between hypersphere normalization and optimization geometry, corroborated by literature on weight normalization favoring flatter minima and enhanced generalization. Future research directions include investigating the origins and scaling behavior of positive signal correlation in normalized architectures, its impact on transferability and generalization, and extension to more diverse model families.

Conclusion

This work formally establishes that nGPT—by constraining weights and activations to the unit hypersphere—achieves native robustness to 4-bit quantization. The analysis elucidates that this robustness is due not to noise suppression but to constructive signal accumulation, with empirical and theoretical validation across dense and MoE transformer configurations. The scaling laws favor nGPT as model width increases, and practical experiments demonstrate both reduced training overhead and substantial acceleration. The implications for ultra-low precision training are profound: normalization should be seen as an architectural cornerstone rather than an auxiliary algorithmic tool, and future LLM designs would benefit from prioritizing structural quantization readiness to enable efficient scaling and deployment.