HyperP Transfer Law in Deep Models

• The paper presents the HyperP Transfer Law, offering a closed-form prediction for optimal learning rates based on model depth, token count, and inherent hypersphere invariance.
• It reveals that weight decay becomes a first-order no-op under Frobenius-sphere constraints, ensuring stable and regularized parameter updates without extra tuning.
• Depth-multipliers and the SqrtGate mechanism enhance stability and compute efficiency, achieving R² > 0.99 accuracy and transferring hyperparameters effectively across scales.

The term "HyperP Transfer Law" encompasses a range of rigorous transfer laws and scaling formalisms emerging in diverse fields, unified by their connection to hyperbolic or hypersphere optimization, hyperbolic transport equations, or hyperbolic medium properties. Several research frontiers use this term to denote closed-form scaling relations or governing equations that fundamentally mediate the transfer (of information, energy, or heat) in systems with hyperbolic structure or constraints.

1. Learning Rate Scaling Laws in Hypersphere-Constrained Deep Models

The central instance of the HyperP Transfer Law arises in large-scale neural network training, specifically within the Hypersphere Parameterization ("HyperP") framework employing the Muon optimizer under a Frobenius-sphere constraint (Ren et al., 30 Mar 2026). Here, the law provides a closed-form prediction for the optimal base learning rate $\alpha^*$, incorporating model depth $d$, training token count $N$, and Mixture-of-Experts (MoE) granularity $g$: $$\boxed{ \alpha^*(w,d,N,g) \approx \alpha_0 \, w^0 \, d^{-1/2}\, N^{-0.32} \, g^0 }$$ with $\alpha_0 \approx 24.27$. Notably, this law exhibits the following key properties:

• Separable Power-Law Exponents: Zero dependence on width ($w^0$) and MoE granularity ($g^0$) due to Frobenius-norm invariance (Thm 2) and SqrtGate-ensured output RMS regularization, respectively.
• Depth and Data Scaling: $d^{-1/2}$ scaling arises from depth-wise update normalization, whereas the "magic" data exponent $-0.32$ aligns with that empirically observed for AdamW optimizers.
• Universality and Transferability: A single $d$0 identified at low ($d$1) scale enables transfer of hyperparameters across diverse model and data scales without retuning, maintaining optimality up to $d$2 FLOPs.

Empirical validation confirms $d$3 on this law's predictions for $d$4 across extensive depth and data sweeps (Ren et al., 30 Mar 2026).

2. Structural Constraints and No-Op Property of Weight Decay

Within the HyperP framework, weight matrices are projected onto a fixed-norm hypersphere, resulting in a tangential-only update rule: $d$5 where $d$6 denotes the Frobenius-sphere tangent plane projection. Crucially, weight decay in the optimizer update is shown to be a first-order no-op under this constraint: $d$7 rendering the $d$8 contribution null. This structural feature provides stability and obviates the need for explicit weight decay regularization (Ren et al., 30 Mar 2026).

3. Depth-$d$9P Scaling and Per-Parameter LR Multipliers

Despite optimizer-induced update normalization, model-depth sensitivity persists, necessitating "Depth-$N$0P" scaling. Explicitly, per-parameter learning-rate multipliers scale as $N$1 for embedding and unembedding matrices, and as $N$2 for hidden weight matrices—mirroring the primary HyperP learning rate law. This ensures stability and transferability of optimization dynamics across architectures of varying depth (Ren et al., 30 Mar 2026).

4. Empirical Verification and Compute Efficiency

Methodical grid sweeps and cross-validation demonstrate the law's predictive reliability:

• Learning Rate Fit: $N$3 for $N$4 (tokens), median absolute error under 1.5%.
• Compute Leverage: Compute-efficiency leverage of $N$5 over a strong Muon baseline at $N$6 FLOPs.
• Stability Across Depths: Transfer of a single tuned base learning rate across $N$7 maintains optimality, whereas non-HyperP approaches display up to $N$8 variation in optimal rate.

These findings validate the law’s robustness and practical scalability (Ren et al., 30 Mar 2026).

5. SqrtGate for MoE Granularity-Invariant Scaling

The SqrtGate mechanism addresses the instability of output root-mean-square (RMS) values in standard MoE gating under variable granularity $N$9. While naive routing yields

$g$0

SqrtGate modifies gating weights via $g$1: $g$2 achieving granularity invariance and improving both performance and expert balance for large auxiliary load-balancing coefficients (robust up to $g$3) (Ren et al., 30 Mar 2026).

6. Stability Metrics Under Frobenius-Sphere Constraint

The HyperP formalism enforces non-increasing and bounded instability metrics with scaling:

• Attention/router Z-values, indicating logit nonlinearity, decrease with increasing depth.
• Output RMS and outlier activation fraction both decline with scale.
• Theoretical estimates confirm that the maximum output and logit values are $g$4 and bounded in $g$5 due to sphere-constrained norm preservation (Ren et al., 30 Mar 2026).

The empirical evidence demonstrates that six critical metrics (Z-values, output RMS for residual branches, outlier fraction) remain stable or improve under compute scaling, a significant advance for high-scaling MoE architectures.

Table: Core Elements of HyperP Transfer Law for LLM Scaling

Component	Scaling Law / Principle	Empirical Result
Base learning rate	$g$6	$g$7 on $g$8 prediction
Weight decay	Null on Frobenius sphere	Eliminated in optimizer
Depth sensitivity	Depth-$g$9P multipliers $$\boxed{ \alpha^*(w,d,N,g) \approx \alpha_0 \, w^0 \, d^{-1/2}\, N^{-0.32} \, g^0 }$$0	Essential for stability
SqrtGate (MoE)	Gating weights $$\boxed{ \alpha^*(w,d,N,g) \approx \alpha_0 \, w^0 \, d^{-1/2}\, N^{-0.32} \, g^0 }$$1	Output RMS granularity-invariant
Stability indicators	Bounded/non-increasing with depth/scale	Outlier %, Z-values decline with $$\boxed{ \alpha^*(w,d,N,g) \approx \alpha_0 \, w^0 \, d^{-1/2}\, N^{-0.32} \, g^0 }$$2

7. Broader Context and Related Definitions

The term "HyperP Transfer Law" is also used in several other domains, including:

• MHD Turbulence: Describes the third-order Yaglom law in hyperviscous MHD turbulence, relating structure functions to energy transfer (Jiang et al., 2022).
• Hyperbolic Heat Conduction: Governs causal finite-speed heat transfer via hyperbolic (Cattaneo–Vernotte) equations and generalizations for compressible fluids (Scharf, 2016, Dhaouadi et al., 2023).
• Radiative Transfer in Hyperbolic Metamaterials: Quantifies thermal hyper-conductivity surpassing the Stefan–Boltzmann law due to diverging photonic DOS in hyperbolic bands (Narimanov et al., 2011, Messina et al., 2016).

While details of the underlying functional forms, fields, or operators differ across fields, the root principle of the HyperP Transfer Law is to provide a closed, often universal solution (power law or PDE) governing optimal transfer rates or stability conditions in hyperbolic-structured systems.

• "Rethinking LLM Scaling under Transferable Hypersphere Optimization" (Ren et al., 30 Mar 2026)
• "Energy transfer and third-order law in forced anisotropic MHD turbulence with hyperviscosity" (Jiang et al., 2022)
• "Approach to steady state in the heat equation and the hyperbolic heat transfer equation" (Scharf, 2016)
• "Beyond Stefan-Boltzmann Law: Thermal Hyper-Conductivity" (Narimanov et al., 2011)
• "Hyperbolic waveguide for long-distance transport of near-field heat flux" (Messina et al., 2016)
• "An Eulerian hyperbolic model for heat transfer derived via Hamilton's principle" (Dhaouadi et al., 2023)