Papers
Topics
Authors
Recent
Search
2000 character limit reached

HyperP Transfer Law in Deep Models

Updated 1 April 2026
  • The paper presents the HyperP Transfer Law, offering a closed-form prediction for optimal learning rates based on model depth, token count, and inherent hypersphere invariance.
  • It reveals that weight decay becomes a first-order no-op under Frobenius-sphere constraints, ensuring stable and regularized parameter updates without extra tuning.
  • Depth-multipliers and the SqrtGate mechanism enhance stability and compute efficiency, achieving R² > 0.99 accuracy and transferring hyperparameters effectively across scales.

The term "HyperP Transfer Law" encompasses a range of rigorous transfer laws and scaling formalisms emerging in diverse fields, unified by their connection to hyperbolic or hypersphere optimization, hyperbolic transport equations, or hyperbolic medium properties. Several research frontiers use this term to denote closed-form scaling relations or governing equations that fundamentally mediate the transfer (of information, energy, or heat) in systems with hyperbolic structure or constraints.

1. Learning Rate Scaling Laws in Hypersphere-Constrained Deep Models

The central instance of the HyperP Transfer Law arises in large-scale neural network training, specifically within the Hypersphere Parameterization ("HyperP") framework employing the Muon optimizer under a Frobenius-sphere constraint (Ren et al., 30 Mar 2026). Here, the law provides a closed-form prediction for the optimal base learning rate α∗\alpha^*, incorporating model depth dd, training token count NN, and Mixture-of-Experts (MoE) granularity gg: α∗(w,d,N,g)≈α0 w0 d−1/2 N−0.32 g0\boxed{ \alpha^*(w,d,N,g) \approx \alpha_0 \, w^0 \, d^{-1/2}\, N^{-0.32} \, g^0 } with α0≈24.27\alpha_0 \approx 24.27. Notably, this law exhibits the following key properties:

  • Separable Power-Law Exponents: Zero dependence on width (w0w^0) and MoE granularity (g0g^0) due to Frobenius-norm invariance (Thm 2) and SqrtGate-ensured output RMS regularization, respectively.
  • Depth and Data Scaling: d−1/2d^{-1/2} scaling arises from depth-wise update normalization, whereas the "magic" data exponent −0.32-0.32 aligns with that empirically observed for AdamW optimizers.
  • Universality and Transferability: A single α0\alpha_0 identified at low (d=8d=8) scale enables transfer of hyperparameters across diverse model and data scales without retuning, maintaining optimality up to 6×10216\times 10^{21} FLOPs.

Empirical validation confirms R2>0.99R^2 > 0.99 on this law's predictions for η∗\eta^* across extensive depth and data sweeps (Ren et al., 30 Mar 2026).

2. Structural Constraints and No-Op Property of Weight Decay

Within the HyperP framework, weight matrices are projected onto a fixed-norm hypersphere, resulting in a tangential-only update rule: W+−W=ΠT(Δ)+O(∥Δ∥F2)W^+ - W = \Pi_T(\Delta) + O(\|\Delta\|_F^2) where ΠT\Pi_T denotes the Frobenius-sphere tangent plane projection. Crucially, weight decay in the optimizer update is shown to be a first-order no-op under this constraint: Δ=−ηG−ηλW  ⟹  ΠT(Δ)=−ηΠT(G)\Delta=-\eta G - \eta\lambda W \implies \Pi_T(\Delta) = -\eta\Pi_T(G) rendering the −ηλW-\eta\lambda W contribution null. This structural feature provides stability and obviates the need for explicit weight decay regularization (Ren et al., 30 Mar 2026).

3. Depth-μ\muP Scaling and Per-Parameter LR Multipliers

Despite optimizer-induced update normalization, model-depth sensitivity persists, necessitating "Depth-μ\muP" scaling. Explicitly, per-parameter learning-rate multipliers scale as 1/d1/\sqrt{d} for embedding and unembedding matrices, and as 1/(T0.32d)1/(T^{0.32}\sqrt{d}) for hidden weight matrices—mirroring the primary HyperP learning rate law. This ensures stability and transferability of optimization dynamics across architectures of varying depth (Ren et al., 30 Mar 2026).

4. Empirical Verification and Compute Efficiency

Methodical grid sweeps and cross-validation demonstrate the law's predictive reliability:

  • Learning Rate Fit: η∗=24.27 T−0.32\eta^* = 24.27\, T^{-0.32} for TT (tokens), median absolute error under 1.5%.
  • Compute Leverage: Compute-efficiency leverage of 1.58×1.58\times over a strong Muon baseline at 6×10216\times10^{21} FLOPs.
  • Stability Across Depths: Transfer of a single tuned base learning rate across d=8→20d=8\to 20 maintains optimality, whereas non-HyperP approaches display up to 2×2\times variation in optimal rate.

These findings validate the law’s robustness and practical scalability (Ren et al., 30 Mar 2026).

5. SqrtGate for MoE Granularity-Invariant Scaling

The SqrtGate mechanism addresses the instability of output root-mean-square (RMS) values in standard MoE gating under variable granularity kk. While naive routing yields

∥yclassical∥rms≈rk\|y_{\text{classical}}\|_{\rm rms} \approx \frac{r}{\sqrt{k}}

SqrtGate modifies gating weights via gi→gig_i \to \sqrt{g_i}: ∥ysqrt∥rms≈r\|y_{\rm sqrt}\|_{\rm rms} \approx r achieving granularity invariance and improving both performance and expert balance for large auxiliary load-balancing coefficients (robust up to γ∼0.1\gamma \sim 0.1) (Ren et al., 30 Mar 2026).

6. Stability Metrics Under Frobenius-Sphere Constraint

The HyperP formalism enforces non-increasing and bounded instability metrics with scaling:

  • Attention/router Z-values, indicating logit nonlinearity, decrease with increasing depth.
  • Output RMS and outlier activation fraction both decline with scale.
  • Theoretical estimates confirm that the maximum output and logit values are O(1)O(1) and bounded in dd due to sphere-constrained norm preservation (Ren et al., 30 Mar 2026).

The empirical evidence demonstrates that six critical metrics (Z-values, output RMS for residual branches, outlier fraction) remain stable or improve under compute scaling, a significant advance for high-scaling MoE architectures.

Table: Core Elements of HyperP Transfer Law for LLM Scaling

Component Scaling Law / Principle Empirical Result
Base learning rate α∗=24.27 d−1/2N−0.32\alpha^* = 24.27\, d^{-1/2} N^{-0.32} R2>0.99R^2>0.99 on η∗\eta^* prediction
Weight decay Null on Frobenius sphere Eliminated in optimizer
Depth sensitivity Depth-μ\muP multipliers ∝1/d\propto 1/\sqrt{d} Essential for stability
SqrtGate (MoE) Gating weights →gi\to \sqrt{g_i} Output RMS granularity-invariant
Stability indicators Bounded/non-increasing with depth/scale Outlier %, Z-values decline with dd

The term "HyperP Transfer Law" is also used in several other domains, including:

  • MHD Turbulence: Describes the third-order Yaglom law in hyperviscous MHD turbulence, relating structure functions to energy transfer (Jiang et al., 2022).
  • Hyperbolic Heat Conduction: Governs causal finite-speed heat transfer via hyperbolic (Cattaneo–Vernotte) equations and generalizations for compressible fluids (Scharf, 2016, Dhaouadi et al., 2023).
  • Radiative Transfer in Hyperbolic Metamaterials: Quantifies thermal hyper-conductivity surpassing the Stefan–Boltzmann law due to diverging photonic DOS in hyperbolic bands (Narimanov et al., 2011, Messina et al., 2016).

While details of the underlying functional forms, fields, or operators differ across fields, the root principle of the HyperP Transfer Law is to provide a closed, often universal solution (power law or PDE) governing optimal transfer rates or stability conditions in hyperbolic-structured systems.

References

  • "Rethinking LLM Scaling under Transferable Hypersphere Optimization" (Ren et al., 30 Mar 2026)
  • "Energy transfer and third-order law in forced anisotropic MHD turbulence with hyperviscosity" (Jiang et al., 2022)
  • "Approach to steady state in the heat equation and the hyperbolic heat transfer equation" (Scharf, 2016)
  • "Beyond Stefan-Boltzmann Law: Thermal Hyper-Conductivity" (Narimanov et al., 2011)
  • "Hyperbolic waveguide for long-distance transport of near-field heat flux" (Messina et al., 2016)
  • "An Eulerian hyperbolic model for heat transfer derived via Hamilton's principle" (Dhaouadi et al., 2023)

Topic to Video (Beta)

No one has generated a video about this topic yet.

Whiteboard

No one has generated a whiteboard explanation for this topic yet.

Follow Topic

Get notified by email when new papers are published related to HyperP Transfer Law.