Phase Transitions in the Fluctuations of Functionals of Random Neural Networks

Abstract: We establish central and non-central limit theorems for sequences of functionals of the Gaussian output of an infinitely-wide random neural network on the d-dimensional sphere . We show that the asymptotic behaviour of these functionals as the depth of the network increases depends crucially on the fixed points of the covariance function, resulting in three distinct limiting regimes: convergence to the same functional of a limiting Gaussian field, convergence to a Gaussian distribution, convergence to a distribution in the Qth Wiener chaos. Our proofs exploit tools that are now classical (Hermite expansions, Diagram Formula, Stein-Malliavin techniques), but also ideas which have never been used in similar contexts: in particular, the asymptotic behaviour is determined by the fixed-point structure of the iterative operator associated with the covariance, whose nature and stability governs the different limiting regimes.

• The paper establishes phase transitions in normalized functional fluctuations, delineating regimes that yield Gaussian and non-Gaussian limits.
• It employs Hermite expansions and Wiener chaos decomposition alongside Malliavin calculus to quantify convergence rates and variance scaling.
• The analysis reveals that kernel dynamics and singularity parameters crucially determine limiting laws, impacting deep network statistical behavior.

Phase Transitions in the Fluctuations of Functionals of Random Neural Networks

Overview and Context

This paper addresses the asymptotic fluctuation behavior of nonlinear functionals of Neural Network Gaussian Fields (NNGF), specifically in the regime where both width ($n \to \infty$) and depth ($L \to \infty$) diverge, and the domain is the $d$-dimensional sphere, $\mathbb{S}^d$. The analysis is performed on functionals of the form

$$F_L = \int_{\mathbb{S}^d} \varphi(\widehat{T}_L(x))\,dx,$$

where $\widehat{T}_L$ is a suitably normalized and centered version of the NNGF of depth $L$, corresponding to a given activation function $\sigma$, and $\varphi$ is a general nonlinear transformation with Hermite rank $Q \geq 2$.

The principal result is a phase transition in the limiting distribution of $L \to \infty$0 as the depth $L \to \infty$1. The limit can be either Gaussian, non-Gaussian, or a potentially random functional of a limiting Gaussian field, depending on explicit analytical properties of the activation kernel. This study connects and significantly extends classical approaches (Breuer-Major theorem, Dobrushin-Major theory, high-frequency CLTs on spheres) to a "deep" regime for random neural networks.

Mathematical Setting and Problem

Random Neural Networks and NNGFs

Fully-connected random neural networks, in the infinite-width limit, yield isotropic Gaussian random fields on $L \to \infty$2 with covariance kernel $L \to \infty$3 ($L \to \infty$4 times), where $L \to \infty$5 depends on the activation $L \to \infty$6. All considered fields are centered, have unit variance, and are isotropic.

Functionals and Their Fluctuations

For any fixed $L \to \infty$7, consider the nonlinear integral functional $L \to \infty$8, as above. The core question is: what is the limit (in distribution) of the normalized fluctuation $L \to \infty$9 as $d$0?

Three Fluctuation Regimes

The order parameter is $d$1:

• Low-disorder: $d$2
• Sparse (critical): $d$3
• High-disorder: $d$4

Within each regime, the limiting behavior of $d$5 (including the nature of phase transitions) is precisely characterized in terms of the fixed-point structure of $d$6, the Hermite rank $d$7, and a "singularity order" $d$8 determined by the kernel's dynamics.

Analytical Results and Phase Transition

Hermite Expansion and Chaos Decomposition

The functionals admit a Wiener chaos expansion:

$d$9

where $\mathbb{S}^d$0 is the Hermite rank of $\mathbb{S}^d$1. The variance of each chaos is governed by the $\mathbb{S}^d$2 power of the covariance function $\mathbb{S}^d$3 and its scaling as $\mathbb{S}^d$4 is critical for identifying the limiting type of fluctuation.

Detailed Results by Regime

Low-Disorder ($\mathbb{S}^d$5)

In this regime, after normalization, the covariance kernel stabilizes. The sequence of fields $\mathbb{S}^d$6 converges in distribution (as a field) to a limiting Gaussian field $\mathbb{S}^d$7, and thus

$\mathbb{S}^d$8

where the limiting law is typically non-Gaussian, with exact structure depending on the choice of $\mathbb{S}^d$9. In particular, for polynomial $$F_L = \int_{\mathbb{S}^d} \varphi(\widehat{T}_L(x))\,dx,$$0 of degree $$F_L = \int_{\mathbb{S}^d} \varphi(\widehat{T}_L(x))\,dx,$$1, the limit is in higher-order Wiener chaos and is explicitly non-Gaussian.

High-Disorder and Sparse ($$F_L = \int_{\mathbb{S}^d} \varphi(\widehat{T}_L(x))\,dx,$$2)

Here, the recursive kernel develops a singularity, and the spectral mass of the field escapes towards high frequencies. The critical parameters are:

• $$F_L = \int_{\mathbb{S}^d} \varphi(\widehat{T}_L(x))\,dx,$$3, a singularity exponent determined by the dynamics of the fixed points of $$F_L = \int_{\mathbb{S}^d} \varphi(\widehat{T}_L(x))\,dx,$$4,
• $$F_L = \int_{\mathbb{S}^d} \varphi(\widehat{T}_L(x))\,dx,$$5, the dimension,
• $$F_L = \int_{\mathbb{S}^d} \varphi(\widehat{T}_L(x))\,dx,$$6, the Hermite rank.

There is a sharp phase transition at $$F_L = \int_{\mathbb{S}^d} \varphi(\widehat{T}_L(x))\,dx,$$7:

• If $$F_L = \int_{\mathbb{S}^d} \varphi(\widehat{T}_L(x))\,dx,$$8 (supercritical): Central Limit Theorem holds. The fluctuation $$F_L = \int_{\mathbb{S}^d} \varphi(\widehat{T}_L(x))\,dx,$$9 is asymptotically Gaussian, with explicit variance scaling (e.g., $\widehat{T}_L$0 in high-disorder, or $\widehat{T}_L$1 in sparse regime).
• If $\widehat{T}_L$2 (critical): Gaussian fluctuations with different normalization (including logarithmic corrections in the scaling of the variance).
• If $\widehat{T}_L$3 (subcritical): The limit is non-Gaussian, explicitly constructed in terms of the limiting singular kernel $\widehat{T}_L$4 and the $\widehat{T}_L$5-admissibility of its power. The law is given by a $\widehat{T}_L$6th-chaos random variable $\widehat{T}_L$7 with explicit moment structure, and this family of limit laws is new in the context of random fields on the sphere.

All statements include rates of convergence, with quantitative bounds in Wasserstein distance derived via the Malliavin-Stein method.

Methodological Approach

The analysis blends:

• Hermite expansions and the Wiener chaos decomposition: for precise control of the nonlinear functionals.
• Diagram formulae and Malliavin calculus: for quantitative CLTs and explicit bounds.
• Spectral geometry of Gaussian fields on the sphere: leveraging spherical harmonics and the behavior of power spectra.
• Fixed-point theory for the kernel iteration: providing the mechanism for singularity formation and regime differentiation.

Notably, the study introduces the use of analytic fixed-point dynamics to determine limiting behavior – a methodological novelty in this area.

Comparison with Classical Limit Theorems

The main results share formal structural analogies with classical Breuer-Major/Dobrushin-Major theorems for nonlinear functionals of stationary Gaussian fields in $\widehat{T}_L$8, especially regarding the role of the Hermite rank and decay rates/exponents in specifying central vs non-central limit regimes. However, key technical differences arise because the asymptotics here are in the depth of composition (the "deep" regime), not domain size or frequency. Furthermore, the compactness of $\widehat{T}_L$9 creates fundamentally new structure, particularly in the emergence of non-Gaussian limits on compact domains, which are precluded in high-frequency asymptotics for random fields on spheres.

Numerical and Bold Claims

• The phase transition threshold at $L$0 is established sharply, with variance scaling laws and explicit convergence rates for the CLT and non-CLT regimes.
• In the subcritical regime, the limiting random variable is non-Gaussian even on the sphere, and is of a new type (not the classical Hermite distribution for $L$1), determined by $L$2 and the singularity of the limiting kernel.
• The appearance of both Gaussian and non-Gaussian asymptotics in the deep regime for functionals of random networks is an assertion that contradicts prevalent intuition from both the central limit and high-frequency limits in earlier random field literature.

Implications and Future Prospects

This work provides a rigorous theoretical framework for understanding the behavior of high-dimensional random neural network architectures at initialization and their limiting structure in the infinite-depth limit. The identification of sharp phase transitions in fluctuation regimes has potential implications for the statistical behavior of random feature models, kernel methods arising from neural tangent kernels, and the study of Bayesian priors in function space induced by network architectures. The explicit new family of non-Gaussian limits on the sphere suggests further unexplored phenomena for random architectures in functional and geometric data analysis.

Methodologically, the paper opens the way for the use of analytic kernel dynamics in the study of deep learning's probabilistic and statistical properties, and provides techniques of substantial potential for geometric functional analysis in stochastic settings.

Conclusion

This paper characterizes the full asymptotic fluctuation theory for nonlinear functionals of random deep neural networks on the sphere, revealing a rich phase diagram consisting of Gaussian, non-Gaussian, and functional-fluctuation regimes, separated by explicit thresholds dependent on kernel dynamics, Hermite rank, and geometric dimension. The results generalize and connect to classical probabilistic limit theorems but reveal fundamentally new behaviors distinctive to the deep asymptotic regime. The analytic techniques developed hold potential for broader applications in the theoretical study of neural network models and high-dimensional stochastic processes.

Reference:

"Phase Transitions in the Fluctuations of Functionals of Random Neural Networks" (2604.19738)