Stochastic Scaling Limits and Synchronization by Noise in Deep Transformer Models

Abstract: We prove pathwise convergence of the layerwise evolution of tokens in a finite-depth, finite-width transformer model with MultiLayer Perceptron (MLP) blocks to a continuous-time stochastic interacting particle system. We also identify the stochastic partial differential equation describing the evolution of the tokens' distribution in this limit and prove propagation of chaos when the number of such tokens is large. The bounds we establish are quantitative and the limits we consider commute. We further prove that the limiting stochastic model displays synchronization by noise and establish exponential dissipation of the interaction energy on average, provided that the common noise is sufficiently coercive relative to the deterministic self-attention drift. We finally characterize the activation functions satisfying the former condition.

• The paper establishes a rigorous stochastic scaling limit for deep transformer models by jointly scaling width, depth, and number of tokens.
• It demonstrates, via a detailed stochastic differential framework, that MLP-induced common noise leads to exponential synchronization of token states.
• The analysis is supported by numerical simulations and quantitative bounds, providing insights into the interplay between attention mechanisms and MLP noise.

Stochastic Scaling Limits and Noise-Induced Synchronization in Deep Transformers

Introduction and Motivation

This work establishes a rigorous stochastic scaling limit for deep transformer models, incorporating both self-attention and MLP blocks, and analyzes the emergent phenomenon of synchronization by common noise. While prior analyses of transformers in mean-field or continuum limits have largely focused on the attention mechanism, omitting the stochastic contributions from randomly initialized MLPs, this paper closes the gap by formalizing the joint scaling of width, depth, and number of tokens, and characterizing the resulting continuum-limit dynamics as a stochastic interacting particle system with explicit mean-field and noise structure. The theoretical analysis connects the discrete-layerwise evolution of tokens to a nonlinear stochastic differential system and, in the large-token limit, to a nonlinear SPDE on probability measures over the sphere.

Transformer Model and Dynamical Setup

The model tracks $N$ tokens, each a vector on the unit sphere $\mathbb{S}^{d-1}$, through $L$ layers of a transformer alternating between attention and MLP blocks. The self-attention uses fixed, random weight matrices, while the MLP block at each layer is a single-hidden-layer, wide, randomly initialized perceptron with activation $\mathrm{act}$. Layer normalization is implemented as radial projection back to the sphere, ensuring all token vectors have unit norm.

The key scaling regime is characterized by:

• Depth $L\to\infty$ (layers as time steps)
• Width $m\to\infty$ (wide MLPs, yielding Gaussian process limits)
• Number of tokens $N\to\infty$ (mean-field limit)
• Layer step size $\alpha_1(L)=T/L$, MLP step $\alpha_2(L)=\sqrt{T/L}$

The parameters of the MLP at each layer ($U^\ell, W^\ell$) are i.i.d. Gaussian, and the randomness across layers induces layerwise-independent random fields operating on the tokens.

Stochastic Scaling Limit and Mean-Field SPDE

The main result establishes a quantitative coupling between the discrete residual stream evolution and a system of SDEs for the token vectors, valid in the simultaneous deep and wide limit. For fixed $\mathbb{S}^{d-1}$0, the limiting system for the $\mathbb{S}^{d-1}$1-th token is:

$\mathbb{S}^{d-1}$2

Here, the deterministic interaction drift arises from the mean-field attention term, while the MLP block induces a common, isotropic Gaussian field noise with explicit covariance kernel determined by the activation and the initialization. The projection guarantees dynamics constrained to the sphere. There is a uniform, quantitative bound on the $\mathbb{S}^{d-1}$3 difference between the discrete-layer system and the SDE system, with rates $\mathbb{S}^{d-1}$4.

Passing to the large-token limit, the empirical law of the tokens converges to the distribution-valued solution of a nonlinear SPDE (of McKean–Vlasov or Vlasov–Fokker–Planck type) on the sphere:

$\mathbb{S}^{d-1}$5

The analysis establishes propagation of chaos and convergence rates for empirical distributions to the SPDE solution, with optimal scaling in $\mathbb{S}^{d-1}$6.

Noise-Induced Synchronization: Mechanisms and Results

A principal theoretical finding is the exponential synchronization of token states induced by the MLP-driven common noise. The authors prove that under explicit spectral/coercivity conditions on the kernel (which depend on the activation function and MLP bias), the stochastic dynamics force the empirical distribution to collapse toward a Dirac measure, corresponding to all tokens synchronizing on the sphere. This is quantified through a Lyapunov-functional framework, using the strict, exponential decay of a self-interaction energy functional:

$\mathbb{S}^{d-1}$7

The result holds as long as the MLP-induced noise is sufficiently strong relative to any counteracting (repulsive) drift introduced by the attention part, and the energy decay rate is computable via the spectral properties of the activation kernel.

Key claims:

• The dissipation rate decomposes into MLP and attention contributions, clarifying the interplay of architectural components.
• Under mild regularity and non-oddness assumptions on the activation and initialization, every standard activation (e.g., ReLU, Tanh, SiLU) yields strictly negative top Lyapunov exponents governing synchronization, especially when bias is present.

Numerical simulations (see below) confirm these findings and elucidate the roles of activation type, MLP bias, feature dimension, and the attention sign on synchronization phenomena.

Figure 1: Noise-induced synchronization without attention. Evolution of the first two components of $\mathbb{S}^{d-1}$8 tokens on $\mathbb{S}^{d-1}$9 as the MLP noise induces token clustering and synchronization.

Numerical Analysis and Activation Dependence

Extensive numerical experiments validate the theoretical predictions across a range of parameters:

• Without explicit MLP bias, synchronization is slow and tokens retain diversity for extended depths.
• Introducing nonzero bias in MLP initialization accelerates synchronization dramatically. For activations not odd at the origin (e.g., Sigmoid), even vanishing small bias leads to rapid collapse to a single point.
• The synchronization rate (related to the top Lyapunov exponent of the associated stochastic flow) is activation-dependent but is always negative—hence synchronization is inevitable—for all standard initializations and commonly used activations.
• High feature dimension slows synchronization for unbiased settings, but the presence of bias counteracts this effect.

Numerical experiments also confirm that attractive attention accelerates collapse, while repulsive attention delays it, but cannot halt synchronization if the MLP-driven noise is sufficiently strong.

Theoretical Proof Structure

The analysis is based on a multi-scale coupling argument, comparing the discrete residual system to an intermediate Euler–Maruyama process and ultimately to the limiting SDE, with an explicit construction that allows for pathwise (not only in law) bounds on the Wasserstein distance of the corresponding empirical measures. The proofs leverage:

• Functional central limit theorems for wide MLP layers (neural tangent kernels),
• Karhunen–Loève decomposition of the limiting GP noise,
• Uniform bounds and local Lipschitz properties for the regularized drift and diffusion,
• Propagation of chaos techniques for McKean–Vlasov systems with common noise.

Implications and Future Directions

This framework provides the first mathematically rigorous characterization of transformer residual stream dynamics—including MLPs—in the stochastic scaling regime. The findings clarify how architectural choices and initialization affect representation dynamics and synchronization, suggesting that current initialization schemes may bias very deep transformers toward rapid loss of token diversity—a phenomenon relevant for practical model design, especially as architectures scale up.

The theoretical techniques introduced here—quantitative strong approximation with explicit rates, Lyapunov analyses for stochastic flows, and identification of synchronization thresholds—create tools for further study of the role of randomness in deep representation learning, the effect of training (gradient flow), and possible interventions to maintain diversity or inject inductive bias in large models.

Conclusion

The paper rigorously establishes stochastic scaling limits for deep transformers, identifies the key role of MLP-induced common noise in driving synchronization, and provides explicit quantitative rates for both the scaling approximation and the exponential collapse of diversity. This advances the theoretical understanding of transformer architectures and highlights synchronization as a structural emergent phenomenon, potentially informing future model design and analysis strategies.