Rank Collapse in Deep Learning

• Rank Collapse is a phenomenon in deep learning where intermediate representations degenerate to a low-rank subspace, reducing input distinguishability and hindering learning.
• It emerges across architectures—including MLPs, Transformers, and GNNs—due to spectral instabilities from repeated matrix multiplications.
• Architectural remedies such as batch normalization, skip connections, and tuned activations help preserve full-rank representations and stabilize training.

Rank collapse is a phenomenon in deep learning wherein intermediate or output representations of a neural network lose nearly all their degrees of freedom, degenerating toward a low-dimensional (often rank-one) subspace. In effect, this means that distinct inputs are mapped to nearly identical network states, destroying discriminative capacity, causing vanishing gradients (when the singular spectrum tightly concentrates), and severely limiting information propagation. Rank collapse manifests across architectures—MLPs, convolutional nets, transformers, state-space models, GNNs, federated low-rank adaptation, multi-modal systems, and even learning rule variants such as feedback alignment—each with distinct but mechanistically related presentations. The phenomenon, originating from spectral instabilities in products of random matrices, has driven a substantial body of theoretical and empirical research into both its mathematical foundations and architectural remedies.

1. Theoretical Foundations and Canonical Mechanisms

Classically, rank collapse is rooted in the repeated multiplication of random matrices, as appears in deep linear or ReLU networks. Consider $H_L = W_L W_{L-1}\cdots W_1 X$ under i.i.d. zero-mean, unit-variance initialization. Random matrix theory reveals that the singular values of $H_L$ evolve as $\sigma_i \sim \exp(\lambda_i L)$, with Lyapunov exponents $\lambda_1 > \lambda_2 > \cdots > \lambda_d$ (Daneshmand et al., 2020). As $L\to\infty$, $\sigma_2/\sigma_1,\ldots,\sigma_d/\sigma_1$ vanish exponentially, causing $H_L$ to concentrate onto a rank-one subspace regardless of initialization. For any $\epsilon > 0$, the probability $P[\sigma_2/\sigma_1 < \epsilon] \to 1$ exponentially, so $P[\mathrm{rank}(H_L) = 1]\to 1$.

This spectral instability is not confined to linear nets. In pure self-attention transformers devoid of skips or pointwise nonlinearity, the action of (random, stochastic) attention matrices $H_L$0 repeatedly contracts signal directions: with row-stochastic or doubly stochastic attention, it is possible to show the residual (non-rank-one) operator norm of the token representation also decays doubly-exponentially with depth. These results generalize to other architectures where repeated smoothing, mixing, or aggregation occurs (Daneshmand et al., 2020, Saada et al., 2024, Lapenna et al., 9 Apr 2026).

Table 1: Characteristic Forms of Rank Collapse

Architecture/Class	Rank Collapse Manifestation
Deep MLPs (linear/ReLU)	$H_L$1 rank-one, exponential singular-value decay
Pure Transformers	Token matrix $H_L$2 rank-one across depth and/or width
State-space/sequence models	Output matrix $H_L$3 approaches rank-one under certain recurrences
Graph Neural Networks	Node representations collapse to low-dim eigenspaces (over-smoothing)
Federated LoRA	Adapter energy collapses to shared-minimum client rank
Feedback alignment	Error signals become low-rank, update directions degenerate

The loss of rank not only eliminates input distinguishability but also yields vanishing gradients for many parameter sets—e.g., Q/K in transformers (Noci et al., 2022).

2. Rank Collapse across Model Classes

Transformers: Rank collapse arises both in depth and width. With depth (many attention/composition layers), stacking row-stochastic (softmax) or doubly stochastic attention matrices leads to the tokens converging to a single vector, as the product contracts onto the leading singular direction (Lapenna et al., 9 Apr 2026, Noci et al., 2022). With width, as sequence length grows and the bulk spectrum of random softmax attention matrices remains $H_L$4, only one outlier singular value persists (“width collapse”) (Saada et al., 2024). Layer normalization is affine-rank-neutral and does not mitigate collapse; residual connections slow or generically obstruct collapse, but only large weights fully prevent it—small attention weights permit “layer collapse,” where the full network's representation is approximable by a shallow model (2505.16284, Cirrincione, 26 Apr 2026).

State Space Models and SSM–Transformer Hybrids: Similar collapse afflicts deep SSMs; $H_L$5-skip (scaled skip connection) mechanisms provide sufficient conditions to sustain representation diversity, formalized as a positive lower bound on Frobenius norm residuals relative to the mean embedding (Joseph et al., 2024). LayerNorm and gating mechanisms also contribute to preventing collapse.

Implicit Neural Representations (INRs): In coordinate MLPs, "inlet rank collapse" refers to the input-to-first-layer mapping: low-dim input coordinates (e.g., 2D/3D) cannot populate a wide first hidden layer, bounding the Jacobian rank and ultimately the NTK rank, thereby bottlenecking the network’s expressive power (Zheng et al., 2 Feb 2026). Early remedies, such as positional encoding, SIREN, and batch norm, act by restoring full numerical rank at the inlet.

GNNs: Message-passing networks stack neighborhood smoothers; the product structure implies that node representations are asymptotically confined to the maximal eigenspace of the aggregation operator. This degeneracy implies both over-smoothing and feature over-correlation (Roth et al., 2023). Preventing collapse requires structural splitting (multi-relation graphs) to ensure that aggregation subspaces cannot universally dominate, thus preserving linearly independent signal paths (Roth et al., 2024).

Federated LoRA/Adapters: In federated low-rank adaptation, client heterogeneity (unequal LoRA rank) means that averaging suppresses higher-rank directions—energy decays geometrically everywhere except the shared minimum rank, so the global update (after enough rounds) becomes concentrated on these minimal directions (“FedLoRA rank collapse”) (Wu et al., 13 Feb 2026). The raFLoRA algorithm mitigates this by rank-wise partitioned aggregation.

Gradient Directions and Feedback Alignment: Low-rank gradients (“gradient rank collapse”) constrain the effective learning subspace. This is pronounced in feedback alignment, where the error signal's effective rank is much lower than in backpropagation, limiting exploration of the parameter space. Orthogonalizing the updates (Muon) and promoting high-rank activations (BatchNorm) can restore high-rank update geometry (Baker et al., 2024, Boeshertz et al., 9 Jun 2026).

3. Mathematical Characterization and Measures

Multiple quantitative proxies for rank collapse are in use, all based on singular-value analysis:

• Residual Norm: For representations $H_L$6, the key statistic is $H_L$7, which measures deviation from the rank-one mean state (Joseph et al., 2024).
• Soft Rank/Stable Rank: The number of singular values above a threshold, or $H_L$8 for $H_L$9 (Daneshmand et al., 2020).
• Effective Rank: Entropy-based, $\sigma_i \sim \exp(\lambda_i L)$0 for normalized singular spectrum $\sigma_i \sim \exp(\lambda_i L)$1; robust to tail perturbations and widely used for both network weights and activations (Rangamani et al., 25 Mar 2026, Kim et al., 9 Nov 2025, Haggi-Mani et al., 9 Jun 2026, Boeshertz et al., 9 Jun 2026).
• Rank Ratios: $\sigma_i \sim \exp(\lambda_i L)$2 as an indicator of singular value collapse; $\sigma_i \sim \exp(\lambda_i L)$3 (average cosine/pairwise correlation); $\sigma_i \sim \exp(\lambda_i L)$4 (energy share outside the shared minimum rank) in federated LoRA (Wu et al., 13 Feb 2026).

As an example, in deep networks with BN, the main theorems guarantee that the expected soft rank is $\sigma_i \sim \exp(\lambda_i L)$5 at any depth when $\sigma_i \sim \exp(\lambda_i L)$6 (residual scale) is sufficiently small, contrasting the asymptotic $\sigma_i \sim \exp(\lambda_i L)$7 in vanilla settings (Daneshmand et al., 2020). In transformers without skips or normalization, residual norms decay doubly-exponential in depth (Lapenna et al., 9 Apr 2026).

4. Architectural and Algorithmic Remedies

Rank collapse is both a pathway and obstacle: it enforces representational simplicity but can destroy expressivity when unmediated. Approaches to prevent it are diverse and architecture-dependent:

• Batch Normalization: BN (or full whitening) constrains the singular spectrum, actively maintaining soft rank $\sigma_i \sim \exp(\lambda_i L)$8 and preventing collapse even for infinite depth (Daneshmand et al., 2020, Zheng et al., 2 Feb 2026).
• Skip Connections / Lambda-skips: Appropriately scaled skip connections introduce an identity component into each layer’s operator, slowing or even halting collapse. The $\sigma_i \sim \exp(\lambda_i L)$9-skip framework formalizes a sufficient condition on skip strength $\lambda_1 > \lambda_2 > \cdots > \lambda_d$0 ensuring the residual norm does not decay with depth (Joseph et al., 2024).
• LayerNorm: LayerNorm is affine-rank-neutral; while it does not restore rank itself, it stabilizes normalization and is instrumental when used in conjunction with other mechanisms (Cirrincione, 26 Apr 2026).
• Large Weights: In transformers, only sufficiently large attention weight magnitudes prevent contraction (layer collapse). Small weights allow even residual-equipped models to degenerate to the effective expressivity of a single layer (2505.16284).
• Structural Heterogeneity: In GNNs and federated adaptation, enforcing multi-relation paths (e.g., via computational graph splitting, as in DAGs), or partitioned aggregation (raFLoRA), allows higher-rank directions to persist (Roth et al., 2024, Wu et al., 13 Feb 2026).
• Activation/Initialization Design: SIREN activations, positional encodings, and special weight initializations are functionally equivalent in restoring full inlet rank in continuous MLPs (Zheng et al., 2 Feb 2026).
• Rank-Preserving Optimizers and Losses: Training-time interventions, such as Muon (orthogonalizing optimizer), hidden activity normalization, and explicit entropy- or rank-promoting regularizations, enhance the effective rank of gradients and activations (Baker et al., 2024, Boeshertz et al., 9 Jun 2026).

Table 2: Empirical Evidence and Failures without Remedies

System	Collapse Symptom	Rank-preserving Intervention and Effect
Vanilla MLP	Rank$\lambda_1 > \lambda_2 > \cdots > \lambda_d$1, loss to chance	BN or pretrain on soft-rank proxy: high rank, trainability
Transformers	$\lambda_1 > \lambda_2 > \cdots > \lambda_d$2 doubly-exp fast	Skip conn., large weights, LayerNorm (with $\lambda_1 > \lambda_2 > \cdots > \lambda_d$3 selection)
GNN	Node rank$\lambda_1 > \lambda_2 > \cdots > \lambda_d$41, over-smoothing	Multi-relation graphs, sum-of-Kronecker layers
LoRA/FedLoRA	Update energy at min rank	raFLoRA blockwise aggregation: high-rank update, higher perf.

5. Problematic Regimes and Limitations

The generality of the product-of-random-matrices argument means any architecture exhibiting repeated mixing of features is vulnerable in the absence of specific countermeasures.

• Small batch sizes: BN loses efficacy, requiring pretraining or alternative normalization.
• Aggressive weight decay / severe regularization: High regularization drives weight and activation rank ever lower, sometimes to the detriment of learning even when strong regularization is not warranted by intrinsic task complexity (Zangrando et al., 2024).
• Skip-connections with small weights: Residuals copy input, but if attention weights are much smaller than unity, the whole stack acts as an identity, eliminating nontrivial depth dynamics—network collapses to input or to a shallow analog (2505.16284).
• Large-scale multi-modal fusion: Without explicit rank enhancement, a single modality can dominate, or joint representations develop low-feature diversity (Kim et al., 9 Nov 2025).

6. Implications for Generalization, Compression, and Representation Learning

Rank collapse links mechanism (spectral structure) to interpretability and function:

• Implicit Regularization: Low-rank solutions, enforced by weight decay, batch averaging, or learning dynamics, bias DNNs toward compressed, parsimonious representations, directly connecting to generalization bounds that scale as function of feature/weight rank rather than parameter count (Rangamani et al., 25 Mar 2026).
• Intrinsic Dimension Discovery: Deep neural regression collapse matches learned subspace dimension to the intrinsic rank of the data, enabling feature disentanglement and model compression (Rangamani et al., 25 Mar 2026).
• Efficient Model Editing: Once collapse occurs at a layer, it is possible to prune or edit only the low-dimensional principal subspace, opening paths for interpretability and surgical adaptation.

7. Connections, Open Problems, and Future Directions

Rank collapse unifies failure modes previously labeled as vanishing gradients, over-smoothing, signal loss, modality/feature collapse, or loss of expressivity. It emerges in both forward and backward propagation, in representations, weights, and gradients. Modern work explores further:

• The boundary between necessary compression (for generalization) and excessive collapse (causing underfitting or expressivity loss).
• Adaptation of spectral countermeasures to multi-modal, federated, and feedback-alignment paradigms.
• Task-adaptive selection of rank-targeting interventions, guided by spectral statistics.
• Symmetry-breaking frameworks that relate collapse phenomena across architectures, including the group-theoretic identification of unbroken symmetries and their breaking via skip, gating, or gating mechanisms (Cirrincione, 26 Apr 2026).
• The role that selective, task-driven or RG-inspired coarse-graining (as in MLP residual networks) plays in determining the beneficial vs harmful nature of rank reduction (Haggi-Mani et al., 9 Jun 2026).

In sum, rank collapse represents not simply a theoretical or pathological artifact, but a central axis along which depth, width, normalization, weight scale, connectivity, and data structure interact to define the effective capacity, trainability, and generalization of modern deep architectures. The interplay of spectral theory, random matrix tools, and practical architectural design underlies ongoing progress in large-scale representation learning.