Covariance Collapse in Representation Learning

• Covariance collapse is the degeneration of second-order feature statistics, where many eigenvalues become near-zero and the effective rank is significantly reduced.
• Regularization techniques like VICReg and VCReg counteract collapse by maintaining high variance and reducing redundant feature correlations, thus improving transfer performance.
• This phenomenon is observed across multiple settings—from self-supervised models and supervised transfer to text embeddings and stochastic processes—with distinct diagnostic criteria for each case.

Covariance collapse denotes a family of degeneracy phenomena in which second-order structure becomes low-rank, highly correlated, or otherwise uninformative. In joint-embedding and supervised representation learning, it refers to embeddings whose sample covariance matrix has many near-zero eigenvalues or whose features occupy only a tiny subspace of the available representation space (Mialon et al., 2022, Zhu et al., 2023). Related formulations quantify within-class covariance collapse through the NC1 ratio in neural collapse (Kothapalli et al., 2024), second-order information loss under mean pooling in text encoders (Hara et al., 30 Apr 2026), and rank-one concentration of quadratic covariation matrices in high-dimensional semimartingales (Petkevicius, 24 Jun 2026). These usages are not identical, but they all concern a loss of informative diversity in second-order statistics.

1. Definitions and diagnostic criteria

Recent work uses closely related but non-equivalent notions of covariance collapse across several settings.

Setting	Second-order object	Collapse criterion
Joint-embedding SSL	$\operatorname{Cov}(Z)$	$\operatorname{rank}\operatorname{Cov}(Z)\ll D$ or many $\lambda_i(\operatorname{Cov}(Z))\to 0$ (Mialon et al., 2022)
Supervised penultimate features	Feature covariance across a batch	Many dimensions have near-zero variance; surviving dimensions become highly correlated (Zhu et al., 2023)
Neural collapse	$\Sigma_W(H)$ and $\Sigma_B(H)$	$$NC_1(H)=\operatorname{tr}[\Sigma_W(H)]/\operatorname{tr}[\Sigma_B(H)]\to 0$$ (Kothapalli et al., 2024)
Mean-pooled text embeddings	Token covariance $\Sigma$	$\mu_1\approx\mu_2$ but $\Sigma_1\neq\Sigma_2$, yielding high SOCM (Hara et al., 30 Apr 2026)
High-dimensional semimartingales	Quadratic covariation $Q_n$	$\operatorname{rank}\operatorname{Cov}(Z)\ll D$0 and $\operatorname{rank}\operatorname{Cov}(Z)\ll D$1 (Petkevicius, 24 Jun 2026)

For a batch of features $\operatorname{rank}\operatorname{Cov}(Z)\ll D$2, covariance collapse in SSL is formalized by the conditions $\operatorname{rank}\operatorname{Cov}(Z)\ll D$3 or many eigenvalues $\operatorname{rank}\operatorname{Cov}(Z)\ll D$4 (Mialon et al., 2022). In supervised training, a more refined decomposition distinguishes variance collapse, where many feature dimensions acquire near-zero variance across a batch, from covariance collapse, where the remaining non-degenerate dimensions become highly correlated with one another (Zhu et al., 2023).

The neural-collapse literature isolates within-class covariance collapse through the metric

$\operatorname{rank}\operatorname{Cov}(Z)\ll D$5

where $\operatorname{rank}\operatorname{Cov}(Z)\ll D$6 is the within-class covariance and $\operatorname{rank}\operatorname{Cov}(Z)\ll D$7 is the between-class covariance (Kothapalli et al., 2024). In the original neural-collapse description, within-class features collapse to their class mean, while class means become equi-angular and maximal-distance apart; this structure yields zero training error but destroys feature diversity and harms transfer (Zhu et al., 2023).

A distinct but related definition appears in text embedding models that use mean pooling. There, second-order collapse means that two token clouds with nearly identical means but different covariances map to nearly identical text embeddings, because mean pooling discards the covariance structure (Hara et al., 30 Apr 2026). In stochastic-process settings, collapse is instead spectral: a quadratic covariation matrix becomes effectively rank one when its top eigenvalue dominates the trace and the effective rank converges to one (Petkevicius, 24 Jun 2026).

2. Regularization mechanisms: VICReg and VCReg

The principal anti-collapse construction in self-supervised learning is VICReg, which combines an invariance loss with separate variance and covariance penalties. For a batch $\operatorname{rank}\operatorname{Cov}(Z)\ll D$8, the covariance term is

$\operatorname{rank}\operatorname{Cov}(Z)\ll D$9

where

$\lambda_i(\operatorname{Cov}(Z))\to 0$0

The variance term is a hinge on each coordinate’s standard deviation with target $\lambda_i(\operatorname{Cov}(Z))\to 0$1. Ablations show that “Inv only” collapses, “Inv+Cov” collapses, “Inv+Var” reaches $\lambda_i(\operatorname{Cov}(Z))\to 0$2 top-1, and “Inv+Var+Cov” reaches $\lambda_i(\operatorname{Cov}(Z))\to 0$3 top-1, establishing that covariance regularization alone does not prevent norm-collapse and variance preservation alone does not prevent rank-1 informational collapse (Bardes et al., 2021).

The supervised adaptation is VCReg, which omits the invariance term and applies the two statistical penalties directly to supervised representations. Writing a batch of representations as $\lambda_i(\operatorname{Cov}(Z))\to 0$4 with covariance matrix $\lambda_i(\operatorname{Cov}(Z))\to 0$5,

$\lambda_i(\operatorname{Cov}(Z))\to 0$6

and the supervised objective is

$\lambda_i(\operatorname{Cov}(Z))\to 0$7

or, when applied to multiple layers,

$\lambda_i(\operatorname{Cov}(Z))\to 0$8

Empirically, VCReg layers are inserted after each block or residual rather than only on the final feature; for convolutional features of shape $\lambda_i(\operatorname{Cov}(Z))\to 0$9, each channel is demeaned over batch$\Sigma_W(H)$0spatial locations and each spatial location is treated as an independent sample; the squared-entry penalty $\Sigma_W(H)$1 is replaced by a smooth-L1 variant to guard against unstable gradients from rare large covariances; and a backward hook computes covariance and variance only in the backward pass. This optimized implementation is only $\Sigma_W(H)$2 slower than identity and comparable to a BatchNorm in wall-clock time (Zhu et al., 2023).

The underlying mechanism is explicit. The high-variance term keeps each axis alive by pushing each dimension’s standard deviation above $\Sigma_W(H)$3, while the low-covariance term discourages redundant features and forces the model to spread information across dimensions. In the VICReg formulation, this blocks rank-1 solutions in which all coordinates track the same scalar; in the VCReg formulation, it is proposed as a general regularization framework for supervised transfer learning (Bardes et al., 2021, Zhu et al., 2023).

3. Neural collapse, gradient starvation, and transferability

In supervised deep networks, covariance collapse is treated as one component of a broader representational degeneracy that includes neural collapse and gradient starvation. The account developed for VCReg states that a network trained only to minimize cross-entropy can focus greedily on a minimal set of source-task features, causing within-class collapse, covariance collapse, and poor transfer to downstream tasks. VCReg is reported to counter this by enforcing high-variance, low-covariance embeddings at intermediate layers, thereby keeping new inter-class directions available and ensuring that gradients exist for each feature dimension rather than allowing one “win” dimension to shut off the rest (Zhu et al., 2023).

The empirical record in that study is extensive. On ImageNet-supervised pretraining followed by linear probing across 9 datasets, ResNet-50 with VCReg improves the average by $\Sigma_W(H)$4 points versus the baseline and beats DeCov and WLD-Reg by $\Sigma_W(H)$5 points. ConvNeXt-T improves by $\Sigma_W(H)$6 point and ViT-Base improves by $\Sigma_W(H)$7 point. On HMDB51 video finetuning for VideoMAE and ViViT backbones, VCReg adds $\Sigma_W(H)$8 point consistently. Collapse-oriented metrics move in the same direction: Class-Distance Normalized Variance rises from $\Sigma_W(H)$9, Nearest-Class-Center accuracy falls from $\Sigma_B(H)$0, and Mutual Information estimated by MINE increases from $\Sigma_B(H)$1 bits. In the synthetic two-moon setting of Pezeshki et al., VCReg is reported as the only regularizer that learns the curved decision boundary rather than starving out all but a linear feature. Additional gains appear in long-tail CIFAR10-LT/CIFAR100-LT, with $\Sigma_B(H)$2 and $\Sigma_B(H)$3 points, in hierarchical subclass probing for ConvNeXt, where CIFAR100 superclass-to-subclass linear probing rises from $\Sigma_B(H)$4 to $\Sigma_B(H)$5, and in noisy HMDB51 finetuning under Gaussian corruption $\Sigma_B(H)$6 (Zhu et al., 2023).

These results are presented as evidence for a three-way link: collapse, gradient starvation, and feature transferability. The paper states that neural collapse and gradient starvation both reflect a network’s greedy focus on a minimal feature set, that this produces a degenerate representation space, and that reversing the collapse metrics aligns nearly one-for-one with improved downstream transfer across images, videos, long-tail learning, and hierarchical tasks (Zhu et al., 2023).

4. Independence and kernel-based explanations

A more theoretical line of work analyzes covariance collapse through independence criteria. In self-supervised joint-embedding models, VCReg applied to the output of a wide MLP projector is shown to enforce pairwise independence between the features of the learned representation. The central result links the covariance penalty on the projector output $\Sigma_B(H)$7 to Hilbert–Schmidt Independence Criterion terms on the projector input $\Sigma_B(H)$8, implying that driving $\Sigma_B(H)$9 toward the identity forces $$NC_1(H)=\operatorname{tr}[\Sigma_W(H)]/\operatorname{tr}[\Sigma_B(H)]\to 0$$0 for all $$NC_1(H)=\operatorname{tr}[\Sigma_W(H)]/\operatorname{tr}[\Sigma_B(H)]\to 0$$1 in the large-width limit. The construction relies on alternating random linear maps and elementwise nonlinearities such as BatchNorm and ReLU, with projector width $$NC_1(H)=\operatorname{tr}[\Sigma_W(H)]/\operatorname{tr}[\Sigma_B(H)]\to 0$$2 and weights remaining near random. Empirically, increasing projector width improves pairwise independence, adding depth beyond 3 layers can hurt HSIC, keeping the projector random or resampling it each step lowers HSIC relative to fully learned projectors, and the same mechanism suffices to solve linear ICA but fails in post-nonlinear ICA, consistent with the claim that VCReg enforces pairwise rather than higher-order independence (Mialon et al., 2022).

Kernel methods provide a complementary analysis of within-class covariance collapse. The NC1 ratio can be written entirely in terms of the Gram matrix $$NC_1(H)=\operatorname{tr}[\Sigma_W(H)]/\operatorname{tr}[\Sigma_B(H)]\to 0$$3, so that

$$NC_1(H)=\operatorname{tr}[\Sigma_W(H)]/\operatorname{tr}[\Sigma_B(H)]\to 0$$4

is computed from kernel evaluations alone. Specializing this expression to the NNGP and NTK of shallow infinite-width networks, one study shows that the NTK does not represent more collapsed features than the NNGP for Gaussian data of arbitrary dimensions. In the prototypical 1-D two-class Gaussian mixture model, the expected NC1 values coincide for the ReLU NNGP and the ReLU NTK, indicating that lazy training does not further decrease NC1 beyond initialization. The same study reports that a data-aware Gaussian Process kernel based on Equations of State yields lower NC1 than NNGP at smaller effective widths, and that the choice of nonlinearity matters: ERF yields lower NC1 than ReLU, both theoretically and empirically (Kothapalli et al., 2024).

Taken together, these analyses separate two questions that are often conflated. One concerns whether covariance regularization decorrelates features strongly enough to approach pairwise independence; the other concerns whether simplified kernel regimes can explain the data dependence of neural collapse. The available results support the first claim for wide random projectors and qualify the second by showing clear limitations of data-independent kernels such as NTK (Mialon et al., 2022, Kothapalli et al., 2024).

5. Structured manifestations in graphs and text embeddings

In graph neural networks, covariance collapse is partial rather than exact in the regimes studied so far. For node-wise classification on sparse stochastic block model graphs, the trace of the within-class covariance, $$NC_1(H)=\operatorname{tr}[\Sigma_W(H)]/\operatorname{tr}[\Sigma_B(H)]\to 0$$5, decreases steadily during the terminal phase of training but plateaus at a positive level, reported as approximately $$NC_1(H)=\operatorname{tr}[\Sigma_W(H)]/\operatorname{tr}[\Sigma_B(H)]\to 0$$6, rather than collapsing nearly to zero as in plain image classifiers. The same plateau behavior appears for both the raw features $$NC_1(H)=\operatorname{tr}[\Sigma_W(H)]/\operatorname{tr}[\Sigma_B(H)]\to 0$$7 and the neighborhood-aggregated features $$NC_1(H)=\operatorname{tr}[\Sigma_W(H)]/\operatorname{tr}[\Sigma_B(H)]\to 0$$8. An “optimistic” graph-UFM shows that exact collapse, $$NC_1(H)=\operatorname{tr}[\Sigma_W(H)]/\operatorname{tr}[\Sigma_B(H)]\to 0$$9, occurs only under a strict structural requirement: for every class $\Sigma$0, the vector of neighbor fractions $\Sigma$1 must be independent of node index $\Sigma$2 within that class. Under exact collapse, the KKT conditions force the class means to form a simplex ETF. Gradient-flow analysis for $\Sigma$3 and a single graph yields $\Sigma$4 and $\Sigma$5, but finite graph noise and $\Sigma$6 make the collapse stall at a positive plateau (Kothapalli et al., 2023).

Text embedding models expose a different second-order failure mode. If a text encoder produces token embeddings $\Sigma$7 with mean $\Sigma$8 and covariance $\Sigma$9, then mean pooling discards $\mu_1\approx\mu_2$0 entirely. To quantify the resulting second-order collapse, the SOCM metric is defined by

$\mu_1\approx\mu_2$1

where $\mu_1\approx\mu_2$2 and $\mu_1\approx\mu_2$3 are normalized first- and second-order components of the squared $\mu_1\approx\mu_2$4-Wasserstein distance between Gaussian approximations to two token clouds. By construction, SOCM is maximal when means coincide and covariances differ, and vanishes when the means differ maximally or the covariances coincide. On 1,000 Wikipedia texts, covering all 499,500 pairwise comparisons, average SOCM drops from $\mu_1\approx\mu_2$5 for BERT to $\mu_1\approx\mu_2$6 for Unsup-SimCSE-mean, $\mu_1\approx\mu_2$7 for E5_base, and $\mu_1\approx\mu_2$8 for GTE_base; for MiniLM it drops from $\mu_1\approx\mu_2$9 to $\Sigma_1\neq\Sigma_2$0 for E5_small and $\Sigma_1\neq\Sigma_2$1 for GTE_small, but rises to $\Sigma_1\neq\Sigma_2$2 for all-MiniLM-L12-v2; for MPNet it drops from $\Sigma_1\neq\Sigma_2$3 to $\Sigma_1\neq\Sigma_2$4 for all-mpnet-base-v2; and for nomic-bert-2048 it drops from $\Sigma_1\neq\Sigma_2$5 to $\Sigma_1\neq\Sigma_2$6 for nomic-embed-text-v1.5. The same work proves that if each text’s normalized spread $\Sigma_1\neq\Sigma_2$7, then $\Sigma_1\neq\Sigma_2$8, and reports that average SOCM correlates with MTEB score with Spearman’s $\Sigma_1\neq\Sigma_2$9 and $Q_n$0 (Hara et al., 30 Apr 2026).

A plausible implication is that “collapse” in structured domains is highly architecture-dependent. In GNNs, graph heterogeneity and finite-sample randomness obstruct exact within-class collapse except under a strict neighbor-fraction condition. In contrastive text encoders, mean pooling remains effective because token embeddings within each text become sufficiently concentrated that little second-order information remains to be lost (Kothapalli et al., 2023, Hara et al., 30 Apr 2026).

6. Spectral collapse in stochastic systems and terminological boundaries

Outside representation learning, covariance collapse appears as a spectral property of high-dimensional semimartingales. Let

$Q_n$1

be the quadratic covariation matrix, with diffusion component $Q_n$2 and jump component

$Q_n$3

Two diagnostics are used:

$Q_n$4

Spectral collapse means that both converge to $Q_n$5. The main theorem states that this occurs if and only if the jump directions are geometrically aligned in a weighted sense and the background diffusion is asymptotically negligible, specifically $Q_n$6 and $Q_n$7. In the stochastic extension, the same conclusion holds in probability under natural conditions, and a scalar diagnostic

$Q_n$8

tends to $Q_n$9 exactly when $\operatorname{rank}\operatorname{Cov}(Z)\ll D$00 grows unboundedly. The proposed operational interpretation is that values of $\operatorname{rank}\operatorname{Cov}(Z)\ll D$01 or $\operatorname{rank}\operatorname{Cov}(Z)\ll D$02 near $\operatorname{rank}\operatorname{Cov}(Z)\ll D$03 signal a rank-one extreme-event phase (Petkevicius, 24 Jun 2026).

A separate terminological issue arises in gravitational-collapse models inspired by loop quantum gravity. There, “covariance” refers to closure of the deformed constraint algebra and underlying $\operatorname{rank}\operatorname{Cov}(Z)\ll D$04 diffeomorphism invariance, not to covariance-matrix degeneracy. In the effective spherically symmetric dust model, one has

$\operatorname{rank}\operatorname{Cov}(Z)\ll D$05

which closes with the diffeomorphism constraint $\operatorname{rank}\operatorname{Cov}(Z)\ll D$06, and the theory remains fully $\operatorname{rank}\operatorname{Cov}(Z)\ll D$07 diffeo-invariant even when one later imposes the areal gauge $\operatorname{rank}\operatorname{Cov}(Z)\ll D$08. The same analysis argues that shock solutions proposed in some earlier phenomenological models are absent because the dynamics depend on $\operatorname{rank}\operatorname{Cov}(Z)\ll D$09 rather than $\operatorname{rank}\operatorname{Cov}(Z)\ll D$10, so coordinate changes cannot create discontinuous shock layers across the bounce (Giesel et al., 2023).

This distinction is important because the phrase combines two overloaded terms. In machine learning and stochastic-process theory, covariance collapse concerns degeneration of second-order structure. In canonical gravity, covariance is a statement about symmetry and constraint closure during gravitational collapse, and the relevant question is whether an effective theory remains covariant across a quantum bounce (Giesel et al., 2023).