Radial-VCReg: Gaussianization in Self-Supervised Learning

• The paper introduces Radial-VCReg, a framework that augments VCReg by enforcing feature norms to follow a chi distribution.
• It employs a radial Gaussianization loss using KL divergence and m-spacing to incorporate higher-order moment constraints.
• Empirical evaluations demonstrate improved representation diversity and accuracy on both synthetic and real-world tasks.

Radial-VCReg is a self-supervised learning (SSL) framework that augments Variance-Invariance-Covariance Regularization (VCReg) with a radial Gaussianization penalty. This regularization enforces that feature norms match the chi distribution associated with high-dimensional standard Gaussians, introducing a higher-order constraint absent in existing methods. The result is a consistent improvement in the informativeness and diversity of learned representations, with concrete benefits demonstrated in both synthetic and real-world tasks (Kuang et al., 15 Feb 2026).

1. Background and Motivation

Self-supervised learning algorithms seek to learn maximally informative representations without requiring labeled data. Two core principles guide these methods:

• Invariance: Representations of two augmented views $Z$ and $Z'$ of the same input should be similar.
• Diversity: The representation’s distribution must have high entropy to avoid collapse to degenerate (constant) codes.

Explicit information maximization, for example maximizing the mutual information $I(Z; Z')$, is infeasible in high dimensions due to the difficulty of estimating and maximizing the marginal entropy $H(Z)$ ("curse of dimensionality"). VICReg circumvents this by constraining only first- and second-order feature statistics—mean and covariance—using:

• Variance loss $v(Z)$: enforces nontrivial per-dimension variance.
• Covariance loss $c(Z)$: penalizes linear correlations through off-diagonal entries of the sample covariance.

While this approach avoids trivial solutions, it does not ensure that the entire representation distribution achieves high entropy (i.e., becomes Gaussian), leaving higher-order dependencies uncontrolled. Radial-VCReg addresses this gap by enforcing that the lengths of feature vectors are distributed according to the chi distribution, a property characterizing the norm of samples from a standard multivariate normal (Kuang et al., 15 Feb 2026).

2. Radial Gaussianization Loss

2.1 Motivation

For a $d$-dimensional standard Gaussian $\mathcal{N}(0, I_d)$, the Euclidean norm $r = \|z\|_2$ follows the chi distribution with $d$ degrees of freedom:

$Z'$0

Any spherically symmetric distribution whose radii are chi-distributed is necessarily a standard normal.

2.2 Formal Definition

The radial Gaussianization loss is defined as the Kullback–Leibler divergence between the empirical radius distribution $Z'$1 and the target chi density $Z'$2:

$Z'$3

This loss is estimated in practice via:

• Cross-entropy estimator:

$Z'$4

• Empirical entropy estimator (m-spacing):

Given sorted radii $Z'$5, for some $Z'$6,

$Z'$7

The radial loss term on a batch $Z'$8 is:

$Z'$9

2.3 Gaussianity via Radial Matching

Lemma 1 states that for a spherically symmetric density $I(Z; Z')$0, if $I(Z; Z')$1, then $I(Z; Z')$2 is the standard normal. Thus, after whitening via mean-centering and covariance normalization, matching the feature-norm distribution to the chi law is a sufficient condition for achieving Gaussianity within the class of elliptical distributions. This introduces a necessary high-order moment constraint absent in prior work (Kuang et al., 15 Feb 2026).

3. Theoretical Properties

The Gaussianization properties of VCReg and Radial-VCReg are formalized by comparing their effect on the representational distribution:

• VCReg applies an affine whitening map $I(Z; Z')$3, Gaussianizing only if the original distribution is Gaussian.
• Radial-VCReg employs a composite transformation:

$I(Z; Z')$4

where $I(Z; Z')$5 and $I(Z; Z')$6 are the CDFs of the empirical and target chi distributions, respectively.

The set of distributions Gaussianized by VCReg, $I(Z; Z')$7, is strictly contained in the set covered by Radial-VCReg, $I(Z; Z')$8, which can handle all elliptical-symmetric distributions rather than just the Gaussian. This extended mapping enforces maximum-entropy structure for a broader class of distributions.

4. Combined Objective Function and Optimization

The Radial-VCReg total objective for a batch of paired views $I(Z; Z')$9 is:

$H(Z)$0

where:

• Similarity (invariance) loss:

$H(Z)$1

• Variance–Covariance (VCReg) loss:

$H(Z)$2

with $H(Z)$3 the sample covariance.

$H(Z)$4

• Radial Gaussianization loss:

$H(Z)$5

Key hyperparameters $H(Z)$6 tune the relative weighting of each regularization.

A stepwise optimization pseudocode is provided in the original work. The batch-based computation incorporates augmentations, similarity and VCReg losses, radius extraction and sorting, m-spacing estimation, and total loss backpropagation for gradient updates (Kuang et al., 15 Feb 2026).

5. Empirical Evaluation

5.1 Synthetic Data

• X-distribution: A 2D "X"-shaped distribution with identity covariance but non-elliptical structure. VCReg fails to Gaussianize; Radial-VCReg progressively transforms the sample towards a spherical Gaussian, as quantified by 2-Wasserstein distance.
• Sunshine distribution: A non-elliptical distribution with correct covariance and radial law. Radial-VCReg does not fully Gaussianize this case but further reduces higher-order dependencies than VCReg, highlighting that the radial constraint is necessary but not sufficient for full normality outside the elliptical-symmetric family.

5.2 Real-world Datasets

• CIFAR-100 (ResNet-18, ViT-Tiny): At projector dimensions $H(Z)$7 and $H(Z)$8, Radial-VCReg exhibits consistent improvements in linear-probe Top-1 accuracy over VICReg by approximately $H(Z)$9–$v(Z)$0 points, confirmed by MLP probes.
• ImageNet-10 (ResNet-18): Similar accuracy gains (up to $v(Z)$1 points) across a wide projector dimension sweep (up to $v(Z)$2).
• CelebA Multi-label: Ablation studies show that the cross-entropy component of the radial loss is critical for performance, while emphasizing the entropy term alone can degrade results.
• Radius matching: Post-training, the distribution of feature norms closely aligns with the chi law, with improvement in chi-fit (lower Wasserstein-1 distance) positively correlating with downstream task accuracy.

6. Practical Recommendations and Limitations

Hyperparameter selection is problem dependent:

• $v(Z)$3 (cross-entropy weight): Larger for lower $v(Z)$4. Typical values: $v(Z)$5 for $v(Z)$6; $v(Z)$7 for $v(Z)$8.
• $v(Z)$9 (entropy weight): Small values ($c(Z)$0–$c(Z)$1) may enhance m-spacing stability.
• $c(Z)$2 (radial loss weight): Set such that $c(Z)$3 and $c(Z)$4 are balanced at initialization.
• Batch size: Larger batches improve the empirical m-spacing estimation.
• Projector dimension $c(Z)$5: Low $c(Z)$6 amplifies the need for strong radial enforcement.

Limitations include:

• The radial loss is necessary but not sufficient for normality outside the elliptical class, as evidenced by specific pathologies (e.g., the Sunshine distribution).
• KL and Wasserstein approaches to radial matching perform comparably in practice, with the latter offering easier parameter tuning.
• Additional computational overhead arises from sorting and m-spacing calculations but remains minor.
• Open directions include integrating radial constraints with higher-order cumulant matching or adversarial Gaussianization, and generalizations to multi-modal or structured representation domains.

Radial-VCReg constitutes a principled higher-order regularization, efficiently steering feature norms toward the Gaussian shell structure and consistently yielding greater representational diversity, higher entropy, and measurable improvements in downstream tasks (Kuang et al., 15 Feb 2026).