---
title: 'Radial-VCReg: Gaussianization in Self-Supervised Learning'
url: https://www.emergentmind.com/topics/radial-vcreg
type: topic
---

# Radial-VCReg: Gaussianization in Self-Supervised Learning

Radial-VCReg is a self-supervised learning (SSL) framework that augments Variance-Invariance-Covariance Regularization (VCReg) with a radial Gaussianization penalty. This regularization enforces that feature norms match the chi distribution associated with high-dimensional standard Gaussians, introducing a higher-order constraint absent in existing methods. The result is a consistent improvement in the informativeness and diversity of learned representations, with concrete benefits demonstrated in both synthetic and real-world tasks [2602.14272].

## 1. Background and Motivation

Self-supervised learning algorithms seek to learn maximally informative representations without requiring labeled data. Two core principles guide these methods:

- **Invariance**: Representations of two augmented views $Z$ and $Z'$ of the same input should be similar.
- **Diversity**: The representation’s distribution must have high entropy to avoid collapse to degenerate (constant) codes.

Explicit information maximization, for example maximizing the mutual information $I(Z; Z')$, is infeasible in high dimensions due to the difficulty of estimating and maximizing the marginal entropy $H(Z)$ ("curse of dimensionality"). VICReg circumvents this by constraining only first- and second-order feature statistics—mean and covariance—using:

- **Variance loss** $v(Z)$: enforces nontrivial per-dimension variance.
- **Covariance loss** $c(Z)$: penalizes linear correlations through off-diagonal entries of the sample covariance.

While this approach avoids trivial solutions, it does not ensure that the entire representation distribution achieves high entropy (i.e., becomes Gaussian), leaving higher-order dependencies uncontrolled. Radial-VCReg addresses this gap by enforcing that the lengths of feature vectors are distributed according to the chi distribution, a property characterizing the norm of samples from a standard multivariate normal [2602.14272].

## 2. Radial Gaussianization Loss

### 2.1 Motivation

For a $d$-dimensional standard Gaussian $\mathcal{N}(0, I_d)$, the Euclidean norm $r = \|z\|_2$ follows the chi distribution with $d$ degrees of freedom:
$$
p_{\chi_d}(r) = \frac{1}{2^{\frac{d}{2}-1}\,\Gamma(d/2)} r^{d-1} \exp\left( -\frac{r^2}{2} \right),\quad r \geq 0.
$$
Any spherically symmetric distribution whose radii are chi-distributed is necessarily a standard normal.

### 2.2 Formal Definition

The **radial Gaussianization loss** is defined as the Kullback–Leibler divergence between the empirical radius distribution $p_\theta(r)$ and the target chi density $p_{\chi_d}(r)$:
$$
L_{\text{radial}} = D_{KL}\left( p_\theta(r)\,\|\; p_{\chi_d}(r) \right)
= \int p_\theta(r)\,\log \frac{p_\theta(r)}{p_{\chi_d}(r)}\,dr.
$$
This loss is estimated in practice via:

- **Cross-entropy estimator**:
  $$
  \mathbb{E}_{r \sim p_\theta}\left[-\log p_{\chi_d}(r)\right]
    \approx
  \frac{\beta_1}{N} \sum_{i=1}^N \left( \frac{1}{2}r_i^2 - (d-1)\log r_i \right) + \text{const.}
  $$
- **Empirical entropy estimator (m-spacing)**:
  Given sorted radii $r_{(1)} \leq \cdots \leq r_{(N)}$, for some $m$,
  $$
  H\bigl(p_\theta(r)\bigr) \approx \frac{\beta_2}{N-m} \sum_{i=1}^{N-m} \log\left( \frac{N+1}{m} (r_{(i+m)} - r_{(i)}) \right).
  $$

The radial loss term on a batch $Z = \{z_i\}$ is:
$$
r(Z; \beta_1, \beta_2) =
\frac{\beta_1}{N}\sum_i \left( \frac{1}{2}\|z_i\|^2 - (d-1)\log\|z_i\| \right)
- \frac{\beta_2}{N-m} \sum_{i=1}^{N-m} \log\left( \frac{N+1}{m}(r_{(i+m)} - r_{(i)}) \right).
$$

### 2.3 Gaussianity via Radial Matching

Lemma 1 states that for a spherically symmetric density $p(x) = g(\|x\|)$, if $\|x\| \sim \chi_d$, then $p(x)$ is the standard normal. Thus, after whitening via mean-centering and covariance normalization, matching the feature-norm distribution to the chi law is a sufficient condition for achieving Gaussianity within the class of elliptical distributions. This introduces a necessary high-order moment constraint absent in prior work [2602.14272].

## 3. Theoretical Properties

The Gaussianization properties of VCReg and Radial-VCReg are formalized by comparing their effect on the representational distribution:

- **VCReg** applies an affine whitening map $T_{\text{VCReg}}(x) = \Sigma^{-1/2}(x-\mu)$, Gaussianizing only if the original distribution is Gaussian.
- **Radial-VCReg** employs a composite transformation:
  $$
  T_{\text{RVCReg}}(x) =
  \frac{ \Sigma^{-1/2}(x-\mu) }{ \|\Sigma^{-1/2}(x-\mu)\| }
  F_\chi^{-1}\left( F_{ \|\Sigma^{-1/2}(x-\mu)\| }( \|\Sigma^{-1/2}(x-\mu)\| ) \right)
  $$
  where $F$ and $F_\chi$ are the CDFs of the empirical and target chi distributions, respectively.

The set of distributions Gaussianized by VCReg, $\mathcal{F}_{\text{VCReg}}$, is strictly contained in the set covered by Radial-VCReg, $\mathcal{F}_{\text{RVCReg}}$, which can handle all elliptical-symmetric distributions rather than just the Gaussian. This extended mapping enforces maximum-entropy structure for a broader class of distributions.

## 4. Combined Objective Function and Optimization

The Radial-VCReg total objective for a batch of paired views $(Z, Z')$ is:
$$
L_{\text{RVCReg}} = L_{\text{sim}}(Z,Z') + L_{\text{VC}}(Z,Z') + L_{\text{radial}}(Z,Z'),
$$
where:

- **Similarity (invariance) loss**:
  $$
  L_{\text{sim}}(Z, Z') = \frac{1}{N}\sum_{i=1}^N \|z_i - z'_i\|^2
  $$
- **Variance–Covariance (VCReg) loss**:
  $$
  v(Z) = \sum_{j=1}^d \max\big(0, \gamma - \sqrt{ \operatorname{Var}(z^j) + \epsilon } \big), \quad
  c(Z) = \sum_{i \neq j} [C(Z)]_{i, j}^2,
  $$
  with $C(Z)$ the sample covariance.
  $$
  L_{\text{VC}} = \lambda_2[v(Z) + v(Z')] + \lambda_3[c(Z) + c(Z')]
  $$
- **Radial Gaussianization loss**:
  $$
  L_{\text{radial}} = \lambda_r [ r(Z; \beta_1, \beta_2) + r(Z'; \beta_1, \beta_2) ]
  $$

Key hyperparameters $(\lambda_2, \lambda_3, \lambda_r, \beta_1, \beta_2)$ tune the relative weighting of each regularization.

A stepwise optimization pseudocode is provided in the original work. The batch-based computation incorporates augmentations, similarity and VCReg losses, radius extraction and sorting, m-spacing estimation, and total loss backpropagation for gradient updates [2602.14272].

## 5. Empirical Evaluation

### 5.1 Synthetic Data

- **X-distribution**: A 2D "X"-shaped distribution with identity covariance but non-elliptical structure. VCReg fails to Gaussianize; Radial-VCReg progressively transforms the sample towards a spherical Gaussian, as quantified by 2-Wasserstein distance.
- **Sunshine distribution**: A non-elliptical distribution with correct covariance and radial law. Radial-VCReg does not fully Gaussianize this case but further reduces higher-order dependencies than VCReg, highlighting that the radial constraint is necessary but not sufficient for full normality outside the elliptical-symmetric family.

### 5.2 Real-world Datasets

- **CIFAR-100 (ResNet-18, ViT-Tiny)**: At projector dimensions $d=512$ and $d=2048$, Radial-VCReg exhibits consistent improvements in linear-probe Top-1 accuracy over VICReg by approximately $1.0$–$1.5$ points, confirmed by MLP probes.
- **ImageNet-10 (ResNet-18)**: Similar accuracy gains (up to $1.5$ points) across a wide projector dimension sweep (up to $8192$).
- **CelebA Multi-label**: Ablation studies show that the cross-entropy component of the radial loss is critical for performance, while emphasizing the entropy term alone can degrade results.
- **Radius matching**: Post-training, the distribution of feature norms closely aligns with the chi law, with improvement in chi-fit (lower Wasserstein-1 distance) positively correlating with downstream task accuracy.

## 6. Practical Recommendations and Limitations

Hyperparameter selection is problem dependent:

- **$\beta_1$ (cross-entropy weight)**: Larger for lower $d$. Typical values: $\beta_1 = 100$ for $d=512$; $\beta_1=1$ for $d=2048$.
- **$\beta_2$ (entropy weight)**: Small values ($0.1$–$0.5$) may enhance m-spacing stability.
- **$\lambda_r$ (radial loss weight)**: Set such that $L_{\text{radial}}$ and $L_{\text{VC}}$ are balanced at initialization.
- **Batch size**: Larger batches improve the empirical m-spacing estimation.
- **Projector dimension $d$**: Low $d$ amplifies the need for strong radial enforcement.

Limitations include:

- The radial loss is necessary but not sufficient for normality outside the elliptical class, as evidenced by specific pathologies (e.g., the Sunshine distribution).
- KL and Wasserstein approaches to radial matching perform comparably in practice, with the latter offering easier parameter tuning.
- Additional computational overhead arises from sorting and m-spacing calculations but remains minor.
- Open directions include integrating radial constraints with higher-order cumulant matching or adversarial Gaussianization, and generalizations to multi-modal or structured representation domains.

Radial-VCReg constitutes a principled higher-order regularization, efficiently steering feature norms toward the Gaussian shell structure and consistently yielding greater representational diversity, higher entropy, and measurable improvements in downstream tasks [2602.14272].

Source: https://www.emergentmind.com/topics/radial-vcreg