Radial-VCReg: More Informative Representation Learning Through Radial Gaussianization

Abstract: Self-supervised learning aims to learn maximally informative representations, but explicit information maximization is hindered by the curse of dimensionality. Existing methods like VCReg address this by regularizing first and second-order feature statistics, which cannot fully achieve maximum entropy. We propose Radial-VCReg, which augments VCReg with a radial Gaussianization loss that aligns feature norms with the Chi distribution-a defining property of high-dimensional Gaussians. We prove that Radial-VCReg transforms a broader class of distributions towards normality compared to VCReg and show on synthetic and real-world datasets that it consistently improves performance by reducing higher-order dependencies and promoting more diverse and informative representations.

• The paper introduces a radial Gaussianization loss that aligns empirical feature norms with the χ-distribution, addressing higher-order dependencies.
• It leverages KL divergence and Wasserstein-1 objectives to transform non-Gaussian distributions towards standard normality.
• Empirical tests with ResNet18 and ViT backbones show improved classification accuracy and effective Gaussianization compared to traditional VCReg.

Radial-VCReg: More Informative Representation Learning Through Radial Gaussianization

Motivation and Background

Self-supervised learning (SSL) seeks to maximize informativeness and diversity in learned representations without labels, with InfoMax-driven objectives being foundational. However, direct information maximization is computationally prohibitive due to the curse of dimensionality, particularly when attempting to match representations to maximum-entropy distributions. Covariance-based methods (e.g., VCReg, VICReg) regularize only linear dependencies, which is insufficient for ensuring true maximum entropy when higher-order dependencies persist. This insufficiency is demonstrated in synthetic settings, where certain non-Gaussian distributions evade rectification under classic VCReg constraints (Figure 1).

Figure 1: Samples from the $\mathrm{X}$-distribution demonstrate that, despite identity covariance, higher-order non-Gaussian structure persists.

Radial Gaussianization Objective

Radial-VCReg generalizes VCReg by introducing a radial Gaussianization loss. The method explicitly aligns the empirical feature norm distribution to the $\chi$-distribution, leveraging the property that high-dimensional Gaussians are characterized by their norms concentrating tightly around $\sqrt{d}$.

This is achieved by minimizing the KL divergence between the empirical radii distribution, $p_\theta(\|\mathbf{z}\|_2)$, and the analytical Chi distribution, $p_\chi(\|\mathbf{z}\|_2)$. The KL objective is a sum of Monte Carlo-estimated cross-entropy and entropy terms, enabling practical batch-wise optimization. Alternatively, the paper shows that a Wasserstein-1-based objective yields qualitatively similar geometry (Figure 2).

Figure 2: Wasserstein-1 optimization demonstrates effective quantitative alignment with the $\chi$-distribution for feature norms.

When included as an auxiliary loss in the full VICReg setting, this regularizer encourages not just whitening or linear decorrelation, but true Gaussianity along the radial component. This distinction is theoretically critical: as established in the paper, standard VCReg (covariance regularization) only ensures Gaussianity for input distributions that are themselves Gaussian. In contrast, Radial-VCReg can "Gaussianize" any elliptically symmetric distribution, as the optimality set of targetable distributions is strictly enlarged by the norm constraint (Figure 3).

Figure 3: Samples from the Sunshine distribution, which allows a non-trivial test of the effect of radial Gaussianization.

Synthetic and Theoretical Results

The methodology is validated on synthetic distributions such as the $\mathrm{X}$- and Sunshine-distributions. Despite having identical covariances, these distributions are not elliptically symmetric and exhibit substantial higher-order dependencies. Gradient-based minimization of the Radial-VCReg loss robustly transforms such distributions toward standard normality, as established via empirical Wasserstein-1 distances to the standard normal (Figure 4). This effect is unattainable by VCReg alone, confirming the necessity of radial constraints for removing persistent non-linear structure.

Figure 4: Initial feature norm distribution and corresponding Wasserstein-1 distances to the $\chi$-distribution, demonstrating significant non-normality at initialization.

Theoretical results include:

• Radial-VCReg loss is a consistent estimator of the KL divergence between feature norm distribution and the $\chi$-distribution (when hyperparameters $\beta_1, \beta_2 = 1$).
• For elliptically symmetric input features, imposing whitening and radial $\chi$-distribution is a sufficient condition for standard multivariate normality (by Lemma 1 in the appendix).
• The Gaussianizability set of Radial-VCReg strictly contains that of VCReg, handling non-Gaussian but elliptically symmetric distributions.

Empirical Evaluation

Radial-VCReg is evaluated with ResNet18 and ViT backbones on CIFAR-100, ImageNet-10, and CelebA. It consistently outperforms VICReg baselines, with robust improvements in Top-1 and Top-5 classification accuracies (e.g., ResNet18 on CIFAR-100: 65.99% vs. 64.23% Top-1 for $d=512$; ViT on CIFAR-100: 61.33% vs. 60.30% Top-1 for $d=512$).

Crucially, qualitative analysis of feature norm histograms shows that Radial-VCReg successfully enforces a match between the learned radii and the target $\chi$-distribution, with Wasserstein-1 distances dropping from ~17 at initialization, to <1 post-optimization (Figure 5).

Figure 5: The optimal performance of Radial-VICReg can be obtained with $\beta_1 \neq \beta_2$, empirically indicating asymmetric weighting of loss terms can improve downstream performance.

Additionally, observed correlation between precise radial alignment and downstream accuracy suggests that effective enforcement of higher-order constraints directly translates into more useful semantics, as reflected by validation performance.

Practical and Theoretical Implications

Practically, Radial-VCReg provides a drop-in regularizer that can be applied to existing InfoMax/VCReg pipelines, with modest computational overhead for one-dimensional statistics. The strict improvement over vanilla covariance-based regularization addresses persistent failure modes in SSL, especially under non-Gaussian, multi-modal, or other non-elliptically symmetric conditions.

Theoretically, this work highlights the value of breaking the reliance on only first and second-order moments: representation collapse and non-Gaussian effects can be robustly countered by enforcing higher-order shape constraints. The extension to families of distributions outside the Gaussian class opens the way for next-generation SSL objectives. Future work may adapt similar radial, angular, or other invariant-based losses to further characterize structural properties, or to extend InfoMax-style objectives to more general semiparametric, manifold, or equivariant representations.

Conclusion

Radial-VCReg constitutes a substantive advancement in non-contrastive self-supervised learning via its introduction of a radial Gaussianization loss. By explicitly enforcing the feature norm distribution to match the $\chi$-distribution, the method robustly reduces higher-order dependencies not amenable to variance or covariance regularization. Both theoretical coverage and empirical improvements are demonstrated, establishing the benefit of higher-order regularizers in maximizing representation informativeness. This approach paves the way for more expressive, information-rich representation learning objectives beyond linear statistics.