VISReg: Variance-Invariance-Sketching Regularization

• VISReg is a self-supervised JEPA regularizer that separates embedding center, coordinate-wise scale, distributional shape, and view invariance to prevent collapse and improve representation quality.
• Its shape objective matches sorted random projections of variance-normalized embeddings to standard-Gaussian quantiles using sliced-Wasserstein distance, capturing higher-order properties beyond covariance.
• VISReg scales through projection and sorting costs of O(NDK) and O(KN log N), and reports strong out-of-distribution, transfer, long-tailed, and low-rank performance, though results depend on batch size and projection count.

VISReg, or Variance-Invariance-Sketching Regularization, is a self-supervised representation-learning regularizer for joint-embedding predictive architectures (JEPAs). It combines a view-invariance or prediction objective with explicit control of embedding scale, distributional shape, and mean. VISReg retains VICReg’s separable variance-based scale control while replacing covariance regularization with a Sliced-Wasserstein-based sketching objective that matches random one-dimensional projections of normalized embeddings to standard-Gaussian quantiles. The method is designed to prevent embedding collapse, improve distributional regularity beyond second-order decorrelation, and provide a computationally scalable alternative to covariance- and characteristic-function-based regularizers (Wu et al., 1 Jun 2026).

1. Motivation and conceptual foundations

A JEPA predicts the embedding of one image view from another. If $z_i$ denotes an embedding of view $i$ and $\mu_g$ is the mean embedding of global views, the prediction objective is

$$\mathcal{L}_{\mathrm{pred}} = \frac{1}{V}\sum_{i=1}^{V} \left\|\mu_g-z_i\right\|_2^2.$$

This objective encourages agreement between views but admits a collapsed solution in which all samples map to a constant vector. VISReg therefore supplements prediction with explicit distributional regularization.

The method is positioned relative to two principal precedents:

• VICReg decomposes regularization into invariance, variance, and covariance terms. Its covariance term discourages correlations between feature dimensions, but covariance constrains only second-order statistics. Distributions with identical means and covariances can still differ in skewness, kurtosis, multimodality, and other higher-order properties. Covariance computation has cost approximately $O(ND^2)$ for a batch of $N$ samples and feature dimension $D$.
• SIGReg, used in the LeJEPA framework, matches random one-dimensional projections of embeddings to a standard Gaussian through an Epps–Pulley characteristic-function statistic. SIGReg provides distributional control and linear dependence on feature dimension, but it does not explicitly decouple scale from shape and can exhibit vanishing gradients near embedding collapse.

VISReg addresses these limitations by treating the embedding distribution as a combination of independently controlled properties:

1. Center: the batch mean should approach zero.
2. Scale: each feature coordinate should have standard deviation one.
3. Shape: normalized random projections should match standard-Gaussian quantiles.
4. Invariance: embeddings of different views should agree.

The central methodological distinction is that scale and shape are not optimized through one undifferentiated distributional statistic. The variance term controls coordinate-wise magnitude, while the sketching term acts on variance-normalized embeddings.

2. Regularization objective

Let $Z\in\mathbb{R}^{N\times D}$ be a batch of embeddings, with batch mean

$\mu=\frac{1}{N}\sum_{i=1}^{N}z_i$

and centered representation

$\widehat Z=Z-\mu.$

The VISReg regularizer is

$i$0

The complete training objective interpolates between prediction and regularization:

$i$1

The center loss is

$i$2

with the implementation using a coordinate-wise mean-square form. It encourages the representation distribution to be centered at the origin.

For coordinate-wise standard deviations $i$3, the scale loss is

$i$4

The target standard deviation is one. Unlike the hinge-style variance term commonly associated with VICReg, this squared objective penalizes both standard deviations below and above the target.

The scale term supplies a direct anti-collapse signal. For a scalar standard deviation $i$5,

$i$6

As $i$7, the derivative approaches $i$8, so the scale objective continues to push the representation away from zero variance. The paper reports that VISReg and Barlow Twins retain a strong gradient as feature norm decreases, whereas the SIGReg gradient diminishes in the collapse regime.

3. Scale-normalized distributional sketching

VISReg separates the shape objective from coordinate-wise scale. The normalized representation is

$i$9

where $\mu_g$0 contains the coordinate-wise standard deviations, $\mu_g$1 denotes stop-gradient, and $\mu_g$2 is a numerical stabilizer.

The stop-gradient prevents the shape term from modifying feature scale through the normalization denominator. Consequently, the scale term changes magnitude, while the shape term operates on approximately scale-invariant features. This is not presented as an asymmetric collapse-prevention mechanism of the type used in BYOL or SimSiam; it is used to decouple optimization of scale from optimization of distributional shape.

Random projections

VISReg samples $\mu_g$3 normalized random directions,

$\mu_g$4

The projection associated with direction $\mu_g$5 is

$\mu_g$6

The Cramér–Wold/Radon result motivates this construction: a multivariate probability distribution is determined by all of its one-dimensional projections. VISReg approximates this infinite family using a finite set of random directions.

For each projected batch, the method sorts the scalar samples and compares them with fixed quantiles of a standard normal distribution,

$\mu_g$7

The shape loss is

$\mu_g$8

with the implementation using mean normalization over both projections and batch elements. Equivalently,

$\mu_g$9

This is a one-dimensional empirical Wasserstein matching procedure. In one dimension, the Wasserstein distance between equally weighted empirical distributions is obtained by sorting both samples and pairing equal ranks. The resulting objective constrains projected distributional properties beyond covariance, including skewness, heavy tails, and multimodality.

The default component weights are generally

$$\mathcal{L}_{\mathrm{pred}} = \frac{1}{V}\sum_{i=1}^{V} \left\|\mu_g-z_i\right\|_2^2.$$0

For low-quality, long-tailed, or effectively low-rank datasets, increasing the relative shape weight can improve performance.

4. Optimization, implementation, and computational complexity

A practical VISReg implementation proceeds by centering a batch, computing coordinate-wise standard deviations, normalizing with detached standard deviations, sampling random projection vectors, projecting the normalized embeddings, sorting each projection, and comparing the sorted values with fixed Gaussian quantiles.

For $$\mathcal{L}_{\mathrm{pred}} = \frac{1}{V}\sum_{i=1}^{V} \left\|\mu_g-z_i\right\|_2^2.$$1 samples, feature dimension $$\mathcal{L}_{\mathrm{pred}} = \frac{1}{V}\sum_{i=1}^{V} \left\|\mu_g-z_i\right\|_2^2.$$2, and $$\mathcal{L}_{\mathrm{pred}} = \frac{1}{V}\sum_{i=1}^{V} \left\|\mu_g-z_i\right\|_2^2.$$3 projections, the dominant operations have costs:

• Projection: $$\mathcal{L}_{\mathrm{pred}} = \frac{1}{V}\sum_{i=1}^{V} \left\|\mu_g-z_i\right\|_2^2.$$4.
• Sorting: $$\mathcal{L}_{\mathrm{pred}} = \frac{1}{V}\sum_{i=1}^{V} \left\|\mu_g-z_i\right\|_2^2.$$5.

The total regularization cost is therefore

$$\mathcal{L}_{\mathrm{pred}} = \frac{1}{V}\sum_{i=1}^{V} \left\|\mu_g-z_i\right\|_2^2.$$6

and is approximately $$\mathcal{L}_{\mathrm{pred}} = \frac{1}{V}\sum_{i=1}^{V} \left\|\mu_g-z_i\right\|_2^2.$$7 when $$\mathcal{L}_{\mathrm{pred}} = \frac{1}{V}\sum_{i=1}^{V} \left\|\mu_g-z_i\right\|_2^2.$$8. This avoids the covariance cost $$\mathcal{L}_{\mathrm{pred}} = \frac{1}{V}\sum_{i=1}^{V} \left\|\mu_g-z_i\right\|_2^2.$$9 associated with VICReg and the $O(ND^2)$0 batch dependence of kernel MMD methods such as KerJEPA.

The nominal linear scaling requires qualification because useful accuracy can require $O(ND^2)$1 to grow with $O(ND^2)$2. In some experiments, good performance required $O(ND^2)$3 for a constant $O(ND^2)$4, yielding an effective cost of $O(ND^2)$5. VISReg addresses this practically through distributed random projections: each GPU generates an independent subset of projections, allowing the total number of projections to increase while keeping per-device computation manageable.

The paper reports that with eight GPUs, accuracy gaps between small and large projection counts were substantially reduced. For a one-GPU comparison between $O(ND^2)$6 and $O(ND^2)$7, the reported gaps were $O(ND^2)$8 for SIGReg, $O(ND^2)$9 for SWD, and $N$0 for VISReg. With eight GPUs, the corresponding gaps fell to $N$1, $N$2, and $$0.22\%.</p> <p>The main ImageNet-1K models use a three-layer projection <a href="https://www.emergentmind.com/topics/multi-layer-perceptron-mlp-based-visual-models" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">MLP</a> with structure</p> <p>$$N$3

batch normalization, and GELU activations. The projection operates on concatenated class tokens from the last two backbone layers.

Model	Backbone	Learning rate	Projection dimension	Projections
VISReg-B	ViT-B/16	$N$4	$N$5	$N$6
VISReg-L	ViT-L/14	$N$7	$N$8	$N$9

ImageNet-1K training uses 400 epochs, AdamW, weight decay $D$0, bfloat16 mixed precision, five-epoch linear warmup, cosine decay, DINO-style multi-crop augmentation, four global crops, six local crops, 32 H100 GPUs, and effective batch size 512. VISReg-B and VISReg-L require approximately 1,120 and 2,060 GPU-hours, respectively.

5. Empirical performance

VISReg has been evaluated on ImageNet-1K, ImageNet-22K, ImageNet-LT, Galaxy10, ImageNette, standard transfer datasets, out-of-distribution datasets, ADE20K segmentation, and generative-model guidance.

Classification and transfer

On ImageNet-1K linear probing, reported results include:

Method	Linear-probe accuracy
VISReg-B/16	75.7%
VISReg-L/14	77.0%
LeJEPA-L/14	75.6%
DINO-B/16	78.2%
iBOT-B/16	79.8%
iBOT-L/16	81.0%
MoCoV3-B/16	76.7%

VISReg is not the strongest method on all in-domain ImageNet linear-probe comparisons. Its principal reported advantage is out-of-distribution generalization and transfer.

Average OOD accuracies include 70.19 for VISReg-B/16 and 70.63 for VISReg-L/14, compared with 69.56 for DINO-B/16, 69.64 for iBOT-B/16, and 69.46 for MoCoV3-B/16. VISReg-L/14 trained on ImageNet-22K reaches 72.94 average OOD accuracy, matching the reported 72.93 of DINOv2-L/14 trained on LVD-142M.

For fine-tuning, VISReg-B/16 reports:

Method	CIFAR-10	CIFAR-100	Flowers	ImageNet-1K	Galaxy10
DINO	99.1	91.7	98.8	82.8	86.6
VISReg	99.2	91.8	99.0	83.0	87.0

Long-tailed and low-rank data

On ImageNet-LT with ViT-S/8, VISReg reaches 32.11 overall accuracy, while increasing the shape weight yields 35.14. The increased shape weight improves many-, medium-, and few-shot categories. On Galaxy10, the corresponding scores are 80.51 for standard VISReg and 80.76 with increased shape weight.

The component ablation on ImageNette reports 91.82 for the full method, compared with 20.80 without scale, 33.42 without shape, and 91.41 without center. These results indicate that scale and shape terms are both central to the method, while the center term has a smaller effect on final accuracy but improves convergence speed.

Dense prediction and generative guidance

On ADE20K linear segmentation, VISReg obtains mIoU 30.16, compared with 31.69 for MoCoV3, 29.40 for DINO, 23.60 for MAE, and 21.99 for data2vec. VISReg is competitive but does not lead this comparison.

When used as guidance for a SiT-B/2 generator, VISReg yields IS 33.48, gFID 40.36, precision 51.38, and recall 61.26, compared with IS 33.47, gFID 41.15, precision 50.51, and recall 60.70 for DINO guidance.

6. Limitations, interpretation, and later planning applications

VISReg does not guarantee exact recovery of an isotropic Gaussian distribution because the Cramér–Wold characterization applies to all projection directions, whereas the method uses a finite random subset. Its approximation quality depends on batch size, projection count, feature dimension, and distributed sampling.

The shape loss has a batch-dependent residual floor. The paper reports an expected minimum behavior of approximately

$D$1

where $D$2 is batch size, while the scale-term floor is $D$3. Consequently, batch size acts partly as an implicit distribution-matching parameter. Larger batches provide finer empirical Gaussian approximation, whereas small batches produce noisier standard-deviation and quantile estimates.

Other limitations include hyperparameter sensitivity, the restriction to a standard isotropic Gaussian target, sorting overhead, and the absence of uniformly leading performance on ImageNet linear probing and dense prediction. Increasing the shape weight benefits long-tailed and low-rank settings but can reduce performance on ordinary ImageNette. No single projection dimension is optimal simultaneously for in-domain classification, OOD classification, and dense prediction.

A later study, “VIScore: Diagnosing Planning-Relevant Quality in Latent World Models” (Wu et al., 11 Aug 2026), examines VISReg in latent world models by replacing SIGReg with VISReg in the LeWorldModel architecture. The study finds that VISReg’s flexible component reweighting, while useful in self-supervised representation learning, does not consistently improve in-domain planning. Its principal OOD planning advantage emerges when larger batch sizes provide a more accurate finite-sample approximation to the Gaussian target. VISReg with batch size 512 obtains PushObj success rates of $D$4, $D$5, and $D$6 for horizons $D$7, compared with $D$8, $D$9, and $Z\in\mathbb{R}^{N\times D}$0 for LeWorldModel with SIGReg and batch size 128.

The same study introduces VIScore, or Veracity–Influence–Sobriety score, to evaluate the encoder, predictor, and planner jointly. It reports pooled Spearman correlation of 0.88 on development checkpoints and 0.91 on held-out checkpoints, exceeding straightness, physical-state probing, and raw empowerment in the reported settings. This result qualifies the interpretation of VISReg: a statistically regular latent space is not sufficient for planning success. Predictor rollout fidelity, action influence, and resistance to planner exploitation also matter.

VISReg can therefore be characterized as a distributional regularizer rather than a complete representation-quality guarantee. Its strongest methodological contribution is the explicit separation of coordinate scale from normalized distributional shape, combined with sliced-Wasserstein quantile matching and a non-vanishing anti-collapse variance gradient. Empirically, this produces strong OOD, transfer, long-tailed, and low-rank performance, while its effect on dense prediction and planning remains task-dependent.