SIGReg: Isotropic Gaussian Regularization

• SIGReg is a differentiable distribution-matching regularizer that encourages embeddings to follow N(0,I), preventing both complete and low-dimensional collapse in joint-embedding architectures.
• It uses random one-dimensional projections and the differentiable Epps–Pulley characteristic-function statistic to match high-dimensional distributions with computational cost that scales approximately linearly in batch size and embedding dimension.
• Researchers apply SIGReg alongside predictive, supervised, temporal, multimodal, and reinforcement-learning objectives, while variants such as Weak-SIGReg and Temporally Centered SIGReg adapt the method for cheaper optimization or structured latent dynamics.

SIGReg, or Sketched Isotropic Gaussian Regularization, is a differentiable distribution-matching regularizer designed to prevent complete and dimensional representation collapse in joint-embedding predictive architectures (JEPAs). Rather than enforcing only nonzero variance, decorrelation, or whitening, SIGReg encourages encoder embeddings to follow an isotropic Gaussian distribution, typically $\mathcal N(0,I_K)$. It approximates high-dimensional distribution matching through one-dimensional random projections and a scalar goodness-of-fit statistic, most commonly the Epps–Pulley characteristic-function statistic. Introduced in the LeJEPA framework, SIGReg has subsequently been applied to multimodal learning, supervised optimization, time-series representation learning, world models, reinforcement learning, language-model hidden states, and other settings (Balestriero et al., 11 Nov 2025).

1. Motivation and theoretical target

A JEPA learns an encoder $f_\theta$ whose embeddings of related views are predictive of one another. For views $x_{n,v}$ and $x_{n,v'}$, a predictive objective encourages

$f_\theta(x_{n,v})\approx f_\theta(x_{n,v'}).$

Prediction or invariance alone admits trivial solutions. Complete collapse occurs when every input is mapped to the same vector. Dimensional collapse occurs when embeddings vary but occupy only a low-dimensional subspace. Both solutions can yield low predictive loss while discarding information.

SIGReg imposes the stronger population-level condition

$Z=f_\theta(X)\sim\mathcal N(0,I_K),$

where $K$ is the embedding dimension. The target has zero mean, unit variance in every direction, identity covariance, and no preferred latent subspace. A point mass, a lower-dimensional distribution, or a non-Gaussian distribution with identity covariance fails to satisfy the full target condition.

The isotropic Gaussian is motivated as a task-agnostic representation geometry. For frozen embeddings $X\in\mathbb R^{N\times K}$ and a downstream target $y\in\mathbb R^N$, a ridge probe solves

$$\hat{\beta}=\arg\min_{\beta}\|y-X\beta\|_2^2+\lambda\|\beta\|_2^2.$$

If two representations have the same span and total energy, but one has anisotropic covariance with eigenvalues $f_\theta$0, anisotropy can increase downstream estimator bias when $f_\theta$1 and estimator variance when $f_\theta$2. In the unregularized case,

$f_\theta$3

For fixed total energy, convexity gives

$f_\theta$4

with equality only under isotropy. Thus isotropy minimizes worst-direction variance for linear probes.

The analysis extends to radial $f_\theta$5-nearest-neighbor and kernel-regression probes. Their integrated squared-bias criteria involve the Fisher information

$f_\theta$6

Among distributions with fixed covariance, Fisher information is minimized uniquely by a Gaussian; among covariance matrices satisfying fixed trace, determinant, Frobenius norm, or spectral-radius constraints, the minimum occurs at an isotropic covariance. For fixed covariance $f_\theta$7,

$f_\theta$8

with equality if and only if $f_\theta$9. Under a trace constraint,

$x_{n,v}$0

with equality at $x_{n,v}$1. The resulting claim is that, without knowledge of future tasks, an isotropic Gaussian provides a well-conditioned geometry with low downstream bias and variance (Balestriero et al., 11 Nov 2025).

This target is not universally task-optimal by logical necessity. The theoretical argument concerns broad classes of downstream probes under specified covariance and smoothness assumptions. In multi-task world models, a single Gaussian marginal can instead compress task-dependent cluster separation. Temporally Centered SIGReg addresses this issue by applying the regularizer to residuals rather than to the full latent marginal (Liu et al., 29 Jul 2026).

2. Random-projection distribution matching

Directly comparing two $x_{n,v}$2-dimensional empirical distributions is expensive. SIGReg uses the Cramér–Wold principle: two multivariate distributions are equal if and only if all one-dimensional projections are equal in distribution,

$x_{n,v}$3

For the encoder distribution $x_{n,v}$4 and Gaussian target $x_{n,v}$5, a unit direction $x_{n,v}$6 produces projected distributions

$x_{n,v}$7

Because $x_{n,v}$8 is isotropic Gaussian,

$x_{n,v}$9

for every unit vector $x_{n,v'}$0. The multivariate problem therefore becomes a collection of scalar standard-normality tests.

For a finite set of directions $x_{n,v'}$1, SIGReg averages the directional statistics:

$x_{n,v'}$2

A maximum over directions would be statistically natural but would provide sparse gradients, because only the worst direction would receive gradient. Averaging produces denser optimization signals.

Directions are normally independently sampled uniformly from the sphere. A practical construction samples

$x_{n,v'}$3

Normalized Gaussian vectors are uniformly distributed on $x_{n,v'}$4. Low-discrepancy alternatives, including Sobol sequences transformed through the inverse standard-normal CDF and normalized, have also been discussed, although reported implementations commonly use normalized Gaussian directions.

The finite projection set does not test every direction at each optimization step. For an embedding density with Sobolev smoothness $x_{n,v'}$5, the reported worst-case rate is

$x_{n,v'}$6

The rate is slow in high dimension, but directions are resampled during training. The cumulative number of directions encountered across optimization can therefore be much larger than the number used in one minibatch. Synthetic experiments reported that resampling as few as $x_{n,v'}$7 directions can outperform a fixed set containing thousands of directions in tested settings (Balestriero et al., 11 Nov 2025).

A related exact-analysis result shows that an infinite-direction Epps–Pulley objective can be written as a full-dimensional MMD after analytically integrating over directions. This observation has been developed for hyperspherical regularization, where deterministic MMD, KSD, and KDE-based KL objectives remove projection-sampling variance (Nicollier et al., 16 Jun 2026).

3. Epps–Pulley characteristic-function statistic

SIGReg can use any differentiable one-dimensional goodness-of-fit statistic, but the recommended statistic is Epps–Pulley because it is differentiable, bounded, reduction-friendly, suitable for distributed training, nonparametric, and sensitive to more than finitely many moments (Balestriero et al., 11 Nov 2025).

For projected samples $x_{n,v'}$8, the empirical characteristic function is

$x_{n,v'}$9

The characteristic function of the target standard normal is

$f_\theta(x_{n,v})\approx f_\theta(x_{n,v'}).$0

The Epps–Pulley statistic is

$f_\theta(x_{n,v})\approx f_\theta(x_{n,v'}).$1

with a Gaussian weighting function, commonly parameterized as

$f_\theta(x_{n,v})\approx f_\theta(x_{n,v'}).$2

The statistic compares empirical and target characteristic functions across frequencies. The complex exponential has unit modulus, $f_\theta(x_{n,v})\approx f_\theta(x_{n,v'}).$3, so the objective avoids the exploding gradients associated with high-order moment matching.

Discrepancies in mean, variance, skewness, heavy tails, multimodality, or dependence revealed by a projection increase the statistic. Matching only finitely many moments does not identify a distribution:

$f_\theta(x_{n,v})\approx f_\theta(x_{n,v'}).$4

can be minimized by non-Gaussian distributions that match the selected moments. Higher moments improve identifiability but can produce unstable gradients, whose Monte Carlo variance can grow as

$f_\theta(x_{n,v})\approx f_\theta(x_{n,v'}).$5

CDF-based tests such as Cramér–von Mises and Anderson–Darling require sorting, which is expensive and synchronization-heavy in distributed training and requires differentiable sorting relaxations. Epps–Pulley requires averages of complex exponentials and can be aggregated with all_reduce.

The reported stability bounds include

$f_\theta(x_{n,v})\approx f_\theta(x_{n,v'}).$6

and

$f_\theta(x_{n,v})\approx f_\theta(x_{n,v'}).$7

for a constant $f_\theta(x_{n,v})\approx f_\theta(x_{n,v'}).$8. The practical interpretation is that gradients and curvature with respect to projected samples are uniformly bounded, although the precise parameter dependence is expressed inconsistently in different parts of the manuscript (Balestriero et al., 11 Nov 2025).

Some later studies identify a limitation near complete collapse: the Epps–Pulley discrepancy can remain nonzero for a point-mass representation while its gradient diminishes as the projected samples become nearly identical. VISReg addresses this reported weakness by retaining an explicit variance pathway and replacing characteristic-function matching with sliced $f_\theta(x_{n,v})\approx f_\theta(x_{n,v'}).$9-Wasserstein matching for the shape component (Wu et al., 1 Jun 2026).

4. Numerical approximation and computational properties

The Fourier integral is approximated by deterministic quadrature. For a grid $Z=f_\theta(X)\sim\mathcal N(0,I_K),$0 and quadrature weights $Z=f_\theta(X)\sim\mathcal N(0,I_K),$1,

$Z=f_\theta(X)\sim\mathcal N(0,I_K),$2

A commonly reported configuration uses an integration interval of $Z=f_\theta(X)\sim\mathcal N(0,I_K),$3, $Z=f_\theta(X)\sim\mathcal N(0,I_K),$4 quadrature points, and $Z=f_\theta(X)\sim\mathcal N(0,I_K),$5 or $Z=f_\theta(X)\sim\mathcal N(0,I_K),$6 projections. Symmetry of the integrand can be exploited.

The computational structure is approximately:

• projection: $Z=f_\theta(X)\sim\mathcal N(0,I_K),$7;
• ECF evaluation: $Z=f_\theta(X)\sim\mathcal N(0,I_K),$8;
• storage: linear in the working tensors, without pairwise $Z=f_\theta(X)\sim\mathcal N(0,I_K),$9 distances.

With fixed or modest $K$0, SIGReg scales linearly in batch size and embedding dimension rather than quadratically in the batch or embedding dimension. Representative Tesla V100 timings reported for $K$1 include $K$2 ms for $K$3, $K$4 ms for $K$5, $K$6 ms for $K$7, $K$8 ms for $K$9, and $X\in\mathbb R^{N\times K}$0 ms for $X\in\mathbb R^{N\times K}$1 (Balestriero et al., 11 Nov 2025).

The empirical characteristic function is a finite-sample estimator. The associated V-statistic introduces $X\in\mathbb R^{N\times K}$2 bias in both the loss and its gradient. The reported analysis gives

$X\in\mathbb R^{N\times K}$3

An unbiased U-statistic correction or sample splitting is possible in principle but is not used. The authors report that the bias is small even for batches as small as $X\in\mathbb R^{N\times K}$4.

Distributed training requires synchronized projection directions. Implementations seed the random generator with global_step, so all workers use the same directions. ECF averages can be combined using all_reduce; global sorting and pairwise distance matrices are unnecessary. In one reported implementation, the core method requires approximately fifty lines of PyTorch code, excluding model definitions, data loading, and optimization.

SIGReg’s practical cost is not only projection. Frequency quadrature, projection count, embedding dimension, and the axis over which samples are collected all affect memory and runtime. In symbolic-music experiments, $X\in\mathbb R^{N\times K}$5 slices are divided into chunks of $X\in\mathbb R^{N\times K}$6 to reduce peak GPU memory (Hawley, 16 Jul 2026). In autoregressive language-model experiments, the main hidden-state configuration uses batch size $X\in\mathbb R^{N\times K}$7, $X\in\mathbb R^{N\times K}$8 directions, and $X\in\mathbb R^{N\times K}$9 quadrature knots (Sun et al., 19 Jul 2026).

5. Integration into predictive and representation-learning objectives

SIGReg alone shapes an embedding distribution but does not ensure that the representation preserves task-relevant information or that related views agree. It is therefore combined with a predictive or invariance term.

In LeJEPA, the objective combines SIGReg with a predictive loss. For $y\in\mathbb R^N$0 global views and $y\in\mathbb R^N$1 local views, with $y\in\mathbb R^N$2, the mean global embedding is

$y\in\mathbb R^N$3

The prediction loss can be written as

$y\in\mathbb R^N$4

The full objective is

$y\in\mathbb R^N$5

with $y\in\mathbb R^N$6 as the principal loss trade-off parameter. A practical default uses two global views, six local views, eight total views, $y\in\mathbb R^N$7, batch size at least $y\in\mathbb R^N$8, $y\in\mathbb R^N$9 slices, and $$\hat{\beta}=\arg\min_{\beta}\|y-X\beta\|_2^2+\lambda\|\beta\|_2^2.$$0 integration points over $$\hat{\beta}=\arg\min_{\beta}\|y-X\beta\|_2^2+\lambda\|\beta\|_2^2.$$1 (Balestriero et al., 11 Nov 2025).

The anti-collapse division of labor is:

• the predictive term makes semantically related views agree;
• SIGReg prevents the agreement from degenerating into a constant or low-rank representation.

There is no inherent requirement for stop-gradient, a teacher–student architecture, EMA targets, negative samples, prototypes, explicit whitening layers, or special anti-collapse schedules in the basic LeJEPA formulation. Ordinary optimizer learning-rate schedules may still be used in experiments.

SIGReg has been adapted to multiple application-specific objectives:

• NOVA applies SIGReg jointly to predicted image embeddings and projected text embeddings while using MSE alignment to a frozen ClinicalBERT text encoder. The objective is $$\hat{\beta}=\arg\min_{\beta}\|y-X\beta\|_2^2+\lambda\|\beta\|_2^2.$$2 with $$\hat{\beta}=\arg\min_{\beta}\|y-X\beta\|_2^2+\lambda\|\beta\|_2^2.$$3 (Kuhn et al., 31 Jan 2026).
• Le MuMo JEPA applies SIGReg to the projected joint multimodal CLS embedding after RGB and depth or thermal information has passed through fusion tokens. A three-pass variant additionally regularizes joint, RGB-only, and companion-only CLS embeddings (Cornelissen et al., 25 Mar 2026).
• HQ-JEPA applies SIGReg after projection, bidirectional cross-modal attention, and fusion of Sentinel-1 and Sentinel-2 representations. Its full objective combines token prediction, cross-modal alignment, SIGReg, and Fidelity Quantum Similarity (Hossain et al., 29 May 2026).
• AV-JEPA applies SIGReg to the shared projected embeddings of two global audio-video views, one audio-only local view, and one video-only local view. Random directions are resampled at every optimization step (Robson et al., 1 Jul 2026).
• Mine-JEPA uses SIGReg with four views and a $$\hat{\beta}=\arg\min_{\beta}\|y-X\beta\|_2^2+\lambda\|\beta\|_2^2.$$4-dimensional projector for side-scan sonar, avoiding negative samples, momentum teachers, EMA targets, and stop-gradient mechanisms in the stated formulation (Kwon et al., 1 Apr 2026).
• LeNEPA applies SIGReg temporally to projected token representations within each individual time series, rather than only across the batch. It uses layers $$\hat{\beta}=\arg\min_{\beta}\|y-X\beta\|_2^2+\lambda\|\beta\|_2^2.$$5 and a projected dimension of $$\hat{\beta}=\arg\min_{\beta}\|y-X\beta\|_2^2+\lambda\|\beta\|_2^2.$$6 (Chemeris et al., 1 Jul 2026).
• MIDI-RAE-JEPA applies SIGReg to student embeddings at coarser Swin hierarchy levels, using $$\hat{\beta}=\arg\min_{\beta}\|y-X\beta\|_2^2+\lambda\|\beta\|_2^2.$$7 random projections. The teacher is EMA-updated separately for equivariance and masked embedding prediction (Hawley, 16 Jul 2026).
• LeWorldModel applies SIGReg independently to per-camera CLS representations at every temporal position. Temporally Centered SIGReg instead applies it to residuals $$\hat{\beta}=\arg\min_{\beta}\|y-X\beta\|_2^2+\lambda\|\beta\|_2^2.$$8, preserving task-dependent latent means while retaining anti-collapse pressure (Liu et al., 29 Jul 2026).
• Visual quadruped navigation applies a covariance-style SIGReg formulation directly to a $$\hat{\beta}=\arg\min_{\beta}\|y-X\beta\|_2^2+\lambda\|\beta\|_2^2.$$9-dimensional recurrent policy state, together with an auxiliary next-state prediction loss (Zhu et al., 20 Jul 2026).
• Autoregressive language modeling adds SIGReg to next-token cross-entropy. Hidden-state regularization reduces hidden-state pairwise-cosine anisotropy, whereas direct regularization of keys and values is required to substantially alter KV-cache geometry (Sun et al., 19 Jul 2026).

These uses differ in whether they employ full characteristic-function SIGReg, a covariance-only approximation, or a task-specific population such as temporal residuals. The term “SIGReg” therefore covers a family of related implementations, and the exact estimator, regularized tensor, sample axis, normalization, and projection scheme must be specified for reproducibility.

6. Empirical findings and applications

The original LeJEPA validation covers more than ten datasets and more than sixty architectures. On ImageNet-1K with frozen-backbone linear evaluation, reported examples include approximately $f_\theta$00 for ViT-Large, $f_\theta$01 for ConvNeXtV2-Huge, and $f_\theta$02 for ViT-H/14. On ImageNet-100, performance is broad across $f_\theta$03 and view counts, with a recommended $f_\theta$04 near $f_\theta$05–$f_\theta$06. Architecture experiments include ResNets, ViTs, ConvNeXts, Swin, MaxViT, and other ConvNet and Transformer families (Balestriero et al., 11 Nov 2025).

In-domain applications demonstrate the use of SIGReg under restricted data and domain shift. Mine-JEPA pretrains on 1,170 side-scan-sonar images and approximately 153,000 extracted real patches, with an optional approximately 256,000 synthetic patches. It reaches binary mine-versus-non-mine F1 of $f_\theta$07 and three-class macro-F1 of $f_\theta$08 with synthetic data, compared with fine-tuned DINOv3 values of $f_\theta$09 and $f_\theta$10, respectively (Kwon et al., 1 Apr 2026).

Multimodal applications use SIGReg to regularize a shared geometry rather than to force token-wise equality. Le MuMo JEPA reports Waymo patch-probe results of $f_\theta$11 XY mAP, $f_\theta$12 depth MAE, $f_\theta$13 XZ mAP, and $f_\theta$14 segmentation mIoU. Its default joint-CLS SIGReg configuration outperforms the focused pruned-fusion VICReg comparison on all four reported metrics (Cornelissen et al., 25 Mar 2026). HQ-JEPA reports improvements over token-prediction-only configurations when SIGReg is added, including gains in linear probing and fine-tuning across remote-sensing datasets (Hossain et al., 29 May 2026). NOVA reports zero-shot chest X-ray classification performance with ViT-Small and ViT-Base of $f_\theta$15 and $f_\theta$16 average AUC, respectively, although its manuscript does not provide a complete SIGReg implementation specification (Kuhn et al., 31 Jan 2026).

Weak-SIGReg repurposes the principle for supervised optimization. It projects hidden representations $f_\theta$17 into a $f_\theta$18-dimensional sketch space, computes the centered covariance $f_\theta$19, and penalizes

$f_\theta$20

On CIFAR-100, Weak-SIGReg raises a collapsing AdamW-trained ViT from $f_\theta$21 to $f_\theta$22 top-1 accuracy, compared with $f_\theta$23 for Strong-SIGReg. It also raises a six-layer vanilla MLP trained with pure SGD from $f_\theta$24 to $$42.17\% (<a href="/papers/2603.05924" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Akbar, 6 Mar 2026</a>). This variant is weaker than characteristic-function matching because identity covariance does not imply Gaussianity, but it is cheaper and can be effective as an optimization stabilizer.</p> <p>Several studies examine alternatives and limitations. VISReg replaces the Epps–Pulley statistic with a sliced$$f_\theta$25-Wasserstein shape objective while retaining an explicit variance term and a center penalty. Its stated motivation is to separate scale from shape and preserve gradients near collapse (Wu et al., 1 Jun 2026). UR-JEPA replaces the full-dimensional Gaussian prior with multiscale local uniform-rectifiability regularization. On ImageNet-10 it reports $f_\theta$26, compared with $f_\theta$27 for LeJEPA, while producing a global PCA spectrum with a four- to five-order-of-magnitude drop after approximately 20–25 active dimensions rather than LeJEPA’s near-flat spectrum (Le, 31 May 2026). These results frame isotropic Gaussianity as a modeling choice rather than an invariant requirement.

Theoretical interpretations have extended SIGReg beyond anti-collapse. One information-bottleneck account treats $f_\theta$28 as a Euclidean relaxation of a maximum-entropy flat Dirichlet distribution on the predictive simplex (Domb, 30 Apr 2026). Another interprets SIGReg as closing an entropy-estimation gap in an Active Inference variational free energy under a constant-noise encoder model and successful population-level Gaussian enforcement (Arnez et al., 15 Jul 2026). A dissipative-learning framework interprets SIGReg as regulating spatial belief precision toward an isotropic Gaussian, but explicitly presents this as a theoretical interpretation without SIGReg experiments or a fully specified algorithm (Caraffa, 25 Jan 2026).

7. Limitations, variants, and open issues

SIGReg’s theoretical distribution-matching guarantee requires all directions and frequencies, whereas implementations use finite projection sets, finite minibatches, truncated frequency intervals, and numerical quadrature. Finite projection coverage relies on random sampling, resampling, and smoothness assumptions. The Epps–Pulley V-statistic also has $f_\theta$29 loss and gradient bias.

The isotropic Gaussian target is task-agnostic rather than universally task-optimal. It can conflict with structured, multimodal, or manifold-like representations. UR-JEPA argues that a full-dimensional Gaussian is in tension with the manifold hypothesis (Le, 31 May 2026). Temporally centered residual regularization offers a separate response for multi-task world models: it preserves low-frequency task and context structure while regularizing local temporal variation (Liu et al., 29 Jul 2026).

SIGReg does not determine semantic sufficiency. A representation can have an approximately Gaussian marginal while discarding task-relevant information or encoding arbitrary distribution-preserving transformations. The predictive, invariance, supervised, or control objective remains necessary. Poorly chosen views, augmentations, temporal neighborhoods, or modality pairings can still produce poor representations.

The exact name is also used for mathematically distinct variants. Full SIGReg uses projected characteristic-function matching. Weak-SIGReg uses sketched covariance matching. The navigation application defines a variance-and-correlation penalty analogous to VICReg or Barlow Twins rather than the full Epps–Pulley objective (Zhu et al., 20 Jul 2026). Some application papers omit the frequency grid, projection count, normalization, weighting function, or estimator details. Consequently, reporting only the label “SIGReg” is insufficient for exact replication.

Finally, SIGReg does not guarantee improved downstream performance, optimization stability, quantization, or physical efficiency in every regime. Direct KV-cache regularization improves coarse group-free quantization in a 110M LLM, but its advantage disappears under the tested KIVI-style combination of token-local grouping, mixed K/V scaling, and zero-points (Sun et al., 19 Jul 2026). The strongest established role of SIGReg is narrower: it supplies a principled, differentiable, distribution-level anti-collapse constraint that can replace or complement heuristic mechanisms when combined with an objective that preserves predictive or task-relevant structure.