---
title: 'Rectified LpJEPA: Sparse, Nonnegative JEPA'
url: https://www.emergentmind.com/topics/rectified-lpjepa
type: topic
---

# Rectified LpJEPA: Sparse, Nonnegative JEPA

Rectified LpJEPA is a self-supervised learning framework that extends Joint-Embedding Predictive Architectures (JEPAs) to produce sparse, nonnegative, and view-invariant representations by introducing a novel rectified distribution matching regularization. This approach addresses the limitations of Gaussian-based JEPA regularization by explicitly controlling sparsity through a rectified generalized Gaussian prior, resulting in feature distributions capable of representing efficient, interpretable, and robust latent codes while maintaining competitive performance across canonical vision benchmarks [2602.01456].

## 1. Foundations of JEPA and Projection-Based Distribution Matching

JEPAs learn representations by predicting one randomized augmentation of an input from another within the latent space, entirely avoiding pixel-level reconstruction. Given augmentations $x, x' \sim \mathbb P_{x,x'}$ and a shared encoder $f_\theta$, JEPAs generate embeddings $z = f_\theta(x)$ and $z' = f_\theta(x')$. View invariance is promoted via a penalty $\|z-z'\|_2^2$, but this term alone causes representational collapse (embeddings become constant). 

To mitigate collapse, recent JEPA variants, notably LeJEPA [1], employ distribution matching as a regularizer: they align all one-dimensional projections $c^\top z$ with a univariate reference distribution using random projection vectors $c \in \mathbb S^{d-1}$. The process leverages Cramér–Wold theorem guarantees, practically enforcing high-entropy, isotropic latent laws via sliced one-dimensional two-sample tests.

## 2. Motivation: Limitations of Gaussian Latent Targets in Sparsity

Traditional Gaussian priors in JEPA settings maximize entropy under quadratic (energy) constraint, resulting in latent embeddings that are dense and sign-symmetric. However, sparse nonnegative codes have proven important for efficiency, interpretability, and robustness—motivated by findings in neuroscience, compressed sensing, and ReLU-based deep models. Gaussian targets preclude exact zeros and nonnegativity, making them suboptimal for learning efficient sparse representations [2602.01456].

## 3. Rectified Generalized Gaussian (RGG) Distribution: Theory and Parameterization

To directly encode sparsity and nonnegativity, Rectified LpJEPA introduces the rectified generalized Gaussian family $\mathcal{RGN}_p(\mu,\sigma)$. Each $X \sim \mathcal{RGN}_p$ is given by $X = \max\{0, G\}$, where $G \sim \mathcal{GN}_p(\mu,\sigma)$ and the density of $\mathcal{GN}_p$ is
$$
f_{\mathcal{GN}_p}(g) = \frac{p^{1-1/p}}{2\,\sigma\,\Gamma(1/p)} \exp\left(-\frac{|g-\mu|^p}{p\,\sigma^p}\right), \quad g \in \mathbb R.
$$
$\mathcal{RGN}_p$ is a mixture, with a point mass at zero (Dirac) and a truncated generalized Gaussian over $(0,\infty)$. The explicit control of expected sparsity arises from the closed form of the expected $\ell_0$ norm:
$$
\mathbb E[\|X\|_0] = \Pr[X \neq 0] = \Phi_{\mathcal{GN}_p(0,1)} \left( \frac{\mu}{\sigma} \right) = \frac12\left(1 + \mathrm{sgn}(\frac{\mu}{\sigma})\,P\left(\frac1p, \frac{|\mu/\sigma|^p}{p}\right) \right)
$$
where $P(s, t)$ is the regularized Gamma function. This formulation enables explicit, independent adjustment of both the moment ($p$) and the sparsity ($\mu/\sigma$). The rectified case strictly generalizes Gaussians, which are recovered when $p=2$ and $\mu/\sigma \to +\infty$.

## 4. Rectified Distribution Matching Regularization (RDMReg)

RDMReg introduces the RGG prior into the JEPA training objective via a sliced two-sample matching loss. The encoder and projection network output $z = \mathrm{ReLU}(f_\theta(x))$ (imposing nonnegativity), and synthetic targets $y \sim \prod_{i=1}^d \mathcal{RGN}_p(\mu,\sigma)$. For each of $N$ random projections $c_i \in \mathbb S^{d-1}$, the slices $Z_j = c_i^\top z_j$ and $Y_j = c_i^\top y_j$ are compared batchwise using the empirical sliced-2-Wasserstein distance,
$$
\mathcal L_{\mathrm{RDMReg}} = \mathbb{E}_{\{c_i\}}\left[\frac{1}{B}\|Z^{\uparrow} - Y^{\uparrow}\|_2^2\right],
$$
where $(\cdot)^{\uparrow}$ denotes sorting. The overall loss combines view-invariance and distribution matching:
$$
\min_{\theta} \mathbb{E}_{x,x'} \|z-z'\|_2^2 + \lambda_{\rm RDM} \left( \mathcal{L}_{\rm RDMReg}(z, y) + \mathcal{L}_{\rm RDMReg}(z', y) \right).
$$
This critically makes use of nonparametric sliced-matching, as the RGG is not closed under arbitrary projections and thus precludes closed-form parametric matching.

## 5. Theoretical Properties and Sparsity–Performance Trade-off

Rectified LpJEPA achieves maximum-entropy latent distributions up to rescaling under fixed sparsity and moment constraints by invoking Rényi’s information dimension $d(\xi)$ and dimension-appropriate entropies. For $X \sim \mathcal{RGN}_p$,
$$
d(X) = \Pr[X > 0], \quad \mathbb{H}_d(X) = d(X)\, \mathbb{H}_1( \mathcal{TGN}_p(\mu, \sigma) ) + \mathbb{H}_0 \left[ \mathbbm{1}_{(0, \infty)}(X) \right]
$$
and $\mathcal{TGN}_p$ maximizes entropy under an $\ell_p$ moment. The mechanism produces a provable trade-off: decreasing $\mu/\sigma$ or $p$ increases sparsity (decreases $d(X)$), yielding lower information dimension and adjusted $d$-entropy. In the fully dense (non-rectified) limit, this reduces to the classical Gaussian maximum entropy principle. Sliced matching via Wasserstein distance is necessary, as RGG distributions are not closed under linear combinations.

## 6. Architecture and Training Procedures

The Rectified LpJEPA implementation comprises:

- Encoder $f_{\theta_1}$ (ResNet-50 or ViT-Small)
- Projector $f_{\theta_2}$, a 2–3-layer MLP (hidden dim 2048, output $\mathbb{R}^d$)
- ReLU nonlinearity enforcing nonnegativity and sparsity: $z = \mathrm{ReLU}(f_{\theta_2}(f_{\theta_1}(x)))$
- A sampling block drawing random directions $c_i \in \mathbb S^{d-1}$ and optionally using empirical covariance eigenvectors for accelerated decorrelation

Hyperparameterization outlines:
- Pretraining on ImageNet-100, 1000 epochs, batch 128, LARS or AdamW optimizer
- $\lambda_{\rm sim} = 25.0$, $\lambda_{\rm RDM} = 125.0$
- $N = 8192$ projections for sliced Wasserstein distance
- Generalized Gaussian scale $\sigma_{\rm GN} := (\Gamma(1/p)/\Gamma(3/p))^{1/2} p^{-1/p}$ fixes pre-ReLU variance; $\sigma_{\rm RGN}$ (post-ReLU variance) as alternative
- Standard image augmentations: crop, flip, color jitter, blur, solarize
- Single-GPU, mixed-precision, training duration 2–3 days for 1000 epochs

Ablation confirms the necessity of rectifying both feature and target distributions, and shows that mixing eigenvector and random projections accelerates decorrelation and convergence.

## 7. Empirical Performance and Ablations

Rectified LpJEPA demonstrates high sparsity–performance flexibility and competitive accuracy across benchmarks:

- On ImageNet-100 (ResNet-50, linear probe): encoder top-1 ≈ 84–85 %; projector top-1 ≈ 80 %; $\ell_0$ sparsity efficiently tunable between 2 % and 0 % zeros, yielding a full sparsity–performance Pareto frontier
- In contrast, LeJEPA (Gaussian) yields no exact zeros (always dense)
- On CIFAR-100, empirical $\ell_0$ closely tracks the theoretical $\mathbb{E}[\|x\|_0]$; accuracy degrades only when sparsity exceeds 95 % zeros
- HSIC independence metrics show Rectified LpJEPA achieves lower higher-order statistical dependence than VICReg or NVICReg
- Ablations reveal Rectified LpJEPA outperforms or matches VICReg/LeJEPA for both small/large projector dimension, and both $\sigma_{\rm GN}$ and $\sigma_{\rm RGN}$ deliver similar trade-offs
- Direct RDMReg matching significantly outperforms continuous-mapping approaches; joint rectification is essential for true sparsity
- Transfer learning across downstream classification tasks retains competitive accuracy, with independent control of sparsity levels

## 8. Implications, Limitations, and Future Directions

Rectified LpJEPA generalizes isotropic Gaussian priors to a broad class of sparse, nonnegative distributions, maintaining strong collapse-prevention and predictive-view invariance properties central to JEPA. The explicit control of sparsity under fixed moment constraint produces a continuum of operational points for sparsity–performance trade-offs, enabling learning of factorial, maximum-entropy, and interpretable sparse codes.

Proposed future research directions include:
- Tighter theoretical bounds on sliced-matching sample complexity for RGG
- Exploration of alternative two-sample tests (e.g., MMD, energy distance)
- Extensions to structured sparsity (group, top-$k$)
- Application to domains demanding structured sparse codes (e.g., RL, multimodal)
- Meta-learning of hyperparameters $(\mu, p)$ for dataset-adaptive priors
- Investigating connections between RGG-induced sparsity and neural coding in biological networks

A plausible implication is that the mechanisms encoded in Rectified LpJEPA could inform both artificial and theoretical models where efficient, sparse, high-entropy representations are critical [2602.01456].

Source: https://www.emergentmind.com/topics/rectified-lpjepa