---
title: Rectified Distribution Matching Reg.
url: https://www.emergentmind.com/topics/rectified-distribution-matching-regularization-rdmreg
type: topic
---

# Rectified Distribution Matching Reg.

Rectified Distribution Matching Regularization (RDMReg) is a regularization technique designed to induce explicit $\ell_0$-controlled sparsity while maintaining maximum-entropy properties in representations learned by Joint-Embedding Predictive Architectures (JEPAs). RDMReg aligns learned non-negative representations to samples from a Rectified Generalized Gaussian (RGG) prior using a sliced two-sample distribution-matching loss. This method enables precise tuning of expected sparsity and generalizes prior Gaussian-based regularizations by introducing rectification, thus producing representations that are sparse, non-negative, and information-rich under $\ell_p$ norm constraints [2602.01456].

## 1. Mathematical Formulation

RDMReg centers on matching representations to the RGG distribution. For $p > 0$, mean $\mu \in \mathbb{R}$, and scale $\sigma > 0$, the univariate RGG, denoted $\mathrm{RGN}_p(\mu,\sigma)$, is defined as a mixture: a point mass at zero with weight $\Phi_{\mathrm{GN}_p(0,1)}(-\mu/\sigma)$ (the CDF of the unrectified generalized Gaussian at $-\mu/\sigma$) and a truncated generalized Gaussian on $(0,\infty)$. The density for $X \sim \mathrm{RGN}_p(\mu,\sigma)$ with respect to $\nu := \delta_0 + \text{Lebesgue}$ is:
\[
f_{\mathrm{RGN}_p(\mu,\sigma)}(x) =
\underbrace{\Phi_{\mathrm{GN}_p(0,1)}\!\bigl(-\tfrac{\mu}{\sigma}\bigr)}_{\Pr[X=0]}\delta_0(x)
+
\underbrace{\frac{p^{1-1/p}}{2\,\sigma\,\Gamma(1/p)} \exp\!\Bigl(-\tfrac{|x-\mu|^p}{p\,\sigma^p}\Bigr)}_{\text{Truncated Gen. Gaussian on $(0,\infty)$}}.
\]
For independent RGG coordinates $\mathbf{x} \in \mathbb{R}^d$, the expected $\ell_0$ sparsity is $\mathbb{E}[\|\mathbf{x}\|_0] = d \,\Phi_{\mathrm{GN}_p(0,1)}(\mu/\sigma)$, allowing direct control over the fraction of nonzero activations by tuning $\mu$.

## 2. Distribution Matching Objective

Given encoder-projector $f_\theta$, two augmented views $v, v'$, and non-negative representations $\mathbf{z} = \mathrm{ReLU}(f_\theta(v))$, the RDMReg loss matches the feature distribution to the RGG prior using a sliced Wasserstein approach. For random projection $c \sim \mathrm{Unif}(\mathbb{S}^{d-1})$:
- Project both learned and RGG target features: $\mathbf{Z}c,\, \mathbf{Y}c \in \mathbb{R}^B$.
- Compute sliced 2-Wasserstein:
  \[
  \mathcal{D}(\mathbf{Z}c \Vert \mathbf{Y}c) = \frac{1}{B} \|(\mathbf{Z}c)^\uparrow - (\mathbf{Y}c)^\uparrow\|_2^2,
  \]
  where ${}^\uparrow$ denotes sorting.

The regularization is the expectation over projections and view-pairs:
\[
L_{\mathrm{RDMReg}}(\theta) = \mathbb{E}_{v, v'}\Bigl[ \mathbb{E}_c[ \mathcal{D}(\mathbf{Z}c\Vert\mathbf{Y}c ) + \mathcal{D}(\mathbf{Z}'c\Vert\mathbf{Y}c ) ] \Bigr].
\]
The JEPA loss becomes
\[
L(\theta) = \mathbb{E}_{v,v'}[\|\mathbf{Z} - \mathbf{Z}'\|_2^2] + \lambda_{\mathrm{RDM}} L_{\mathrm{RDMReg}}(\theta).
\]

## 3. Maximum-Entropy and Sparsity Guarantees

Among all distributions on $S \subset \mathbb{R}^d$ with fixed expected $\ell_p^p$ norm, the truncated product Generalized Gaussian achieves maximum differential entropy. For $S=(0,\infty)^d$, each coordinate is a positive-orthant truncated generalized Gaussian. The RGG, while not absolutely continuous (due to its atom at zero), retains this maximum-entropy property on its support.

The effective entropy is characterized via the Rényi information dimension. Theorem 3.8 provides that the information dimension of an RGG distributed vector is $D \cdot \Phi_{\mathrm{GN}_p(0,1)}(\mu/\sigma)$, where $D$ denotes ambient dimension. The entropy decomposes into a Bernoulli ($\ell_0$) term plus the entropy of the truncated-GG on the active coordinates, enabling explicit $\ell_0$ control without sacrificing information content.

## 4. Algorithmic Implementation

The implementation of RDMReg within JEPA consists of the following steps:

1. **Forward computation**: Given batch size $B$ of view pairs $(v_i, v'_i)$, compute $\mathbf{Z} = \mathrm{ReLU}(f_\theta(V)) \in \mathbb{R}^{B \times d}$, $\mathbf{Z}' = \mathrm{ReLU}(f_\theta(V'))$.
2. **Invariance loss**: $L_{\mathrm{inv}} = \frac{1}{B}\sum_i \|\mathbf{Z}_i - \mathbf{Z}_i'\|^2$.
3. **Sample RGG targets**: Draw $\mathbf{Y} \in \mathbb{R}^{B \times d}$ by sampling i.i.d. $\mathrm{GN}_p(\mu, \sigma)$ then applying $x \mapsto \max(0,x)$.
4. **Sliced Wasserstein regularization**: Sample projection directions $c_j \sim \mathrm{Unif}(\mathbb{S}^{d-1})$ (for $N$ projections). For each, compute $\mathcal{D}_j$ for both $\mathbf{Z}$ and $\mathbf{Z}'$, average to get $L_{\mathrm{RDMReg}}$.
5. **Backpropagation**: Update via $L = L_{\mathrm{inv}} + \lambda_{\mathrm{RDM}} L_{\mathrm{RDMReg}}$.

Typical runtime is determined by sorting operations, with $O(B \log B)$ per slice and practical $N$ in the few thousands even for $d \sim 2048$.

## 5. Hyperparameterization and Sparsity–Performance Trade-off

RDMReg affords direct and interpretable control over sparsity and representational properties:

- **Norm order $p>0$**: $p=2$ (rectified Gaussian) yields dense codes; $p=1$ (rectified Laplace) produces sparser codes; $p<1$ further approximates $\ell_0$ sparsity.
- **Mean shift $\mu$**: Controls expected active fraction, corresponding to $\Phi(\mu/\sigma)$ per dimension; empirically, $\mathbb{E}\|\mathbf{z}\|_0/B \approx \Phi(\mu/\sigma)$.
- **Scale $\sigma$**: Two options: (i) $\sigma_{\mathrm{GN}}$ for unit variance of pre-ReLU Gaussian, (ii) $\sigma_{\mathrm{RGN}}$ for unit variance of RGG; (i) is recommended due to simplicity and empirical parity on performance frontiers.
- **Number of slices $N$**: Effective performance with $N \approx 2,000$–$8,000$ largely independent of feature dimension.
- **Trade-off**: Performance remains stable up to approximately 95% zeros; higher sparsity induces sharp accuracy drops. Optimal Pareto fronts are attained with $p=1$ and moderate $\mu \approx -1$ or $-2$.

## 6. Comparison to Gaussian-Based Regularizations

Prior approaches, notably LeJEPA (SIGReg), regularize toward isotropic Gaussian targets via similar sliced-Wasserstein losses, enforcing only $\ell_2$ constraints and producing dense, non-negative activations without control over $\ell_0$ sparsity.

RDMReg strictly generalizes these methods:
- If $p=2$ and rectification is absent, the method recovers the Gaussian case.
- Adding ReLU and RGG targets introduces an explicit $\ell_0$ handle, preserving maximum entropy in the nonzero coordinates.

Empirical results demonstrate RDMReg learns representations with 5–10% density while matching or exceeding accuracy of dense baselines on linear-probe and transfer tasks, and yields more factorial codes with lower nHSIC. Qualitative probes indicate preservation of semantic content even at extreme sparsity levels.

## 7. Recommendations and Practical Use

Recommended settings for RDMReg in practice:
- Choose $p=1$ (rectified Laplace) or intermediate $p \in [1,2]$ for moderate sparsity.
- Adjust $\mu \in [-2,0]$ to control expected fraction of zeros (50–90%).
- Use $\sigma = \sigma_{\mathrm{GN}}$ for simplicity.
- Set the number of projections $N \approx 4,000$–$8,000$ for $d \approx 2,048$.
- Balance invariance and RDMReg losses with $\lambda_{\mathrm{RDM}} \approx 5$–$10 \times \lambda_{\mathrm{inv}}$.

RDMReg can be integrated as a “plug-in” into any JEPA variant, enabling explicit, controlled transitions between dense maximum-entropy and sparse maximum-entropy codes while preserving high information capacity under $\ell_p$ constraints [2602.01456].

Source: https://www.emergentmind.com/topics/rectified-distribution-matching-regularization-rdmreg