---
title: Rectified Generalized Gaussian (RGG)
url: https://www.emergentmind.com/topics/rectified-generalized-gaussian-rgg
type: topic
---

# Rectified Generalized Gaussian (RGG)

The Rectified Generalized Gaussian (RGG) refers to a family of distributions, defined by the rectification (thresholding and truncation) of the classical Generalized Gaussian (GG) family, which plays a central role in high-dimensional representation learning and image modeling as both a probabilistic prior and a regularization tool. The RGG provides explicit control over coding sparsity while maintaining maximum-entropy properties under expected $\ell_p$-norm constraints. Recent formulations highlight its mixture structure, entropy dimension, parametrization, and its use in algorithms for both predictive representation learning and statistical image restoration.

## 1. Mathematical Definitions

The Generalized Gaussian distribution (GN/GGD) is parameterized by a shape parameter $p>0$, location $\mu \in \mathbb{R}$, and scale $\sigma>0$. Its density:
$$
f_{\mathrm{GN}_p(\mu, \sigma)}(x) = \frac{p^{1-1/p}}{2\,\sigma\,\Gamma(1/p)} \exp\left(-\frac{|x-\mu|^p}{p\,\sigma^p}\right)
$$
Special cases: $p=1$ yields Laplace$(\mu,\sigma)$, $p=2$ yields Gaussian $\mathcal{N}(\mu,\sigma^2)$.

The truncated (rectified, or "half") GN is the restriction of $f_{\mathrm{GN}_p(\mu, \sigma)}$ to $x>\theta$, normalized:
$$
f_{\mathrm{TGN}_p(\mu, \sigma)}(x) = \frac{1_{x > \theta}}{Z} \exp\left(-\frac{|x-\mu|^p}{p\,\sigma^p}\right), \quad Z = \int_{\theta}^{\infty} \exp\left(-\frac{|x-\mu|^p}{p\,\sigma^p}\right) dx
$$
The Rectified Generalized Gaussian (RGG), as formalized in [2602.01456], introduces a rectification threshold $\theta$ (commonly $\theta=0$), defining the random variable $X$ as:
$$
X = \max(0, Z - \theta) + \theta,\qquad Z \sim \mathrm{GN}_p(\mu, \sigma)
$$
The RGG law is a mixture:
$$
P_X = \kappa\,\delta_\theta + (1-\kappa)\,P_{\mathrm{TGN}_p(\mu, \sigma)}
$$
with
$$
\kappa = P(Z \leq \theta) = \Phi_{\mathrm{GN}_p(\mu, \sigma)}(\theta)
$$
The corresponding density is
$$
f_{\mathrm{RGG}_p(\mu, \sigma; \theta)}(x) = 
\kappa\,1_{x=\theta} +
(1-\kappa)\,\frac{1_{x > \theta}}{Z} \exp\left(-\frac{|x-\mu|^p}{p\,\sigma^p}\right)
$$
In the literature on image regularization [1906.10517], this is often formulated for $\theta=0$, $\mu=0$:
$$
p_{\mathrm{RGG}}(g;\alpha,p) = \frac{p}{\alpha\,\Gamma(1/p)} \exp\left(-\left(\frac{g}{\alpha}\right)^p\right), \quad g \ge 0
$$

## 2. Derivation and Maximum-Entropy Principles

Rectification transforms a GN variable $Z$ by thresholding, mixing a Dirac mass at $\theta$ and the truncated GN on $(\theta, \infty)$. For any Borel measurable $B \subset [\theta, \infty)$:
$$
P(X \in B) = P(Z \in B \setminus \{\theta\}) + P(Z \leq \theta)\,1_{\theta \in B}
$$
Under entropy maximization with support $S$ and constraint $\mathbb{E}[r(X)] = \alpha$, the solution is:
$$
p^*(x) \propto \exp\left(-\lambda\,r(x)\right)\,1_S(x)
$$
For $S=(0,\infty)$, $r(x)=|x|^p$, $\lambda=1/(p\sigma^p)$, yielding the $\mathrm{TGN}_p$; subsequent rectification adds the Dirac mass. The rectified construction preserves maximum-entropy under $\ell_p$-moment constraints up to rescaling and dimensional adjustment [2602.01456].

## 3. Structural and Information-Theoretic Properties

The RGG supports explicit $\ell_0$ norm (sparsity) control. For $X\sim\mathrm{RGG}_p(\mu,\sigma;0)$ in $d$ dimensions,
$$
\mathbb{E}\|X\|_0 = d\,(1-\kappa) = d\,\Phi_{\mathrm{GN}_p(\mu, \sigma)}(\mu/\sigma)
$$
Adjustment of $\mu$ thus permits fine-grained control of expected sparsity.

The RGG variable $X$ is a mixture of a discrete spike and continuous density, so classical differential entropy is ill-posed. Instead, the effective entropy is described by the Rênyi information dimension $d(X) = 1 - \kappa$ and $d(X)$-dimensional entropy
$$
H_{d(X)}(X) = (1-\kappa) H_0(\mathrm{Bernoulli}(1-\kappa)) + (1-\kappa)\,h(\mathrm{TGN}_p(\mu,\sigma))
$$
where $H_0$ is Shannon entropy and $h$ is the differential entropy of the continuous part (see Theorem 4.3 in [2602.01456]).

As $\mu \to +\infty$, $\kappa \to 0$, and $\mathrm{RGG}_p$ approaches the standard $\mathrm{GN}_p$; as $\mu \to -\infty$, the distribution mass is concentrated at zero.

## 4. Integration into Machine Learning and Image Processing

### 4.1 Predictive Representation Learning

The RGG is leveraged as a target distribution for internal representations in self-supervised joint-embedding architectures [2602.01456]. The Rectified Distribution Matching Regularization (RDMReg) loss aligns the empirical distribution of learned representations $z \in \mathbb{R}^D$ to the product RGG:
$$
\mathcal{L}_{\mathrm{RDMReg}}(Z; Y) = \mathbb{E}_{c}\; W_2^2(\mathrm{Law}(c^\top Z), \mathrm{Law}(c^\top Y))
$$
where $Y$ are i.i.d. synthetic RGG samples, $c$ are random projection vectors, and $W_2$ refers to the $2$-Wasserstein distance in one dimension. The surrogate implementation sorts projected values and matches their empirical distributions batchwise.

The full self-supervised loss combines a view-invariance term and the RDMReg penalty:
$$
\mathcal{L} = \mathbb{E}\|z - z'\|_2^2 + \lambda_{\mathrm{dist}}\,(\mathcal{L}_{\mathrm{RDMReg}}(Z;Y) + \mathcal{L}_{\mathrm{RDMReg}}(Z';Y))
$$
where $z = \mathrm{ReLU}(f_\theta(x))$ for two augmentations $x,x'$ of the same data point.

### 4.2 Regularization in Image Restoration

In variational image restoration, the assumption that image gradients locally follow a half-Generalized Gaussian (i.e., RGG) motivates a space-variant regularizer [1906.10517]:
$$
R(u) = \sum_i \alpha_i \|\nabla u_i\|_2^{p_i}
$$
with per-pixel parameters $\alpha_i, p_i$ estimated from neighborhood moment statistics. This regularizer is coupled to loss terms for different noise models, and the total energy is minimized using an alternating direction method of multipliers (ADMM) scheme, maintaining explicit treatment of the non-convex, non-smooth regularization term.

## 5. Parameterization, Special Cases, and Practical Guidance

### Special Cases

| $p$ Value | Name                    | Atom Mass $\kappa$                                      |
|-----------|-------------------------|--------------------------------------------------------|
| 1         | Rectified Laplace       | $\kappa = \Phi_{\mathrm{Laplace}}(-\mu/\sigma)$        |
| 2         | Rectified Gaussian      | $\kappa = \Phi_{\mathcal{N}(\mu,\sigma^2)}(0)$         |

With $\mu \to +\infty$, $\mathrm{RGG}_2 \to \mathcal{N}(\mu, \sigma^2)$.

### Parameter Selection

For “pre-ReLU unit variance,” set $\sigma$ so that $\mathrm{Var}[\mathrm{GN}_p(0,\sigma)] = 1$, leading to
$$
\sigma_{\mathrm{GN}} = \left[p^{2/p}\, \frac{\Gamma(3/p)}{\Gamma(1/p)}\right]^{-1/2}
$$
Alternatively, post-ReLU unit variance may be enforced by numerical inversion $\mathrm{Var}[\mathrm{RGG}_p(\mu,\sigma)] = 1$.

### Trade-off Control

Varying $\mu$ allows explicit control of the sparsity/information trade-off, modulating the expected fraction of zeros and the Rênyi-dimensional entropy $H_{d(X)}$. Empirically, models tolerate sparsity rates up to about 95% zeros before significant loss in downstream accuracy is observed [2602.01456].

### Visualization

For $p=1$ or $2$, $\mu=-0.5$, and $\sigma=1$, the 1D RGG exhibits an atom at zero and heavy-tailed decay for $x > 0$, illustrating the distribution's decomposability into a sparse component and a continuous part.

## 6. Connections, Generalizations, and Use in Literature

The RGG generalizes both the half-Gaussian and half-Laplace cases, and strictly generalizes prior isotropic Gaussian-based approaches in joint-embedding architectures. As demonstrated in [2602.01456], enforcing RGG target distributions yields nonnegative, sparse latent codes with competitive downstream performance and tunable sparsity.

In image processing, space-variant RGG regularization enables local adaptivity and flexible modeling of gradient distributions [1906.10517]. The model allows for robust restoration under a range of noise types, including both Gaussian and impulsive noise, by adapting per-pixel smoothness and edge-preservation properties.

The RGG framework can subsume regularizers targeting second-order statistics (as in nonnegative VCReg) and naturally interfaces with ADMM for efficient optimization in imaging problems.

## 7. Summary Table of RGG Variants

| Distribution            | Density (support)                                                      | Parameter Regime                              |
|-------------------------|------------------------------------------------------------------------|-----------------------------------------------|
| GN (Generalized Gaussian) | $\frac{p^{1-1/p}}{2\sigma \Gamma(1/p)} e^{-|x-\mu|^p/(p\sigma^p)}$  | $x\in\mathbb{R}$, $p>0, \sigma>0$             |
| Rectified GN (RGG)        | $\kappa\,\delta_\theta + (1-\kappa)\frac{1_{x>\theta}}{Z}e^{-|x-\mu|^p/(p\sigma^p)}$ | $x\geq \theta$, $p>0, \sigma>0, \theta\in\mathbb{R}$ |
| Half-GGD [1906.10517]     | $\frac{p}{\alpha\Gamma(1/p)} e^{-(g/\alpha)^p}$                      | $g\geq0$, $p>0, \alpha>0$                     |

The RGG provides a unifying, parametrically tunable foundation for sparse, nonnegative, and maximum-entropy modeling in both high-dimensional representation learning and adaptive image regularization.

Source: https://www.emergentmind.com/topics/rectified-generalized-gaussian-rgg