Rectified Generalized Gaussian (RGG)

• Rectified Generalized Gaussian (RGG) is a distribution defined by thresholding the generalized Gaussian, creating a mixture of a discrete spike and a continuous tail to enable explicit sparsity control.
• It maintains maximum-entropy properties under ℓp-norm constraints while providing a flexible parameterization that adjusts the balance between sparsity and information retention.
• RGG is applied in predictive representation learning and variational image restoration to regularize latent codes and adapt local gradient distributions for robust performance.

The Rectified Generalized Gaussian (RGG) refers to a family of distributions, defined by the rectification (thresholding and truncation) of the classical Generalized Gaussian (GG) family, which plays a central role in high-dimensional representation learning and image modeling as both a probabilistic prior and a regularization tool. The RGG provides explicit control over coding sparsity while maintaining maximum-entropy properties under expected $\ell_p$-norm constraints. Recent formulations highlight its mixture structure, entropy dimension, parametrization, and its use in algorithms for both predictive representation learning and statistical image restoration.

1. Mathematical Definitions

The Generalized Gaussian distribution (GN/GGD) is parameterized by a shape parameter $p>0$, location $\mu \in \mathbb{R}$, and scale $\sigma>0$. Its density:

$$f_{\mathrm{GN}_p(\mu, \sigma)}(x) = \frac{p^{1-1/p}}{2\,\sigma\,\Gamma(1/p)} \exp\left(-\frac{|x-\mu|^p}{p\,\sigma^p}\right)$$

Special cases: $p=1$ yields Laplace$(\mu,\sigma)$, $p=2$ yields Gaussian $\mathcal{N}(\mu,\sigma^2)$.

The truncated (rectified, or "half") GN is the restriction of $f_{\mathrm{GN}_p(\mu, \sigma)}$ to $p>0$0, normalized:

$p>0$1

The Rectified Generalized Gaussian (RGG), as formalized in (Kuang et al., 1 Feb 2026), introduces a rectification threshold $p>0$2 (commonly $p>0$3), defining the random variable $p>0$4 as:

$p>0$5

The RGG law is a mixture:

$p>0$6

with

$p>0$7

The corresponding density is

$p>0$8

In the literature on image regularization (Lanza et al., 2019), this is often formulated for $p>0$9, $\mu \in \mathbb{R}$0:

$\mu \in \mathbb{R}$1

2. Derivation and Maximum-Entropy Principles

Rectification transforms a GN variable $\mu \in \mathbb{R}$2 by thresholding, mixing a Dirac mass at $\mu \in \mathbb{R}$3 and the truncated GN on $\mu \in \mathbb{R}$4. For any Borel measurable $\mu \in \mathbb{R}$5:

$\mu \in \mathbb{R}$6

Under entropy maximization with support $\mu \in \mathbb{R}$7 and constraint $\mu \in \mathbb{R}$8, the solution is:

$\mu \in \mathbb{R}$9

For $\sigma>0$0, $\sigma>0$1, $\sigma>0$2, yielding the $\sigma>0$3; subsequent rectification adds the Dirac mass. The rectified construction preserves maximum-entropy under $\sigma>0$4-moment constraints up to rescaling and dimensional adjustment (Kuang et al., 1 Feb 2026).

3. Structural and Information-Theoretic Properties

The RGG supports explicit $\sigma>0$5 norm (sparsity) control. For $\sigma>0$6 in $\sigma>0$7 dimensions,

$\sigma>0$8

Adjustment of $\sigma>0$9 thus permits fine-grained control of expected sparsity.

The RGG variable $$f_{\mathrm{GN}_p(\mu, \sigma)}(x) = \frac{p^{1-1/p}}{2\,\sigma\,\Gamma(1/p)} \exp\left(-\frac{|x-\mu|^p}{p\,\sigma^p}\right)$$0 is a mixture of a discrete spike and continuous density, so classical differential entropy is ill-posed. Instead, the effective entropy is described by the Rênyi information dimension $$f_{\mathrm{GN}_p(\mu, \sigma)}(x) = \frac{p^{1-1/p}}{2\,\sigma\,\Gamma(1/p)} \exp\left(-\frac{|x-\mu|^p}{p\,\sigma^p}\right)$$1 and $$f_{\mathrm{GN}_p(\mu, \sigma)}(x) = \frac{p^{1-1/p}}{2\,\sigma\,\Gamma(1/p)} \exp\left(-\frac{|x-\mu|^p}{p\,\sigma^p}\right)$$2-dimensional entropy

$$f_{\mathrm{GN}_p(\mu, \sigma)}(x) = \frac{p^{1-1/p}}{2\,\sigma\,\Gamma(1/p)} \exp\left(-\frac{|x-\mu|^p}{p\,\sigma^p}\right)$$3

where $$f_{\mathrm{GN}_p(\mu, \sigma)}(x) = \frac{p^{1-1/p}}{2\,\sigma\,\Gamma(1/p)} \exp\left(-\frac{|x-\mu|^p}{p\,\sigma^p}\right)$$4 is Shannon entropy and $$f_{\mathrm{GN}_p(\mu, \sigma)}(x) = \frac{p^{1-1/p}}{2\,\sigma\,\Gamma(1/p)} \exp\left(-\frac{|x-\mu|^p}{p\,\sigma^p}\right)$$5 is the differential entropy of the continuous part (see Theorem 4.3 in (Kuang et al., 1 Feb 2026)).

As $$f_{\mathrm{GN}_p(\mu, \sigma)}(x) = \frac{p^{1-1/p}}{2\,\sigma\,\Gamma(1/p)} \exp\left(-\frac{|x-\mu|^p}{p\,\sigma^p}\right)$$6, $$f_{\mathrm{GN}_p(\mu, \sigma)}(x) = \frac{p^{1-1/p}}{2\,\sigma\,\Gamma(1/p)} \exp\left(-\frac{|x-\mu|^p}{p\,\sigma^p}\right)$$7, and $$f_{\mathrm{GN}_p(\mu, \sigma)}(x) = \frac{p^{1-1/p}}{2\,\sigma\,\Gamma(1/p)} \exp\left(-\frac{|x-\mu|^p}{p\,\sigma^p}\right)$$8 approaches the standard $$f_{\mathrm{GN}_p(\mu, \sigma)}(x) = \frac{p^{1-1/p}}{2\,\sigma\,\Gamma(1/p)} \exp\left(-\frac{|x-\mu|^p}{p\,\sigma^p}\right)$$9; as $p=1$0, the distribution mass is concentrated at zero.

4. Integration into Machine Learning and Image Processing

4.1 Predictive Representation Learning

The RGG is leveraged as a target distribution for internal representations in self-supervised joint-embedding architectures (Kuang et al., 1 Feb 2026). The Rectified Distribution Matching Regularization (RDMReg) loss aligns the empirical distribution of learned representations $p=1$1 to the product RGG:

$p=1$2

where $p=1$3 are i.i.d. synthetic RGG samples, $p=1$4 are random projection vectors, and $p=1$5 refers to the $p=1$6-Wasserstein distance in one dimension. The surrogate implementation sorts projected values and matches their empirical distributions batchwise.

The full self-supervised loss combines a view-invariance term and the RDMReg penalty:

$p=1$7

where $p=1$8 for two augmentations $p=1$9 of the same data point.

4.2 Regularization in Image Restoration

In variational image restoration, the assumption that image gradients locally follow a half-Generalized Gaussian (i.e., RGG) motivates a space-variant regularizer (Lanza et al., 2019):

$(\mu,\sigma)$0

with per-pixel parameters $(\mu,\sigma)$1 estimated from neighborhood moment statistics. This regularizer is coupled to loss terms for different noise models, and the total energy is minimized using an alternating direction method of multipliers (ADMM) scheme, maintaining explicit treatment of the non-convex, non-smooth regularization term.

5. Parameterization, Special Cases, and Practical Guidance

Special Cases

$(\mu,\sigma)$2 Value	Name	Atom Mass $(\mu,\sigma)$3
1	Rectified Laplace	$(\mu,\sigma)$4
2	Rectified Gaussian	$(\mu,\sigma)$5

With $(\mu,\sigma)$6, $(\mu,\sigma)$7.

Parameter Selection

For “pre-ReLU unit variance,” set $(\mu,\sigma)$8 so that $(\mu,\sigma)$9, leading to

$p=2$0

Alternatively, post-ReLU unit variance may be enforced by numerical inversion $p=2$1.

Trade-off Control

Varying $p=2$2 allows explicit control of the sparsity/information trade-off, modulating the expected fraction of zeros and the Rênyi-dimensional entropy $p=2$3. Empirically, models tolerate sparsity rates up to about 95% zeros before significant loss in downstream accuracy is observed (Kuang et al., 1 Feb 2026).

Visualization

For $p=2$4 or $p=2$5, $p=2$6, and $p=2$7, the 1D RGG exhibits an atom at zero and heavy-tailed decay for $p=2$8, illustrating the distribution's decomposability into a sparse component and a continuous part.

6. Connections, Generalizations, and Use in Literature

The RGG generalizes both the half-Gaussian and half-Laplace cases, and strictly generalizes prior isotropic Gaussian-based approaches in joint-embedding architectures. As demonstrated in (Kuang et al., 1 Feb 2026), enforcing RGG target distributions yields nonnegative, sparse latent codes with competitive downstream performance and tunable sparsity.

In image processing, space-variant RGG regularization enables local adaptivity and flexible modeling of gradient distributions (Lanza et al., 2019). The model allows for robust restoration under a range of noise types, including both Gaussian and impulsive noise, by adapting per-pixel smoothness and edge-preservation properties.

The RGG framework can subsume regularizers targeting second-order statistics (as in nonnegative VCReg) and naturally interfaces with ADMM for efficient optimization in imaging problems.

7. Summary Table of RGG Variants

Distribution	Density (support)	Parameter Regime
GN (Generalized Gaussian)	$p=2$9	$\mathcal{N}(\mu,\sigma^2)$0, $\mathcal{N}(\mu,\sigma^2)$1
Rectified GN (RGG)	$\mathcal{N}(\mu,\sigma^2)$2	$\mathcal{N}(\mu,\sigma^2)$3, $\mathcal{N}(\mu,\sigma^2)$4
Half-GGD (Lanza et al., 2019)	$\mathcal{N}(\mu,\sigma^2)$5	$\mathcal{N}(\mu,\sigma^2)$6, $\mathcal{N}(\mu,\sigma^2)$7

The RGG provides a unifying, parametrically tunable foundation for sparse, nonnegative, and maximum-entropy modeling in both high-dimensional representation learning and adaptive image regularization.