Generalized Kullback–Leibler Divergence

• Generalized Kullback–Leibler divergence is a family of extensions that modifies the classical KL to handle nonextensive systems, robust inference, and unnormalized models.
• It incorporates deformation parameters and tailored functionals to adjust properties like convexity and (pseudo)additivity for diverse statistical and physical contexts.
• The approach finds practical applications in nonextensive thermostatistics, robust learning methods, and energy-based model optimization in machine learning.

Generalized Kullback–Leibler Divergence

The Generalized Kullback–Leibler (KL) divergence refers to a spectrum of extensions of the classical KL divergence, each designed to accommodate different probabilistic, statistical, and physical scenarios beyond the constraints of standard information theory. These generalizations are indispensable in nonextensive thermodynamics, robust inference, learning with unnormalized densities, deformed exponential family theory, and set-valued uncertainty quantification. Typically, these divergences interpolate or extrapolate the functional form of the classical KL, preserving or modifying property sets such as convexity, metric consistency, and information-geometric interpretations.

1. Formulations and Algebraic Structure

A variety of generalized KL divergences exists, each arising from a distinct theoretical motivation. Principal examples include:

• q-Generalized/Tsallis KL Divergence: For $q \neq 1$, the Tsallis divergence is defined for discrete $p=(p_i)$ and $r=(r_i)$ as

$$D_{K\!-\!L}^q\bigl[p\|r\bigr] = \frac{1}{q-1}\sum_i p_i\left[\left(\frac{p_i}{r_i}\right)^{q-1} - 1\right] = \sum_i p_i\ln_q\left(\frac{p_i}{r_i}\right)$$

where the "q-logarithm" is $\ln_q(x) = \frac{x^{1-q}-1}{1-q}$ (Venkatesan et al., 2011).

• Deformed Exponential/$(h, \tau)$-Divergence: This general form is

$$D_{h,\tau}(p\|q) = \int d_{h,\tau}(p(x),q(x))\,d\mu(x), \quad d_{h,\tau}(t,s) = h(\tau(t)) - h(\tau(s)) - (\tau(t)-\tau(s)) h'(\tau(s))$$

unifying the KL, Tsallis, $\beta$- and Rényi divergences via specific choices of $(h,\tau)$ (Matsuzoe et al., 25 Dec 2025).

• Generalized $\varphi$-divergence/Tsallis-Related: For a strictly increasing "deformed exponential" $p=(p_i)$0 with inverse $p=(p_i)$1,

$p=(p_i)$2

recovers Tsallis' case when $p=(p_i)$3 (Vigelis et al., 2018).

• Scaled Bregman Divergences: Defined with generating function $p=(p_i)$4 and scaling measure $p=(p_i)$5,

$p=(p_i)$6

with the dual Tsallis divergence realized for $p=(p_i)$7, $p=(p_i)$8 (Venkatesan et al., 2011).

• Generalized KL for Sets: For $p=(p_i)$9 sets of probability measures,

$r=(r_i)$0

addresses robust or distributional uncertainty (Li et al., 30 Oct 2025).

• Unnormalized Density Generalization: For nonnegative (unnormalized) $r=(r_i)$1, $r=(r_i)$2,

$r=(r_i)$3

linking directly to energy-based model learning (Miller et al., 2023).

2. Key Properties and Comparison with Classical KL

• Reduction to KL: In each scheme, taking $r=(r_i)$4, or the specific parameter limit, recovers the classical KL divergence

$r=(r_i)$5

or its measure-theoretic analog (Erven et al., 2012, Venkatesan et al., 2011).

• Convexity: Most generalizations (e.g., scaled Bregman and Tsallis KL) retain convexity in the first argument and, under appropriate conditions, joint convexity or quasi-convexity. Strict convexity is inherited from the underlying convex generator or entropy function (Vigelis et al., 2018, Beretta et al., 5 Feb 2026).
• Additive/Pseudoadditive Structure: Tsallis-type divergences exhibit pseudoadditivity under product measures, in contrast to the additivity of KL. For example,

$r=(r_i)$6

(Okamura, 2024).

• Pinsker-type Inequalities: Generalized Pinsker inequalities relate divergence lower bounds to $r=(r_i)$7 or total variation distances. For $r=(r_i)$8-Tsallis Bregman divergences,

$r=(r_i)$9

with explicit, regime-dependent $$D_{K\!-\!L}^q\bigl[p\|r\bigr] = \frac{1}{q-1}\sum_i p_i\left[\left(\frac{p_i}{r_i}\right)^{q-1} - 1\right] = \sum_i p_i\ln_q\left(\frac{p_i}{r_i}\right)$$0 (Beretta et al., 5 Feb 2026, Vigelis et al., 2018).

• Information-Geometric Interpretation: Bregman and $$D_{K\!-\!L}^q\bigl[p\|r\bigr] = \frac{1}{q-1}\sum_i p_i\left[\left(\frac{p_i}{r_i}\right)^{q-1} - 1\right] = \sum_i p_i\ln_q\left(\frac{p_i}{r_i}\right)$$1-divergences induce dually flat structures on the corresponding (deformed) exponential families, supporting Pythagorean theorems and generalized projection principles (Matsuzoe et al., 25 Dec 2025, Venkatesan et al., 2011, Venkatesan et al., 2011).

3. Applications and Consequences in Theory and Practice

• Nonextensive Statistical Physics: Generalized KL divergences, particularly the Tsallis and its "dual," are central in nonextensive thermostatistics, entropic variational principles with normal or $$D_{K\!-\!L}^q\bigl[p\|r\bigr] = \frac{1}{q-1}\sum_i p_i\left[\left(\frac{p_i}{r_i}\right)^{q-1} - 1\right] = \sum_i p_i\ln_q\left(\frac{p_i}{r_i}\right)$$2-expectation constraints, and the corresponding maximum-entropy (MaxEnt) inference (Venkatesan et al., 2011, Venkatesan et al., 2011, Okamura, 2024).
• Robust and Distributionally Ambiguous Inference: Generalized KLs for sets and the $$D_{K\!-\!L}^q\bigl[p\|r\bigr] = \frac{1}{q-1}\sum_i p_i\left[\left(\frac{p_i}{r_i}\right)^{q-1} - 1\right] = \sum_i p_i\ln_q\left(\frac{p_i}{r_i}\right)$$3-divergence family enable minimax or robust estimation, uncertainty quantification, and learning under ambiguous or complex non-i.i.d. sources (Li et al., 30 Oct 2025).
• Machine Learning Losses: The generalized KL loss has been used to improve robustness and regularization in adversarial training, knowledge distillation, and semi-supervised learning. The decoupled structure in (Cui et al., 11 Mar 2025) shows that GKL combines weighted-MSE with cross-entropy and incorporates global class-level weighting for stability and fairness.
• Simulation-based Inference with Unnormalized Models: The GKL divergence provides a tractable, unified loss for flow-based, ratio-based, and hybrid energy-based models in SBI, enabling accurate inference of complex posteriors without requiring normalization of the surrogate density (Miller et al., 2023).
• Information Bottleneck and Rate-Distortion Theory: The Bregman structure of dual-Tsallis KL allows extending clustering (e.g., k-means), bottleneck, and rate-distortion methods to the nonextensive regime, with matching variational and projection properties (Venkatesan et al., 2011).

4. Parametric Flexibility: The Role of Deformation Parameters

The crucial innovation of many generalized KL divergences is their dependency on a deformation parameter (e.g., $$D_{K\!-\!L}^q\bigl[p\|r\bigr] = \frac{1}{q-1}\sum_i p_i\left[\left(\frac{p_i}{r_i}\right)^{q-1} - 1\right] = \sum_i p_i\ln_q\left(\frac{p_i}{r_i}\right)$$4, $$D_{K\!-\!L}^q\bigl[p\|r\bigr] = \frac{1}{q-1}\sum_i p_i\left[\left(\frac{p_i}{r_i}\right)^{q-1} - 1\right] = \sum_i p_i\ln_q\left(\frac{p_i}{r_i}\right)$$5). This parameter:

• Interpolate between regimes: $$D_{K\!-\!L}^q\bigl[p\|r\bigr] = \frac{1}{q-1}\sum_i p_i\left[\left(\frac{p_i}{r_i}\right)^{q-1} - 1\right] = \sum_i p_i\ln_q\left(\frac{p_i}{r_i}\right)$$6 gives compact support or outlier-suppressing behavior; $$D_{K\!-\!L}^q\bigl[p\|r\bigr] = \frac{1}{q-1}\sum_i p_i\left[\left(\frac{p_i}{r_i}\right)^{q-1} - 1\right] = \sum_i p_i\ln_q\left(\frac{p_i}{r_i}\right)$$7 admits fat-tailed distributions and power-law behaviors.
• Control robustness and sensitivity: Small $$D_{K\!-\!L}^q\bigl[p\|r\bigr] = \frac{1}{q-1}\sum_i p_i\left[\left(\frac{p_i}{r_i}\right)^{q-1} - 1\right] = \sum_i p_i\ln_q\left(\frac{p_i}{r_i}\right)$$8 or $$D_{K\!-\!L}^q\bigl[p\|r\bigr] = \frac{1}{q-1}\sum_i p_i\left[\left(\frac{p_i}{r_i}\right)^{q-1} - 1\right] = \sum_i p_i\ln_q\left(\frac{p_i}{r_i}\right)$$9 weights rare events more heavily; larger values yield more "uniformizing" behavior (Beretta et al., 5 Feb 2026, Okamura, 2024).
• Allow family extensions: The $\ln_q(x) = \frac{x^{1-q}-1}{1-q}$0-generalized multinomial/divergence correspondence spins off an entire family $\ln_q(x) = \frac{x^{1-q}-1}{1-q}$1, with Tsallis relative entropy as the lead term (Okamura, 2024).
• Support statistical regularization: The modulus of strong convexity for regularization in online or bandit learning is explicit in terms of the deformation parameter (Beretta et al., 5 Feb 2026).

5. Generalizations: Sets, Unnormalized Densities, and New Statistical Families

• Sets of Measures: $\ln_q(x) = \frac{x^{1-q}-1}{1-q}$2 accommodates robust hypothesis testing and learning under sublinear expectation and is fundamental in the study of weak convergence in the context of model uncertainty (Li et al., 30 Oct 2025).
• Unnormalized Densities: The generalized divergence $\ln_q(x) = \frac{x^{1-q}-1}{1-q}$3 encapsulates normalization matching, enabling hybrid parameterizations, and is central to energy-based model optimization without recourse to intractable normalization gradients (Miller et al., 2023).
• Deformed Exponential Families: The $\ln_q(x) = \frac{x^{1-q}-1}{1-q}$4-divergence underpins a vast family of deformed exponential families, each carrying a Hessian information-geometric structure, and extending large-sample laws and information projection principles into non-Shannonian regimes (Matsuzoe et al., 25 Dec 2025).

6. Information Geometry, Variational Principles, and Pythagorean Relations

Generalized KL divergences, particularly those with a scaled Bregman or $\ln_q(x) = \frac{x^{1-q}-1}{1-q}$5 form, retain much of the information geometry and variational machinery of their classical prototype:

• Scaled Bregman Structure: The dual Tsallis KL is a scaled Bregman divergence, allowing the full suite of Pythagorean, projection, and geometric inference tools (Venkatesan et al., 2011, Venkatesan et al., 2011).
• Generalized Pythagorean Theorems: Variational minimization under normal-averaged constraints leads to nonadditive Pythagorean theorems whose triangle relations depend on the value of $\ln_q(x) = \frac{x^{1-q}-1}{1-q}$6 (direction of inequality flips at $\ln_q(x) = \frac{x^{1-q}-1}{1-q}$7) (Venkatesan et al., 2011).
• Extended MaxEnt and Law of Large Numbers: In deformed exponential families, maximum-entropy principles and strong laws extend beyond the i.i.d. classical context, with non-classical convergence properties and entropy rates (Matsuzoe et al., 25 Dec 2025).

7. Summary Table of Prominent Generalized KL Diversion Types

Name	Prototype Formula	Specialization(s)
Tsallis-KL (Naudts form)	$\ln_q(x) = \frac{x^{1-q}-1}{1-q}$8	$\ln_q(x) = \frac{x^{1-q}-1}{1-q}$9: KL divergence
Generalized Bregman KL	$(h, \tau)$0 as Bregman divergence for $(h, \tau)$1	$(h, \tau)$2: classical Bregman–KL
$(h, \tau)$3-divergence	$(h, \tau)$4	KL: $(h, \tau)$5, $(h, \tau)$6
Scaled Bregman (dual Tsallis)	$(h, \tau)$7 with $(h, \tau)$8	Duality: $(h, \tau)$9
$$D_{h,\tau}(p\|q) = \int d_{h,\tau}(p(x),q(x))\,d\mu(x), \quad d_{h,\tau}(t,s) = h(\tau(t)) - h(\tau(s)) - (\tau(t)-\tau(s)) h'(\tau(s))$$0-divergence (deformed exp)	$$D_{h,\tau}(p\|q) = \int d_{h,\tau}(p(x),q(x))\,d\mu(x), \quad d_{h,\tau}(t,s) = h(\tau(t)) - h(\tau(s)) - (\tau(t)-\tau(s)) h'(\tau(s))$$1	$$D_{h,\tau}(p\|q) = \int d_{h,\tau}(p(x),q(x))\,d\mu(x), \quad d_{h,\tau}(t,s) = h(\tau(t)) - h(\tau(s)) - (\tau(t)-\tau(s)) h'(\tau(s))$$2 yields Tsallis
GKL for sets	$$D_{h,\tau}(p\|q) = \int d_{h,\tau}(p(x),q(x))\,d\mu(x), \quad d_{h,\tau}(t,s) = h(\tau(t)) - h(\tau(s)) - (\tau(t)-\tau(s)) h'(\tau(s))$$3	Reduces to KL if $$D_{h,\tau}(p\|q) = \int d_{h,\tau}(p(x),q(x))\,d\mu(x), \quad d_{h,\tau}(t,s) = h(\tau(t)) - h(\tau(s)) - (\tau(t)-\tau(s)) h'(\tau(s))$$4 are singletons
GKL for unnormalized densities	$$D_{h,\tau}(p\|q) = \int d_{h,\tau}(p(x),q(x))\,d\mu(x), \quad d_{h,\tau}(t,s) = h(\tau(t)) - h(\tau(s)) - (\tau(t)-\tau(s)) h'(\tau(s))$$5	Equals KL when densities normalized

Each family carries a precise set of monotonicity, convexity, and variational characteristics, and the choice among them is dictated by the statistical, physical, or algorithmic scenario under consideration (Venkatesan et al., 2011, Matsuzoe et al., 25 Dec 2025, Vigelis et al., 2018, Miller et al., 2023, Cui et al., 11 Mar 2025, Li et al., 30 Oct 2025, Okamura, 2024, Beretta et al., 5 Feb 2026, Venkatesan et al., 2011).

---

References:

(Venkatesan et al., 2011, Venkatesan et al., 2011, Okamura, 2024, Li et al., 30 Oct 2025, Miller et al., 2023, Cui et al., 11 Mar 2025, Matsuzoe et al., 25 Dec 2025, Vigelis et al., 2018, Beretta et al., 5 Feb 2026)