Geometric Jensen–Shannon Divergence (G-JSD)

• Geometric Jensen–Shannon Divergence (G-JSD) is an information-theoretic measure that symmetrizes the KL divergence using the geometric mean for enhanced analytic tractability.
• It provides closed-form expressions for exponential families, notably offering explicit formulas for Gaussian distributions and infinite-dimensional Gaussian measures.
• G-JSD is widely used in machine learning, statistical inference, and clustering, delivering robust theoretical guarantees and improved computational efficiency.

The geometric Jensen–Shannon divergence (G-JSD) is a class of information-theoretic divergences that symmetrize the Kullback–Leibler divergence using the geometric mean rather than the arithmetic mean of probability densities. Initially proposed to address the lack of closed-form expressions for the classical Jensen–Shannon divergence (JSD) on exponential families, G-JSD admits exact analytic formulas, notably for Gaussian distributions and Gaussian measures on infinite-dimensional spaces. G-JSD and its generalizations form a principal tool for statistical inference, machine learning, and information geometry due to their computational tractability and strong theoretical guarantees.

1. Definition and Mathematical Formulation

Let $p$ and $q$ be probability densities (or positive measures) on some measurable space $(X, \mu)$. The classical JSD considers the arithmetic mean mixture, $m_A = (1-\alpha)p + \alpha q$, as the reference, and defines

$$\mathrm{JS}_\alpha(p:q) = (1-\alpha)\,\mathrm{KL}(p\|m_A) + \alpha\,\mathrm{KL}(q\|m_A).$$

The geometric Jensen–Shannon divergence, in contrast, uses the normalized geometric mean

$$(pq)^G_\alpha(x) = \frac{p(x)^{1-\alpha} q(x)^\alpha}{Z^G_\alpha(p, q)}, \quad Z^G_\alpha(p, q) = \int p(x)^{1-\alpha} q(x)^\alpha\, d\mu(x),$$

and sets

$$\mathrm{JS}^{G_\alpha}(p:q) = (1-\alpha)\,\mathrm{KL}(p\|(pq)^G_\alpha) + \alpha\,\mathrm{KL}(q\|(pq)^G_\alpha).$$

When $\alpha = 1/2$, the symmetric form is obtained. For discrete distributions $P = (p_1, \ldots, p_n)$ and $Q = (q_1, \ldots, q_n)$,

$q$0

where $q$1 is the coordinate-wise geometric mean: $q$2 (Costa et al., 2011, Nielsen, 2019, Nielsen, 7 Aug 2025).

A notable variant, the extended G-JSD, is defined for unnormalized positive densities via the extended KL divergence: $q$3 yielding

$q$4

where $q$5 denotes the unnormalized geometric mean (Nielsen, 7 Aug 2025).

2. Closed-Form Expressions for Exponential Families and Gaussians

A central feature of G-JSD is that, for members of an exponential family, the geometric mixture remains within the family. For an exponential family $q$6, the geometric mean can be written as

$q$7

with $q$8. This yields the closed-form (Nielsen, 2019): $q$9 where $(X, \mu)$0 is the Bregman divergence for the log-partition $(X, \mu)$1.

For multivariate Gaussians, $(X, \mu)$2, the symmetric case ($(X, \mu)$3) gives

$(X, \mu)$4

where $(X, \mu)$5 is the Jeffreys divergence and $(X, \mu)$6 the Bhattacharyya distance. The geometric barycenter $(X, \mu)$7 is given by: $(X, \mu)$8 and the G-JSD admits explicit evaluation in terms of traces and determinants involving these parameters (Nielsen, 2019, Nielsen, 7 Aug 2025, Quang et al., 12 Jun 2025).

For Gaussian measures on infinite-dimensional Hilbert space $(X, \mu)$9, the closed form extends by using covariance operators and appropriate determinant constructs (logdet₂, extended Log-Det divergence), with a regularization for non-equivalent measures (Quang et al., 12 Jun 2025).

3. Properties and Theoretical Guarantees

G-JSD inherits several crucial properties:

• Non-negativity and definiteness: $m_A = (1-\alpha)p + \alpha q$0, with equality iff $m_A = (1-\alpha)p + \alpha q$1.
• Symmetry: Holds exactly at $m_A = (1-\alpha)p + \alpha q$2; otherwise, G-JSD is skewed.
• Closed-form computability: For exponential families—including Gaussians—the divergence is available analytically, unlike the ordinary JSD where the arithmetic mixture exits the family.
• Relation to classical divergences: G-JSD is bounded above and below by functions of the Jeffreys divergence, Hellinger distance, and Bhattacharyya distance; specifically, for symmetric G-JSD,

$m_A = (1-\alpha)p + \alpha q$3

$m_A = (1-\alpha)p + \alpha q$4

• Metricity: On the probability simplex, the square-root $m_A = (1-\alpha)p + \alpha q$5 is a true metric (Costa et al., 2011). In the exponential family case, $m_A = (1-\alpha)p + \alpha q$6 recovers metric-like behavior in natural parameter space (Nielsen, 2019). For the extended G-JSD, the metric property is retained in the discrete case.

A summary table of comparison:

Property	Classical JSD	G-JSD (normalized)	Extended G-JSD
Closed-form for Gaussians	No	Yes	Yes
Bounded ($m_A = (1-\alpha)p + \alpha q$7)	Yes	No	No
Metric ($m_A = (1-\alpha)p + \alpha q$8 )	Yes	Yes (discrete case)	Yes (discrete case)
Applicable to unnorm. dens.	No	No	Yes

4. Regularization and Relationship to Other Divergences

G-JSD can be interpreted as a regularization or augmentation of the ordinary JSD. Explicitly, for normalized densities,

$m_A = (1-\alpha)p + \alpha q$9

The extended G-JSD further adds a "mass-gap" penalty,

$$\mathrm{JS}_\alpha(p:q) = (1-\alpha)\,\mathrm{KL}(p\|m_A) + \alpha\,\mathrm{KL}(q\|m_A).$$0

where $$\mathrm{JS}_\alpha(p:q) = (1-\alpha)\,\mathrm{KL}(p\|m_A) + \alpha\,\mathrm{KL}(q\|m_A).$$1. This structure clarifies how G-JSD acts as a strict symmetrization and penalizer, with the geometric mean enforcing more stringent similarity than the arithmetic mean (Nielsen, 7 Aug 2025).

For Gaussian processes and measures on Hilbert space, the regularized G-JSD (with covariance regularization parameter $$\mathrm{JS}_\alpha(p:q) = (1-\alpha)\,\mathrm{KL}(p\|m_A) + \alpha\,\mathrm{KL}(q\|m_A).$$2) extends the divergence to non-equivalent covariances, recovering the exact G-JSD in the $$\mathrm{JS}_\alpha(p:q) = (1-\alpha)\,\mathrm{KL}(p\|m_A) + \alpha\,\mathrm{KL}(q\|m_A).$$3 limit (Quang et al., 12 Jun 2025).

5. Applications and Practical Considerations

G-JSD is widely applied in:

• Variational inference and generative modeling: G-JSD is used as a divergence regularizer in variational autoencoders (VAE) to interpolate between forward and reverse KL, enhancing controllability of mode-seeking vs. mode-covering (Deasy et al., 2020). The interpretability and closed-form for Gaussians allow efficient and effective optimization.
• Clustering and information geometry: Lloyd-style and $$\mathrm{JS}_\alpha(p:q) = (1-\alpha)\,\mathrm{KL}(p\|m_A) + \alpha\,\mathrm{KL}(q\|m_A).$$4-means++ clustering algorithms operate directly in the space of distributions using G-JSD, with cluster centers computable via convex combinations of natural parameters (Nielsen, 2019).
• Unnormalized models and Monte Carlo: Extended G-JSD enables divergence measurement when normalization constants are unavailable, relevant in unnormalized posterior sampling and annealed MCMC (Nielsen, 7 Aug 2025).
• Gaussian processes on function spaces: The G-JSD between Gaussian measures on infinite-dimensional Hilbert spaces underpins theoretical analysis and regularization for Gaussian process models (Quang et al., 12 Jun 2025).

Empirical evaluations indicate that, in VAE settings, tuning the skew parameter $$\mathrm{JS}_\alpha(p:q) = (1-\alpha)\,\mathrm{KL}(p\|m_A) + \alpha\,\mathrm{KL}(q\|m_A).$$5 yields a controllable trade-off between reconstruction loss and generative capacity—with $$\mathrm{JS}_\alpha(p:q) = (1-\alpha)\,\mathrm{KL}(p\|m_A) + \alpha\,\mathrm{KL}(q\|m_A).$$6 recovering forward KL, and $$\mathrm{JS}_\alpha(p:q) = (1-\alpha)\,\mathrm{KL}(p\|m_A) + \alpha\,\mathrm{KL}(q\|m_A).$$7 the reverse KL (Deasy et al., 2020).

6. Open Problems and Extensions

• Metric geometry beyond the simplex: While the square-root of G-JSD is a metric in the probability simplex and for discrete distributions, extensions to continuous densities and quantum/state space remain partially open. For quantum G-JSDs, analogous metricity properties are studied but depend on the operator setting (Virosztek, 2019).
• Hilbert embedding and geometry: For the quantum Jensen–Shannon divergence, a Hilbert space embedding is known for qubits but not for higher-dimensional states, suggesting a limitation in geometric interpretation (Virosztek, 2019).
• Information geometry for G-JSD: The infinitesimal structure (curvature, geodesics, invariants) of G-JSD is not fully characterized, particularly in the quantum and infinite-dimensional contexts (Virosztek, 2019, Quang et al., 12 Jun 2025).

A plausible implication is that further understanding of the geometric and differential structure of G-JSD would yield new insights into statistical manifold theory and quantum information geometry.

7. Summary of Comparative Advantages

G-JSD provides a robust, computationally tractable alternative to the classical JSD for parametric families, notably Gaussians, addressing several key limitations:

• Analytic tractability: Exact closed-forms in exponential families ensure practical efficiency.
• Rich regularization structure: Its role as a higher-order regularizer of the ordinary JSD underlines its utility in penalized statistical learning.
• Applicability to function spaces: Regularized extensions define divergences between Gaussian processes/fields on infinite-dimensional spaces.
• Suitability in unsupervised learning: Empirically favorable properties for generative modeling, clustering, and representation learning.

Collectively, these properties underpin the increasing popularity and application of G-JSD and its variants in contemporary information theory, machine learning, and stochastic modeling (Nielsen, 7 Aug 2025, Nielsen, 2019, Quang et al., 12 Jun 2025, Costa et al., 2011, Deasy et al., 2020).