A family of statistical symmetric divergences based on Jensen's inequality (1009.4004v2)

Abstract: We introduce a novel parametric family of symmetric information-theoretic distances based on Jensen's inequality for a convex functional generator. In particular, this family unifies the celebrated Jeffreys divergence with the Jensen-Shannon divergence when the Shannon entropy generator is chosen. We then design a generic algorithm to compute the unique centroid defined as the minimum average divergence. This yields a smooth family of centroids linking the Jeffreys to the Jensen-Shannon centroid. Finally, we report on our experimental results.

• The paper introduces a novel parametric family of symmetric divergences derived from Jensen's inequality that bridges Jeffreys and Jensen-Shannon metrics.
• It provides closed-form solutions and an iterative centroid algorithm using CCCP, enhancing computations in exponential families such as Gaussian models.
• Experimental validation in binary classification tasks shows that tuning the divergence parameter optimizes performance in high-dimensional settings.

A Family of Statistical Symmetric Divergences Based on Jensen's Inequality

The paper presented is an in-depth exploration of a novel parametric family of symmetric information-theoretic distances, founded on Jensen's inequality and leveraging a convex functional generator. This innovative family offers a unifying framework that bridges the Jeffreys divergence with the Jensen-Shannon divergence, using the Shannon entropy as a generator.

Conceptual Foundations and Mathematical Formulation

The exposition begins with a review of key information-theoretic concepts central to the paper, including Shannon differential entropy, cross-entropy, and the Kullback-Leibler (KL) divergence. The discussion emphasizes the asymmetric nature of the KL divergence when applied to statistical measures and introduces symmetrization techniques essential for the later development of symmetric divergences.

The paper proposes a family of symmetrized divergences derived from Jensen's inequality. The symmetrized $\alpha$-Jensen divergences, represented as $sJ_F^{(\alpha)}(p, q)$, offer a generalized form that encapsulates both the Jeffreys and Jensen-Shannon divergences as limit cases for specific parameter settings. Through comprehensive mathematical treatment, the authors elucidate that as $\alpha$ approaches different values, these divergences asymptotically approach distinct known divergences, providing a continuous interpolation between Jeffreys and Jensen-Shannon metrics.

Implications and Applications in Exponential Families

One significant result of the paper is the demonstration of closed-form solutions for computing divergences in parametric families, particularly within statistical exponential families. This has substantial implications for computations involving Gaussian distributions and other similar statistical models where mixture calculations are traditionally complex.

The paper further explores the implications of these divergences within the space of exponential families via $\alpha$-skew Bhattacharyya divergences. Theoretical results show that these divergences can be related back to the parameter space through natural parameters, offering practical computational benefits. Notably, in the exponential family, symmetrized $\alpha$-Bhattacharyya divergences align with symmetrized Jensen divergences, yielding efficient calculation methods for information measures within this context.

Algorithmic Contributions

The paper presents a generic algorithm for computing centroids, defined as the minimum average divergence, with respect to these symmetrized divergences. The iterative approach proposed leverages the concave-convex procedure (CCCP), ensuring convergence to a solution without onerous hyperparameter tuning. This provides a robust mechanism for applications such as $k$-means clustering and other centroid-based methods in high-dimensional, data-driven environments.

Experimental Validation

An empirical paper backs the theoretical contributions, showcasing the performance of this divergence family in a binary classification task using image intensity histograms. Results suggest that adjusting the $\alpha$ parameter can optimize classification performance, highlighting the adaptable nature of this divergence in practical applications.

Conclusion and Future Directions

In conclusion, the paper effectively establishes a broad framework for symmetric information divergences, with clear computational advantages in information-theoretic measures across various fields, including signal processing and machine learning. Future research could explore further applications of these divergences in large-scale data environments and investigate the potential for dynamic $\alpha$ adjustments in real-time systems. Additionally, extending these concepts to anisotropic divergences or integrating deep learning modalities may provide further versatility in emerging applications across diverse data modalities.