Fast proxy centers for Jeffreys centroids: The Jeffreys-Fisher-Rao and the inductive Gauss-Bregman centers

Abstract: The symmetric Kullback-Leibler centroid also called the Jeffreys centroid of a set of mutually absolutely continuous probability distributions on a measure space provides a notion of centrality which has proven useful in many tasks including information retrieval, information fusion, and clustering in image, video and sound processing. However, the Jeffreys centroid is not available in closed-form for sets of categorical or normal distributions, two widely used statistical models, and thus need to be approximated numerically in practice. In this paper, we first propose the new Jeffreys-Fisher-Rao center defined as the Fisher-Rao midpoint of the sided Kullback-Leibler centroids as a plug-in replacement of the Jeffreys centroid. This Jeffreys-Fisher-Rao center admits a generic formula for uni-parameter exponential family distributions, and closed-form formula for categorical and normal distributions, matches exactly the Jeffreys centroid for same-mean normal distributions, and is experimentally observed in practice to be close to the Jeffreys centroid. Second, we define a new type of inductive centers generalizing the principle of Gauss arithmetic-geometric double sequence mean for pairs of densities of any given exponential family. This center is shown experimentally to approximate very well the Jeffreys centroid and is suggested to use when the Jeffreys-Fisher-Rao center is not available in closed form. Moreover, this Gauss-Bregman inductive center always converges and matches the Jeffreys centroid for sets of same-mean normal distributions. We report on our experiments demonstrating the use of the Jeffreys-Fisher-Rao and Gauss-Bregman centers instead of the Jeffreys centroid. Finally, we conclude this work by reinterpreting these fast proxy centers of Jeffreys centroids under the lens of dually flat spaces in information geometry.

• The paper introduces novel proxy centers—the JFR and GB centers—to efficiently approximate the Jeffreys centroid for various probability distributions.
• It details how the JFR center leverages a Fisher-Rao midpoint in uniparametric exponential families, matching the Jeffreys centroid for same-mean normals.
• Experimental results show that both proxy centers reduce computational complexity while maintaining high accuracy, enhancing clustering and signal processing applications.

Overview of the Paper on Proxy Centers for Jeffreys Centroids

This paper addresses the challenge of computing the Jeffreys centroid for a set of probability distributions, focusing particularly on the practical difficulties due to the lack of closed-form solutions for many common distribution families, such as categorical and normal distributions. Two new proxy centers, the Jeffreys-Fisher-Rao (JFR) center and the Gauss-Bregman (GB) inductive center, are proposed as efficient approximations or replacements for the Jeffreys centroid in practical applications.

Jeffreys Centroid and Its Challenges

The Jeffreys centroid, based on the symmetric Kullback-Leibler divergence (Jeffreys divergence), provides a centrality notion that is highly useful in fields such as information retrieval and clustering. However, obtaining the Jeffreys centroid in closed form is often infeasible for many distribution types, necessitating numerical approximation methods that can be computationally expensive.

Proposed Proxy Centers

1. Jeffreys-Fisher-Rao Center (JFR Center):
   • This center defines the Fisher-Rao midpoint between the left and right KL centroids. It is formulated explicitly for uniparametric exponential family distributions and is shown to admit closed-form solutions for certain categorical and normal distributions.
   • Notably, it coincides with the Jeffreys centroid in the case of same-mean normal distributions.
2. Gauss-Bregman Inductive Center (GB Center):
   • Generalizing the arithmetic-geometric mean, this inductive center is constructed via a double sequence of arithmetic and quasi-arithmetic means. It is observed experimentally to closely approximate the Jeffreys centroid.
   • It always converges to the Jeffreys centroid in cases of same-mean normal distributions.

Numerical and Experimental Results

The paper reports strong numerical results indicating that both the JFR and GB centers effectively approximate the Jeffreys centroid, providing computational efficiency without significant loss of accuracy. Experiments demonstrate that these proxy centers perform well across different distribution sets, reducing computational complexity and time.

Implications and Future Directions

The introduction of these proxy centers presents significant implications for practical applications that rely on computing centroids across various domains such as image and sound processing. These centers offer more efficient computation without the loss of the theoretical robustness provided by the Jeffreys divergence.

Theoretically, the study opens avenues for exploring further generalizations within the framework of dually flat spaces in information geometry. Future work may extend these ideas to other distribution families and examine the theoretical properties and limitations of these proxy centers more rigorously.

In summary, the paper contributes new methods for approximating the computationally challenging Jeffreys centroid, enhancing the applicability of Jeffreys divergence in real-world applications through efficient and practical solutions.