Rao’s Quadratic Entropy: Theory & Applications

• Rao’s Quadratic Entropy is a concave functional that measures distribution diversity through pairwise relationships, forming the basis for geometry-aware statistical distances.
• It generalizes techniques like Maximum Mean Discrepancy by leveraging conditionally negative semi-definite kernels and enables closed-form expressions for mixtures of Gaussians.
• Its interpretable hyperparameters facilitate fine-tuning for practical machine learning tasks, including density estimation, generative modeling, and mixture simplification.

Rao’s quadratic entropy is a concave entropy functional that enables the derivation of statistical distances for probability distributions by leveraging the geometry of their sample spaces. In the context of machine learning, it supports the construction of geometry-aware divergence measures, notably those that generalize the popular Maximum Mean Discrepancy (MMD) framework to allow a broader class of kernels, specifically, conditionally negative semi-definite kernels. Distances derived from Rao’s quadratic entropy, such as those termed Schoenberg-Rao distances, provide interpretable hyperparameters and exhibit theoretical properties suitable for machine learning applications, including cases where probability distributions have disjoint supports (Hadjeres et al., 2020).

1. Definition and Entropic Properties

Rao’s quadratic entropy is a concave functional on the space of probability distributions. Unlike the Shannon entropy, which is logarithmic, the quadratic entropy is based on measuring the dispersion or diversity of a distribution through pairwise relationships. This entropy provides a foundation for defining statistical distances that are sensitive to the geometry of the probability space. The concavity of Rao’s quadratic entropy implies that linear combinations of distributions generally have higher entropy, a property that can be exploited in constructing divergence measures (Hadjeres et al., 2020).

2. Generalization of MMD via Conditioned Kernels

Distances such as MMD traditionally rely on positive definite kernels and are widely used for distribution comparison. The Schoenberg-Rao distance, derived from Rao’s quadratic entropy, extends the MMD paradigm by supporting a broader class of kernels—specifically, the conditionally negative semi-definite class. This extension is crucial as it enables practitioners to construct kernel-based distances capable of incorporating richer geometric information about the underlying probability space, especially in scenarios where standard MMD may not be optimal (Hadjeres et al., 2020).

3. Construction of Statistical Hilbert Distances

Statistical Hilbert distances are constructed using kernels derived in a principled manner from Rao’s quadratic entropy. The methodology involves carefully formulating distances that respect the concavity of the entropy functional, while also ensuring that the derived kernel satisfies the requisite conditional negative semi-definiteness. These constructions permit deriving closed-form expressions for distances between specific classes of distributions, such as mixtures of Gaussians (Hadjeres et al., 2020). This suggests a practical advantage by reducing the computational cost and improving interpretability in concrete applications.

4. Closed-Form Expressions for Gaussian Mixtures

A notable outcome of leveraging Rao’s quadratic entropy in the kernel construction is the ability to derive closed-form distances between mixtures of Gaussian distributions. These formulae avoid the need for costly numerical approximations, thereby making the distances particularly attractive in high-dimensional density estimation and generative modeling contexts. A plausible implication is that efficiency gains and numerical stability are achieved compared to approaches relying on Wasserstein distances (Hadjeres et al., 2020).

5. Hyperparameter Interpretability and Tuning

Distances derived from Rao’s quadratic entropy possess hyperparameters that are interpretable within their geometric framework, allowing fine control over the metric’s sensitivity and scale. This interpretability facilitates tuning for specific machine learning applications, such as optimizing for density estimation accuracy or enhancing generative modeling capabilities (Hadjeres et al., 2020). This suggests a practical advantage: hyperparameter choice can be informed by domain knowledge, leading to better empirical performance.

6. Applications in Machine Learning

The entropy-based statistical distances exhibiting geometry-awareness have demonstrated efficiency across various machine learning tasks. These include density estimation, generative modeling, and mixture simplification. The principled framework underlying these Schoenberg-Rao distances provides a practical alternative to Wasserstein distances, especially in scenarios where standard metrics may falter, such as comparing distributions with disjoint supports (Hadjeres et al., 2020). A plausible implication is that practitioners may opt for these distances when computational cost or geometric fidelity is a practical concern.