A Hierarchical Decomposition of Kullback-Leibler Divergence: Disentangling Marginal Mismatches from Statistical Dependencies (2504.09029v3)

Abstract: The Kullback-Leibler (KL) divergence is a foundational measure for comparing probability distributions. Yet in multivariate settings, its single value often obscures the underlying reasons for divergence, conflating mismatches in individual variable distributions (marginals) with effects arising from statistical dependencies. We derive an algebraically exact, additive, and hierarchical decomposition of the KL divergence between a joint distribution P(X1,...,Xn) and a standard product reference distribution Q(X1,...,Xn) = product_i q(Xi), where variables are assumed independent and identically distributed according to a common reference q. The total divergence precisely splits into two primary components: (1) the summed divergence of each marginal distribution Pi(Xi) from the common reference q(Xi), quantifying marginal deviations; and (2) the total correlation (or multi-information), capturing the total statistical dependency among variables. Leveraging Mobius inversion on the subset lattice, we further decompose this total correlation term into a hierarchy of signed contributions from distinct pairwise, triplet, and higher-order statistical interactions, expressed using standard Shannon information quantities. This decomposition provides an algebraically complete and interpretable breakdown of KL divergence using established information measures, requiring no approximations or model assumptions. Numerical validation using hypergeometric sampling confirms the decomposition's exactness to machine precision across diverse system configurations.