Finite Mixture Modeling with Riemannian Gaussian Distributions on Hyperbolic Space

Abstract: Hyperbolic space is increasingly used for hierarchical, tree-like, and network-structured data, but likelihood-based density modeling on hyperbolic space remains relatively limited. This paper develops finite mixture modeling with isotropic Riemannian Gaussian distributions on hyperbolic space under the hyperboloid model. We derive the density, radial normalizing constant, and a finite-sum representation involving the complementary error function. We then formulate weighted maximum likelihood estimation, which is the fundamental subproblem in mixture fitting: the location estimator is the weighted Fréchet mean, while the inverse-scale estimator is obtained from a one-dimensional strictly convex profile problem. For finite mixtures, we derive exact EM and generalized EM algorithms. The generalized version replaces exact barycenter solves with truncated hyperbolic majorization-minimization updates. We establish existence and uniqueness of the weighted single-component estimator, singularity of the unrestricted mixture likelihood, existence of a constrained mixture estimator, and monotonicity properties of the EM-type algorithms. Simulations show accurate weighted estimation, reliable mixture recovery, effective model selection, and substantial computational savings from generalized EM. Real network examples based on hyperbolic embeddings illustrate the method as an exploratory likelihood-based clustering tool for non-Euclidean data.