Multi-Curvature Expert Mixtures

• Multi-curvature expert mixtures are models that integrate experts from diverse curved spaces to capture local complexities in data.
• They use context-sensitive gating mechanisms to adaptively blend contributions from experts in curved-exponential families and Riemannian manifolds.
• Optimization via KL-minimization and online EM ensures efficient parameter updates, improving performance in SMC and graph representation tasks.

Multi-curvature expert mixtures denote a family of mixture-of-experts models where the individual experts operate in spaces of differing geometric curvature, or in probability families with varying curvature in the sense of the exponential family. These architectures have emerged independently in the context of sequential Monte Carlo (SMC) proposal adaptation and geometric representation learning for graphs. Central to both is the notion of capturing local or context-specific complexity by adaptively combining information from a set of heterogeneous expert models, each attuned to a specific curvature, with gating mechanisms that learn to select or blend these contributions in a data-dependent way (Cornebise et al., 2011, Guo et al., 2024).

1. Foundations of Curvature in Mixture-of-Experts

Curvature arises as a critical attribute either of the statistical manifold underlying a distribution or of the geometric structure of a representation space. In the context of SMC, experts are instantiated as integrated curved-exponential distributions—statistical models generalizing the (linear) exponential family to non-zero curvature strata, including Student’s $t$ and Gaussian cases (Cornebise et al., 2011). For geometric machine learning, especially on graphs, curvature refers to the sectional curvature of Riemannian manifolds, where node embeddings can exploit negative, zero, or positive curvatures to capture diverse topological features (Guo et al., 2024).

The following table details the instantiations of expert curvature in these domains:

Context	Type of Curvature	Expert Example
SMC proposal adaptation	Curved exponential family	Gaussian, Student’s $t$
Graph geometric embeddings	Riemannian manifold	Poincaré ball, Sphere, Euclidean

2. Curved-Exponential and Riemannian Experts

In SMC, the objective is to approximate an intractable optimal proposal kernel $r^*(x_{k-1}, x_k)$ via a mixture of flexible curved-exponential distributions, each parameterized by natural parameters $\eta_j$ and sufficient statistics $T(\cdot)$. Gaussian and Student’s $t$ (via Normal–Gamma mixing) represent concrete expert instantiations within this framework. The marginalization over auxiliary latent variables enables the representation of heavy-tailed or skewed distributions (Cornebise et al., 2011).

For graph representation learning, each expert encodes a low-dimensional constant-curvature manifold $M^d_{\kappa_k}$ (with sectional curvature $\kappa_k$). Node embeddings $z_i^{(k)}$ are produced by independent Riemannian GNNs operating on their respective manifolds, leveraging the metric geometry (exponential and logarithmic maps, Möbius addition) to encode local graph topology and global geometric patterns (Guo et al., 2024).

3. Data-Dependent Gating Mechanisms

The mixture weights are modulated via gating networks that adaptively favor experts in accordance with the “context”:

• SMC context: Mixture weights $\omega_j(x_{k-1};\beta)$ depend on the ancestor particle $t$0 through a multinomial logistic model. This allows adaptation to local state-space geometry and distributional multimodality (Cornebise et al., 2011).
• Graph context: Node-specific weights $t$1 are determined by passing encoded summaries of a node’s multi-scale subgraphs through an MLP followed by softmax. Training guides the gating network towards expert configurations that yield minimal embedding distortion, targeting alignment with local topological characteristics (Guo et al., 2024).

Both approaches produce soft assignments, promoting a smooth adaptation to heterogeneity—in state-space transitions or in graph topology.

4. Optimization via KL-Minimization and Online EM

In the sequential Monte Carlo setting, mixture parameters and gating weights are optimized by minimizing the Kullback–Leibler divergence between the auxiliary target $t$2 and the instrumental distribution $t$3. An online EM algorithm, leveraging importance-weighted samples, updates both mixing coefficients and expert parameters through running averages. Closed-form M-step updates are possible for certain exponential-family members, such as Gaussians. The algorithm is computationally efficient: the adaptation overhead is $t$4 with $t$5 EM iterations and $t$6 mini-particles per step (Cornebise et al., 2011).

For graph mixtures, optimization involves a distortion criterion $t$7 aligning embedding geodesic distances with ground-truth graph distances, combined with task-specific losses. The gating network is regularized and possibly trained with weight-decay and curvature parameters learned via Riemannian Adam optimization (Guo et al., 2024).

5. Distance Alignment and Heterogeneous Space Fusion

In graph representation learning, embeddings resulting from different curvature experts must be consistently fused. This is achieved by:

• Mapping expert embeddings to a common alignment via weighted sums of geodesic distances, where joint expert assignment weights $t$8 are computed as softmax-normalized products of node-specific gate weights.
• Scalar multiplication operations $t$9 are utilized to blend embeddings in each curvature space, and the fused mixed-curvature embedding is obtained via direct concatenation or product-manifold construction.
• Explicit alignment losses may be employed, but in practice distortion minimization suffices (Guo et al., 2024).

In SMC, such alignment is inherent in the KL-divergence minimization between proposal and target.

6. Applications and Computational Considerations

In SMC, multi-curvature mixtures allow proposal kernels to flexibly handle multimodal or ill-conditioned filtering problems in nonlinear state-space models, maintaining near-linear scaling with the number of particles. Mixture initialization, covariance pooling, and regularization (e.g., fixed degrees of freedom for t-experts) address stability and identifiability (Cornebise et al., 2011).

In geometric graph embedding, these mixtures provide a principled framework to represent topological heterogeneity. Applications span node classification, link prediction, and foundation graph modeling. Per-node curvature adaptation leads to lower distortion and improved performance compared to embeddings in homogeneous or product-manifold spaces. Regularization, expert parameterization (fixed versus trainable curvature), and architectural design choices (number of curvature experts, scale of subgraph sampling) directly impact effectiveness and cost (Guo et al., 2024).

7. Connections and Broader Implications

The multi-curvature expert mixture framework formalizes the intuition that real-world sequential processes and data manifolds often exhibit non-uniform, locally varying complexity or curvature. By fitting expert mixtures using context-sensitive gating and explicit geometric priors, these approaches generalize classical mixture-of-experts and product manifolds. In SMC, this controls the trade-off between expressivity and computational feasibility of the proposal mechanism, while in geometric deep learning, it underpins scalable and flexible geometric inductive biases for heterogeneous data.

A plausible implication is that similar architectural patterns—mixtures of locally specialized, curvature-adaptive experts—may be advantageous in other domains characterized by heterogeneity, non-Gaussianity, or nonconstant curvature in latent or data spaces.