Spherical Barycentric Aggregation in MoE Models

• SBA is a geometry-preserving aggregation operator that decomposes expert outputs into radial and angular components, maintaining the hyperspherical structure.
• It replaces standard linear summation in MoE models by aggregating magnitudes and directions separately, preventing inward collapse of output norms.
• Empirical results show SBA enhances embedding comparability, improving performance in semantic similarity, clustering, and duplicate question detection.

Spherical Barycentric Aggregation (SBA) denotes an aggregation paradigm in which data supported on a sphere or hypersphere are combined in a manner that respects spherical geometry rather than treating the ambient space as an unconstrained Euclidean vector space. In contemporary machine learning usage, SBA refers most specifically to the geometry-preserving operator introduced for Mixture-of-Experts (MoE) embedding models, where expert outputs are empirically observed to lie on a shared hyperspherical manifold and are therefore aggregated by separating radial and angular components rather than by weighted linear summation (Kachuee et al., 15 Feb 2026). Closely related geometric constructions also appear in the literature on spherical barycentric coordinates, spherical area coordinates, and intrinsic barycenters on curved spaces, where aggregation is formulated through area ratios on $S^2$, generalized coordinates on spherical polygons, conformal barycenters in the Poincaré ball, or minimax barycenters in CAT$(k)$ spaces [(Fillmore et al., 2014); (Aitelhad, 2022); (Hamilton, 2021); (Cantarella et al., 2020); (Girolo, 2008)].

1. Definition and terminological scope

In "Geometry-Preserving Aggregation for Mixture-of-Experts Embedding Models" (Kachuee et al., 15 Feb 2026), SBA is introduced as a drop-in replacement for the standard weighted linear summation used to combine expert outputs in MoE embedding models. Its defining feature is explicit respect for the hyperspherical structure empirically observed in expert representations: each expert output is decomposed into a magnitude and a direction, the magnitudes are aggregated separately from the directions, and the final output is reconstructed on the hypersphere. Routing, expert computation, and training objectives remain unchanged.

The term is not used uniformly across the broader spherical-geometry literature. In particular, the paper on spherical point-set generation states that the exact term “Spherical Barycentric Aggregation (SBA)” does not appear there, but that the underlying procedure is conceptually an aggregation of points determined by spherical barycentric, or area-based, coordinates over spherical triangles (Hamilton, 2021). This suggests that “SBA” is best treated as a context-dependent label rather than a single canonical construction. In machine learning, it presently denotes a specific MoE aggregation operator; in geometric processing and approximation theory, related constructions emphasize interpolation, quadrature, or point placement on $S^2$.

2. Geometric motivation in MoE embedding models

The motivation for SBA in MoE embeddings is a geometric inconsistency between conventional MoE aggregation and the structure of learned embeddings. Standard MoE layers combine selected expert outputs by weighted linear summation,

$$y_{\text{linear}}=\sum_{i\in \text{Top-}K} w_i e_i.$$

This assumes that expert outputs reside in compatible linear subspaces and that linear interpolation preserves semantic structure. The paper argues that this assumption is nontrivial in embedding models where cosine similarity and angular relations are fundamental (Kachuee et al., 15 Feb 2026).

The reported geometric analysis of the sparse MoE embedding model nomic-embed-text-v2-moe extracts expert outputs before aggregation across MoE layers on a large corpus (MMLU) and identifies two empirical regularities. First, the relative norm ratios $r_1/r_2$ between the top-2 active experts are tightly concentrated around $1$, where $r_i=\|e_i\|$. Second, pairwise angular distances between active experts are mostly above $40^\circ$, with angles defined by

$$\theta_{ij}=\arccos(\hat{u}_i^\top \hat{u}_j), \qquad \hat{u}_i=\frac{e_i}{\|e_i\|}.$$

Taken together, these observations imply a shared hyperspherical manifold $S^{d-1}$ on which specialization is encoded primarily by direction rather than by norm.

Under this geometry, linear aggregation induces inward collapse. If expert norms are comparable, $(k)$0, then

$(k)$1

Unless all expert directions are aligned, angular diversity causes partial cancellation, so that $(k)$2. The resulting output is shifted toward the interior of the hypersphere, with both magnitude and direction distorted. The paper argues that this reduces embedding comparability when cosine similarity is used downstream, and it validates the claim by measuring the relative output norm ratio $(k)$3, where $(k)$4: linear aggregation produces systematically reduced norms, whereas SBA keeps ratios tightly around unity (Kachuee et al., 15 Feb 2026).

3. Operator definition and integration

SBA decomposes each routed expert output $(k)$5 into radial and angular components,

$(k)$6

The aggregation then respects the hyperspherical structure by combining magnitudes and directions separately. In the top-2 case used for exposition in the paper, the radius is aggregated as

$(k)$7

the angular coordinate is aggregated as

$(k)$8

and the output is reconstructed as

$(k)$9

For general top-$S^2$0 routing, the paper gives

$S^2$1

A central point of interpretation is that this construction is called “Spherical Barycentric Aggregation,” but it is not formulated through a Fréchet mean objective on $S^2$2, and the reported implementation does not use Riemannian log/exp maps, geodesic distances, slerp, or Karcher-mean optimization. Its novelty lies instead in a lightweight, norm-aware angular averaging together with a weighted barycenter of magnitudes (Kachuee et al., 15 Feb 2026).

Operationally, SBA is a drop-in integration into each MoE layer. For each active expert in top-$S^2$3, the model computes $S^2$4, extracts $S^2$5 and $S^2$6, maps $S^2$7 to the angular coordinate $S^2$8, aggregates $S^2$9 and $$y_{\text{linear}}=\sum_{i\in \text{Top-}K} w_i e_i.$$0, reconstructs $$y_{\text{linear}}=\sum_{i\in \text{Top-}K} w_i e_i.$$1, and outputs $$y_{\text{linear}}=\sum_{i\in \text{Top-}K} w_i e_i.$$2. Subsequent network layers and loss computation proceed as usual. The implementation is differentiable, because it uses norms, weighted averages, and conversions between angular coordinates and unit vectors, so it is compatible with standard backpropagation without modifying routing or gating weights. The paper states that the extra computation is negligible relative to linear summation, with $$y_{\text{linear}}=\sum_{i\in \text{Top-}K} w_i e_i.$$3 work per token, and reports identical training cost and full stability (Kachuee et al., 15 Feb 2026).

4. Empirical performance and ablations

The experimental study compares a Linear MoE baseline with an SBA MoE variant on nomic-ai/nomic-embed-text-v2-moe with top-2 routing. The implementation uses PyTorch and HuggingFace, fine-tunes for $$y_{\text{linear}}=\sum_{i\in \text{Top-}K} w_i e_i.$$4 epochs on $$y_{\text{linear}}=\sum_{i\in \text{Top-}K} w_i e_i.$$5K triplets on a single NVIDIA T4 GPU, and keeps all settings identical except the aggregation operator: AdamW, learning rate $$y_{\text{linear}}=\sum_{i\in \text{Top-}K} w_i e_i.$$6, linear learning-rate schedule with $$y_{\text{linear}}=\sum_{i\in \text{Top-}K} w_i e_i.$$7 warmup, and gradient accumulation for effective batch size $$y_{\text{linear}}=\sum_{i\in \text{Top-}K} w_i e_i.$$8. The loss is Multiple Negatives Ranking Loss (InfoNCE) with cosine similarity and $$y_{\text{linear}}=\sum_{i\in \text{Top-}K} w_i e_i.$$9. Evaluation uses embeddings from the final MoE layer on selected MTEB tasks with cosine similarity, and results are averaged over three seeds (Kachuee et al., 15 Feb 2026).

Task	Linear MoE	SBA MoE
STSBenchmark	71.18	71.51
StackExchangeClustering	85.66	86.19
SprintDuplicateQuestions	83.66	89.06

The reported pattern is consistent across semantic similarity, clustering, and duplicate question detection. The largest gain, greater than $r_1/r_2$0 points, occurs on SprintDuplicateQuestions. The paper interprets this as evidence that geometry-preserving aggregation improves pairwise discrimination, while the gains on STSBenchmark and StackExchangeClustering indicate broader improvements in embedding quality (Kachuee et al., 15 Feb 2026).

The ablation study separates the effect of radial and angular treatment. A norm-free angular aggregation variant uses

$r_1/r_2$1

Its reported scores are $r_1/r_2$2 on STSBenchmark, $r_1/r_2$3 on StackExchangeClustering, and $r_1/r_2$4 on SprintDuplicateQuestions. The paper states that removing norm information from angular averaging slightly degrades most tasks compared to full SBA, but slightly improves duplicate question detection, likely because pure angular averaging emphasizes directional contrast for cosine-based ranking. By contrast, a unit-normalized output variant,

$r_1/r_2$5

severely harms clustering, with scores $r_1/r_2$6 on STSBenchmark, $r_1/r_2$7 on StackExchangeClustering, and $r_1/r_2$8 on SprintDuplicateQuestions. This is presented as evidence that the aggregated radius carries discriminative information for structured embedding spaces (Kachuee et al., 15 Feb 2026).

5. Spherical barycentric coordinates and area-based aggregation on $r_1/r_2$9

Outside MoE embeddings, spherical barycentric aggregation is closely related to area-ratio coordinates on spherical simplices and polygons. For spherical triangles, one construction defines barycentric coordinates by subtriangle area ratios. If $1$0 are the triangle vertices and $1$1 lies in or on the triangle, then

$1$2

with $1$3. The corresponding spherical interpolant is

$1$4

The same framework develops Whitney $1$5-, $1$6-, and $1$7-forms on spherical triangles, including an exactly normalized Whitney $1$8-form satisfying $1$9, all expressed directly through dot products and $r_i=\|e_i\|$0 determinants without requiring a coordinate chart or projection (Fillmore et al., 2014).

For general spherical polygons contained in a common hemisphere, another construction derives spherical barycentric coordinates by transporting $r_i=\|e_i\|$1-dimensional generalized barycentric coordinates of the origin to the unit sphere. In the mean-value case this yields normalized spherical mean value coordinates

$r_i=\|e_i\|$2

where $r_i=\|e_i\|$3 and $r_i=\|e_i\|$4 is the angle between consecutive great-circle directions around $r_i=\|e_i\|$5. These coordinates satisfy the Lagrange property at vertices, edge linearity along geodesic edges, positivity under the common-hemisphere assumption, and $r_i=\|e_i\|$6 evaluation complexity per query point. The same paper proves that spherical mean value coordinates from the $r_i=\|e_i\|$7-dimensional origin-based construction coincide with the classical projection-induced spherical mean value coordinates, जबकि spherical Wachspress coordinates obtained by the two approaches are in general different (Aitelhad, 2022).

A further use of barycentric ideas on $r_i=\|e_i\|$8 appears in spherical point-set generation. There, planar barycentric coordinates on a Caspar–Klug lattice are reinterpreted as spherical area coordinates on each spherical triangle of a triangulation, and the induced points are aggregated globally across all faces. The paper reports that recursive spherical area coordinate sequences such as $r_i=\|e_i\|$9, $40^\circ$0, and $40^\circ$1 achieve the lowest mesh ratios and appear bounded, with $40^\circ$2 seeming bounded by approximately $40^\circ$3 in the reported computations (Hamilton, 2021). In this setting, “aggregation” refers not to combining feature vectors but to assembling a global spherical configuration from local barycentric placements.

6. Intrinsic barycenters, misconceptions, and limitations

SBA is often associated with more intrinsic notions of barycenter on curved spaces, but these constructions are mathematically distinct. One prominent example is the conformal barycenter of Douady–Earle, defined for a probability measure $40^\circ$4 on the unit sphere $40^\circ$5 at infinity of the Poincaré ball model $40^\circ$6 by the zero of the vector field

$40^\circ$7

This barycenter lies in the interior of the Poincaré ball, is conformally natural under Möbius isometries, is unique for stable measures, and can be computed with Newton or regularized Newton methods that have explicit convergence guarantees under Newton–Kantorovich conditions (Cantarella et al., 2020). It is therefore an intrinsic hyperbolic construction, not merely an angular average on a Euclidean embedding.

A different notion appears in CAT$40^\circ$8 geometry, where the barycenter of a bounded set $40^\circ$9 relative to a bounded non-negative weight function $$\theta_{ij}=\arccos(\hat{u}_i^\top \hat{u}_j), \qquad \hat{u}_i=\frac{e_i}{\|e_i\|}.$$0 is defined as the unique minimizer of the minimax functional

$$\theta_{ij}=\arccos(\hat{u}_i^\top \hat{u}_j), \qquad \hat{u}_i=\frac{e_i}{\|e_i\|}.$$1

When $$\theta_{ij}=\arccos(\hat{u}_i^\top \hat{u}_j), \qquad \hat{u}_i=\frac{e_i}{\|e_i\|}.$$2, this reduces to the circumcenter. On a sphere $$\theta_{ij}=\arccos(\hat{u}_i^\top \hat{u}_j), \qquad \hat{u}_i=\frac{e_i}{\|e_i\|}.$$3, the cited result gives existence and uniqueness under the sufficient condition $$\theta_{ij}=\arccos(\hat{u}_i^\top \hat{u}_j), \qquad \hat{u}_i=\frac{e_i}{\|e_i\|}.$$4 in the weighted case, with the unweighted circumcenter allowing the weaker bound $$\theta_{ij}=\arccos(\hat{u}_i^\top \hat{u}_j), \qquad \hat{u}_i=\frac{e_i}{\|e_i\|}.$$5 (Girolo, 2008). This is again a fundamentally different object from the MoE operator in (Kachuee et al., 15 Feb 2026): it is a weighted spherical $$\theta_{ij}=\arccos(\hat{u}_i^\top \hat{u}_j), \qquad \hat{u}_i=\frac{e_i}{\|e_i\|}.$$6-center defined by a worst-case geodesic criterion.

A common misconception is therefore to identify every “spherical barycentric” construction with a Fréchet mean or another intrinsically geodesic optimizer. The MoE SBA paper explicitly states that its operator does not employ the Riemannian Fréchet/Karcher mean, slerp, or log/exp maps on $$\theta_{ij}=\arccos(\hat{u}_i^\top \hat{u}_j), \qquad \hat{u}_i=\frac{e_i}{\|e_i\|}.$$7, and its claimed advantage is instead full compatibility with existing routing mechanisms and pretrained weights together with improved embedding comparability at identical training cost (Kachuee et al., 15 Feb 2026). Another misconception is that preserving only direction is sufficient on hyperspherical manifolds: the unit-normalized ablation shows that discarding the aggregated radius can severely damage clustering.

The present evidence base for the MoE formulation is also delimited in the paper itself. The study evaluates one MoE embedding architecture, assumes a hyperspherical structure supported by the reported analysis, and focuses on top-$$\theta_{ij}=\arccos(\hat{u}_i^\top \hat{u}_j), \qquad \hat{u}_i=\frac{e_i}{\|e_i\|}.$$8, with behavior for larger $$\theta_{ij}=\arccos(\hat{u}_i^\top \hat{u}_j), \qquad \hat{u}_i=\frac{e_i}{\|e_i\|}.$$9 left for additional study. This suggests that future work will need to determine how broadly the reported hyperspherical regularities and performance gains transfer across architectures, objectives, and routing regimes (Kachuee et al., 15 Feb 2026).