Hierarchical Adaptive Mixture of Experts (HA-MoE)

• Hierarchical Adaptive Mixture of Experts (HA-MoE) is a family of models that organizes multiple levels of experts to better align with structured data, tasks, and hardware.
• HA-MoE architectures implement tree-structured, two-level, and multigranular routing mechanisms that enable adaptive specialization and efficient feature aggregation.
• Empirical and theoretical studies demonstrate that adaptive gating and hierarchical partitioning in HA-MoE enhance performance, though challenges like training stability and expert polarization remain.

Taken together, recent literature suggests that Hierarchical Adaptive Mixture of Experts (HA-MoE) is best understood as a family of mixture-of-experts architectures in which expert organization, routing, or feature aggregation is structured over multiple levels and adapts to signals such as joint input-output separability, environmental context, modality, time, spatial locality, program granularity, or expert groups. In this family, hierarchy can appear as a binary tree of classifier and regression nodes, a grouped two-level router, a local-to-global token-routing stack, or a nested multilevel statistical model. A plausible unifying interpretation is that HA-MoE separates specialization decisions across levels so that experts can align with structure in data, tasks, or hardware, rather than relying on a single flat gate (Zhao et al., 2019, Molodtsov et al., 8 May 2026, Fung et al., 2022).

1. Genealogy and conceptual scope

Early ingredients of the HA-MoE perspective appear in architectures that adaptively aggregate hierarchical representations rather than using fixed fusion. In scene parsing, MoE-SPNet introduced a convolutional mixture-of-experts layer that assesses the importance of features from different levels and at different spatial locations, and its adaptive hierarchical feature aggregation (AHFA) variant re-weights hierarchical convolutional features according to local structure and surrounding contextual information (Fu et al., 2018). In regression, the hierarchical routing mixture of experts (HRME) formalized a binary tree with classifiers at non-leaf nodes and simple regression experts at leaves, explicitly arguing that partitioning the joint input-output space can simplify multimodal regression (Zhao et al., 2019).

Subsequent work broadened the meaning of hierarchy. For multilevel statistics, mixed mixture of experts (MMoE) extended MoE to multilevel data by embedding random effects in the gating functions, and a nested MMoE was shown to universally approximate a broad range of dependence structures for hierarchical data (Fung et al., 2022). For high-level synthesis, a two-level hierarchical MoE organized experts over node, basic block, and graph granularities, then used a high-level MoE to aggregate those predictions for improved domain generalization on new kernels (Li et al., 2024). In large-scale HMoE analysis, the question of hierarchy shifted from structure alone to the gating function itself, with Laplace gating proposed as an alternative to Softmax at both levels (Nguyen et al., 2024).

Recent 2025–2026 work further diversified the design space. MixER framed hierarchical MoE for dynamical system reconstruction around environmental context vectors and sparse top-1 routing (Nzoyem et al., 7 Feb 2025); HAEMSA used hierarchical adaptive experts for multimodal sentiment analysis with evolutionary optimization and multi-task learning (Qin et al., 25 Mar 2025); HM4SR organized recommendation through an Interactive MoE followed by a Temporal MoE (Zhang et al., 24 Jan 2025); HoME implemented local and global Soft Mixture-of-Experts layers for 3D medical image segmentation (Płotka et al., 8 Jul 2025); and Hi-MoE decomposed routing control into inter-group balancing and intra-group specialization for sparse expert models (Molodtsov et al., 8 May 2026). The literature therefore uses “hierarchical” in several non-equivalent but related senses: recursive partitioning, multi-stage fusion, grouped routing, multigranular aggregation, and nested probabilistic structure.

2. Canonical architectural patterns

Several recurring architectural motifs define HA-MoE systems. One motif is tree-structured routing, where non-leaf nodes decide increasingly specialized partitions and leaves execute task experts. Another is two-stage or two-level routing, in which a first level performs local, modality-specific, or coarse grouping, and a second level performs temporal, global, or higher-level aggregation. A third motif is multigranular composition, where separate expert modules operate on different resolutions or structural units and a higher-level gate combines them. A fourth is nested statistical gating, where hierarchical dependencies enter through random effects in the gating functions (Zhao et al., 2019, Płotka et al., 8 Jul 2025, Li et al., 2024, Fung et al., 2022).

System	Hierarchical organization	Adaptive signal
HRME	Binary tree; classifiers at non-leaf nodes; regression experts at leaves	Joint input-output partitioning via local threshold $t$ and classifier $h_\beta$
HM4SR	Interactive MoE followed by Temporal MoE	Modality embedding, then explicit temporal embeddings
HoME	Local SMoE followed by global SMoE	Grouped slot assignment and soft token routing
Hi-MoE	Inter-group and intra-group routing control	Group balancing and within-group specialization
Hierarchical MoE for HLS	Low-level MoEs at node, block, and graph levels plus a high-level MoE	Program granularity and learned aggregation

In HRME, the hierarchy is literal: internal nodes are classifiers $h_\beta$ with a threshold $t$, and leaves are regressors $r_\theta$. In HM4SR, the hierarchy is pipeline-like: the first MoE extracts essential user interest-related information from the multi-modal data of each item, and the second MoE captures user dynamic interests by introducing explicit temporal embeddings from timestamps in modality encoding (Zhang et al., 24 Jan 2025). In HoME, hierarchy is local-to-global: a first-stage SMoE partitions input sequences into local groups for specialized per-group experts, while a second-stage global SMoE aggregates those outputs for cross-group information fusion (Płotka et al., 8 Jul 2025). In high-level synthesis, hierarchy is multigranular rather than spatial or temporal: low-level MoEs operate at node, basic block, and graph scales, and the high-level MoE learns to aggregate the three granularities for the final decision (Li et al., 2024).

A plausible implication is that HA-MoE is less a single topology than a principle of staged specialization. What remains stable across variants is the attempt to avoid a flat expert pool whose routing burden is concentrated in one gate.

3. Routing, gating, and optimization mechanisms

Routing mechanisms in HA-MoE differ substantially. HRME uses probabilistic hierarchical routing through a binary tree. For a node $n_i$, the routing probability is written as

$$q(z_{n_i} \mid \mathbf{x}) \longleftarrow h_{\beta_{n_{i-1}}^*}(\mathbf{x}),$$

the path probability to a leaf $l_k$ is

$$q(z_{l_k} \mid \mathbf{x}) = \prod_{j=1}^{k-1} q(z_{l_{j+1}} \mid z_{l_j}, \mathbf{x}),$$

and prediction is the leaf-wise expectation

$$\hat{y} = \sum_{l_k \in \textnormal{leaves}} r_{\theta_{l_k}}^*(\mathbf{x}) \, q(z_{l_k} \mid \mathbf{x}).$$

Its learning procedure is a recursive EM-based algorithm that jointly learns tree structure and expert models, with a practical alternative split criterion $h_\beta$0 (Zhao et al., 2019).

Other HA-MoE systems replace or augment gradient-based gating. MixER argues that naive MoEs are inadequate for hierarchical dynamical system reconstruction because gradient descent-based gating updates lead to slow updates and conflicted routing. Its sparse top-1 MoE layer routes on an environmental context vector $h_\beta$1 and updates gating through K-means clustering, expert-cluster pairing, and a least-squares fit of a linear gate $h_\beta$2 to one-hot assignments (Nzoyem et al., 7 Feb 2025). HAEMSA pushes adaptivity further by using evolutionary algorithms to dynamically optimize network architectures and modality combinations, while also combining cross-modal knowledge transfer and multi-task learning (Qin et al., 25 Mar 2025).

Grouped sparse MoE introduces yet another optimization regime. Hi-MoE factorizes routing as

$h_\beta$3

and couples standard task and load-balancing losses with an inter-group regularizer and an intra-group anti-overlap regularizer: $h_\beta$4 This decomposition is explicitly intended to separate global load balancing from local specialization (Molodtsov et al., 8 May 2026).

Training stability is a recurring issue. The hierarchical MoE for HLS reports “expert polarization” when naive end-to-end training causes the high-level gate to over-favor the fastest-converging low-level MoE; its remedy is a two-stage training strategy with low-level warmup followed by alternation between joint and individual training (Li et al., 2024). In HMoE estimation theory, the gating function itself becomes the optimization variable of interest: replacing Softmax with Laplace gating is argued to reduce undesirable parameter interactions and accelerate expert convergence and specialization (Nguyen et al., 2024).

4. Domain-specific instantiations

HA-MoE has been instantiated across a wide range of problem classes. In regression, HRME targets multimodal-distributed data by jointly soft-partitioning the input-output space so that simple leaf experts, such as linear regressors or SVR, become effective for prediction. Experiments were reported on a synthetic “3-lines” dataset and on Boston Housing, Concrete, Energy, CCPP, and Kin40k, with the model described as effective compared to a variety of other regression models (Zhao et al., 2019).

In scientific machine learning, MixER addresses dynamical system reconstruction from time series with multi-level data structure in which environments are grouped into families. It is validated on systems of up to ten parametric ordinary differential equations and is reported to outperform vanilla MoE and to match or beat flat meta-learners when environments are dissimilar, while underperforming state-of-the-art meta-learners in high-data regimes where experts only process fractions of highly related datasets (Nzoyem et al., 7 Feb 2025). This sharply conditions the usefulness of hierarchy on the presence of actual hierarchical structure in the data.

In multimodal affective computing, HAEMSA uses fine-grained modality-specific experts, a unified modality-shared expert, hierarchical fusion layers, task-driven attention gates, multi-task output layers, evolutionary architecture search, and KL-divergence-based knowledge transfer. Reported gains include a 2.6% increase in 7-class accuracy and a 0.059 decrease in MAE on CMU-MOSEI, a 6.3% improvement in 7-class accuracy and a 0.058 reduction in MAE on CMU-MOSI, and a 2.84% weighted-F1 gain on IEMOCAP (Qin et al., 25 Mar 2025). In recommendation, HM4SR uses an Interactive MoE and a Temporal MoE together with three auxiliary tasks—category prediction, ID contrastive learning, and placeholder contrastive learning—to address redundant multimodal information, explicit temporal signals, and data sparsity (Zhang et al., 24 Jan 2025).

In computer vision and medical imaging, hierarchical expert fusion appears in both convolutional and state-space settings. MoE-SPNet and AHFA adaptively re-weight hierarchical convolutional features at each spatial location for scene parsing on PASCAL VOC 2012 and SceneParse150 (Fu et al., 2018). HoME, built on the Mamba state-space model backbone, introduces a two-level token-routing layer for 3D medical image segmentation, with a local SMoE for localized feature extraction and a global SMoE for cross-group information fusion; it is described as surpassing state-of-the-art results across datasets from CT, MRI, and US and evaluated with Dice and Hausdorff metrics in the detailed summary (Płotka et al., 8 Jul 2025).

In systems and foundation-model settings, hierarchy serves generalization, interpretability, and efficiency. For high-level synthesis, the hierarchical MoE is evaluated on HLSyn with 42 diverse kernels; when trained on all but six target kernels and fine-tuned with $h_\beta$5 examples per target kernel, it is reported to outperform baselines in both MSE and online search speedup, achieving a 26.6% higher geometric mean speedup than the best non-MoE baseline (Li et al., 2024). In HMoE with Laplace gating, empirical studies span MIMIC-IV multimodal prediction, CIFAR-10, ImageNet, and latent domain discovery on MIMIC-IV and eICU (Nguyen et al., 2024). For MoE LLMs, hierarchical sparse dictionary learning (HSDL) identifies expert collaboration patterns, and the contribution-aware expert pruning (CAEP) algorithm improves overall performance by 2.5\% on average while pruning low-contribution experts (Tang et al., 16 Apr 2025).

5. Theoretical perspectives

Several theoretical programs support the HA-MoE literature, but they address different questions. For multilevel data, MMoE provides an approximation-theoretic account. Its conditional model embeds random effects only in the gate: $h_\beta$6 with

$h_\beta$7

Under regularity conditions, the class of MMoE models is compactly dense in the class of continuous mixed-effects models, and nested MMoE is compactly dense in the class of generalized hierarchical mixed models in the sense of weak convergence (Fung et al., 2022). This means that hierarchy can be justified not only architecturally but also as a universal approximation framework for multilevel dependence.

A different theoretical line studies the estimation behavior of hierarchical gating. For two-level Gaussian HMoE,

$h_\beta$8

and the analysis compares Softmax–Softmax, Softmax–Laplace, and Laplace–Laplace gating. In exact-specified settings, all three achieve $h_\beta$9, whereas in over-specified settings Laplace–Laplace achieves $h_\beta$0. The paper attributes this to fewer and simpler parameter interactions, faster convergence, and improved expert specialization under Laplace gating (Nguyen et al., 2024).

A third line emphasizes the balance-specialization trade-off in sparse MoE systems. Hi-MoE formalizes an inter-group regularizer that controls group-level coefficient of variation and an intra-group anti-overlap regularizer that reduces gradient coupling. Its analysis states that

$h_\beta$1

and

$h_\beta$2

With balanced marginals, maximizing $h_\beta$3 is also linked to conditional mutual information between tokens and expert identity (Molodtsov et al., 8 May 2026). This suggests that hierarchy can be used not merely to add experts, but to decompose optimization objectives that would otherwise conflict in a flat router.

6. Empirical regularities, limitations, and interpretive issues

Across the literature, HA-MoE is repeatedly associated with improved specialization, better alignment to structured heterogeneity, and greater control over routing. HRME argues that simple leaf experts can be effective when routing partitions the joint input-output space well (Zhao et al., 2019). MixER reports that context-based gating becomes meaningful earlier than gradient-based soft-gated models and that the quality of contextual representations is closely linked to the presence of hierarchical structure in the data (Nzoyem et al., 7 Feb 2025). Hi-MoE reports consistent improvements over sparse-routing and grouped-MoE baselines, including a 5.6% perplexity reduction and a 40% improvement in expert balance over OLMoE-7B in 58B-token pre-training (Molodtsov et al., 8 May 2026). HSDL analysis in MoE LLMs further indicates that expert collaboration patterns are task-dependent and semantically meaningful, and that pruning domain-specific experts disproportionately harms the corresponding domains (Tang et al., 16 Apr 2025).

The same literature also places clear limits on hierarchical adaptivity. MixER underperforms state-of-the-art meta-learners in high-data regimes, particularly when each expert is constrained to process only a fraction of a dataset composed of highly related data points (Nzoyem et al., 7 Feb 2025). The HLS model requires a two-stage training method specifically to avoid expert polarization (Li et al., 2024). Hi-MoE is motivated by the observation that strong load balancing can suppress expert specialization while aggressive diversity can cause routing collapse (Molodtsov et al., 8 May 2026). HAEMSA’s gains come with computational overhead from population-based evolutionary optimization, and the detailed summary identifies missing or noisy modalities as a logical next step (Qin et al., 25 Mar 2025). These results jointly caution against the assumption that “more hierarchy” or “more experts” is automatically beneficial.

Several common misconceptions can therefore be corrected. First, HA-MoE is not synonymous with a hard decision tree: the literature includes binary trees, soft local-to-global routers, grouped sparse selection, multigranular fusion, and nested mixed-effects gates. Second, adaptivity is not reducible to a Softmax gate trained by gradient descent: K-means plus least squares, Laplace gating, evolutionary search, context clustering, and task-driven attention all appear as adaptive mechanisms. Third, expert specialization is not purely an architectural property; it depends on optimization, regularization, and the strength of structure in the data.

A plausible synthesis is that HA-MoE becomes most effective when the hierarchy in the model matches a hierarchy already present in the problem—modal, temporal, spatial, environmental, statistical, semantic, or hardware-level. Where that match is weak, forced partitioning can waste capacity, fragment supervision, or destabilize training.