Hierarchical Mixture of Experts (HMoE)

• HMoE is a hierarchical model that recursively combines gating networks and expert modules to enable fine-grained specialization and efficient capacity scaling.
• It employs diverse training paradigms, including EM, variational Bayesian inference, and gradient descent, to optimize expert routing and overall performance.
• HMoE applications span language model fine-tuning, multimodal fusion, meta-learning, and other tasks requiring management of complex hierarchical data.

A Hierarchical Mixture of Experts (HMoE) is a multi-level extension of the classical Mixture of Experts paradigm, enabling modular partitioning of input-space or representational-space complexity across nested layers of gating and expert modules. HMoE architectures are characterized by the recursive composition of gating functions and distributed expert subnetworks, providing capacity scaling, fine-grained specialization, and improved efficiency for high-dimensional or hierarchically structured tasks. HMoE models are widely deployed in regression, classification, large-scale LLM fine-tuning, multi-modal fusion, sequential modeling, meta-learning, and diverse domain-generalization tasks.

1. Fundamentals of Hierarchical Mixture of Experts

At the core of HMoE is the recursive application of gating networks that determine expert activation at each level of the hierarchy. In its canonical form, HMoE models are represented as soft decision trees or nested mixtures. In a two-level HMoE, the conditional density or prediction for input $x$ can be expressed as

$$y(x) = \sum_{i=1}^{K_1} \pi_1(i|x) \left[ \sum_{j=1}^{K_2} \pi_2(j|i,x) f_{ij}(x) \right],$$

where $\pi_1$ and $\pi_2$ are softmax (or alternative) gating functions and $f_{ij}$ are leaf-level expert models (Bishop et al., 2012, Fung et al., 2022, Nguyen et al., 3 Oct 2024). This structure extends naturally to arbitrary depth, recursively mixing over children at each internal node: $$y(x) = \sum_{p_1} \cdots \sum_{p_L} \prod_{\ell=1}^L \pi_{\ell}(p_\ell | p_{<\ell}, x) \, f_{p_1,\ldots,p_L}(x),$$ with each gating function possibly depending on the current or all previous gating choices.

Two principal forms of HMoE are prominent:

• Tree-structured HMoE: Arranged as a binary or multi-way tree, where internal nodes serve as gating functions and leaves are the expert models. Each data point is softly or probabilistically routed to a unique or mixture of experts (İrsoy et al., 2018, Zhao et al., 2019, Bishop et al., 2012).
• Hierarchical block MoE: Used in deep neural architectures, involving nested blocks of gating and expert modules at different network layers (e.g., local and global MoEs, or multi-granularity modules in GNNs and transformers) (Li et al., 25 Oct 2024, Cong et al., 6 Feb 2025, Li et al., 27 May 2025).

HMoE generalizes single-layer Mixture-of-Experts (MoE) by introducing multiple levels of gating, enabling coarse-to-fine specialization and layer-adaptive allocation of capacity.

2. Mathematical Structure and Gating Mechanisms

All variants of HMoE utilize gating networks, typically implemented as softmaxes over affine or nonlinear projections of inputs, but with recent work investigating alternative gating such as Laplace (L1 distance-based) mechanisms (Nguyen et al., 3 Oct 2024). The two-level Gaussian HMoE can be formulated as: $$p(y|x) = \sum_{i_1=1}^{K_1} \pi_1(i_1|x)\! \sum_{i_2=1}^{K_2} \pi_2(i_2|i_1,x) \, \mathcal N(y; \eta_{i_1i_2}^Tx+\tau_{i_1i_2},\,\nu_{i_1i_2})$$ The gating weights may be defined as: $$\pi_L(i|x) = \frac{\exp(-\|\mathbf{w}_i - x\|_1 + \beta_i)}{\sum_j \exp(-\|\mathbf{w}_j - x\|_1 + \beta_j)}$$ or in standard softmax form: $$\pi(i|x) = \frac{\exp(a_i^Tx + b_i)}{\sum_j \exp(a_j^Tx + b_j)}$$ which are combined hierarchically (Nguyen et al., 3 Oct 2024).

HMoEs in deep architectures often employ sparse routing: for each token or example, only a subset (top-K) of experts at each level is activated, and their probabilities renormalized (Cong et al., 6 Feb 2025, Kim et al., 11 Feb 2025). Adaptive strategies for expert counts, rank allocation within experts, and group-based top-level routing (e.g., by modality or granularity) provide further efficiency and specialization.

3. Model Training and Regularization

HMoE training typically follows one of three main paradigms:

• Maximum Likelihood (EM): Recursive EM updates are employed to jointly optimize gating parameters and expert models, fitting responsibilities at each expert leaf (Zhao et al., 2019, Bishop et al., 2012). For tree-based HMoEs, EM alternates between responsibility computation (E-step) and weighted regression/classification at experts (M-step).
• Variational Bayesian Inference: For Bayesian HMoEs, a structured variational posterior is defined over all gating variables and expert weights, enabling rigorous model selection and complexity control via evidence maximization (Bishop et al., 2012).
• End-to-End Gradient Descent: In transformer-based and neural HMoE settings, all gating and expert weights are jointly optimized by SGD or related optimizers, with loss functions including classification, regression, load balancing, and regularizers to avoid expert collapse or polarization (Cong et al., 6 Feb 2025, Li et al., 25 Oct 2024, Kim et al., 11 Feb 2025).

Specialized regularization approaches are often required for HMoE:

• Hierarchy-aware dropout: Dropping whole subtrees or routing paths to prevent co-adaptation in deep trees (İrsoy et al., 2018).
• Expert-polarization prevention: Regularization terms to ensure balanced usage of all experts at each node or level (Li et al., 25 Oct 2024, Bishop et al., 2012).
• Parameter-budget constraints: Particularly in LLM fine-tuning, constrained parameter allocation via layerwise expert and rank schedules (Cong et al., 6 Feb 2025).

4. Adaptive Capacity Allocation and Matching Hierarchical Complexity

Many contemporary HMoE architectures exploit adaptive allocation of capacity to best match the task's intrinsic hierarchical complexity. For instance, in parameter-efficient LLM fine-tuning, experts and their internal ranks are both varied across layers according to the representational demand of each layer, yielding two-dimensional hierarchical allocation

$\{ (E_i, r_i) \}_{i=1}^N$

where $E_i$ is the number of experts and $r_i$ their rank in layer $i$ (Cong et al., 6 Feb 2025). Empirical analysis shows that shallow layers in transformers require less adapter capacity, whereas deeper layers require more expressivity. Hierarchical scheduling outperforms flat or single-dimensional strategies by efficiently allocating resources where most needed.

Similarly, multimodal and multigranular architectures group experts by modality, spatial region, or program granularity, with superordinate gating networks adaptively fusing the outputs (Kim et al., 11 Feb 2025, Li et al., 25 Oct 2024). This adaptive specialization at each level is critical for domain generalization, multi-modal fusion, and the decoding of complex structured data.

5. Applications and Empirical Insights

HMoE models are deployed in a wide range of domains:

• LLM Fine-Tuning: Hierarchical mixtures of adapters with controlled expert count and rank scheduling (e.g., HiLo) significantly reduce parameter footprint while exceeding performance of prior PEFT and MoE strategies. On Llama 2-7B, HiLo achieves higher accuracy with 37.5% fewer active parameters compared to MoLA and related methods (Cong et al., 6 Feb 2025).
• Multimodal and Sequential Learning: Grouped experts per modality with hierarchical routers (e.g., MoHAVE for audio-visual speech recognition), two-level mixtures for multi-modal sequential recommendation (HM4SR), and expert grouping for generalist vision-language-action policies (HiMoE-VLA) yield improved robustness, dynamic fusion, and group-aware adaptation (Kim et al., 11 Feb 2025, Du et al., 5 Dec 2025, Zhang et al., 24 Jan 2025).
• Meta-learning and Scientific Modeling: Hierarchical gating and explicit cluster routing (e.g., MixER) enable scalable family-wise specialization for heterogeneous dynamical system reconstruction (Nzoyem et al., 7 Feb 2025).
• High-Level Synthesis and Graph Learning: Multi-granularity expert routing on graph structures improves capacity to generalize across unseen program kernels in FPGA performance estimation (Li et al., 25 Oct 2024).
• Multilevel Data and Mixed Effects: Nested HMoEs provably approximate broad classes of mixed-effects models for hierarchical or grouped data (Fung et al., 2022), with established universal approximation properties.

Empirical findings universally support the use of hierarchical mixtures for enhanced generalization, parameter efficiency, and alignment with task structure. Hierarchical expert-rank scheduling, multi-level gating with Laplace or softmax functions, and cross-modal cross-attention enable fine adaptation to the complexity and heterogeneity of modern data.

6. Limitations, Open Problems, and Design Considerations

Despite their efficacy, HMoE models present several design and deployment considerations:

• Computational Overhead: Additional gating networks and dynamic routing logic may increase inference time and memory requirements, though sparsification (top-K routing) and batch grouping help mitigate this (Cong et al., 6 Feb 2025).
• Expert Collapse and Load Imbalance: Without regularization, some experts may be underutilized ("dead"); purpose-built regularizers are necessary to enforce load balancing (Li et al., 25 Oct 2024, İrsoy et al., 2018, Du et al., 5 Dec 2025).
• Hyperparameter Sensitivity: Performance depends on expert count, rank (in neural MoEs), hierarchy depth, and allocation strategy, often requiring cross-validation or policy search (Cong et al., 6 Feb 2025, Nguyen et al., 3 Oct 2024).
• Scaling Depth: Theoretical results are mature for depth-2 (two-level) HMoE, but extension to deeper hierarchies and nonlinear experts remains open (Nguyen et al., 3 Oct 2024).
• Interpretability: In HMoE tree models, path-based explanations are tractable, but hierarchically nested neural or transformer-MoEs may obscure interpretability.

Emerging topics include hierarchical gating in the absence of softmax parameter interaction (Laplace gating), policy-gradient–style self-supervised preference alignment via internal router signals, plug-and-play hierarchical adaptation for multi-objective model alignment, and hierarchical mixture approaches in nonstandard domains such as 3D medical image segmentation and program synthesis (Gao et al., 24 Nov 2025, Li et al., 27 May 2025, Płotka et al., 8 Jul 2025).

7. Theoretical Foundations and Universal Approximation

HMoE architectures possess rich theoretical properties. Nested MoE models approximate any continuous multilevel mixed-effects distribution in the sense of weak convergence, subject to basic regularity assumptions (Fung et al., 2022). For tree-structured models, the recursive EM and Bayesian variational approaches provide principled mechanisms for structure learning, model selection, and uncertainty quantification (Bishop et al., 2012). When Laplace gating is employed, parameter interaction chains can be eliminated, leading to improved convergence rates and sharper specialist experts (Nguyen et al., 3 Oct 2024).

In summary, Hierarchical Mixture of Experts constitutes a versatile, theoretically grounded, and empirically validated family of models for handling complex, high-dimensional, and hierarchically structured learning problems in modern machine learning and artificial intelligence.