---
title: Heterogeneous 4-Expert Mixture-of-Experts
url: https://www.emergentmind.com/topics/heterogeneous-4-expert-mixture-of-experts-moe4
type: topic
---

# Heterogeneous 4-Expert Mixture-of-Experts

Searching arXiv for the cited papers to ground the article in current literature.
Heterogeneous 4-Expert Mixture-of-Experts (MoE4) denotes a four-expert Mixture-of-Experts system in which the experts are not treated as interchangeable identical subnetworks. In the most direct recent instantiation, MoE4 is a dense four-expert classifier whose experts are full backbone networks drawn from different architecture families in the LEMUR database, and whose output is the weighted sum of four expert logits under a temperature-scaled softmax gate [2606.23739]. Related literature uses “heterogeneous” in broader senses, including diversity by temporal pathway, expert size, routing multiplicity, and hardware placement; accordingly, MoE4 is best understood as a family of four-expert designs rather than a single canonical architecture [2605.28229; 2409.12210; 2202.09368; 2603.02633].

## 1. Definition and semantic scope

The term “heterogeneous” is not used uniformly across the MoE literature. In some papers it refers to experts with different backbone families; in others it refers to different temporal streams, different widths, different routing multiplicities, or different compute substrates. A precise reading of “Heterogeneous 4-Expert Mixture-of-Experts” therefore depends on which axis of heterogeneity is being emphasized.

| Work | Heterogeneity notion | 4-expert status |
|---|---|---|
| [2606.23739] | Four different backbone families with a learned gate | Direct MoE4 |
| [2605.28229] | Four temporal-rate pathways \(\{2,4,8,16\}\) with dedicated experts | Direct 4-expert configuration |
| [2402.00893] | Mutual distillation among experts | Exact 4-expert ablation reported |
| [2505.17553] | Top-2 routed PEFT experts trained with contrastive specialization | Default setting uses 4 experts |
| [2409.12210] | Experts of different FFN sizes | Evaluated with 8 experts, not 4 |
| [2601.21866] | One shared convolutional expert plus routed Fourier experts | Not MoE4 |

The most explicit MoE4 paper defines a classifier built from “four different backbone networks drawn from distinct architecture families in the LEMUR database,” all receiving the same image input and all contributing to every prediction through a dense soft mixture [2606.23739]. By contrast, VidPrism’s best setting is a four-expert heterogeneous temporal architecture with experts bound to different temporal rates rather than different backbone families [2605.28229]. Other adjacent works are directly informative for four-expert design, but they do not instantiate a canonical MoE4; for example, Mixture of Diverse Size Experts studies heterogeneous expert widths with \(8\) experts [2409.12210], and MoHETS is explicitly described as an \(8\)-routed-expert design with one shared expert rather than a four-expert system [2601.21866].

A central conceptual consequence is that MoE4 should not be reduced to “four different expert types.” Some works use heterogeneity to denote structural diversity across experts, while others use it to denote non-uniform routing or adaptive computation across inputs [2202.09368]. This distinction matters for both interpretation and comparison.

## 2. Architectural realizations

In the systematic MoE4 exploration, each candidate is assembled into a wrapper derived from the hand-crafted reference model “MoE-hetero4-Alex-Dense-Air-Bag,” and each expert is a full backbone model taken from a distinct LEMUR family [2606.23739]. If \(f_i(x)\) is the logits vector of expert \(i\) and \(g_i(x)\) is its gate weight, the ensemble output is

\[
y(x)=\sum_{i=1}^{4} g_i(x) f_i(x).
\]

This is a dense weighted average of expert logits rather than sparse top-\(k\) routing. The gate, called **HeterogeneousGate**, has the architecture “Conv – BN – ReLU – Conv – BN – ReLU – AvgPool – Linear – ReLU – Dropout – Linear,” and its logits are converted to mixture weights by a learned temperature-scaled softmax with temperature clamped to \([0.5,5.0]\); Gaussian noise is added to the gate logits during training [2606.23739].

A second architectural pattern appears in MixtureKit, which is a general framework for composing MoE models from existing checkpoints rather than training an MoE from scratch [2512.12121]. It supports three composition methods: Traditional MoE, BTX (Branch-Train-Mix), and BTS (Branch-Train-Stitch). Traditional MoE uses one router per FFN block; BTX introduces separate routers for specified FFN sub-layers such as `mlp.gate_proj`, `mlp.up_proj`, and `mlp.down_proj`; BTS keeps experts fully intact and inserts trainable stitch layers for hub–expert information exchange. The framework is parameterized by the number of experts, so a four-expert system is obtained by specifying four entries in the expert list and choosing the routing sparsity through `num_experts_per_tok` [2512.12121]. Within the paper’s own experimental evidence, BTX has the strongest support.

VidPrism provides a different direct realization of heterogeneous four-expert design in video understanding [2605.28229]. Its final selected configuration uses four experts at temporal rates \(\{2,4,8,16\}\). Each expert is “a standard Transformer layer” trained on one temporal-rate stream, so the heterogeneity is produced primarily by input pathway and functional role rather than by distinct internal block types. After content-aware multi-rate aggregation and dynamic bidirectional interaction, the expert outputs are concatenated and fused by global attention through a learnable query. The best ablation result is obtained with this four-expert configuration, which the paper states is selected for the final model [2605.28229].

These architectural instances imply two stable design lineages. One lineage treats heterogeneity as diversity across full expert backbones under a dense gate. The other treats it as diversity across structured pathways or submodules, with experts bound to different temporal resolutions or to different transformed checkpoints. Both lineages are directly represented in four-expert form.

## 3. Routing, specialization, and regularization

Routing in heterogeneous MoE4 spans a spectrum from dense soft mixing to sparse and even variable-multiplicity routing. The direct MoE4 classifier in the LEMUR study is fully dense: all four experts contribute to every input, with the gate controlling only relative weights [2606.23739]. This contrasts with sparse MoE traditions in which only a subset of experts is active.

Expert Choice routing is relevant here because it makes heterogeneity appear at the token level rather than the expert-architecture level [2202.09368]. Instead of each token selecting a fixed top-\(k\) set of experts, each expert selects its top-scoring tokens up to its capacity. In a four-expert system, a token may therefore be selected by \(0\), \(1\), \(2\), \(3\), or all \(4\) experts. The paper explicitly reports that restricting the number of experts to \(2\) degrades perplexity relative to the base Expert Choice method, while capping at \(3\) is nearly as good as uncapped Expert Choice, suggesting that variable experts-per-token can be advantageous even in a fixed four-expert pool [2202.09368].

Training methods for MoE4 specialization have also been studied directly. MoDE introduces mutual distillation among experts to address what it calls “narrow vision,” the tendency of routed experts to learn too narrowly from the samples emphasized by the gate [2402.00893]. For more than two experts, each expert is distilled toward the mean expert output:

\[
e_{\text{avg}}=\frac{1}{K}\sum_{i=1}^{K} e_i, \qquad
L_{KD}=\frac{1}{K}\sum_{i=1}^{K}\mathrm{mean}\left((e_i-e_{\text{avg}})^2\right),
\]

and the total loss is

\[
L=L_{\text{task}}+\alpha L_{KD}.
\]

The paper includes an exact \(4\)-expert ablation on tabular datasets. For example, on Isolet, the \(4\)-expert MoE improves from \(0.9352\) to \(0.9483\) under MoDE, and on Mfeat-factors from \(0.9620\) to \(0.9742\) [2402.00893]. The paper also emphasizes that distillation must remain moderate: excessive \(\alpha\) causes experts to become too similar.

CoMoE studies another exact small-expert regime by using \(4\) experts with top-\(2\) routing in parameter-efficient fine-tuning [2505.17553]. It adds a contrastive objective that treats activated experts as positives and inactivated experts as negatives for the same input, with total objective

\[
\mathcal{L}_{\text{total}}=\mathcal{L}_{CE}+\lambda \mathcal{L}_{\text{con}}.
\]

Its default configuration uses \(n=4\) experts, top-\(2\) routing, LoRA rank \(16\), and the ablation over \(\lambda \in \{0.0, 0.001, 0.01, 0.1, 1.0\}\) identifies \(\lambda=10^{-2}\) as best in the reported settings [2505.17553]. The paper’s central claim is that sparse routing alone does not guarantee modularization on heterogeneous data; the contrastive objective is introduced precisely to prevent experts from learning redundant representations.

Together, these works show that specialization in MoE4 is not solely a matter of choosing four experts. It depends on the routing rule, on whether every input sees all experts or only a subset, and on whether explicit auxiliary objectives are used to maintain diversity without expert collapse.

## 4. Empirical landscape and search-based evidence

The most extensive direct exploration of heterogeneous MoE4 is the automated pipeline search over LEMUR families [2606.23739]. The full theoretical search space is

\[
\binom{29}{4}=23{,}751
\]

possible four-family combinations. Over a \(28\)-day campaign on an NVIDIA RTX \(4090\), the pipeline generated \(4{,}463\) candidate models across \(197\) batches, and \(1{,}021\) were successfully evaluated [2606.23739]. Every generated model was passed through three validation stages before GPU training: syntax checking with `compile()`, a CPU forward-pass probe at shape \((2,3,256,256)\) with expected output shape \((2,10)\), and MD5-based source deduplication.

The campaign’s central methodological result is a severe search-space coverage bias. Because quartets were enumerated with `itertools.combinations(sorted(model_names), 4)`, and because `sorted()` places AirNet first, the explored slice remained entirely AirNet-anchored [2606.23739]. Exactly \(1{,}146\) combinations were attempted, all \(1{,}146\) contained AirNet, and this corresponded to only about \(4.8\%\) of the theoretical space. The paper therefore states that all empirical family-level conclusions are conditioned on AirNet always being one of the four experts [2606.23739].

Within that constrained slice, the paper reports broad performance dispersion across the \(1{,}021\) successful models: mean accuracy \(0.5221\), median \(0.5388\), \(25\)th percentile \(0.4891\), \(75\)th percentile \(0.6059\), maximum \(0.6801\), and minimum \(0.1250\) [2606.23739]. The best model is **AirNet + AlexNet + DPN68 + ResNet**, which reaches \(0.6801\) top-1 accuracy on CIFAR-10 after a single training epoch. At the family level, ShuffleNet and MobileNetV3 are the strongest co-experts within the explored slice, with mean accuracies up to \(0.632\) and \(0.621\), respectively, while FractalNet is identified as a “practical dead end” with only \(23\) successful evaluations out of \(3{,}273\) appearances, and MNASNet as a “consistent accuracy degrader” with mean accuracy \(0.218\) [2606.23739].

VidPrism contributes a second direct empirical argument for MoE4, but in video rather than image classification [2605.28229]. Its ablation over number of experts finds that the best configuration uses four experts with rates \(\{2,4,8,16\}\), achieving \(95.9\) on UCF-101 and \(76.3\) on HMDB-51. The paper states that this four-expert configuration is selected for the final model [2605.28229]. The result is significant because it shows a concrete four-expert sweet spot in a heterogeneous temporal architecture, not merely a four-expert simplification of a larger model.

The current empirical picture is therefore bifurcated. One line of evidence studies MoE4 as four heterogeneous full backbones under extremely shallow screening. Another studies MoE4 as four temporally differentiated pathways inside a transfer architecture for video. Both provide direct four-expert results, but their assumptions, tasks, and meanings of heterogeneity are materially different.

## 5. Capacity, hardware, and efficiency

A major heterogeneous design question is whether the four experts should differ only in identity or also in size. Mixture of Diverse Size Experts addresses this question by making expert FFN hidden dimensions unequal while preserving total budget through pairwise constraints [2409.12210]. In the reported models, the paper evaluates \(8\) experts rather than \(4\), but its core finding is directly relevant: difficult tokens, defined using baseline cross-entropy, route disproportionately to larger experts, and the harder the token bucket, the larger the average loss reduction. For the bucket with baseline CE \(>2.0\), the average loss reduction is \(0.58\), and top-1 difficult-token routing favors larger experts by \(6215\) versus \(3085\) when aggregating the reported size groups [2409.12210]. This suggests that a four-expert heterogeneous design may usefully vary capacity as well as specialization, though the paper’s direct experiments are not at \(4\) experts.

Hardware heterogeneity introduces a separate axis of MoE4 design. In the analog–digital deployment framework, the recommended rule is to keep all densely activated modules digital and rank sparse experts by the maximum neuron norm score

\[
\mathrm{MaxNNScore}^{(s)} := \prod_{* \in \{\mathrm{up,down,gate}\}} \mathrm{MaxNNorm}(\mathbf{W}_*^{(s)}),
\]

placing the top \(\Gamma\) fraction digitally while executing the rest on AIMC hardware [2603.02633]. The paper explicitly states how this transfers to a four-expert system: \(\Gamma=0.25\) corresponds to \(1\) digital and \(3\) analog experts, \(\Gamma=0.5\) to \(2\) digital and \(2\) analog, and \(\Gamma=0.75\) to \(3\) digital and \(1\) analog [2603.02633]. It also reports that dense modules such as attention layers, the LM head, and shared experts are disproportionately sensitive to analog noise and should remain digital.

Training MoE on mixed GPU generations is addressed by HeterMoE [2504.03871]. Its key systems claim is that newer GPUs are disproportionately better at attention, while older GPUs remain relatively efficient for experts, so attention and experts should be disaggregated across GPU types. HeterMoE introduces zebra parallelism to overlap attention and expert computation across microbatches and reports up to \(2.3\times\) speed-up compared to existing MoE training systems and \(1.4\times\) compared to an optimally balanced heterogeneity-aware solution; it also reports \(95\%\) average throughput retention even when half of the GPUs in a homogeneous A40 cluster are replaced with V100 [2504.03871]. The evaluated models use \(12\), \(18\), \(24\), and \(40\) experts rather than \(4\), so the direct evidence is not MoE4-specific, but the disaggregation principle is independent of expert count.

Efficiency claims for sparse MoE must also be treated cautiously. In a controlled vision study with matched parameter budgets of \(11.44\)M parameters, SparseMoE with \(8\) experts and top-\(2\) routing is slower than both Dense and SoftMoE on an NVIDIA A100 and on CPU: on A100 at batch size \(256\), SparseMoE takes \(13.86\) ms/batch versus \(9.14\) for Dense and \(9.29\) for SoftMoE [2601.15021]. This does not directly invalidate sparse MoE4, but it does show that conditional routing does not automatically translate into realized latency gains on modern hardware.

## 6. Misconceptions, ambiguities, and boundaries of the term

A recurring misconception is that any heterogeneous MoE paper with a small number of active pathways is automatically a “heterogeneous 4-expert MoE.” MoHETS is an explicit counterexample [2601.21866]. It replaces Transformer FFNs with Mixture-of-Heterogeneous-Experts layers composed of one shared depthwise-convolution expert and \(N\) routed Fourier Analysis FFN experts, with experiments using \(N=8\) routed experts and top-\(K=2\). Counted by routed experts, it is an \(8\)-expert sparse heterogeneous MoE; counted with the always-on shared expert, each layer has \(9\) expert pathways. The paper therefore states that it is not accurate to call the reported model “MoE4” [2601.21866].

A second misconception is that “heterogeneous” necessarily means “architecturally different experts.” Expert Choice routing uses the term in a different sense: heterogeneity is introduced because different tokens may be processed by different numbers of experts, even when the experts themselves are architecturally homogeneous [2202.09368]. MixtureKit adds yet another sense, in which heterogeneity may arise from different domain specializations and training histories of compatible checkpoints that are then composed through Traditional MoE, BTX, or BTS [2512.12121]. In remote sensing super-resolution, heterogeneity is implemented across expert groups with different convolution kernel sizes, while experts within each group remain homogeneous; the paper’s closest four-expert adaptation is \(2\) groups \(\times\) \(2\) experts each rather than four fully distinct expert types [2502.09654].

A third misconception is that empirical rankings over expert families are already global conclusions about MoE4 composition. The automated LEMUR study explicitly rules this out because its search covered only an AirNet-anchored \(4.8\%\) of the theoretical combination space and every candidate was trained for only \(1\) epoch [2606.23739]. Statements such as the strength of ShuffleNet or MobileNetV3 are therefore conditional conclusions within that anchored subset, not universal rankings across all \(29\) architecture families.

The literature also places clear limits on what is directly established for MoE4. Some results are exact four-expert findings, such as the dense four-backbone classifier [2606.23739], the four-rate VidPrism configuration [2605.28229], the four-expert mutual-distillation ablation [2402.00893], and the four-expert top-\(2\) PEFT setting in CoMoE [2505.17553]. Other highly relevant results—expert-size heterogeneity, analog–digital partitioning, and mixed-GPU scheduling—are derived from larger expert counts and transfer to MoE4 by structural argument rather than direct four-expert benchmarking [2409.12210; 2603.02633; 2504.03871].

Taken together, the literature supports a precise but plural understanding of Heterogeneous MoE4. It can denote a dense four-backbone ensemble with a learned gate, a four-path temporal architecture with expert streams at rates \(\{2,4,8,16\}\), a four-expert routed model trained with mutual distillation or contrastive modularization, or a four-expert system deployed under heterogeneous hardware constraints. What unifies these variants is not a single routing law or expert template, but the principle that a four-expert MoE can benefit when the four experts are differentiated by backbone family, pathway, capacity, routing role, or deployment substrate rather than treated as identical replicas.

Source: https://www.emergentmind.com/topics/heterogeneous-4-expert-mixture-of-experts-moe4