Heterogeneous 4-Expert Mixture-of-Experts

• Heterogeneous MoE4 is a four-expert system where experts differ in backbone architecture, temporal rates, or capacity, enabling targeted specialization.
• Design variants use dense soft mixing, variable expert routing, and auxiliary objectives like mutual distillation or contrastive penalties to optimize performance.
• Empirical studies and automated searches highlight that expert selection and gating mechanisms critically impact accuracy and efficiency in image and video tasks.

Searching arXiv for the cited papers to ground the article in current literature. Heterogeneous 4-Expert Mixture-of-Experts (MoE4) denotes a four-expert Mixture-of-Experts system in which the experts are not treated as interchangeable identical subnetworks. In the most direct recent instantiation, MoE4 is a dense four-expert classifier whose experts are full backbone networks drawn from different architecture families in the LEMUR database, and whose output is the weighted sum of four expert logits under a temperature-scaled softmax gate (Lukhi et al., 21 Jun 2026). Related literature uses “heterogeneous” in broader senses, including diversity by temporal pathway, expert size, routing multiplicity, and hardware placement; accordingly, MoE4 is best understood as a family of four-expert designs rather than a single canonical architecture (Lin et al., 27 May 2026, Sun et al., 2024, Zhou et al., 2022, Chowdhury et al., 3 Mar 2026).

1. Definition and semantic scope

The term “heterogeneous” is not used uniformly across the MoE literature. In some papers it refers to experts with different backbone families; in others it refers to different temporal streams, different widths, different routing multiplicities, or different compute substrates. A precise reading of “Heterogeneous 4-Expert Mixture-of-Experts” therefore depends on which axis of heterogeneity is being emphasized.

Work	Heterogeneity notion	4-expert status
(Lukhi et al., 21 Jun 2026)	Four different backbone families with a learned gate	Direct MoE4
(Lin et al., 27 May 2026)	Four temporal-rate pathways $\{2,4,8,16\}$ with dedicated experts	Direct 4-expert configuration
(Xie et al., 2024)	Mutual distillation among experts	Exact 4-expert ablation reported
(Feng et al., 23 May 2025)	Top-2 routed PEFT experts trained with contrastive specialization	Default setting uses 4 experts
(Sun et al., 2024)	Experts of different FFN sizes	Evaluated with 8 experts, not 4
(Ortigossa et al., 29 Jan 2026)	One shared convolutional expert plus routed Fourier experts	Not MoE4

The most explicit MoE4 paper defines a classifier built from “four different backbone networks drawn from distinct architecture families in the LEMUR database,” all receiving the same image input and all contributing to every prediction through a dense soft mixture (Lukhi et al., 21 Jun 2026). By contrast, VidPrism’s best setting is a four-expert heterogeneous temporal architecture with experts bound to different temporal rates rather than different backbone families (Lin et al., 27 May 2026). Other adjacent works are directly informative for four-expert design, but they do not instantiate a canonical MoE4; for example, Mixture of Diverse Size Experts studies heterogeneous expert widths with $8$ experts (Sun et al., 2024), and MoHETS is explicitly described as an $8$-routed-expert design with one shared expert rather than a four-expert system (Ortigossa et al., 29 Jan 2026).

A central conceptual consequence is that MoE4 should not be reduced to “four different expert types.” Some works use heterogeneity to denote structural diversity across experts, while others use it to denote non-uniform routing or adaptive computation across inputs (Zhou et al., 2022). This distinction matters for both interpretation and comparison.

2. Architectural realizations

In the systematic MoE4 exploration, each candidate is assembled into a wrapper derived from the hand-crafted reference model “MoE-hetero4-Alex-Dense-Air-Bag,” and each expert is a full backbone model taken from a distinct LEMUR family (Lukhi et al., 21 Jun 2026). If $f_i(x)$ is the logits vector of expert $i$ and $g_i(x)$ is its gate weight, the ensemble output is

$y(x)=\sum_{i=1}^{4} g_i(x) f_i(x).$

This is a dense weighted average of expert logits rather than sparse top-$k$ routing. The gate, called HeterogeneousGate, has the architecture “Conv – BN – ReLU – Conv – BN – ReLU – AvgPool – Linear – ReLU – Dropout – Linear,” and its logits are converted to mixture weights by a learned temperature-scaled softmax with temperature clamped to $[0.5,5.0]$; Gaussian noise is added to the gate logits during training (Lukhi et al., 21 Jun 2026).

A second architectural pattern appears in MixtureKit, which is a general framework for composing MoE models from existing checkpoints rather than training an MoE from scratch (Chamma et al., 13 Dec 2025). It supports three composition methods: Traditional MoE, BTX (Branch-Train-Mix), and BTS (Branch-Train-Stitch). Traditional MoE uses one router per FFN block; BTX introduces separate routers for specified FFN sub-layers such as mlp.gate_proj, mlp.up_proj, and mlp.down_proj; BTS keeps experts fully intact and inserts trainable stitch layers for hub–expert information exchange. The framework is parameterized by the number of experts, so a four-expert system is obtained by specifying four entries in the expert list and choosing the routing sparsity through num_experts_per_tok (Chamma et al., 13 Dec 2025). Within the paper’s own experimental evidence, BTX has the strongest support.

VidPrism provides a different direct realization of heterogeneous four-expert design in video understanding (Lin et al., 27 May 2026). Its final selected configuration uses four experts at temporal rates $\{2,4,8,16\}$. Each expert is “a standard Transformer layer” trained on one temporal-rate stream, so the heterogeneity is produced primarily by input pathway and functional role rather than by distinct internal block types. After content-aware multi-rate aggregation and dynamic bidirectional interaction, the expert outputs are concatenated and fused by global attention through a learnable query. The best ablation result is obtained with this four-expert configuration, which the paper states is selected for the final model (Lin et al., 27 May 2026).

These architectural instances imply two stable design lineages. One lineage treats heterogeneity as diversity across full expert backbones under a dense gate. The other treats it as diversity across structured pathways or submodules, with experts bound to different temporal resolutions or to different transformed checkpoints. Both lineages are directly represented in four-expert form.

3. Routing, specialization, and regularization

Routing in heterogeneous MoE4 spans a spectrum from dense soft mixing to sparse and even variable-multiplicity routing. The direct MoE4 classifier in the LEMUR study is fully dense: all four experts contribute to every input, with the gate controlling only relative weights (Lukhi et al., 21 Jun 2026). This contrasts with sparse MoE traditions in which only a subset of experts is active.

Expert Choice routing is relevant here because it makes heterogeneity appear at the token level rather than the expert-architecture level (Zhou et al., 2022). Instead of each token selecting a fixed top-$8$0 set of experts, each expert selects its top-scoring tokens up to its capacity. In a four-expert system, a token may therefore be selected by $8$1, $8$2, $8$3, $8$4, or all $8$5 experts. The paper explicitly reports that restricting the number of experts to $8$6 degrades perplexity relative to the base Expert Choice method, while capping at $8$7 is nearly as good as uncapped Expert Choice, suggesting that variable experts-per-token can be advantageous even in a fixed four-expert pool (Zhou et al., 2022).

Training methods for MoE4 specialization have also been studied directly. MoDE introduces mutual distillation among experts to address what it calls “narrow vision,” the tendency of routed experts to learn too narrowly from the samples emphasized by the gate (Xie et al., 2024). For more than two experts, each expert is distilled toward the mean expert output:

$8$8

and the total loss is

$8$9

The paper includes an exact $8$0-expert ablation on tabular datasets. For example, on Isolet, the $8$1-expert MoE improves from $8$2 to $8$3 under MoDE, and on Mfeat-factors from $8$4 to $8$5 (Xie et al., 2024). The paper also emphasizes that distillation must remain moderate: excessive $8$6 causes experts to become too similar.

CoMoE studies another exact small-expert regime by using $8$7 experts with top-$8$8 routing in parameter-efficient fine-tuning (Feng et al., 23 May 2025). It adds a contrastive objective that treats activated experts as positives and inactivated experts as negatives for the same input, with total objective

$8$9

Its default configuration uses $f_i(x)$0 experts, top-$f_i(x)$1 routing, LoRA rank $f_i(x)$2, and the ablation over $f_i(x)$3 identifies $f_i(x)$4 as best in the reported settings (Feng et al., 23 May 2025). The paper’s central claim is that sparse routing alone does not guarantee modularization on heterogeneous data; the contrastive objective is introduced precisely to prevent experts from learning redundant representations.

Together, these works show that specialization in MoE4 is not solely a matter of choosing four experts. It depends on the routing rule, on whether every input sees all experts or only a subset, and on whether explicit auxiliary objectives are used to maintain diversity without expert collapse.

4. Empirical landscape and search-based evidence

The most extensive direct exploration of heterogeneous MoE4 is the automated pipeline search over LEMUR families (Lukhi et al., 21 Jun 2026). The full theoretical search space is

$f_i(x)$5

possible four-family combinations. Over a $f_i(x)$6-day campaign on an NVIDIA RTX $f_i(x)$7, the pipeline generated $f_i(x)$8 candidate models across $f_i(x)$9 batches, and $i$0 were successfully evaluated (Lukhi et al., 21 Jun 2026). Every generated model was passed through three validation stages before GPU training: syntax checking with compile(), a CPU forward-pass probe at shape $i$1 with expected output shape $i$2, and MD5-based source deduplication.

The campaign’s central methodological result is a severe search-space coverage bias. Because quartets were enumerated with itertools.combinations(sorted(model_names), 4), and because sorted() places AirNet first, the explored slice remained entirely AirNet-anchored (Lukhi et al., 21 Jun 2026). Exactly $i$3 combinations were attempted, all $i$4 contained AirNet, and this corresponded to only about $i$5 of the theoretical space. The paper therefore states that all empirical family-level conclusions are conditioned on AirNet always being one of the four experts (Lukhi et al., 21 Jun 2026).

Within that constrained slice, the paper reports broad performance dispersion across the $i$6 successful models: mean accuracy $i$7, median $i$8, $i$9th percentile $g_i(x)$0, $g_i(x)$1th percentile $g_i(x)$2, maximum $g_i(x)$3, and minimum $g_i(x)$4 (Lukhi et al., 21 Jun 2026). The best model is AirNet + AlexNet + DPN68 + ResNet, which reaches $g_i(x)$5 top-1 accuracy on CIFAR-10 after a single training epoch. At the family level, ShuffleNet and MobileNetV3 are the strongest co-experts within the explored slice, with mean accuracies up to $g_i(x)$6 and $g_i(x)$7, respectively, while FractalNet is identified as a “practical dead end” with only $g_i(x)$8 successful evaluations out of $g_i(x)$9 appearances, and MNASNet as a “consistent accuracy degrader” with mean accuracy $y(x)=\sum_{i=1}^{4} g_i(x) f_i(x).$0 (Lukhi et al., 21 Jun 2026).

VidPrism contributes a second direct empirical argument for MoE4, but in video rather than image classification (Lin et al., 27 May 2026). Its ablation over number of experts finds that the best configuration uses four experts with rates $y(x)=\sum_{i=1}^{4} g_i(x) f_i(x).$1, achieving $y(x)=\sum_{i=1}^{4} g_i(x) f_i(x).$2 on UCF-101 and $y(x)=\sum_{i=1}^{4} g_i(x) f_i(x).$3 on HMDB-51. The paper states that this four-expert configuration is selected for the final model (Lin et al., 27 May 2026). The result is significant because it shows a concrete four-expert sweet spot in a heterogeneous temporal architecture, not merely a four-expert simplification of a larger model.

The current empirical picture is therefore bifurcated. One line of evidence studies MoE4 as four heterogeneous full backbones under extremely shallow screening. Another studies MoE4 as four temporally differentiated pathways inside a transfer architecture for video. Both provide direct four-expert results, but their assumptions, tasks, and meanings of heterogeneity are materially different.

5. Capacity, hardware, and efficiency

A major heterogeneous design question is whether the four experts should differ only in identity or also in size. Mixture of Diverse Size Experts addresses this question by making expert FFN hidden dimensions unequal while preserving total budget through pairwise constraints (Sun et al., 2024). In the reported models, the paper evaluates $y(x)=\sum_{i=1}^{4} g_i(x) f_i(x).$4 experts rather than $y(x)=\sum_{i=1}^{4} g_i(x) f_i(x).$5, but its core finding is directly relevant: difficult tokens, defined using baseline cross-entropy, route disproportionately to larger experts, and the harder the token bucket, the larger the average loss reduction. For the bucket with baseline CE $y(x)=\sum_{i=1}^{4} g_i(x) f_i(x).$6, the average loss reduction is $y(x)=\sum_{i=1}^{4} g_i(x) f_i(x).$7, and top-1 difficult-token routing favors larger experts by $y(x)=\sum_{i=1}^{4} g_i(x) f_i(x).$8 versus $y(x)=\sum_{i=1}^{4} g_i(x) f_i(x).$9 when aggregating the reported size groups (Sun et al., 2024). This suggests that a four-expert heterogeneous design may usefully vary capacity as well as specialization, though the paper’s direct experiments are not at $k$0 experts.

Hardware heterogeneity introduces a separate axis of MoE4 design. In the analog–digital deployment framework, the recommended rule is to keep all densely activated modules digital and rank sparse experts by the maximum neuron norm score

$k$1

placing the top $k$2 fraction digitally while executing the rest on AIMC hardware (Chowdhury et al., 3 Mar 2026). The paper explicitly states how this transfers to a four-expert system: $k$3 corresponds to $k$4 digital and $k$5 analog experts, $k$6 to $k$7 digital and $k$8 analog, and $k$9 to $[0.5,5.0]$0 digital and $[0.5,5.0]$1 analog (Chowdhury et al., 3 Mar 2026). It also reports that dense modules such as attention layers, the LM head, and shared experts are disproportionately sensitive to analog noise and should remain digital.

Training MoE on mixed GPU generations is addressed by HeterMoE (Wu et al., 4 Apr 2025). Its key systems claim is that newer GPUs are disproportionately better at attention, while older GPUs remain relatively efficient for experts, so attention and experts should be disaggregated across GPU types. HeterMoE introduces zebra parallelism to overlap attention and expert computation across microbatches and reports up to $[0.5,5.0]$2 speed-up compared to existing MoE training systems and $[0.5,5.0]$3 compared to an optimally balanced heterogeneity-aware solution; it also reports $[0.5,5.0]$4 average throughput retention even when half of the GPUs in a homogeneous A40 cluster are replaced with V100 (Wu et al., 4 Apr 2025). The evaluated models use $[0.5,5.0]$5, $[0.5,5.0]$6, $[0.5,5.0]$7, and $[0.5,5.0]$8 experts rather than $[0.5,5.0]$9, so the direct evidence is not MoE4-specific, but the disaggregation principle is independent of expert count.

Efficiency claims for sparse MoE must also be treated cautiously. In a controlled vision study with matched parameter budgets of $\{2,4,8,16\}$0M parameters, SparseMoE with $\{2,4,8,16\}$1 experts and top-$\{2,4,8,16\}$2 routing is slower than both Dense and SoftMoE on an NVIDIA A100 and on CPU: on A100 at batch size $\{2,4,8,16\}$3, SparseMoE takes $\{2,4,8,16\}$4 ms/batch versus $\{2,4,8,16\}$5 for Dense and $\{2,4,8,16\}$6 for SoftMoE (Rokah et al., 21 Jan 2026). This does not directly invalidate sparse MoE4, but it does show that conditional routing does not automatically translate into realized latency gains on modern hardware.

6. Misconceptions, ambiguities, and boundaries of the term

A recurring misconception is that any heterogeneous MoE paper with a small number of active pathways is automatically a “heterogeneous 4-expert MoE.” MoHETS is an explicit counterexample (Ortigossa et al., 29 Jan 2026). It replaces Transformer FFNs with Mixture-of-Heterogeneous-Experts layers composed of one shared depthwise-convolution expert and $\{2,4,8,16\}$7 routed Fourier Analysis FFN experts, with experiments using $\{2,4,8,16\}$8 routed experts and top-$\{2,4,8,16\}$9. Counted by routed experts, it is an $8$00-expert sparse heterogeneous MoE; counted with the always-on shared expert, each layer has $8$01 expert pathways. The paper therefore states that it is not accurate to call the reported model “MoE4” (Ortigossa et al., 29 Jan 2026).

A second misconception is that “heterogeneous” necessarily means “architecturally different experts.” Expert Choice routing uses the term in a different sense: heterogeneity is introduced because different tokens may be processed by different numbers of experts, even when the experts themselves are architecturally homogeneous (Zhou et al., 2022). MixtureKit adds yet another sense, in which heterogeneity may arise from different domain specializations and training histories of compatible checkpoints that are then composed through Traditional MoE, BTX, or BTS (Chamma et al., 13 Dec 2025). In remote sensing super-resolution, heterogeneity is implemented across expert groups with different convolution kernel sizes, while experts within each group remain homogeneous; the paper’s closest four-expert adaptation is $8$02 groups $8$03 $8$04 experts each rather than four fully distinct expert types (Chen et al., 12 Feb 2025).

A third misconception is that empirical rankings over expert families are already global conclusions about MoE4 composition. The automated LEMUR study explicitly rules this out because its search covered only an AirNet-anchored $8$05 of the theoretical combination space and every candidate was trained for only $8$06 epoch (Lukhi et al., 21 Jun 2026). Statements such as the strength of ShuffleNet or MobileNetV3 are therefore conditional conclusions within that anchored subset, not universal rankings across all $8$07 architecture families.

The literature also places clear limits on what is directly established for MoE4. Some results are exact four-expert findings, such as the dense four-backbone classifier (Lukhi et al., 21 Jun 2026), the four-rate VidPrism configuration (Lin et al., 27 May 2026), the four-expert mutual-distillation ablation (Xie et al., 2024), and the four-expert top-$8$08 PEFT setting in CoMoE (Feng et al., 23 May 2025). Other highly relevant results—expert-size heterogeneity, analog–digital partitioning, and mixed-GPU scheduling—are derived from larger expert counts and transfer to MoE4 by structural argument rather than direct four-expert benchmarking (Sun et al., 2024, Chowdhury et al., 3 Mar 2026, Wu et al., 4 Apr 2025).

Taken together, the literature supports a precise but plural understanding of Heterogeneous MoE4. It can denote a dense four-backbone ensemble with a learned gate, a four-path temporal architecture with expert streams at rates $8$09, a four-expert routed model trained with mutual distillation or contrastive modularization, or a four-expert system deployed under heterogeneous hardware constraints. What unifies these variants is not a single routing law or expert template, but the principle that a four-expert MoE can benefit when the four experts are differentiated by backbone family, pathway, capacity, routing role, or deployment substrate rather than treated as identical replicas.