MoE-Beyond: Extending Mixture-of-Experts Research

• MoE-Beyond is a research framework that extends traditional Mixture-of-Experts by redefining what constitutes an expert and by innovating routing mechanisms across varied domains.
• It incorporates fine-grained specialization methods, parameter-space decomposition, and lightweight adaptations such as PEFT to enhance efficiency while maintaining high performance.
• MoE-Beyond also addresses hardware-aware training and inference challenges by optimizing memory, throughput, and edge execution to enable robust, specialized deployments.

MoE-Beyond denotes a line of Mixture-of-Experts research that extends MoE past the conventional Transformer pattern in which a router sends each token to a small top-$K$ set of feed-forward experts. In this broader literature, the “expert” can be a fine-grained FFN fragment, a rank-one spectral component of a pretrained matrix, a PEFT module, a camera view, a driving skill, or a fused layer; routing can operate in activation space, parameter space, or execution policy; and the objective can shift from parameter scaling alone to specialization, memory efficiency, throughput, edge deployment, robustness, or trustworthiness (Yuan et al., 17 Jun 2025, Zhang et al., 8 Jan 2026, Gavhane et al., 23 Aug 2025).

1. Scope and defining departures from conventional MoE

Conventional MoE in this literature is usually written as $y=\sum_{i\in\tau}R_i(x)E_i(x)$, where $\tau$ is the selected Top-$K$ expert set and $R$ is the router. DeepSeek-style and fine-grained variants preserve this outer form, but they change what counts as an expert, how many experts are activated, and how shared versus routed capacity is organized (Krajewski et al., 3 Jun 2025, Dai et al., 2024).

Taken together, the literature suggests that “beyond” standard MoE can be understood along three main axes: redefining the expert itself, redesigning the systems substrate that executes sparse routing, and transplanting MoE principles into new deployment regimes or application domains.

Axis	Representative mechanism	Example papers
Expert redefinition	Spectral, PEFT, fine-grained, or fused-layer experts	MoORE (Yuan et al., 17 Jun 2025), PEFT-MoE (Zadouri et al., 2023), DeepSeekMoE (Dai et al., 2024), Dense2MoE (Li et al., 26 May 2026)
System redesign	Metadata-only dispatch, expert-specific operators, padding-free training	MoEBlaze (Zhang et al., 8 Jan 2026), HEXA-MoE (Luo et al., 2024), X-MoE (Yuan et al., 18 Aug 2025)
Deployment and application shift	Edge caching, cost-aware execution, autonomous driving, trustworthy edge MoE	MoE-Beyond (Gavhane et al., 23 Aug 2025), CAEE (Zang et al., 29 Jun 2026), DriveMoE (Yang et al., 22 May 2025), B-MoE (Zhu et al., 15 Sep 2025)

A plausible implication is that MoE-Beyond is less a single architecture than a research program: it asks where conditional computation should occur, what object should be selected or combined, and which hardware or application constraints should shape the design.

2. Recasting what an expert is

One major departure replaces coarse routed FFN blocks with finer or structurally constrained units. DeepSeekMoE is the canonical example of fine-grained specialization: it segments experts into $mN$ experts, activates $mK$ of them, and isolates $K_s$ shared experts to capture common knowledge. At 2B parameters, DeepSeekMoE 2B is reported as comparable to GShard 2.9B, and at 16B it is reported as comparable to LLaMA2 7B while using only about 40% of computations (Dai et al., 2024). The fine-grained scaling study extends this logic empirically up to 55.8B total parameters and 17.1B active parameters, finding that $G=8$ fine-grained configurations can improve validation loss and downstream accuracy, and that a fine-grained Top-1-style model can match a standard Top-2 baseline at the largest tested scale (Krajewski et al., 3 Jun 2025).

A more radical shift moves routing from activation-space subnetworks into parameter space. MoORE performs “model MoE-ization” by applying SVD to a pretrained weight matrix $W$, interpreting each rank-one term $y=\sum_{i\in\tau}R_i(x)E_i(x)$0 as an expert, and routing by additively modulating singular values. The adapted operator is

$y=\sum_{i\in\tau}R_i(x)E_i(x)$1

optionally followed by an orthogonal transform $y=\sum_{i\in\tau}R_i(x)E_i(x)$2. The stated advantages are expert orthogonality and preservation of $y=\sum_{i\in\tau}R_i(x)E_i(x)$3, which the paper connects to resistance against task conflict and oblivion in multi-task adaptation (Yuan et al., 17 Jun 2025). Here the MoE is not a collection of added modules; it is a routed decomposition of the original pretrained operator.

Another branch compresses the expert itself into a PEFT object. “Pushing Mixture of Experts to the Limit” replaces full experts with $y=\sum_{i\in\tau}R_i(x)E_i(x)$4 vectors or LoRA updates, yielding Mixture of Vectors and Mixture of LoRA. The dense backbone remains frozen, only the router and lightweight experts are trained, and the paper reports competitive instruction tuning while updating less than 1% of an 11B model (Zadouri et al., 2023). In this regime, soft routing over all experts outperforms sparse top-$y=\sum_{i\in\tau}R_i(x)E_i(x)$5, which is a direct reversal of the usual sparse-MoE intuition.

Dense2MoE introduces a different redefinition of expertise through Layer Fusion UpCycling. Instead of cloning an MLP within one layer, it prunes redundant attention modules from neighboring layers and repurposes those layers’ MLPs as heterogeneous experts attached to a retained base layer. In the reported Qwen2.5-0.5B result, a 24-layer dense model becomes a 19-layer Dense2MoE model, reducing active parameters from 0.50B to 0.42B while improving average benchmark score from 37.53 to 47.31 and lowering theoretical latency from 195.87 ms to 174.95 ms (Li et al., 26 May 2026). This suggests that “expert” can also mean preserved knowledge extracted from inter-layer redundancy rather than an FFN defined in place.

3. Memory-centric and hardware-aware training systems

A second MoE-Beyond trajectory argues that the main obstacle to large sparse models is not arithmetic but memory movement. MoEBlaze makes this claim most explicitly. For a DeepSeek-like configuration with $y=\sum_{i\in\tau}R_i(x)E_i(x)$6 million, $y=\sum_{i\in\tau}R_i(x)E_i(x)$7, and $y=\sum_{i\in\tau}R_i(x)E_i(x)$8, the paper gives a routed activation buffer of about 94GB, and for hidden size $y=\sum_{i\in\tau}R_i(x)E_i(x)$9 it gives expert activation storage of about 98GB for a single MoE layer (Zhang et al., 8 Jan 2026). MoEBlaze therefore replaces routed-token materialization with compact metadata—$\tau$0, $\tau$1, $\tau$2, and $\tau$3—so expert kernels gather from the original activation tensor on the fly and reduce outputs directly back into token order. Combined with fused SwiGLU kernels and selective checkpointing, this yields up to 6.2× speedup and multi-fold activation-memory reduction on H100 in the reported single-GPU experiments (Zhang et al., 8 Jan 2026).

The earlier DeepSpeed MoE work frames the same problem at a larger systems scale. Its contribution is five-dimensional parallelism—data parallelism, tensor/model parallelism, expert parallelism, ZeRO, and ZeRO-Offload—which lets sparse models scale both expert parameters and the dense base model while exploiting heterogeneous GPU and CPU memory. The paper reports support for 3.5 trillion parameters on 512 A100 GPUs and argues that optimizer states and gradients, rather than activations alone, are often the dominant memory burden (Kim et al., 2021). It also introduces Random Token Selection to remove the positional bias induced by capacity-limited token dropping, Aggregation of Experts to merge expert pools across checkpoints, and expert pruning for deployment (Kim et al., 2021).

HEXA-MoE pushes beyond the standard expert-parallel substrate itself. It replaces generic GEMM/grouped-GEMM execution with expert-specific operators—ESMM, ESS, and ESTMM—designed for in-place computation with “ZERO redundancy,” and it supports both data-centric and model-centric configurations rather than assuming homogeneous expert-parallel clusters. On the Swin-MoE benchmark, it reports reducing memory consumption by 10%–48% and achieving 0.5–4.3× speedup relative to current MoE libraries, while also showing that proportional partitioning over heterogeneous devices can reduce latency by 6.3%–25.3% depending on configuration (Luo et al., 2024).

X-MoE extends this hardware-conscious redesign to DeepSeek-style expert-specialized MoEs on AMD-based HPC systems. It introduces Padding-Free Token buffers, redundancy-bypassing dispatch, and sequence-sharded MoE blocks. On Frontier, it trains DeepSeek-style MoEs up to 545.4B parameters across 1024 MI250X GPUs—reported as 10× larger than the largest trainable model with existing methods under the same hardware budget—and reaches 10.44 PFLOPS in the largest configuration (Yuan et al., 18 Aug 2025). Its underlying claim is that once top-$\tau$4 and expert count grow, dispatch/combine activations and all-to-all redundancy become the central bottlenecks, so sparse-model systems must be redesigned around those costs rather than around dense-kernel abstractions.

4. Inference, offloading, and edge execution

MoE-Beyond is equally visible on the inference side, where the main challenge is often no longer FLOPs but bandwidth, scheduling, and memory residency. EPS-MoE addresses multi-GPU inference by treating expert execution as a schedulable pipeline rather than a monolithic GroupGemm invocation. It dynamically switches between GroupGemm and DenseGemm based on token load, overlaps all-to-all communication with expert FFN execution, and reports at most 52.4% improvement in prefill throughput in microbenchmarks; on DeepSeekV2 it raises a claimed 100K tokens/s to at least 120K tokens/s in the reported setting (Qian et al., 2024).

MoE-Lightning targets a different regime: batch inference for large MoEs on memory-constrained GPUs. Its CGOPipe schedule overlaps GPU compute, CPU attention, and CPU-GPU-I/O transfers, while paged weights and a Hierarchical Roofline Model guide placement of weights and KV cache. The paper reports up to 10.3× higher throughput than prior offloading-enabled systems for Mixtral 8x7B on a single T4 GPU, and support for Mixtral 8x22B and DBRX on 2–4 T4 GPUs (Cao et al., 2024). The broader implication is that sparse activation alone does not make MoE deployment cheap; systems must also manage expert weight residency and transfer overlap effectively.

A still more deployment-specific branch predicts or suppresses expert execution itself. The paper “MoE-Beyond” formulates expert activation prediction as a multi-label sequence problem over 64 routed experts, trains a lightweight transformer on 66 million DeepSeek-V2-Lite activation traces, and reports 97.5% accuracy with an 86.6% F1-score on held-out prompts. In simulation, when only 10% of experts fit in GPU cache, it raises cache hit rate from 17% to 72% relative to MoE-Infinity (Gavhane et al., 23 Aug 2025). CAEE, by contrast, treats runtime expert execution as a joint importance-cost optimization: it estimates expert importance from router scores, estimates execution cost with calibrated hardware models, prunes low-importance high-cost experts on the critical path, and compensates by redistributing routing mass to already retained experts. On DeepSeek-R1 671B, it reports 8%–18% end-to-end latency reduction with a model accuracy drop of less than 1% in moderate operating regimes (Zang et al., 29 Jun 2026).

Dense2MoE belongs partly in this inference lineage as well. Guided by Roofline theory, it converts dense checkpoints into top-1 sparse models that are explicitly optimized for on-device latency under unified memory constraints, rather than only for static compression ratios (Li et al., 26 May 2026). Taken together, these works suggest that inference-oriented MoE research is shifting from “which experts are activated?” to “which activated experts should actually be executed, cached, moved, or suppressed under hardware cost constraints.”

5. Multilingual, embodied, and trustworthy extensions

Beyond standard LLM pretraining, MoE has been adapted to multilingual, multitask, embodied, and distrustful settings. The multilingual multitask line is exemplified by Z-code M3, which combines sparse MoE with encoder-decoder multilingual generation over 50 languages. The paper reports that a 10B-parameter model achieves state-of-the-art performance in machine translation and multilingual natural language generation, and that MoE variants can reach the same validation loss as a 0.7B dense baseline in roughly 10× fewer update steps (Kim et al., 2021). This use of MoE is not only about scale; it is about allocating conditional capacity across tasks and languages.

That same setting also revealed a failure mode. “Fixing MoE Over-Fitting on Low-Resource Languages in Multilingual Machine Translation” shows that sparsely gated MoEs can over-fit severely on low-resource and very low-resource directions. It introduces Expert Output Masking, Final Output Masking, Conditional MoE Routing, and step-based Curriculum Learning, and reports about +1 chrF++ improvement on very low-resource language pairs without harming high-resource directions (Elbayad et al., 2022). The important point is that MoE-Beyond includes not only new expert structures but also new regularization strategies for domains where sparse capacity can amplify memorization.

DriveMoE demonstrates a different kind of extension: experts need not correspond to language FFNs at all. Built on a vision-language-action baseline, it adds a Scene-Specialized Vision MoE that dynamically selects relevant camera views and a Skill-Specialized Action MoE that activates behavior experts for driving maneuvers. In Bench2Drive closed-loop evaluation, DriveMoE improves Driving Score from 60.45 to 74.22 and Success Rate from 30.00 to 48.64 over Drive-$\tau$5, with especially large gains on Emergency Brake (Yang et al., 22 May 2025). Here conditional computation is aligned with sensor selection and control specialization rather than with token routing inside a text model.

The most infrastructure-oriented extension is B-MoE, a blockchain-aided trustworthy distributed MoE framework. It consists of an edge layer, a blockchain layer, and a storage layer; the gating network is stored on chain, activated experts are downloaded from storage, and blockchain nodes compare results for the same expert across different edges and accept the most consistent ones as trustworthy. The paper reports that B-MoE is more robust to data manipulation attacks than traditional distributed MoE during both training and inference, though at the cost of higher latency and under the assumption that malicious blockchain power and malicious edge ratio each remain below 50% (Zhu et al., 15 Sep 2025). This broadens MoE from a model architecture into a verifiable distributed service architecture.

6. Evaluation, benchmarking, and unresolved questions

The MoE-Beyond literature has also changed how sparse models are evaluated. “Revisiting MoE and Dense Speed-Accuracy Comparisons for LLM Training” argues that equal-token or FLOP-based comparisons systematically favor MoE because they ignore routing communication overhead. It proposes measured step time and dense Chinchilla-style budget accounting instead, and still finds that MoE generally improves the speed-accuracy frontier, though not uniformly on every task; at the 29.6B-scale budget, for example, the reported MoE is faster and slightly better on CoreEN and MMLU but worse on GSM8K (Du et al., 2024). This establishes a stricter baseline for claims about sparse efficiency.

The fine-grained scaling study adds a second evaluation lesson: architecture comparisons must hold active parameters and non-router FLOPs fixed when possible. Under that protocol, fine-grained MoE with $\tau$6 shows clearer gains as model size and token budget increase, especially at 55.8B total parameters, where the best fine-grained Top-1 configuration matches or slightly exceeds a standard Top-2 baseline while using fewer active parameters (Krajewski et al., 3 Jun 2025). This supports the view that expert granularity is itself a scaling variable, not merely an implementation detail.

LIBMoE systematizes this benchmarking impulse. Built around modular MoE, training, and evaluation modules, it benchmarks five routing algorithms across three multimodal LLM configurations and 11 zero-shot datasets and finds that current MoE algorithms perform roughly similarly on average, despite visibly different routing dynamics, entropy profiles, and specialization behavior (Nguyen et al., 2024). One of its most consequential observations is that the final checkpoint is often not the best checkpoint, which suggests that routing analysis, checkpoint selection, and trajectory-aware evaluation remain underdeveloped.

Taken together, these papers leave several open questions. A plausible implication is that future MoE-Beyond work will be shaped less by a single universal router and more by co-design problems: how expert granularity interacts with hardware topology, how parameter-space and activation-space routing can be combined, how edge caching or cost-aware suppression generalizes beyond batch-size-1 settings, how trustworthy or embodied MoE scales outside controlled benchmarks, and how to benchmark sparse systems when total parameters, active parameters, memory traffic, and communication all tell different stories. The common thread is that MoE is no longer treated merely as a sparse FFN replacement. It has become a general framework for conditional specialization whose central problems are now architectural, systems-level, and domain-specific at once.