UnimpMoe: Unified Sparse MoE Approaches

• UnimpMoe is a diverse label for MoE techniques that employ sparse, input-dependent expert activation to address varying constraints such as hardware efficiency, latency, and multimodal integration.
• In the Dense2MoE variant, the approach converts dense Transformers into sparse MoE models using hardware-aware pruning and upcycling, yielding lower latency and improved benchmark scores.
• Additional UnimpMoe methods include importance-driven expert scheduling for edge inference, multimodal encoders with sparse FFNs, and graph encoders that enhance representational richness and interpretability.

Searching arXiv for the specified papers and related "UnimpMoe" usages. UnimpMoe is an overloaded label in recent arXiv literature rather than the name of a single canonical architecture. In the material at hand, it refers most explicitly to the Unified Pruning & Upcycling framework of Dense2MoE for on-device LLMs, but closely related spellings are also used for an importance-driven expert-scheduling method for edge MoE inference, for the IMP-MoE multimodal pretraining system, and for a UniMP-style graph encoder augmented with a sparse Mixture-of-Experts layer in neural vehicle-routing research (Li et al., 26 May 2026, Zhu et al., 26 Aug 2025, Akbari et al., 2023, Afifi, 5 Jul 2026). Across these usages, the common motif is sparse, input-dependent expert activation, but the optimization target differs sharply: hardware-aware model conversion, runtime offloading, multimodal scaling, or graph-level representational specialization.

1. Nomenclature and scope

The label is best understood as a family resemblance across distinct MoE instantiations rather than a single model lineage. One usage centers on converting dense Transformers into hardware-aware sparse models via pruning and upcycling; another concerns scheduling already-activated experts under GPU-memory constraints; a third denotes a multimodal encoder trained with Alternating Gradient Descent; and a fourth denotes a graph encoder whose last-stage latent is refined by a sparse expert layer (Li et al., 26 May 2026, Zhu et al., 26 Aug 2025, Akbari et al., 2023, Afifi, 5 Jul 2026).

Usage of the label	Domain	Defining mechanism
Dense2MoE “Unified Pruning & Upcycling”	On-device LLMs	Layer-Fusion UpCycling with pruning of attention and upcycling of MLPs
Importance-driven expert scheduling	Edge MoE deployment	Low-importance substitution and GPU-cache reuse scheduling
IMP-MoE / “UnimpMoE”	Multimodal perception	Modality-agnostic encoder with sparse FFN experts and AGD
UniMP-style UnimpMoe	Neural MAVRP	MoE layer after message passing, before decoder attention

A common misconception is to treat these as interchangeable architectures. The evidence instead indicates a naming collision around sparsely gated experts. This suggests that “UnimpMoe” functions more as a local paper-specific designation than as a standardized research term.

2. Dense2MoE: unified pruning and upcycling

In Dense2MoE, “UnimpMoe” denotes a hardware-aware pipeline that converts a pretrained dense Transformer into a sparse MoE model by combining structured pruning with Layer-Fusion UpCycling. The retained “base” layer keeps its shared attention module, while attention modules in structurally redundant successor layers are pruned and their MLP sub-blocks are upcycled into experts inside a fused MoE block (Li et al., 26 May 2026).

The central construction forms an expert pool of size

$N = K + n^* \times M,$

where $K$ copies of the base MLP preserve capacity and $M$ copies of each pruned-layer MLP introduce heterogeneity. At inference, a single shared MHA is executed, and the cascade of pruned layers is replaced by a routed MoE sum over the selected experts. The stated motivation is to remove the main memory-bandwidth bottleneck by pruning bandwidth-heavy attention modules while reusing pretrained MLP weights.

Layer selection is guided by Hardware Roofline Theory. The paper uses a Roofline latency estimate limited by compute or memory bandwidth, and operationalizes pruning through inter-layer decoder-output and MLP-input cosine similarities, a norm-matching constraint

$$\Delta_{\mathrm{norm}} =\frac{\bigl|\|h^{(l)}\|_2-\|h^{(l+n)}\|_2\bigr|}{\|h^{(l+n)}\|_2} <\epsilon,$$

a global greedy search for non-overlapping blocks, and a multi-objective reward

$$\text{Reward}_i =S_i\;\bigl(\tfrac{LaT_i}{LaT_{\mathrm{base}}}\bigr)^{w},$$

with $w\approx -0.15$ on Thor-U. This makes pruning explicitly latency-aware rather than accuracy-only.

Routing is standard sparse MoE. For each token representation $h_t$, a router $W_r$ computes softmax gate values, Top-$k$ experts are selected, per-expert capacities are enforced, and a Switch/FOCUS-style load-balancing loss is added:

$$\mathcal{L}_{\mathrm{load}} =\alpha\sum_{i=1}^N \Bigl(\mathbb{E}_t[g_i(h_t)] - \tfrac1N\Bigr)^2, \quad \alpha=10^{-3}.$$

The continual pretraining workflow uses a corpus of approximately $K$0B tokens, updates $K$1, all MLPs, and LayerNorms, and occasionally fine-tunes a small learning rate on remaining parameters.

The reported Pareto gains are concrete. On Qwen2.5-0.5B, a 19-layer Dense2MoE activates only $K$2B parameters per token, has Roofline-predicted latency $K$3 ms versus $K$4 ms for the dense model, and achieves $K$5 average score across nine benchmarks versus $K$6 for the dense baseline. Table 1 further reports Dense at $K$7B active parameters, $K$8 ms, and $K$9 average score; UIDL at $M$0B, $M$1 ms, and $M$2; ToMoE at $M$3B, $M$4 ms, and $M$5; Llama-MoE at $M$6B, $M$7 ms, and $M$8; and Dense2MoE at $M$9B, $$\Delta_{\mathrm{norm}} =\frac{\bigl|\|h^{(l)}\|_2-\|h^{(l+n)}\|_2\bigr|}{\|h^{(l+n)}\|_2} <\epsilon,$$0 ms, and $$\Delta_{\mathrm{norm}} =\frac{\bigl|\|h^{(l)}\|_2-\|h^{(l+n)}\|_2\bigr|}{\|h^{(l+n)}\|_2} <\epsilon,$$1. The same paper states that across Qwen2.5-1.5B and LLaMA2-7B seeds, Dense2MoE delivers approximately $$\Delta_{\mathrm{norm}} =\frac{\bigl|\|h^{(l)}\|_2-\|h^{(l+n)}\|_2\bigr|}{\|h^{(l+n)}\|_2} <\epsilon,$$2–$$\Delta_{\mathrm{norm}} =\frac{\bigl|\|h^{(l)}\|_2-\|h^{(l+n)}\|_2\bigr|}{\|h^{(l+n)}\|_2} <\epsilon,$$3 point average-score gains at equal or lower latency (Li et al., 26 May 2026).

3. Importance-driven expert scheduling on the edge

A distinct usage of “UnimpMoe” refers to a deployment-time mechanism for running MoE models on consumer-grade edge hardware under strict memory limits. Here the problem is not how to build the experts, but how to decide which activated experts should remain resident in GPU memory, which can be substituted by functionally similar cached experts, and which must be offloaded or computed on CPU (Zhu et al., 26 Aug 2025).

The method starts from router outputs $$\Delta_{\mathrm{norm}} =\frac{\bigl|\|h^{(l)}\|_2-\|h^{(l+n)}\|_2\bigr|}{\|h^{(l+n)}\|_2} <\epsilon,$$4 and defines normalized expert importances

$$\Delta_{\mathrm{norm}} =\frac{\bigl|\|h^{(l)}\|_2-\|h^{(l+n)}\|_2\bigr|}{\|h^{(l+n)}\|_2} <\epsilon,$$5

For each token, the top-$$\Delta_{\mathrm{norm}} =\frac{\bigl|\|h^{(l)}\|_2-\|h^{(l+n)}\|_2\bigr|}{\|h^{(l+n)}\|_2} <\epsilon,$$6 activated experts are partitioned into top-score experts and low-score experts using the $$\Delta_{\mathrm{norm}} =\frac{\bigl|\|h^{(l)}\|_2-\|h^{(l+n)}\|_2\bigr|}{\|h^{(l+n)}\|_2} <\epsilon,$$7-th score threshold $$\Delta_{\mathrm{norm}} =\frac{\bigl|\|h^{(l)}\|_2-\|h^{(l+n)}\|_2\bigr|}{\|h^{(l+n)}\|_2} <\epsilon,$$8 and a substitution margin $$\Delta_{\mathrm{norm}} =\frac{\bigl|\|h^{(l)}\|_2-\|h^{(l+n)}\|_2\bigr|}{\|h^{(l+n)}\|_2} <\epsilon,$$9. Low-score activated experts that are not resident in GPU memory become candidates for replacement by near-threshold experts already cached on GPU or already in the top-score set. The paper’s claim is that one only pays PCIe or CPU cost for the truly irreplaceable low-score experts; the rest are swapped with functionally similar cached experts with negligible effect on final accuracy.

Cache management is cast as a streaming knapsack-style problem that maximizes the number of activated experts already resident in GPU memory subject to a memory capacity $$\text{Reward}_i =S_i\;\bigl(\tfrac{LaT_i}{LaT_{\mathrm{base}}}\bigr)^{w},$$0. The proposed greedy approximation maintains a sliding-window average $$\text{Reward}_i =S_i\;\bigl(\tfrac{LaT_i}{LaT_{\mathrm{base}}}\bigr)^{w},$$1 for each expert, locks in top-score experts, fills remaining GPU slots with inactive experts having highest $$\text{Reward}_i =S_i\;\bigl(\tfrac{LaT_i}{LaT_{\mathrm{base}}}\bigr)^{w},$$2, and evicts the smallest-$$\text{Reward}_i =S_i\;\bigl(\tfrac{LaT_i}{LaT_{\mathrm{base}}}\bigr)^{w},$$3 unlocked expert if capacity is exceeded. The system then coordinates GPU gating, CPU-side routing and load balancing, and asynchronous PCIe transfers in a three-stage pipeline. The CPU-Load Balancer minimizes the maximum of expert-load time and CPU-compute time by sorting CPU-resident experts by batch size and assigning larger batches to GPU so that PCIe cost is amortized.

The quantitative outcome is a deployment-oriented performance profile. The paper reports $$\text{Reward}_i =S_i\;\bigl(\tfrac{LaT_i}{LaT_{\mathrm{base}}}\bigr)^{w},$$4–$$\text{Reward}_i =S_i\;\bigl(\tfrac{LaT_i}{LaT_{\mathrm{base}}}\bigr)^{w},$$5 reduction in decoding-stage latency relative to prior MoE offloading systems, depending on batch size and GPU memory, with a $$\text{Reward}_i =S_i\;\bigl(\tfrac{LaT_i}{LaT_{\mathrm{base}}}\bigr)^{w},$$6 reduction on a $$\text{Reward}_i =S_i\;\bigl(\tfrac{LaT_i}{LaT_{\mathrm{base}}}\bigr)^{w},$$7 GB A6000 at batch $$\text{Reward}_i =S_i\;\bigl(\tfrac{LaT_i}{LaT_{\mathrm{base}}}\bigr)^{w},$$8. GPU-expert cache hit rate rises from approximately $$\text{Reward}_i =S_i\;\bigl(\tfrac{LaT_i}{LaT_{\mathrm{base}}}\bigr)^{w},$$9 in prior work to over $w\approx -0.15$0, and the accuracy effect of low-score substitution is reported as negligible, below $w\approx -0.15$1 on five QA/Reasoning benchmarks for $w\approx -0.15$2–$w\approx -0.15$3. Ablation assigns approximate TPOT reductions of $w\approx -0.15$4 to the expert-cache router, $w\approx -0.15$5 to prefetching, $w\approx -0.15$6 to cache eviction, and $w\approx -0.15$7 to the final CPU-Load Balancer. The explicit trade-off is that larger $w\approx -0.15$8 increases substitution and lowers TPOT at the cost of slight accuracy drift (Zhu et al., 26 Aug 2025).

4. IMP-MoE and the multimodal “UnimpMoE” usage

A third usage appears in Integrated Multimodal Perception, where the sparse multimodal encoder is described as IMP-MoE and the consolidated technical description labels it “UnimpMoE.” The design is a single modality-agnostic Transformer encoder that integrates image, video, text, and audio, with MoE feed-forward blocks replacing dense FFNs in the last $w\approx -0.15$9 of Transformer layers (Akbari et al., 2023).

The encoder shares self-attention and sparse FFN parameters across vision, audio, and text. Vision inputs are patchified into $h_t$0D tokens; text uses a $h_t$1K-piece SentencePiece vocabulary with $h_t$2D positional embeddings; audio can use waveform patches or spectrogram patches. In IMP-L, the encoder has $h_t$3 layers, and the final $h_t$4 layers are sparse MoE. Output heads support both supervised classification and contrastive learning through separate two-layer MLP projection heads with GeLU nonlinearity.

Training uses Alternating Gradient Descent rather than a static weighted sum of all tasks. On each step, one dataset-objective pair is sampled with probability proportional to dataset size, one batch is processed, one task loss is computed, and one gradient is applied. In expectation this optimizes the weighted objective, but avoids gradient accumulation and large static graphs. The implementation emphasizes rematerialization with jax.checkpoint, scan-over-layers with jax.lax.scan, and partitioned compilation with jax.pjit.

The MoE itself uses a small gating network $h_t$5 with softmax outputs over $h_t$6 experts, experts-choose routing with top-capacity assignment, capacity factor $h_t$7, and no auxiliary jitter or enhanced load-balancing losses beyond the capacity constraint. Hyperparameters in the description specify $h_t$8 for IMP-S and $h_t$9 for IMP-B and IMP-L. QK LayerNorm is applied in self-attention to stabilize large-scale MoE training.

The empirical emphasis is multimodal scaling efficiency. IMP-MoE-L is reported with $W_r$0B sparse parameters and achieves $W_r$1 zero-shot accuracy on Kinetics-400, $W_r$2 on Kinetics-600, and $W_r$3 on Kinetics-700, improving the previous state of the art by $W_r$4, $W_r$5, and $W_r$6 percentage points while using only $W_r$7 of the prior training computational cost. Additional numbers include $W_r$8 top-1 on ImageNet-1k, $W_r$9 on UCF101, $k$0 on HMDB51, and $k$1 on ESC-50. The same description states that IMP-MoE-L uses approximately $k$2M dense plus approximately $k$3B sparse parameters, yet per-token compute equivalent to a ViT-L with $k$4M dense parameters, and that alternating multi-resolution plus AGD plus MoE saves $k$5–$k$6 cost versus naive multi-task batching (Akbari et al., 2023).

5. UnimpMoe as a sparse graph encoder in neural MAVRP

In neural combinatorial optimization, “UnimpMoe” names a UniMP-style graph encoder augmented with a sparsely gated MoE layer. The MoE is placed immediately after the last message-passing layer of the Unimp backbone and before the decoder attends over the node latents (Afifi, 5 Jul 2026).

If $k$7 is node $k$8’s embedding after $k$9 message-passing layers, the model defines $$\mathcal{L}_{\mathrm{load}} =\alpha\sum_{i=1}^N \Bigl(\mathbb{E}_t[g_i(h_t)] - \tfrac1N\Bigr)^2, \quad \alpha=10^{-3}.$$0 expert MLPs $$\mathcal{L}_{\mathrm{load}} =\alpha\sum_{i=1}^N \Bigl(\mathbb{E}_t[g_i(h_t)] - \tfrac1N\Bigr)^2, \quad \alpha=10^{-3}.$$1 and a shared gate

$$\mathcal{L}_{\mathrm{load}} =\alpha\sum_{i=1}^N \Bigl(\mathbb{E}_t[g_i(h_t)] - \tfrac1N\Bigr)^2, \quad \alpha=10^{-3}.$$2

with output

$$\mathcal{L}_{\mathrm{load}} =\alpha\sum_{i=1}^N \Bigl(\mathbb{E}_t[g_i(h_t)] - \tfrac1N\Bigr)^2, \quad \alpha=10^{-3}.$$3

This architecture is studied not only as a solver component but as an object of interpretability analysis through encoder probing and decoder attribution.

The encoder-side findings emphasize distributed coding of constraint families. Graph-level clustering yields normalized mutual information approximately $$\mathcal{L}_{\mathrm{load}} =\alpha\sum_{i=1}^N \Bigl(\mathbb{E}_t[g_i(h_t)] - \tfrac1N\Bigr)^2, \quad \alpha=10^{-3}.$$4 for UnimpMoe/Hard-Mask and approximately $$\mathcal{L}_{\mathrm{load}} =\alpha\sum_{i=1}^N \Bigl(\mathbb{E}_t[g_i(h_t)] - \tfrac1N\Bigr)^2, \quad \alpha=10^{-3}.$$5 for UnimpMoe/Recourse, versus $$\mathcal{L}_{\mathrm{load}} =\alpha\sum_{i=1}^N \Bigl(\mathbb{E}_t[g_i(h_t)] - \tfrac1N\Bigr)^2, \quad \alpha=10^{-3}.$$6–$$\mathcal{L}_{\mathrm{load}} =\alpha\sum_{i=1}^N \Bigl(\mathbb{E}_t[g_i(h_t)] - \tfrac1N\Bigr)^2, \quad \alpha=10^{-3}.$$7 for the attention baseline and $$\mathcal{L}_{\mathrm{load}} =\alpha\sum_{i=1}^N \Bigl(\mathbb{E}_t[g_i(h_t)] - \tfrac1N\Bigr)^2, \quad \alpha=10^{-3}.$$8–$$\mathcal{L}_{\mathrm{load}} =\alpha\sum_{i=1}^N \Bigl(\mathbb{E}_t[g_i(h_t)] - \tfrac1N\Bigr)^2, \quad \alpha=10^{-3}.$$9 for Unimp. Silhouette scores are lower, approximately $K$00 for Hard-Mask and $K$01 for Recourse, than the attention baseline’s approximately $K$02–$K$03, because UnimpMoe clusters are broader but more informative. Node-role linear probes reach macro-F1 approximately $K$04 versus approximately $K$05–$K$06 for the other encoders, while edge-level same-route prediction remains approximately $K$07–$K$08 AUC for all encoders.

The stated signature of UnimpMoe is higher richness with less axis alignment. Effective rank is approximately $K$09 for UnimpMoe, approximately $K$10 for Unimp, and approximately $K$11 for the attention baseline at $K$12. In an unsupervised PCA or ICA basis, the best one-axis correlation with a constraint family is lower for UnimpMoe than for Unimp, but allowing a $K$13 dimensional subspace reduces the gap to below $K$14. The paper identifies this gap between one-axis and subspace recoverability as the formal witness to distributed coding. PCA also reveals that $K$15 of UnimpMoe correlates at approximately $K$16 with time-window tightness, while $K$17–$K$18 pick up geometry or capacity. Causal intervention along aligned axes yields $K$19 concept-level success but only approximately $K$20 direction-level success, reinforcing the claim that each axis captures only part of a broader distributed signal.

Decoder attribution extends the picture. With gradient-based explanations, UnimpMoe/Hard-Mask and UnimpMoe/Recourse both reach flip@1 $K$21, flip@3 $K$22, and flip@5 $K$23, with $K$24 approximately $K$25 and $K$26, and $K$27 approximately $K$28 and $K$29, exceeding the attention baseline. Concentration is slightly higher for Hard-Mask than Recourse, while Recourse is more diffuse but provides richer state-dependent signal. Sanity ratios are approximately $K$30 and $K$31 for flip@1 and approximately $K$32 and $K$33 for $K$34, indicating that the attributions are learned rather than architectural artifacts. Actionability differs sharply by decoder regime: availability of any switch or make-feasible counterfactual is approximately $K$35 under Hard-Mask and approximately $K$36 under Recourse, with make-feasible rate $K$37 versus approximately $K$38. The paper’s interpretation is that Recourse training, not merely a softer mask, produces policies that represent infeasibility usefully (Afifi, 5 Jul 2026).

6. Position within broader MoE research

Placed against the wider MoE literature, the several “UnimpMoe” usages occupy distinct points in the design space. Dense2MoE uses experts to repurpose pretrained dense MLPs after pruning attention, so its central object is a hardware-aware converted Transformer. The edge-scheduling variant assumes an existing MoE and optimizes runtime residency, substitution, and overlap across GPU, CPU, and PCIe. IMP-MoE uses sparse FFN experts for modality-agnostic multimodal scaling under Alternating Gradient Descent. The UniMP-style version attaches experts to graph latents in order to alter representational geometry and improve decoder-side faithfulness and actionability (Li et al., 26 May 2026, Zhu et al., 26 Aug 2025, Akbari et al., 2023, Afifi, 5 Jul 2026).

This contrasts with other named MoE systems in the same corpus. UMoE unifies attention and FFN layers by reformulating self-attention to expose a two-layer linear structure and then sharing a single bank of experts across attention and FFN sub-layers; its Base-model ablation reports parameter reduction from approximately $K$39M plus $K$40M to $K$41M overall, with separate routers performing best (Yang et al., 12 May 2025). SC-MoE, by contrast, is a Switch-Conformer-based system for unified streaming and non-streaming code-switching ASR, with encoder-side experts for Mandarin, English, and blank, a CTC-trained LID router, and decoder-side MoE that together improve CER, WER, and MER while keeping multiply-adds at $K$42M through top-1 sparsity (Ye et al., 2024).

The broader significance of the “UnimpMoe” label is therefore not architectural uniformity but the recurrence of sparse expert routing as a solution to different bottlenecks. In one setting the bottleneck is DRAM bandwidth and on-device latency; in another it is PCIe transfer and cache reuse; in another it is multimodal gradient conflict and scaling cost; in another it is the tension between representational richness and interpretability. A plausible implication is that the label’s future meaning will depend less on a fixed topology than on which systems-level constraint or representational objective the next paper chooses to optimize.