Dense2MoE: Efficient MoE Conversion

• Dense2MoE is a framework that transforms dense neural architectures, particularly Transformers, into conditional-sparse Mixture-of-Experts models to enhance efficiency.
• It employs dense training with softmax routing and load-balancing losses to enable sparse inference via top-K expert selection.
• Empirical results show Dense2MoE reduces active parameter usage and inference latency while preserving model expressivity, benefiting large-scale language and vision tasks.

Dense2MoE refers to a spectrum of model conversion frameworks and training regimes that systematically transition dense neural architectures—particularly Transformers—into conditional-sparse Mixture-of-Experts (MoE) models. Dense2MoE approaches aim to combine the parameter and computational efficiency of sparse MoEs with the performance, stability, and simplicity of dense, monolithic training. The primary motivation is to reduce active parameter count and inference latency without the characteristic redundancy or brittle expert initialization of traditional MoE setups. This is achieved through innovations in training protocols, expert construction, routing strategies, and post-hoc conversion algorithms.

1. Dense2MoE Frameworks: Definitions and Core Principles

Dense2MoE denotes strategies that either train neural networks in a dense mode but infer with conditional sparsity, or post-process existing dense model checkpoints to endow MoE properties. In the prototypical DS-MoE approach (Pan et al., 2024), training is performed in the dense regime by activating and backpropagating through all experts in MoE blocks for every token, using standard dense-model optimization protocols and losses. Only at inference is conditional sparsity imposed, with a top-$K$ gating policy determining active experts per token.

Other paradigms implement Dense2MoE via post-hoc conversion algorithms. These include: activation sparsity mining and k-means clustering (Szatkowski et al., 2023), optimal transport partitioning (Bamba et al., 1 Jun 2026), domain-informed pruning with retraining (Feng et al., 11 Jun 2025), or structured block fusion and bandwidth-aware pruning (Li et al., 26 May 2026). All Dense2MoE methods seek to (i) maintain model expressivity and capacity, (ii) minimize inference costs compared to the base dense models, and (iii) avoid the pitfalls of permanent parameter removal or naive MoE overparameterization.

2. Dense2MoE Methodologies

Dense2MoE methods span three principal recipe classes:

A. Dense Training, Sparse Inference (DS-MoE):

• All experts and router scores are computed and updated for every token at every step.
• Routing uses softmax gates, and all experts receive full gradients.
• A mutual information load-balancing loss term is added to encourage both balanced and selective routing:

$$L_{total} = L_{LM} + \lambda \cdot (-H(e) + H(e|X))$$

where $H(e)$ reflects marginal entropy (global expert balance), $H(e|X)$ is the token-level conditional entropy (encourages sparse routing).

• Sparse inference is achieved by top-K or threshold-based truncation of routing weights.

B. Structured Post-hoc Dense-to-MoE Conversion:

• Dense FFN weights are partitioned into experts either by activation patterns (Szatkowski et al., 2023), assignment via balanced optimal transport (Bamba et al., 1 Jun 2026), or domain-specific pruning masks (Feng et al., 11 Jun 2025).
• A router (trained or constructed) enforces sparse token–expert assignment at inference.
• Retraining is conducted only on the router or small adapters (LoRA/PEFT), usually via cross-entropy or KL mimicking the dense model.
• Conversion frameworks such as ToMoE (Gao et al., 25 Jan 2025), LaDiMo (Kim et al., 2024), and ExpertWeaver (Zhao et al., 17 Feb 2026) further introduce knowledge distillation, progressive parameter freezing, and adaptive expert sharing.

C. Hardware- and Application-Aware MoE Upcycling:

• For edge device deployment, models such as "Dense2MoE" (Li et al., 26 May 2026) search for bandwidth-heavy attention block redundancy, prune only the attention modules, and fuse preserved MLPs as MoE experts, maintaining low latency under strict hardware constraints.

3. Model Architecture and Inference Workflow

In DS-MoE and comparable schemes, the Transformer block is equipped with dual MoE sublayers: a Mixture-of-Attention-Heads (“MoA”), and a Mixture-of-Experts MLP block. Each MoE component includes:

• E experts (FFN or attention head submodules).
• A small gating network outputting softmax-ed scores $G_i(x)$ over the experts.
• The pre-activation MoE output for input $x$: $y(x) = \sum_{i=1}^{E} G_i(x) F_i(x)$, where $F_i$ is each expert submodule.

At inference, only a subset $A(x)$ of experts (by top-K or threshold criteria) are evaluated for each token, with final output reconstructed as:

$y(x) \approx \sum_{i \in A(x)} G_i(x) F_i(x)$

Complexity reduces from $$L_{total} = L_{LM} + \lambda \cdot (-H(e) + H(e|X))$$0 to $$L_{total} = L_{LM} + \lambda \cdot (-H(e) + H(e|X))$$1 per token, where $$L_{total} = L_{LM} + \lambda \cdot (-H(e) + H(e|X))$$2 (Pan et al., 2024).

4. Efficiency Analysis and Empirical Performance

Empirical studies across LLM and vision backbones consistently demonstrate that Dense2MoE models:

• Activate only 30–40% (DS-MoE) (Pan et al., 2024), 20–50% (ToMoE, DIVE) (Gao et al., 25 Jan 2025, Feng et al., 11 Jun 2025) of the total dense model parameters at inference.
• Achieve wall-clock speedups of up to 1.91× (DS-MoE-6B on A100 vs. Dense-6B) and 1.86× vs. Mistral-7B on vLLM (Pan et al., 2024).
• Retain or surpass original dense model perplexity and accuracy for active parameter budgets above ~30%; at more extreme sparsities, auxiliary distillation or domain-driven expert learning is crucial to prevent collapse.
• On-device deployments (e.g., Jetson Thor-U) demonstrate that by pruning attention while fusing MLPs as heterogeneous MoE experts, overall Pareto efficiency (latency vs. accuracy) can surpass standard pruning, naive MoE, and even dense baselines (Li et al., 26 May 2026).

5. Routing Strategies and Load-Balancing Losses

Dense2MoE frameworks employ routing networks with both fixed and dynamic sparse selection:

• Softmax routers with mutual information or load-balance losses to avoid expert collapse and ensure balanced utilization.
• Dynamic-$$L_{total} = L_{LM} + \lambda \cdot (-H(e) + H(e|X))$$3 routing (SADMoE (Szatkowski et al., 2023), ToMoE (Gao et al., 25 Jan 2025)) allows token-dependent variable compute allocation, reducing overhead for "easy" inputs.
• Routing thresholds, Gumbel-Softmax annealing schedules, content thresholds, and expert regularization (e.g., in EvoMoE (Nie et al., 2021), LaDiMo (Kim et al., 2024), MoE Jetpack (Zhu et al., 2024)) further stabilize training and deployment.
• Distillation-driven router learning (often using KL between teacher and student logits) is empirically superior to token-level cross-entropy (Gao et al., 25 Jan 2025).

6. Comparative Analysis with Related Paradigms

Dense2MoE methods differ fundamentally from classical model pruning, which irreversibly removes redundant weights and risks severe loss of capacity under high compression. Instead, MoE upcycling preserves all "knowledge repositories" (e.g., MLP modules) and maintains or even increases usable capacity (Li et al., 26 May 2026).

Compared to naive MoE construction (randomly initialized or isotropic experts), Dense2MoE realizes specialized and diverse experts via structured partitioning (domain-informed, activation-driven) and explicit retraining, avoiding redundancy and collapse (Feng et al., 11 Jun 2025).

A comparative summary:

Methodology	Parameter Savings	Performance Retention	Training Data Required	Key References
DS-MoE	~60–70% inactive	High (30–40% active)	Standard LLM	(Pan et al., 2024)
Dynamic-$$L_{total} = L_{LM} + \lambda \cdot (-H(e) + H(e|X))$$4 (SADMoE)	up to 60% FLOPs	High (<0.5% accuracy)	Fine-tuned dense	(Szatkowski et al., 2023)
ToMoE	40–70% inactive	Outperforms pruning	No original weight update	(Gao et al., 25 Jan 2025)
DIVE	50–88% inactive	Outperforms pruning/MoE upcycle	~6 days retrain	(Feng et al., 11 Jun 2025)
Layer-Fusion UpCycling	>1.5x speedup, higher accuracy	Pareto optimal	<1% scratch-train cost	(Li et al., 26 May 2026)

7. Limitations and Practical Considerations

Dense2MoE frameworks present specific trade-offs:

• Dense training of all experts causes elevated memory and compute cost relative to sparse training, though one-to-one token routing dominates at inference (Pan et al., 2024).
• Routing, expert selection, and expert diversity hyperparameters demand careful tuning; insufficient balancing can cause expert redundancy or collapse.
• For extreme resource-constrained scenarios (p < 30%), training may become unstable or expertise starve (Gao et al., 25 Jan 2025).
• Hardware-aware design (e.g., ensuring batch and router communication aligns with accelerator topology) is required for real inference gains (Du et al., 2024, Li et al., 26 May 2026).

Dense2MoE unlocks efficient, scalable conditional computation, often with superior Pareto tradeoffs in latency vs. accuracy vs. memory, and is broadly applicable to large language modeling, vision, and text-to-image diffusion tasks (Pan et al., 2024, Zhu et al., 2024, Zheng et al., 10 Oct 2025).