DualSparse-MoE Acceleration

• DualSparse-MoE is a post-training method that accelerates MoE models by exploiting tensor-level and neuron-level sparsity to reduce inference cost.
• The approach partitions each expert into finer sub-experts and applies dynamic routing and reconstruction to maintain accuracy while trimming computations.
• It leverages dual dropout strategies (1T-Drop and 2T-Drop) with calibrated thresholds to optimally balance efficiency and performance in large language models.

DualSparse-MoE is a post-training method and inference system for accelerating Mixture-of-Experts (MoE) LLMs by coordinating two complementary forms of sparsity already present in pretrained MoE modules: tensor-level sparsity across experts and neuron-level sparsity within experts. In this usage, “dual sparsity” does not denote a new hierarchical router or a training-from-scratch MoE recipe; it denotes a deployment-oriented combination of post-training expert partitioning, dynamic token-expert computation dropping, and static neuron-aware reconstruction, with the stated goal of reducing inference cost while preserving accuracy (Cai et al., 25 Aug 2025).

1. Problem setting and conceptual scope

In the standard MoE formulation used by DualSparse-MoE, an input token representation $\mathbf{x}$ is scored by a gating network

$$\mathbf{s} = G(\mathbf{x}) = \text{Softmax}(\mathbf{x}\mathbf{W}_g),$$

with $\mathbf{W}_g \in \mathbb{R}^{d_{model}\times E}$. After Top-$K$ routing, the expert score is

$$g_e(\mathbf{x}) = \begin{cases} \mathbf{s}_i & \text{if } i \in \text{TopK}(\mathbf{s}, K),\ 0 & \text{otherwise}, \end{cases}$$

and the MoE output is

$$\mathbf{y} = \sum_{e=1}^{E} g_e(\mathbf{x}) \cdot f_e(\mathbf{x}),$$

where $f_e(\mathbf{x})$ is expert $e$’s FFN output. For a SwiGLU expert,

$$f(\mathbf{x}) = \left(\text{Swish}(\mathbf{x}\mathbf{W}_1)\odot (\mathbf{x}\mathbf{W}_3)\right)\mathbf{W}_2,$$

with $$\mathbf{W}_1,\mathbf{W}_3 \in \mathbb{R}^{d_{model}\times d_{ffn}}$$ and $$\mathbf{s} = G(\mathbf{x}) = \text{Softmax}(\mathbf{x}\mathbf{W}_g),$$0 (Cai et al., 25 Aug 2025).

The method is motivated by three deployment difficulties. First, even after sparse expert routing, the selected experts still contain very large FFN computations. Second, expert activation patterns are input-dependent and therefore uneven across devices. Third, expert parallelism (EP) introduces latency sensitivity to load imbalance, because end-to-end response time is often determined by the busiest device. DualSparse-MoE therefore distinguishes two efficiency axes already latent in pretrained MoEs. The first is the familiar tensor-level sparsity of MoE routing, in which only a subset of experts is active for each token. The second is neuron-level sparsity inside each selected expert, where some neurons contribute much more than others across inputs. The framework treats these two sparsity sources as jointly exploitable rather than independent.

This framing places DualSparse-MoE in a distinct part of the MoE design space. It is neither a weight-pruning method that permanently deletes FFN structure nor a new sparse training recipe. Its central claim is that finer-grained sparsity can be induced after pretraining, without retraining the base MoE from scratch, if the transformation preserves the original function up to a deterministic correction and if the runtime can exploit that transformed structure efficiently.

2. Post-training expert partitioning and mathematically consistent transformation

The first stage of DualSparse-MoE is expert partitioning. Given an MoE with $$\mathbf{s} = G(\mathbf{x}) = \text{Softmax}(\mathbf{x}\mathbf{W}_g),$$1 experts, each expert can be partitioned into $$\mathbf{s} = G(\mathbf{x}) = \text{Softmax}(\mathbf{x}\mathbf{W}_g),$$2 finer experts. The method defines two variants.

In the complete transformation, the original gate matrix

$$\mathbf{s} = G(\mathbf{x}) = \text{Softmax}(\mathbf{x}\mathbf{W}_g),$$3

is expanded by repeating each gate vector $$\mathbf{s} = G(\mathbf{x}) = \text{Softmax}(\mathbf{x}\mathbf{W}_g),$$4 times: $$\mathbf{s} = G(\mathbf{x}) = \text{Softmax}(\mathbf{x}\mathbf{W}_g),$$5 with

$$\mathbf{s} = G(\mathbf{x}) = \text{Softmax}(\mathbf{x}\mathbf{W}_g),$$6

If the original logits are $$\mathbf{s} = G(\mathbf{x}) = \text{Softmax}(\mathbf{x}\mathbf{W}_g),$$7, then each partition of expert $$\mathbf{s} = G(\mathbf{x}) = \text{Softmax}(\mathbf{x}\mathbf{W}_g),$$8 has the same logit, and its softmax score becomes

$$\mathbf{s} = G(\mathbf{x}) = \text{Softmax}(\mathbf{x}\mathbf{W}_g),$$9

Each original expert is split evenly across neurons into $\mathbf{W}_g \in \mathbb{R}^{d_{model}\times E}$0 partitioned experts satisfying

$\mathbf{W}_g \in \mathbb{R}^{d_{model}\times E}$1

The transformed output is then

$\mathbf{W}_g \in \mathbb{R}^{d_{model}\times E}$2

To preserve equivalence, the method scales the down-projection $\mathbf{W}_g \in \mathbb{R}^{d_{model}\times E}$3 by $\mathbf{W}_g \in \mathbb{R}^{d_{model}\times E}$4. Operationally, the complete transformation repeats gate weights $\mathbf{W}_g \in \mathbb{R}^{d_{model}\times E}$5 times, changes Top-$\mathbf{W}_g \in \mathbb{R}^{d_{model}\times E}$6 to Top-$\mathbf{W}_g \in \mathbb{R}^{d_{model}\times E}$7, evenly splits expert neurons into $\mathbf{W}_g \in \mathbb{R}^{d_{model}\times E}$8 partitioned experts, and multiplies each partitioned expert’s down-projection by $\mathbf{W}_g \in \mathbb{R}^{d_{model}\times E}$9 (Cai et al., 25 Aug 2025).

In the partial transformation, the original gate network is left unchanged. If selected expert indices are

$K$0

they are remapped to partitioned indices

$K$1

Because the original gate score multiplies the sum of the partitioned outputs, the output remains exactly

$K$2

This version avoids enlarging the gate and is used mainly for inference and system optimization. The complete transformation is instead the basis for a finer-grained mathematically equivalent MoE that can be fine-tuned further.

The importance of these transformations is not merely algebraic. They allow expert granularity to be changed after pretraining, which in turn creates room for runtime dropping policies that operate at a more useful tensor granularity than whole-expert skipping while remaining much coarser than unstructured sparsity.

3. Tensor-level dropping and neuron-level reconstruction

DualSparse-MoE’s core inference mechanism combines a dynamic sparsity operator with a static reconstruction operator. The dynamic component acts at tensor level: low-importance token-expert computations are skipped at runtime. The static component acts at neuron level: each expert is reconstructed into a major and a minor sub-expert so that partial execution remains accurate.

The simplest dropping rule is 1T-Drop. For each token at each MoE layer, the method takes the Top-$K$3 activated experts, normalizes their gate scores, and drops any token-expert pair whose normalized score is below a threshold $K$4. This produces real compute reduction, but the paper reports that accuracy degrades as the threshold increases.

To improve the accuracy–efficiency trade-off, the method introduces 2T-Drop. Before inference, each expert is profiled on calibration data and split into two fixed parts: a major sub-expert containing high-importance neurons and a minor sub-expert containing low-importance neurons. At runtime, two thresholds are applied: $K$5 The decision rule is ternary. If the score is above $K$6, both major and minor parts are computed. If it is below $K$7, the expert is dropped entirely. If it lies between the two thresholds, only the major sub-expert is executed. This introduces an intermediate state between “full compute” and “drop all,” which is the paper’s central mechanism for graceful degradation (Cai et al., 25 Aug 2025).

Neuron importance is measured offline. For SwiGLU experts, the paper evaluates four profiling metrics:

1. accumulated gate value,

$K$8

2. accumulated absolute gate value,

$K$9

3. accumulated gate-up value,

$$g_e(\mathbf{x}) = \begin{cases} \mathbf{s}_i & \text{if } i \in \text{TopK}(\mathbf{s}, K),\ 0 & \text{otherwise}, \end{cases}$$0

4. accumulated absolute gate-up value,

$$g_e(\mathbf{x}) = \begin{cases} \mathbf{s}_i & \text{if } i \in \text{TopK}(\mathbf{s}, K),\ 0 & \text{otherwise}, \end{cases}$$1

The empirical conclusion is that absolute-value metrics are stronger, plausibly because positive and negative contributions do not cancel. The best metric is model-dependent: Mixtral and OLMoE favor accumulated absolute gate value, whereas DeepSeek favors accumulated absolute gate-up value.

The method’s notion of “reconstruction” is therefore structural rather than optimization-based. There is no explicit reconstruction loss. Instead, the original expert is reorganized into two sub-experts using neuron importance statistics from calibration samples. The reported ablations indicate that this reconstruction step, rather than partitioning alone, is the