DS-MoE: Depth-Specialized Mixture Experts

• DS-MoE is a hybrid model that uses dense training (all experts active) with sparse inference (selective activation) to enhance computational efficiency.
• It encompasses two formulations: one focuses on parameter efficiency via dense-to-sparse computation, and the other on dynamic depth-specialized reasoning.
• Empirical results show DS-MoE-6B reduces active parameters to 30–40% and achieves up to 1.86× faster throughput with improved reasoning accuracy.

Searching arXiv for the provided DS-MoE papers and closely related MoE work. to=arxiv_search iluaq്യjson {"^^^^^^^^2query2 OR \2^^^^^^^^"Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts LLMs\"","max_results":5} Searching arXiv by identifier (&&&^^^^2query2^^^^&&&). Depth Specialised Mixture of Experts (DS-MoE) is an overloaded acronym in recent arXiv literature. In "Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts LLMs" (&&&^^^^2query2^^^^&&&), DS-MoE denotes a hybrid MoE framework that performs dense computation across all experts during training and sparse computation during inference. In "Dynamic Reasoning Chains through Depth-Specialized Mixture-of-Experts in Transformer Architectures" (Roy et al., 24 Sep 2025), DS-MoE denotes a modular Transformer framework in which expert modules are optimized for distinct reasoning depths and are assembled into dynamic reasoning chains. The shared abbreviation reflects a common concern with adaptive computation, but the two formulations differ substantially in architectural locus, optimization objective, and efficiency mechanism.

^^^^2id:(Pan et al., 2024) OR \2^^^^. Terminological scope and conceptual distinction

In the cited literature, DS-MoE refers to two related but non-identical families of mechanisms. The first retains the standard MoE substitution of the feed-forward sub-layer with an MoE block and changes the training-inference regime: all experts are active during pre-training, but only a sparse subset is activated at serving time. The second shifts the emphasis from width-based expert selection within layers to depth specialised computation, so that inputs traverse only a task-appropriate chain of expert modules rather than a uniform stack of layers (&&&^^^^2query2^^^^&&&).

Formulation	Core mechanism	Reported effect
DS-MoE in (&&&^^^^2query2^^^^&&&)	Dense training, sparse inference	3^^^^2query2^^^^–4^^^^2query2 active parameters at inference; up to PRESERVED_PLACEHOLDER_^^^^2query2^^^^ faster than Mistral-7B
DS-MoE in (Roy et al., 24 Sep 2025)	Dynamic reasoning chains through depth-specialized experts	Up to ^^^^2id:(Pan et al., 2024) OR \2^^^^6 per cent computational savings; 35 per cent faster inference; 2.8 per cent higher accuracy on complex multi-step reasoning benchmarks

A central misconception is to treat DS-MoE as a single canonical architecture. The available arXiv usage instead indicates two design directions: one addresses the parameter-efficiency gap between sparse-trained MoEs and dense LLMs, while the other addresses the inefficiency of uniform-depth Transformers by routing inputs through a subset of depth-specialized modules. This suggests that DS-MoE is better understood as a label for adaptive expert computation than as a uniquely fixed model family.

2. Dense training and sparse inference in MoE LLMs

In the formulation of (&&&^^^^2query2^^^^&&&), a standard Transformer layer replaces the feed-forward sub-layer with an MoE block comprising PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2^^^^ experts and a router $h(\cdot)$. During pre-training, every token’s forward pass is fed through all $M$ experts; their outputs and gating scores are stored so that back-propagated gradients flow into every expert and into the router for every token, with no sparse gradient masking. During inference, the router computes gating scores for each token, and only the top-$K$ experts, or those above a threshold $\epsilon$, are selected; all other experts are skipped, yielding a $K$-sparse computation per token (&&&^^^^2query2^^^^&&&).

The layer is defined for input $x \in \mathbb{R}^d$ by

$g(x)=\mathrm{softmax}(W_g x) \in \mathbb{R}^M,$

with $W_g \in \mathbb{R}^{M \times d}$, and expert transforms

PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2query2^^^^

where each PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2id:(Pan et al., 2024) OR \2^^^^ is a two-layer MLP or other expert network. The dense-training layer output is

PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \22^^^^

The essential architectural claim is that dense computation across experts during training provides dense gradient propagation, while sparse inference recovers the conventional inference-time benefits of MoE routing.

This formulation is motivated by a specific empirical tension in MoE LLMs. MoE models can reduce computational costs by PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2^^^^3-PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2^^^^4 compared to dense models without sacrificing performance, but they generally require PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2^^^^5-PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2^^^^6 times more parameters to achieve comparable performance to a dense model. The larger parameter count increases GPU memory requirements and makes standard sparse-trained MoEs less efficient in I/O-bounded scenarios such as autoregressive generation. The DS-MoE proposal seeks to combine compute savings with a parameter footprint that is on par with dense models.

3. Objective design, router regularization, and inference sparsification

The training objective in (&&&^^^^2query2^^^^&&&) consists of the standard autoregressive language-modeling cross-entropy

PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2^^^^7

augmented with a Mutual-Information loss intended to prevent router collapse. The regularizer encourages both even marginal usage of experts and peaky per-token distributions. For a minibatch PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2^^^^8, the marginal expert probability is

PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2^^^^9

and the per-token conditional entropy is

$h(\cdot)$^^^^2query2^^^^

The MI objective is

$h(\cdot)$^^^^2id:(Pan et al., 2024) OR \2^^^^

where

$h(\cdot)$2

The full loss is

$h(\cdot)$3

with $h(\cdot)$4 a tunable weight; the paper summary gives examples $h(\cdot)$5 or $h(\cdot)$6 as in Table 4 (&&&^^^^2query2^^^^&&&).

At inference time, sparsification is implemented through either top-$h(\cdot)$7 gating or threshold-based selection. For each token, the router computes raw scores $h(\cdot)$8 and $h(\cdot)$9, then identifies either $M$^^^^2query2^^^^ or $M$^^^^2id:(Pan et al., 2024) OR \2^^^^. Only experts in $M$2 are evaluated, and the output becomes

$M$3

If $M$4, the number of active parameters and FLOPs is reduced by factor $M$5 compared to dense computation; the summary reports $M$6–$M$7. Complexity is correspondingly framed as $M$8 FLOPs per token for dense inference and $M$9 for sparse MoE inference, with $K$^^^^2query2^^^^ denoting the cost of one expert. The distinctive claim for DS-MoE is that it retains the same $K$^^^^2id:(Pan et al., 2024) OR \2^^^^ sparse-inference cost while keeping total parameters approximately equal to a dense model of the same size, thereby improving both compute-bound and I/O-bound efficiency.

4. Parameter efficiency, throughput, and ablation behavior

The parameter-efficiency argument of (&&&^^^^2query2^^^^&&&) is expressed through representative model statistics. Dense-6B is reported as having $K$2 total parameters, $K$3 active parameters, $K$4 hidden-active, and $K$5 GPU memory. SMoE-5B is reported as having $K$6 total parameters, $K$7 active parameters, $K$8 hidden-active, and $K$9 GPU memory. DS-MoE-6B is reported as having $\epsilon$^^^^2query2^^^^ total parameters, $\epsilon$^^^^2id:(Pan et al., 2024) OR \2^^^^ active parameters, $\epsilon$2 hidden-active, and $\epsilon$3 GPU memory. The resulting interpretation is explicit: DS-MoE-6B has approximately $\epsilon$4 total parameters, on par with Dense-6B, while activating only approximately $\epsilon$5 parameters, or approximately $\epsilon$6, achieving 3^^^^2query2^^^^–4^^^^2query2 compute while maintaining dense-model parameter footprint (&&&^^^^2query2^^^^&&&).

Autoregressive generation results using vLLM provide the principal serving-time evidence. DS-MoE-6B versus Mistral-7B is reported as $\epsilon$7 faster throughput on A^^^^2id:(Pan et al., 2024) OR \2query2query2^^^^-8^^^^2query2^^^^GB, with $\epsilon$8 versus $\epsilon$9 req/s and $K$^^^^2query2^^^^ versus $K$^^^^2id:(Pan et al., 2024) OR \2^^^^ TPS, and $K$2 faster on H^^^^2id:(Pan et al., 2024) OR \2query2query2^^^^-8^^^^2query2^^^^GB, with $K$3 versus $K$4 req/s and $K$5 versus $K$6 TPS. Against DeepSeekMoE-^^^^2id:(Pan et al., 2024) OR \2^^^^6B, DS-MoE-6B is reported as $K$7 faster on A^^^^2id:(Pan et al., 2024) OR \2query2query2^^^^, with $K$8 versus $K$9 req/s, and $x \in \mathbb{R}^d$^^^^2query2^^^^ faster on H^^^^2id:(Pan et al., 2024) OR \2query2query2^^^^, with $x \in \mathbb{R}^d$^^^^2id:(Pan et al., 2024) OR \2^^^^ versus $x \in \mathbb{R}^d$2 req/s. Against Qwen^^^^2id:(Pan et al., 2024) OR \2^^^^.5-MoE-A2.7B, it is reported as $x \in \mathbb{R}^d$3 faster on A^^^^2id:(Pan et al., 2024) OR \2query2query2^^^^, with $x \in \mathbb{R}^d$4 versus $x \in \mathbb{R}^d$5 req/s, and $x \in \mathbb{R}^d$6 faster on H^^^^2id:(Pan et al., 2024) OR \2query2query2^^^^, with $x \in \mathbb{R}^d$7 versus $x \in \mathbb{R}^d$8 req/s.

The ablation summary clarifies the role of routing and sparsity control. Increasing the MI-loss weight $x \in \mathbb{R}^d$9 in the MLP router from $g(x)=\mathrm{softmax}(W_g x) \in \mathbb{R}^M,$^^^^2query2^^^^ to $g(x)=\mathrm{softmax}(W_g x) \in \mathbb{R}^M,$^^^^2id:(Pan et al., 2024) OR \2^^^^ yields models that tolerate higher sparsity, that is, fewer active experts, at minimal accuracy loss. Among expert sampling strategies, threshold only $g(x)=\mathrm{softmax}(W_g x) \in \mathbb{R}^M,$2 gives the best compute-accuracy trade-off but complicates batching, whereas top-$g(x)=\mathrm{softmax}(W_g x) \in \mathbb{R}^M,$3 and Threshold-TopK are more deployment-friendly with comparable PPL versus active-param curves. Layer-wise utilization further indicates that self-attention MoA blocks tend to use more than $g(x)=\mathrm{softmax}(W_g x) \in \mathbb{R}^M,$4 of experts, while MLP layers achieve more aggressive sparsity. The reported variability across tokens suggests that a single fixed $g(x)=\mathrm{softmax}(W_g x) \in \mathbb{R}^M,$5 per layer, as in conventional sparse-trained MoEs, under-utilizes the potential of adaptive sparsity.

5. Dynamic reasoning chains and depth-specialized experts

In (Roy et al., 24 Sep 2025), DS-MoE is reformulated around the inefficiency of uniform-depth Transformers. In a standard Transformer with depth $g(x)=\mathrm{softmax}(W_g x) \in \mathbb{R}^M,$6 and sequence length $g(x)=\mathrm{softmax}(W_g x) \in \mathbb{R}^M,$7, every input costs $g(x)=\mathrm{softmax}(W_g x) \in \mathbb{R}^M,$8 in time and memory. The motivating claim is that simple factual questions need only shallow pattern matching, yet in a 24-layer model they are processed exactly as if they were complex legal or scientific inferences. The proposed remedy is to assign expert modules to distinct depth bands or reasoning styles and to let a learned routing network assemble a task-specific chain of computation (Roy et al., 24 Sep 2025).

The expert taxonomy in the paper comprises Shallow Pattern Experts (SPE), for keyword lookup and surface pattern recognition; Compositional Reasoning Experts (CRE), for two- or three-step chains and simple deduction; Logical Inference Experts (LIE), for abstract, symbolic, theorem-style reasoning; Memory Integration Experts (MIE), for long-context tracking and cross-document reference; and Meta-Cognitive Experts (MCE), as supervisory units that monitor and dynamically adjust the chain. Each expert $g(x)=\mathrm{softmax}(W_g x) \in \mathbb{R}^M,$9 implements

$W_g \in \mathbb{R}^{M \times d}$^^^^2query2^^^^

where $W_g \in \mathbb{R}^{M \times d}$^^^^2id:(Pan et al., 2024) OR \2^^^^ are the parameters specialized for reasoning depth $W_g \in \mathbb{R}^{M \times d}$2.

Routing is preceded by a scalar complexity estimate for input sequence $W_g \in \mathbb{R}^{M \times d}$3:

$W_g \in \mathbb{R}^{M \times d}$4

where $W_g \in \mathbb{R}^{M \times d}$5 is parse-tree depth, $W_g \in \mathbb{R}^{M \times d}$6 is unique concepts per sentence, and $W_g \in \mathbb{R}^{M \times d}$7 is estimated minimal reasoning steps via clause-chaining. With feature vector $W_g \in \mathbb{R}^{M \times d}$8, the routing distribution is

$W_g \in \mathbb{R}^{M \times d}$9

and the selected expert set is

PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2query2query2^^^^

Once experts PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2query2id:(Pan et al., 2024) OR \2^^^^ are chosen and ordered by descending PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2query22^^^^, the dynamic reasoning chain is

PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2query2^^^^3

with PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2query2^^^^4 passed to an output head.

The paper explicitly contrasts this design with width-based MoE. In width-based MoE, many experts are placed in each layer but depth remains fixed: all inputs still traverse all PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2query2^^^^5 layers, though each layer may use only a subset of its width. DS-MoE instead places experts at different logical depths and gates entire modules, so that only PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2query2^^^^6 experts are activated. The reported average cost is reduced to roughly PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2query2^^^^7 while preserving or improving reasoning quality. This claim should be read as specific to the framework of (Roy et al., 24 Sep 2025), not as a property of DS-MoE in the dense-training/sparse-inference sense of (&&&^^^^2query2^^^^&&&).

6. Training protocol, evaluation, interpretability, and open issues

The training pipeline in (Roy et al., 24 Sep 2025) begins with pretraining individual experts on data reflecting their specialization: SPE on factual QA corpora, CRE on logic puzzles and arithmetic chains, LIE on legal texts and mathematical proofs, MIE on long narrative corpora from the Books subset of The Pile, and MCE on synthetic curriculum tasks requiring dynamic depth control. Joint fine-tuning then introduces the routing network and optimizes

PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2query2^^^^8

where PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2query2^^^^9 is cross-entropy on the final prediction, PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2id:(Pan et al., 2024) OR \2query2^^^^ is cross-entropy between the router’s PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2id:(Pan et al., 2024) OR \2id:(Pan et al., 2024) OR \2^^^^ and annotated oracle expert hints derived during complexity-aware preprocessing, and PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2id:(Pan et al., 2024) OR \22^^^^ is an entropy or variance penalty on batch-aggregated PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2id:(Pan et al., 2024) OR \2^^^^3 to avoid collapse. The dataset is The Pile, described as an 8^^^^2query2query2^^^^ GB, approximately 4^^^^2query2query2^^^^ B-token corpus covering 22 high-quality sources. The preprocessing pipeline uses parse-tree analysis, concept extraction, and clause-chain detection; a small portion is manually annotated to validate complexity estimates; the sampling balance is approximately 4^^^^2query2^^^^% shallow, 35% compositional, 2^^^^2query2^^^^% inference, and 5% meta-cognitive; and curriculum training ramps from simple examples to higher complexity. Infrastructure details include PyTorch 2.^^^^2id:(Pan et al., 2024) OR \2^^^^, distributed on NVIDIA A^^^^2id:(Pan et al., 2024) OR \2query2query2^^^^ (8^^^^2query2^^^^ GB) clusters, top-PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2id:(Pan et al., 2024) OR \2^^^^4 experts with PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2id:(Pan et al., 2024) OR \2^^^^5, total expert count PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2id:(Pan et al., 2024) OR \2^^^^6 with three in each category, a two-layer MLP routing network with softmax output, and separate optimizers for experts versus routing (Roy et al., 24 Sep 2025).

Evaluation compares a Uniform-Depth Transformer with PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2id:(Pan et al., 2024) OR \2^^^^7 layers, a width-based MoE baseline, and DS-MoE on test splits drawn from five Pile subsets: Wikipedia, GitHub, PubMed, Legal, and Books. On complex multi-step benchmarks formed from Legal and Books, DS-MoE improved accuracy by approximately 2.8 points over the Uniform-Depth Transformer, while on average matching it on shallow tasks. The summary reports up to ^^^^2id:(Pan et al., 2024) OR \2^^^^6% total FLOPs saving versus the Uniform-Depth Transformer, and up to 7^^^^2query2^^^^–8^^^^2query2 in per-case active depth; approximately 35% faster latency on average; and 35–4^^^^2query2^^^^% lower GPU usage. A specific table description states that on Wikipedia DS-MoE achieves 92.5% versus 92.^^^^2id:(Pan et al., 2024) OR \2^^^^% with only PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2id:(Pan et al., 2024) OR \2^^^^8 FLOPs, and on Legal it scores 83.7% versus 78.4% with PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \2id:(Pan et al., 2024) OR \2^^^^9 FLOPs.

The ablation evidence is correspondingly modular. Removing the router and using a fixed chain lowers accuracy by 4% and restores FLOPs to PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \22query2^^^^. Removing meta-cognitive control lowers average accuracy by 2.5%. Removing the memory expert lowers Books accuracy by 7%. SPE+CRE only is strong on shallow tasks but fails on deep tasks, whereas LIE only is good on Legal but wasteful on simple tasks. Routing interpretability is presented as a major property of the system: logging PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \22id:(Pan et al., 2024) OR \2^^^^ at test time makes it possible to display exact chains such as SPEPRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \222^^^^CREPRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \22^^^^3LIEPRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \22^^^^4MCE, and human evaluators rated these chains 4.6/5 on transparency versus approximately 2.2 for baselines.

The limitations are stated directly. Routing overhead adds an extra PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \22^^^^5 cost that must remain small in low-latency settings. New domains may require bespoke experts and pretraining. The paper proposes replacing supervised PRESERVED_PLACEHOLDER_^^^^2id:(Pan et al., 2024) OR \22^^^^6 with a reinforcement learning signal that optimizes end-to-end performance and cost, and it proposes a multimodal extension based on composition of vision, audio, and language experts in a similar depth-specialized fashion. Taken together with (&&&^^^^2query2^^^^&&&), these limitations reinforce a broader interpretive point: DS-MoE currently names two different strategies for adaptive expert computation—one centered on dense gradient propagation with sparse serving, the other on dynamic depth allocation. A plausible implication is that future work may combine both ideas, although that synthesis is not itself established in the cited sources.