Adaptive Shared Experts (ASE)

Updated 12 March 2026

Adaptive Shared Experts (ASE) is a paradigm that integrates globally shared and dynamically selected specialized experts to improve multi-task, multi-domain, and federated learning efficiency.
It employs innovative gating strategies such as Top-K selection, KL and mutual-information regularization, and two-stage adaptive routing to drive expert specialization and effective resource use.
ASE delivers practical benefits including faster convergence, sharper expert-task assignments, and reduced computational overhead, making it valuable in search, language modeling, and federated adaptation.

Adaptive Shared Experts (ASE) is a paradigm for enhancing the Mixture-of-Experts (MoE) architecture by incorporating dedicated mechanisms for sharing, selecting, and specializing experts within multi-task, multi-domain, and federated learning systems. ASE implementations systematically address limitations of static or naively shared architectures by enabling dynamic, instance-wise selection and adaptive utilization of both globally shared and dynamically specialized experts. This results in improved learning efficiency, better expert specialization, robust transfer, and enhanced computational scalability across a range of domains including search, recommendation, language modeling, and federated adaptation.

1. Architectural Foundations and Model Variants

ASE extends the standard MoE framework by introducing explicit support for shared experts, typically implemented alongside dynamically or sparsely routed per-task/domain experts. The architectural instantiations vary but consistently feature two or more classes of experts:

Globally Shared Experts: Always active for all inputs, designed to capture invariant, task-agnostic knowledge (Nguyen et al., 16 May 2025).
Routed/Specialized Experts: Activated on a per-input or per-task/domain basis by dynamic gating networks (Yang et al., 1 Oct 2025, Dong et al., 2024).
Hybrid Routing: A combination wherein shared and routed experts are both included in the aggregation, with normalized or sparsely regularized gating (Yang et al., 1 Oct 2025, Zou et al., 2022, Wang et al., 18 Sep 2025).

Model layers typically consist of a combination of feed-forward neural networks (FFNs) for expert implementation, often employing parameter-efficient adaptation (LoRA) (Yang et al., 1 Oct 2025, Li et al., 2024, Wang et al., 18 Sep 2025). Gating mechanisms select between experts, leveraging softmax or (in recent works) normalized sigmoid functions for improved sample efficiency and compositionality (Nguyen et al., 16 May 2025). Some architectures introduce a two-level routing process, such as in AT-MoE, with both inter-group and intra-group gating to improve both control and interpretability (Li et al., 2024).

Notable variant-specific architectural features include:

ASE Variant	Shared Expert Gating	Specificity Enforcement	Notes
AESM² (Zou et al., 2022)	KL-regularized, sparsified softmax	Layerwise KL loss (targeting one-hot or uniform)	Hierarchical MSL/MTL
CESAA (Dong et al., 2024)	Always-on MLP	Mutual Information loss	MDR, sparse Top-K
AT-MoE (Li et al., 2024)	Layerwise LoRA fusion	Grouped adaptive routing	LLMs, two-stage gating
DeepSeekMoE (Nguyen et al., 16 May 2025)	Normalized sigmoid	Theoretical convergence	Language/Vision LM
LoRA-MoE ASE (Yang et al., 1 Oct 2025)	Jointly normalized	Task-specific router	Multitask ViT/STL→MTL
FedLEASE ASE (Wang et al., 18 Sep 2025)	Client-specific Top-M	Client clustering	Federated PEFT
Expert-merging ASE (Park, 2024)	Usage-frequency tracking	Periodic merging, no regularizer	Task-incremental

2. Expert Selection, Routing, and Gating Mechanisms

ASE frameworks universally deploy advanced gating strategies to achieve instance-adaptive expert combination. Selection is typically subject to sparsity constraints and often governed by both task/domain context and data-derived statistics:

Sparsity and Joint Normalization: Experts are partitioned into shared and sparse sets; the latter are selected by Top-K gating (often with randomized or noisy logits for gate exploration) and combined with shared experts via joint softmax or similar normalization (Yang et al., 1 Oct 2025, Dong et al., 2024).
KL/Mutual Information Regularization: Scenario/task-specific and shared experts are promoted by directly aligning gating distributions with one-hot (for specificity) or uniform (for sharing) targets, measured via KL-divergence (Zou et al., 2022) or by maximizing expert-domain mutual information (Dong et al., 2024).
Two-stage/Grouped Routing: AT-MoE utilizes a temperature-controlled, group-level softmax followed by within-group normalization, supporting multidimensional control and tractable expertise partitioning (Li et al., 2024).
Adaptive Expert Allocation: FedLEASE adaptively determines, for each federated client, the number and identity of experts to mix (always including the client’s “home” expert), solving a locally optimal routing problem for heterogeneous, distributed data (Wang et al., 18 Sep 2025).

A general formulation for joint normalization of shared and sparse experts given logits $z^s_j$ (shared) and $z_i$ (sparse) is:

$g^s_j = \frac{\exp(z^s_j)}{\sum_{i \in T} \exp(z_i) + \sum_\ell \exp(z^s_\ell)} \ g_i = \frac{\exp(z_i)}{\sum_{i \in T} \exp(z_i) + \sum_\ell \exp(z^s_\ell)}$

where $T$ denotes indices of selected sparse experts.

3. Training Objectives, Regularizers, and Optimization

ASE systems employ composite objectives that go beyond task losses, adding terms to govern expert specialization and sharing:

Task Losses: Binary cross-entropy or domain-appropriate task losses are consistently used for each target (Zou et al., 2022, Dong et al., 2024).
Auxiliary/Regularization Losses:
- KL-based losses enforce the expert selection gates to match desired distributions for specificity (one-hot) or sharing (uniform) (Zou et al., 2022).
- Mutual information maximization drives experts and domains to become maximally correlated, ensuring clear domain-expert specialization (Dong et al., 2024).
- Load-balancing penalties (entropic or variance-based) prevent expert collapse and encourage uniform utilization (Nguyen et al., 16 May 2025).

Optimization proceeds via standard gradient methods (Adam/AdamW), with noise in logit calculations and paired router/expert parameter updates (Zou et al., 2022, Dong et al., 2024).

4. Statistical, Empirical, and Computational Benefits

ASE innovations offer both theoretical and empirical improvements:

Sample Efficiency and Convergence: The addition of shared experts and normalized sigmoid gating provably accelerates convergence rates for both shared and routed experts in strongly identifiable FFN settings, guaranteeing parametric rates ( $\sim n^{-1/2}$ ) and mitigating issues in linear regimes (Nguyen et al., 16 May 2025).
Expert Specialization and Transfer: Across multiple scenarios (MDR, STL→MTL, federated adaptation), ASE yields sharper expert-task/domain assignments, improved transfer accuracy, and reduced negative transfer (Yang et al., 1 Oct 2025, Wang et al., 18 Sep 2025, Dong et al., 2024).
Resource Efficiency: Sparse activation (Top-K + shared) reduces FLOPs substantially without compromising accuracy. For example, CESAA achieves a 50% reduction in inference/training FLOPs for $N=8$ , $K=3$ (Dong et al., 2024).
Empirical Performance: ASE consistently outperforms static or naively shared MoE baselines. On PASCAL-Context, joint-normalized ASE achieved the highest $\Delta_m$ performance metric compared to both vanilla LoRA-MoE and classical multitask models, with minimal parameter overhead (Yang et al., 1 Oct 2025). In federated experiments, FedLEASE ASE surpassed prior strong baselines by 1–3 accuracy points (Wang et al., 18 Sep 2025).
Interpretability and Control: Novel routing schemes (grouped, joint, mutual-information-regularized) result in more interpretable and controllable expert assignments, confirmed by human evaluation and router analysis metrics such as router saturation, change rate, and fairness (Li et al., 2024, Nguyen et al., 16 May 2025).
Specialization Dynamics: Training curves reveal an early bias toward shared experts, shifting over time as sparse experts become task/domain-specialized (Yang et al., 1 Oct 2025).

5. Application Domains and Notable Implementations

ASE has been validated in diverse, large-scale, real-world settings:

Search and Recommendation: Deployed in major production systems (e.g., AliPay/AliExpress), ASE achieves substantial online gains (e.g., $+0.10\%$ CTR, $+2.61\%$ CVR, $+7.21\%$ GMV over production rankers) (Zou et al., 2022). Multidomain recommender systems utilize ASE to overcome scalability and negative transfer (Dong et al., 2024).
Language and Vision Modeling: DeepSeekMoE leverages ASE for superior sample efficiency and downstream accuracy in both LM and multimodal VQA tasks (Nguyen et al., 16 May 2025).
Multitask Learning and STL→MTL Transfer: LoRA-based ASE modules accelerate specialization without redundant adaptation, especially when fine-grained, low-rank expert partitioning is optimized (Yang et al., 1 Oct 2025).
Federated Fine-tuning: FedLEASE demonstrates ASE-driven allocations are robust to data and system heterogeneity, with adaptive routing optimizing cross-client collaboration (Wang et al., 18 Sep 2025).
Task-incremental Learning: Periodic merging and replacement of overused experts with their average supports transfer and mitigates catastrophic forgetting, though gains are modest and context-sensitive (Park, 2024).

6. Limitations, Open Problems, and Guidelines

While ASE introduces substantial practical and statistical benefits, several constraints and open questions remain:

Heuristic Expert Merging: Usage-frequency-based merging may not capture semantic similarity, leading in some cases to suboptimal knowledge blending or instability (Park, 2024).
Parameter and Routing Budgeting: Optimal allocations of shared vs. sparse experts, as well as fine-grained vs. coarse experts under fixed LoRA parameter budgets, require workload-dependent empirical tuning (Yang et al., 1 Oct 2025).
Router Regularization Sensitivity: The balance between task performance and expert specialization (e.g., via $\lambda$ or $\alpha$ regularizers) is critical; improper settings can lead to expert collapse or overfragmentation (Nguyen et al., 16 May 2025, Dong et al., 2024).
Interpretability in Complex Routing: While AT-MoE-style grouped routings offer post-hoc analysis, more granular real-time interpretability in deep ASE stacks remains a research frontier (Li et al., 2024).
Empirical Gains in Incremental Learning: Some settings show only marginal accuracy improvements post-merging, with effects sensitive to expert count and merge cycle length (Park, 2024).
Generalization Across MoE Classes: Transferring ASE techniques (e.g., mutual-information regularization, topology-aware clustering) across drastically different expert architectures is an open research avenue.

Empirical and theoretical results unequivocally show that ASE components (shared experts with adaptive routing and appropriate regularization) deliver robust, scalable, and efficient knowledge sharing in modern MoE systems (Zou et al., 2022, Dong et al., 2024, Nguyen et al., 16 May 2025, Yang et al., 1 Oct 2025, Wang et al., 18 Sep 2025). Their deployment, however, entails careful design with respect to sparsity, regularization, and task/domain partitioning.