Flexible Upcycling of Dense Experts

• Flexible upcycling of dense experts is a set of techniques that transforms fully-activated pretrained neural networks into sparse, modular MoE architectures.
• Key methodologies include dense-to-MoE transformation, expert specialization, parameter sharing, and adaptive routing to balance compute, memory, and latency trade-offs.
• Empirical findings indicate that upcycled MoEs can match or outperform dense models in generalization while reducing compute costs by around 50% and supporting diverse applications.

Flexible upcycling of dense experts is a collective term for a set of methodologies in neural network architecture design, particularly for transformers and Mixture-of-Experts (MoE) models, that enable efficient conversion, extension, and composition of dense (fully-activated) models into parameter- and compute-efficient sparse MoE architectures. These techniques exploit pretrained knowledge, promote expert diversity, and support specialization across domains, modalities, or tasks, all while maintaining or improving on the original baseline's computational, memory, and latency constraints. The upcycling paradigm broadly subsumes approaches that initialize MoE models using dense checkpoints, share or specialize expert parameters, and use various expert selection and merging strategies to maximize representational capacity, generalization, and modularity.

1. Motivation and Conceptual Foundations

Large-scale dense models exhibit strong generalization but cannot dynamically focus their capacity; further, their size makes scaling cost-prohibitive. The MoE framework mitigates this by routing each input (token, patch, etc.) through a sparse subset of experts, decoupling total capacity from per-token FLOPs. Flexible upcycling leverages this by beginning with existing dense, pretrained models—either a single model or multiple specialized models—and reconfiguring/fusing them into one or more MoE architectures. The objectives are:

• Reduced sunk compute and data costs versus from-scratch MoE training (Komatsuzaki et al., 2022, He et al., 10 Oct 2024)
• Leveraging diverse domain or modality specialization (Gritsch et al., 28 Aug 2024, Wang et al., 23 Sep 2025)
• Achieving parameter efficiency and modularity (Huang et al., 3 Mar 2025)
• Preserving or improving generalization versus dense baselines (Fu et al., 23 Dec 2024, Wang et al., 3 Feb 2025, Zhang et al., 28 Sep 2024)

The term “flexible” denotes support for variable expert counts, activated experts, expert granularity, and the ability to extend or compress the architecture post hoc.

2. Core Upcycling Methodologies

Flexible upcycling strategies span a common set of architectural and algorithmic modifications:

A. Dense-to-MoE Transformation

• Replace dense FFN sublayers with sparse-activated MoE blocks by duplicating the pretrained weights into $N$ experts (Komatsuzaki et al., 2022, Fu et al., 23 Dec 2024, He et al., 10 Oct 2024).
• Each expert is initialized identically, preserving the output during the first forward pass.
• The MoE router is randomly initialized, and sparsity is enforced by keeping only the Top-$K$ experts active per input.

B. Expert Specialization, Diversity, and Parameter Sharing

• Diversity is promoted by partial parameter re-initialization after duplication (Nakamura et al., 26 Feb 2025), checkpoint merging via genetic algorithms (Hui et al., 2 Oct 2024), or the extraction/finetuning of experts on distinct domains (Wang et al., 23 Sep 2025, Gritsch et al., 28 Aug 2024).
• Parameter sharing mechanisms include reusing base weights and modeling expert "delta" via sparse or low-rank forms (Huang et al., 3 Mar 2025).
• Shared/generalist experts coexist with specialists to avoid forgetting and promote transfer (Liao et al., 24 Jul 2025, Ding et al., 23 Apr 2024, Gritsch et al., 28 Aug 2024).

C. Routing and Merging Techniques

• Routers are trained to distribute tokens adaptively, balancing specialization and load (Komatsuzaki et al., 2022, Fu et al., 23 Dec 2024, Nakamura et al., 26 Feb 2025, He et al., 10 Oct 2024).
• Advanced approaches include projection-based routers informed by domain embeddings (Gritsch et al., 28 Aug 2024) and online/functional alignment for experts from disparate models (Wang et al., 23 Sep 2025).
• For maximizing downstream efficiency, methods such as dynamic merging after fine-tuning can collapse MoE parameters back to a dense form (Ding et al., 23 Apr 2024).

D. Capacity, Granularity, and Extension

• Number of experts, expert width, and activation count per input are systematically varied to trade off between generalization, specialization, and compute (He et al., 10 Oct 2024, Liao et al., 24 Jul 2025).
• Modular upcycling allows extension with new experts via lightweight retraining (Gritsch et al., 28 Aug 2024).

3. Implementation and Architectural Trade-offs

Several practical axes govern the flexible upcycling landscape:

Method or Dimension	Implementation Detail	Typical Trade-off
Expert Initialization	Full copy, partial re-init (“Drop-Upcycling”), cross-checkpoint fusion	Diversity vs. knowledge retention
Router Design	Softmax-then-TopK, TopK-then-Softmax, projection, gating normalization	Routing sharpness, stability, computational
Expert Parameter Efficiency	Full MLP, sparse delta (Huang et al., 3 Mar 2025), low-rank delta	Parameter cost vs. fidelity
Expert Pool Source	Single dense model, multiple dense (disparate) models	Diversity, OOD generalization
Granularity	Coarse (full FFN), fine (sharded FFN, e.g. E×G)	Early vs. late capacity gains
Shared/Generalist Expert	Coexists with specialists; always active	Prevents collapse, preserves coverage

Key Recipes

• Freeze non-expert layers when beneficial: preserves general representations and reduces compute (Fu et al., 23 Dec 2024).
• Load-balancing auxiliary losses: critical to prevent expert under-utilization and collapse (Komatsuzaki et al., 2022).
• Delta-based de/compression: enables aggressive parameter compression with minimal accuracy loss (Huang et al., 3 Mar 2025).
• Parallelization and memory-awareness: sharding and grouping experts to manage memory and bandwidth (Vavre et al., 13 Dec 2024, Liao et al., 24 Jul 2025, He et al., 10 Oct 2024).

4. Empirical Scaling Laws and Performance Characteristics

Scaling laws for upcycled MoEs elucidate how test loss, parameter count, and token budget interact. The core relations (Liew et al., 5 Feb 2025) include:

• Loss scales with both “sunk” (dense pretrain) and “upcycled” (MoE continued-training) tokens:

$$L(D_1, D_2) = A D_1^{-\alpha_1} D_2^{-\alpha_2 + \alpha_3\log D_1} + E$$

where $D_1$ is tokens for dense pretrain, $D_2$ for MoE continued-training.

• Joint scaling in active parameter count ($N_2$) and sparsity ratio ($P$):

$$L(P, N_2) = B P^{-\beta_1} N_2^{-\beta_2 + \beta_3\log P} + E$$

Larger sparsity and more active parameters uniformly improve test loss; diminishing returns set in for large $D_1$, limiting upcycling efficiency.

• Compute-optimal rule: Upcycling provides a win when the additional MoE training budget $D_2 < D^*$, with $D^*$ scaling sublinearly with model size.
• Empirical observations: Upcycling MoEs surpass dense continuation and often match or surpass scratch-trained MoEs at ~50% less compute (Komatsuzaki et al., 2022, He et al., 10 Oct 2024, Liew et al., 5 Feb 2025, Huang et al., 3 Mar 2025, Vavre et al., 13 Dec 2024).

5. Applications in Multimodal, Domain-Specific, and Continual Learning

Flexible upcycling is applied across domains and modalities:

• Multimodal models: Upcycling dense CLIP to sparse MoE enables state-of-the-art retrieval with dramatic reductions in inference cost and resource usage (Wang et al., 3 Feb 2025, Zhang et al., 28 Sep 2024).
• Speech recognition: UME upcycles dense ASR, freezes most modules, and uses load-balancing to deliver $11-16\%$ relative error-rate reduction at $<35\%$ extra latency (Fu et al., 23 Dec 2024).
• Scientific instruction and code LLMs: Fine-grained expert splitting and domain-anchored routers allow targeted scientific knowledge acquisition and avoidance of catastrophic forgetting (Liao et al., 24 Jul 2025, Ding et al., 23 Apr 2024).
• Composable and extendable systems: Frameworks such as Nexus and Symphony-MoE generalize to dynamically append new dense experts, harmonize parameter spaces, or upcycle from disparate dense sources, supporting highly modular assembly and open-ended transfer (Gritsch et al., 28 Aug 2024, Wang et al., 23 Sep 2025).

6. Parameter and Inference Efficiency Mechanisms

Parameter (and sometimes FLOP) efficiency is achieved via several pathways:

• DeRS Paradigm: Experts are expressed as $W_i = W_{\mathrm{base}} + \mathcal{F}(\Delta_i)$ with $\mathcal{F}$ a sparse or low-rank transformation, yielding $>1000\times$ reduction in added expert parameters and $>40\%$ reduction in memory at negligible accuracy loss (Huang et al., 3 Mar 2025).
• Upcycling with fine-grained granularity: Shrinking each expert’s width (while increasing count) permits creation of iso-FLOP MoEs, supporting high capacity without inflating compute (He et al., 10 Oct 2024).
• Partial parameter re-init: Drop-Upcycling randomizes only a fraction $r$ of each expert’s parameters, balancing knowledge transfer and diversity, with the optimum at $r=0.5$ (Nakamura et al., 26 Feb 2025).
• Inference compression: Post hoc compression via DeRS-sparsification/quantization achieves $0.99$ sparsity or $2-4$ bit quantization of deltas, maintaining LLaVA/CLIP/CodeMoE accuracy (Huang et al., 3 Mar 2025).

7. Limitations, Recommendations, and Future Directions

Limitations:

• Upcycling assumes the existence of a pretrained dense model with sound inductive bias; radical domain shifts may require adaptation (Komatsuzaki et al., 2022).
• Over-upcycling (excess MoE training) exhibits diminishing returns due to the interaction term in scaling laws (Liew et al., 5 Feb 2025).
• Expert diversity and router training can become limiting if diversity mechanisms, seed checkpoints, or data clusters are not chosen carefully (Hui et al., 2 Oct 2024, Zhang et al., 28 Sep 2024).

Practitioner Guidance:

• Balance expert count, width, and routing granularity against hardware, memory, and latency constraints (He et al., 10 Oct 2024, Vavre et al., 13 Dec 2024).
• Retain a shared/generalist expert to prevent overly narrow specialization (Liao et al., 24 Jul 2025, Gritsch et al., 28 Aug 2024, Ding et al., 23 Apr 2024).
• Use load-balancing and auxiliary entropy/z-losses consistently to prevent expert collapse (Komatsuzaki et al., 2022, Fu et al., 23 Dec 2024, Gritsch et al., 28 Aug 2024).
• Modularize compressors and Delta representations to enable both flexible training and on-demand inference compression (Huang et al., 3 Mar 2025).
• For compositional and extendable settings, tools such as domain-embedding routers and activation-permutation alignment are critical for harmonizing disparate experts (Gritsch et al., 28 Aug 2024, Wang et al., 23 Sep 2025).

Future Directions:

• Adaptive expert growth and co-upcycling across modalities/languages (Fu et al., 23 Dec 2024, Huang et al., 3 Mar 2025).
• Extending delta parameterization to backbone parameters, recursive compression, or nonlinear expert merging (Huang et al., 3 Mar 2025, Hui et al., 2 Oct 2024).
• Improved theoretical understanding of expert diversity, domain transfer, and load regulation.

Flexible upcycling of dense experts has become a foundational toolbox for parameter-efficient, modular, and extensible deep learning architectures, spanning LLMs, multimodal systems, speech, and beyond. Its continued evolution is likely to further blur the lines between static dense pretraining and dynamically composable sparse inference, supporting both rigorous scaling and practical deployment.