Symphony-MoE: Sparse MoE Upcycling Framework

• The paper demonstrates that functional alignment of diverse pretrained experts significantly enhances multi-domain performance in sparse MoE models.
• Symphony-MoE is a framework that merges dense models with identical architectures using training-free harmonization and lightweight router training.
• Experimental metrics indicate that the method outperforms traditional upcycling, achieving up to 44.07 average in-distribution score and superior out-of-distribution results.

Symphony-MoE is a framework for constructing a sparse Mixture-of-Experts (MoE) LLM from multiple pretrained dense models that share the same architecture but differ in training history and specialization. It addresses the limitation of conventional MoE upcycling, which typically replicates feed-forward network (FFN) layers from a single dense checkpoint and therefore restricts expert diversity. Its central claim is that experts sourced from disparate pretrained models can be combined into a single coherent MoE only if their shared backbone is fused in a layer-aware manner and their FFN experts are functionally aligned before routing is learned (Wang et al., 23 Sep 2025).

1. Problem formulation and motivation

Symphony-MoE is situated in sparse MoE language modeling. In the formulation given in the paper, the MoE output for token input $x$ is

$M_{MoE}(x) = \sum_{i=1}^{N} g_i(x) \cdot E_i(x)$

where $N$ is the number of experts, $E_i(\cdot)$ is expert $i$, and $g_i(x)$ is the router’s gating weight for expert $i$ on token $x$. The routing weights are produced by a linear router followed by softmax,

$g(x) = Softmax(xW_g)$

and, in practice, only the top-$k$ experts are activated. This is the standard conditional-computation setting in which total parameter count can grow while per-token FLOPs remain relatively low because only a small subset of experts is used (Wang et al., 23 Sep 2025).

The specific problem addressed by Symphony-MoE is multi-source upcycling. Prior upcycling methods generally begin from one dense pretrained model and convert its FFNs into experts, often by replication. Symphony-MoE argues that such single-origin construction intrinsically limits expert diversity because all experts begin in the same parameter basin and inherit the same pretrained knowledge source. Its stated alternative is to build an MoE from multiple identically architected but disparate pretrained models, including heterogeneous sources such as general chat, code, math, and science specialists (Wang et al., 23 Sep 2025).

The paper identifies parameter-space and representational misalignment as the main obstacle. Independently pretrained models may occupy dissonant regions of parameter space; corresponding neurons need not match; FFN hidden units may encode different features in different orders; and non-FFN modules such as attention, embeddings, and LayerNorm cannot be safely combined with a single naive merge rule. According to the paper, this leads to interference between experts, incoherent shared representations, loss of specialization, and difficulty for the router to learn meaningful dispatching. Symphony-MoE therefore frames harmonization, rather than routing alone, as the primary technical challenge (Wang et al., 23 Sep 2025).

2. Two-stage architecture and model assembly

Symphony-MoE is explicitly a two-stage framework. Stage 1 is a training-free harmonization step. Stage 2 is lightweight router training. The design principle is to make the source models functionally compatible before any gradient-based coordination is attempted (Wang et al., 23 Sep 2025).

The final model replaces the dense FFN in each Transformer block with a set of experts, one per source model per layer. The shared backbone consists of token embeddings, self-attention modules, and LayerNorm parameters. The experts are the FFN or MLP blocks. A trainable linear router is added to each MoE layer to select experts. This design preserves source-specific FFN specialization while consolidating non-FFN computation into a common backbone (Wang et al., 23 Sep 2025).

Component	Role in final model	Construction strategy
Token embeddings	Shared backbone	Selective linear merge
Self-attention modules	Shared backbone	SLERP on $M_{MoE}(x) = \sum_{i=1}^{N} g_i(x) \cdot E_i(x)$0
LayerNorm parameters	Shared backbone	Simple averaging
FFN / MLP blocks	Experts	Activation-based alignment, then preserved as experts

The framework assumes that source models are identically architected. In the paper, this implies compatibility in layer count, hidden size, FFN dimension, attention dimensions, and parameter shapes. The tokenizer question is treated more narrowly: the selective embedding merge can accommodate vocabulary mismatch at the embedding level, but the framework does not fully analyze composition across models with more substantial tokenizer divergence (Wang et al., 23 Sep 2025).

3. Training-free harmonization

Stage 1 consists of two operations: shared-backbone construction and activation-based functional alignment of experts. No model parameters are gradient-trained in this stage (Wang et al., 23 Sep 2025).

For the shared backbone, Symphony-MoE uses a layer-aware fusion strategy. Token embeddings are merged with the selective linear strategy from MergeKit: shared tokens are averaged linearly, and tokens unique to one model retain their original embeddings. Self-attention modules are merged with Spherical Linear Interpolation (SLERP) applied to the $M_{MoE}(x) = \sum_{i=1}^{N} g_i(x) \cdot E_i(x)$1, $M_{MoE}(x) = \sum_{i=1}^{N} g_i(x) \cdot E_i(x)$2, $M_{MoE}(x) = \sum_{i=1}^{N} g_i(x) \cdot E_i(x)$3, and $M_{MoE}(x) = \sum_{i=1}^{N} g_i(x) \cdot E_i(x)$4 projection matrices. LayerNorm parameters are combined by simple linear averaging. In the reported four-model experiments, source weights are set uniformly to $M_{MoE}(x) = \sum_{i=1}^{N} g_i(x) \cdot E_i(x)$5 for each model (Wang et al., 23 Sep 2025).

The rationale for layer awareness is architectural rather than depth-dependent. Embeddings must tolerate vocabulary overlap and mismatch; attention weights benefit from geometry-preserving interpolation; and LayerNorm behaves as a statistical normalization component for which averaging is stable. The paper’s ablations support this claim: replacing SLERP with naive attention averaging reduces performance, and naive embedding merging also degrades results (Wang et al., 23 Sep 2025).

The second part of Stage 1 is functional alignment of FFN experts. One source model is chosen as an anchor, and the other models are aligned to it layer by layer. In the experiments, the general-purpose model $M_{MoE}(x) = \sum_{i=1}^{N} g_i(x) \cdot E_i(x)$6 serves as the default anchor. A calibration set $M_{MoE}(x) = \sum_{i=1}^{N} g_i(x) \cdot E_i(x)$7 is constructed by sampling $M_{MoE}(x) = \sum_{i=1}^{N} g_i(x) \cdot E_i(x)$8 from each instruction fine-tuning dataset used to train the dense models; in the implementation details this totals $M_{MoE}(x) = \sum_{i=1}^{N} g_i(x) \cdot E_i(x)$9 million tokens (Wang et al., 23 Sep 2025).

For each non-anchor model $N$0 and each layer $N$1, the same calibration samples are passed through the anchor and target model to obtain FFN activation matrices $N$2 and $N$3. Alignment is then posed as a linear assignment problem over permutation matrices:

$N$4

where $N$5 is a permutation matrix, $N$6 is the set of permutation matrices, and $N$7 is the Frobenius norm. The paper states that this is solved with the Hungarian algorithm (Wang et al., 23 Sep 2025).

Once the permutation $N$8 is found, the target FFN is reparameterized into the anchor’s functional coordinate system by remapping the FFN weights as

$N$9

This permutes the output dimension of the first linear layer and the input dimension of the second linear layer, preserving the FFN’s function under hidden-unit reordering. After this step, each layer contains one aligned expert per source model, all intended to operate in a consistent representational space (Wang et al., 23 Sep 2025).

4. Router training and optimization regime

After harmonization, Symphony-MoE freezes the shared backbone and all expert FFN parameters, and trains only the router. The router is described as a simple randomly initialized linear layer. Routing is sparse and top-2, in standard token-level MoE fashion (Wang et al., 23 Sep 2025).

The router objective is

$E_i(\cdot)$0

where $E_i(\cdot)$1 is the causal language modeling loss, $E_i(\cdot)$2 is a load-balancing regularization term encouraging uniform expert utilization, and $E_i(\cdot)$3 in all reported experiments. The paper does not provide the explicit form of $E_i(\cdot)$4, nor does it specify capacity factors, token dropping, or overflow handling (Wang et al., 23 Sep 2025).

An important property of the method is that the router receives no oracle routing labels. It learns dispatching implicitly from the language modeling objective once the architecture has been harmonized. The paper’s broader claim is that heavy end-to-end retraining is unnecessary if the backbone is coherent and the experts are functionally aligned. This suggests a separation of concerns: harmonization resolves representational incompatibility, and the router learns coordination only after that incompatibility has been removed (Wang et al., 23 Sep 2025).

The implementation details reinforce the framework’s emphasis on lightweight adaptation. Dense source models are trained using LLaMA-Factory for 2 epochs. Router training uses 8 NVIDIA V100 GPUs, cutoff length $E_i(\cdot)$5, 6 epochs, batch size 2, learning rate $E_i(\cdot)$6, AdamW, and maximum gradient norm $E_i(\cdot)$7. Evaluation uses lm-evaluation-harness v0.4 (Wang et al., 23 Sep 2025).

5. Experimental setting and empirical findings

The reported experiments construct four dense specialists at two scales, $E_i(\cdot)$8B and $E_i(\cdot)$9B. The foundation checkpoints are Qwen2.5-Base, Qwen2.5-Coder, and Qwen2-Base. From these, the paper instruction-tunes four specialists: a general expert $i$0 on Alpaca, a math expert $i$1 on MetaMathQA, a code expert $i$2 on CodeAlpaca, and a science expert $i$3 on SciQAG (Wang et al., 23 Sep 2025).

Evaluation spans five in-distribution benchmarks and one out-of-distribution benchmark. The in-distribution suite is MMLU, GSM8K, BBH, HumanEval, and TruthfulQA. The out-of-distribution test is MedCQA. Metrics are zero-shot accuracy for MMLU and MedCQA, 8-shot accuracy for GSM8K and BBH, pass@1 for HumanEval, and MC1 for TruthfulQA. Baselines include the dense source models and the upcycling methods BTX, BAM, and Drop (Wang et al., 23 Sep 2025).

Setting	Average in-distribution score	MedCQA
0.5B × 4 Symphony-MoE	31.77	29.07
1.5B × 4 Symphony-MoE	44.07	35.26

At $i$4B$i$5, Symphony-MoE achieves an average in-distribution score of $i$6, compared with $i$7 for Drop, $i$8 for BAM, $i$9 for BTX, and $g_i(x)$0 for the best dense model. On MedCQA, it reaches $g_i(x)$1, compared with $g_i(x)$2 for Drop and $g_i(x)$3 for BAM. Selected task scores are MMLU $g_i(x)$4, GSM8K $g_i(x)$5, BBH $g_i(x)$6, HumanEval $g_i(x)$7, and TruthfulQA $g_i(x)$8 (Wang et al., 23 Sep 2025).

At $g_i(x)$9B$i$0, Symphony-MoE attains an average in-distribution score of $i$1, compared with $i$2 for Drop, $i$3 for BAM, $i$4 for BTX, and $i$5 for the best dense model. On MedCQA it reaches $i$6, compared with $i$7 for Drop and $i$8 for BAM. Selected scores are MMLU $i$9, GSM8K $x$0, BBH $x$1, HumanEval $x$2, and TruthfulQA $x$3 (Wang et al., 23 Sep 2025).

The paper stresses that these gains do not arise from simple averaging into a compromise model. On HumanEval at $x$4B scale, the code specialist $x$5 remains the strongest single model at $x$6, while Symphony-MoE reaches $x$7, only about one point lower while remaining stronger on non-code tasks. This is presented as evidence that the MoE preserves specialist competence while improving multi-domain breadth and out-of-distribution generalization (Wang et al., 23 Sep 2025).

The ablation study identifies functional alignment as the dominant ingredient. In the $x$8B$x$9 setting, the full method scores ID $g(x) = Softmax(xW_g)$0 and OOD $g(x) = Softmax(xW_g)$1, whereas removing functional alignment yields ID $g(x) = Softmax(xW_g)$2 and OOD $g(x) = Softmax(xW_g)$3. Naive attention merge reduces performance to ID $g(x) = Softmax(xW_g)$4, OOD $g(x) = Softmax(xW_g)$5, and naive embedding merge to ID $g(x) = Softmax(xW_g)$6, OOD $g(x) = Softmax(xW_g)$7. Using biased calibration data degrades results to ID $g(x) = Softmax(xW_g)$8, OOD $g(x) = Softmax(xW_g)$9 (Wang et al., 23 Sep 2025).

The paper also uses Centered Kernel Alignment (CKA) to assess specialization. Naive merging without alignment produces high inter-expert CKA around $k$0–$k$1, which the paper interprets as representational collapse. Activation-based alignment restores CKA to levels near those of the original unmerged experts. Additional analyses report that the general-purpose expert $k$2 is the best anchor choice and that increasing the number of experts from 1 to 4 consistently improves both in-distribution and out-of-distribution performance, with the addition of the math expert $k$3 providing the largest gain in the reported progression (Wang et al., 23 Sep 2025).

6. Scope, limitations, and disambiguation

Symphony-MoE’s scope is sharply defined. It requires source models with identical architecture, and its alignment quality depends on the diversity and representativeness of the calibration dataset. The paper does not fully analyze the exact load-balancing formula, capacity management, scaling to much larger expert counts, behavior when source models are extremely far apart semantically, or exact FLOP and wall-clock savings relative to training from scratch. The paper also notes that future work may need alignment methods that can handle minor architectural differences such as different activation functions or normalization strategies (Wang et al., 23 Sep 2025).

The method is distinct from several similarly named systems. It is not the multi-agent planning framework “SYMPHONY: Synergistic Multi-agent Planning with Heterogeneous LLM Assembly,” which concerns heterogeneous language-model agents and Monte Carlo Tree Search rather than MoE upcycling (Zhu et al., 30 Jan 2026). It is also distinct from “From Solo to Symphony,” whose method is Solo-to-Collaborative RL (SoCo); that work contains an MoE-like gating selector inside a policy-fusion module for cooperative MARL, but its “experts” are candidate actions produced from decomposed observations rather than trainable FFN expert blocks in a LLM (Wang et al., 4 Nov 2025).

It is likewise separate from “SymphonyGen,” a hierarchical symbolic orchestral generation system that uses bar-track-event factorization, harmony-skeleton conditioning, GRPO fine-tuning, and dissonance-averse sampling, but does not use sparse expert routing or any MoE mechanism (He et al., 28 Apr 2026). A plausible implication is that the “Symphony” label has been used across several unrelated technical contexts; in the specific case of Symphony-MoE, the term denotes a model-composition method for harmonizing multiple dense pretrained specialists into a coherent sparse MoE, rather than a planning system, a MARL orchestration mechanism, or a music-generation architecture (Wang et al., 23 Sep 2025).