---
title: 'Symphony-MoE: Sparse MoE Upcycling Framework'
url: https://www.emergentmind.com/topics/symphony-moe
type: topic
---

# Symphony-MoE: Sparse MoE Upcycling Framework

Symphony-MoE is a framework for constructing a sparse Mixture-of-Experts (MoE) language model from multiple pretrained dense models that share the same architecture but differ in training history and specialization. It addresses the limitation of conventional MoE upcycling, which typically replicates feed-forward network (FFN) layers from a single dense checkpoint and therefore restricts expert diversity. Its central claim is that experts sourced from disparate pretrained models can be combined into a single coherent MoE only if their shared backbone is fused in a layer-aware manner and their FFN experts are functionally aligned before routing is learned [2509.18542].

## 1. Problem formulation and motivation

Symphony-MoE is situated in sparse MoE language modeling. In the formulation given in the paper, the MoE output for token input \(x\) is

$$
M_{MoE}(x) = \sum_{i=1}^{N} g_i(x) \cdot E_i(x)
$$

where \(N\) is the number of experts, \(E_i(\cdot)\) is expert \(i\), and \(g_i(x)\) is the router’s gating weight for expert \(i\) on token \(x\). The routing weights are produced by a linear router followed by softmax,

$$
g(x) = Softmax(xW_g)
$$

and, in practice, only the top-\(k\) experts are activated. This is the standard conditional-computation setting in which total parameter count can grow while per-token FLOPs remain relatively low because only a small subset of experts is used [2509.18542].

The specific problem addressed by Symphony-MoE is multi-source upcycling. Prior upcycling methods generally begin from one dense pretrained model and convert its FFNs into experts, often by replication. Symphony-MoE argues that such single-origin construction intrinsically limits expert diversity because all experts begin in the same parameter basin and inherit the same pretrained knowledge source. Its stated alternative is to build an MoE from multiple identically architected but disparate pretrained models, including heterogeneous sources such as general chat, code, math, and science specialists [2509.18542].

The paper identifies parameter-space and representational misalignment as the main obstacle. Independently pretrained models may occupy dissonant regions of parameter space; corresponding neurons need not match; FFN hidden units may encode different features in different orders; and non-FFN modules such as attention, embeddings, and LayerNorm cannot be safely combined with a single naive merge rule. According to the paper, this leads to interference between experts, incoherent shared representations, loss of specialization, and difficulty for the router to learn meaningful dispatching. Symphony-MoE therefore frames harmonization, rather than routing alone, as the primary technical challenge [2509.18542].

## 2. Two-stage architecture and model assembly

Symphony-MoE is explicitly a two-stage framework. Stage 1 is a training-free harmonization step. Stage 2 is lightweight router training. The design principle is to make the source models functionally compatible before any gradient-based coordination is attempted [2509.18542].

The final model replaces the dense FFN in each Transformer block with a set of experts, one per source model per layer. The shared backbone consists of token embeddings, self-attention modules, and LayerNorm parameters. The experts are the FFN or MLP blocks. A trainable linear router is added to each MoE layer to select experts. This design preserves source-specific FFN specialization while consolidating non-FFN computation into a common backbone [2509.18542].

| Component | Role in final model | Construction strategy |
|---|---|---|
| Token embeddings | Shared backbone | Selective linear merge |
| Self-attention modules | Shared backbone | SLERP on \(Q,K,V,O\) |
| LayerNorm parameters | Shared backbone | Simple averaging |
| FFN / MLP blocks | Experts | Activation-based alignment, then preserved as experts |

The framework assumes that source models are identically architected. In the paper, this implies compatibility in layer count, hidden size, FFN dimension, attention dimensions, and parameter shapes. The tokenizer question is treated more narrowly: the selective embedding merge can accommodate vocabulary mismatch at the embedding level, but the framework does not fully analyze composition across models with more substantial tokenizer divergence [2509.18542].

## 3. Training-free harmonization

Stage 1 consists of two operations: shared-backbone construction and activation-based functional alignment of experts. No model parameters are gradient-trained in this stage [2509.18542].

For the shared backbone, Symphony-MoE uses a layer-aware fusion strategy. Token embeddings are merged with the selective linear strategy from MergeKit: shared tokens are averaged linearly, and tokens unique to one model retain their original embeddings. Self-attention modules are merged with Spherical Linear Interpolation (SLERP) applied to the \(Q\), \(K\), \(V\), and \(O\) projection matrices. LayerNorm parameters are combined by simple linear averaging. In the reported four-model experiments, source weights are set uniformly to \(0.25\) for each model [2509.18542].

The rationale for layer awareness is architectural rather than depth-dependent. Embeddings must tolerate vocabulary overlap and mismatch; attention weights benefit from geometry-preserving interpolation; and LayerNorm behaves as a statistical normalization component for which averaging is stable. The paper’s ablations support this claim: replacing SLERP with naive attention averaging reduces performance, and naive embedding merging also degrades results [2509.18542].

The second part of Stage 1 is functional alignment of FFN experts. One source model is chosen as an anchor, and the other models are aligned to it layer by layer. In the experiments, the general-purpose model \(M_1\) serves as the default anchor. A calibration set \(D_{\mathrm{cal}}\) is constructed by sampling \(5\%\) from each instruction fine-tuning dataset used to train the dense models; in the implementation details this totals \(10.404\) million tokens [2509.18542].

For each non-anchor model \(M_i\) and each layer \(l\), the same calibration samples are passed through the anchor and target model to obtain FFN activation matrices \(A_1^{(l)}\) and \(A_i^{(l)}\). Alignment is then posed as a linear assignment problem over permutation matrices:

$$
\min_{P \in \mathcal{P}} \|A_1^{(l)} - A_i^{(l)}P\|_F^2
$$

where \(P\) is a permutation matrix, \(\mathcal{P}\) is the set of permutation matrices, and \(\|\cdot\|_F\) is the Frobenius norm. The paper states that this is solved with the Hungarian algorithm [2509.18542].

Once the permutation \(P_i^{(l)}\) is found, the target FFN is reparameterized into the anchor’s functional coordinate system by remapping the FFN weights as

$$
W'_{up,i} = W_{up,i}^{(l)} P_i^{(l)}, \quad W'_{down,i} = (P_i^{(l)})^T W_{down,i}^{(l)}.
$$

This permutes the output dimension of the first linear layer and the input dimension of the second linear layer, preserving the FFN’s function under hidden-unit reordering. After this step, each layer contains one aligned expert per source model, all intended to operate in a consistent representational space [2509.18542].

## 4. Router training and optimization regime

After harmonization, Symphony-MoE freezes the shared backbone and all expert FFN parameters, and trains only the router. The router is described as a simple randomly initialized linear layer. Routing is sparse and top-2, in standard token-level MoE fashion [2509.18542].

The router objective is

$$
\mathcal{L}_{total} = \mathcal{L}_{lm} + \lambda \cdot \mathcal{L}_{bal}
$$

where \(\mathcal{L}_{lm}\) is the causal language modeling loss, \(\mathcal{L}_{bal}\) is a load-balancing regularization term encouraging uniform expert utilization, and \(\lambda = 0.01\) in all reported experiments. The paper does not provide the explicit form of \(\mathcal{L}_{bal}\), nor does it specify capacity factors, token dropping, or overflow handling [2509.18542].

An important property of the method is that the router receives no oracle routing labels. It learns dispatching implicitly from the language modeling objective once the architecture has been harmonized. The paper’s broader claim is that heavy end-to-end retraining is unnecessary if the backbone is coherent and the experts are functionally aligned. This suggests a separation of concerns: harmonization resolves representational incompatibility, and the router learns coordination only after that incompatibility has been removed [2509.18542].

The implementation details reinforce the framework’s emphasis on lightweight adaptation. Dense source models are trained using LLaMA-Factory for 2 epochs. Router training uses 8 NVIDIA V100 GPUs, cutoff length \(2048\), 6 epochs, batch size 2, learning rate \(5\times10^{-5}\), AdamW, and maximum gradient norm \(1.0\). Evaluation uses lm-evaluation-harness v0.4 [2509.18542].

## 5. Experimental setting and empirical findings

The reported experiments construct four dense specialists at two scales, \(0.5\)B and \(1.5\)B. The foundation checkpoints are Qwen2.5-Base, Qwen2.5-Coder, and Qwen2-Base. From these, the paper instruction-tunes four specialists: a general expert \(M_1\) on Alpaca, a math expert \(M_2\) on MetaMathQA, a code expert \(M_3\) on CodeAlpaca, and a science expert \(M_4\) on SciQAG [2509.18542].

Evaluation spans five in-distribution benchmarks and one out-of-distribution benchmark. The in-distribution suite is MMLU, GSM8K, BBH, HumanEval, and TruthfulQA. The out-of-distribution test is MedCQA. Metrics are zero-shot accuracy for MMLU and MedCQA, 8-shot accuracy for GSM8K and BBH, pass@1 for HumanEval, and MC1 for TruthfulQA. Baselines include the dense source models and the upcycling methods BTX, BAM, and Drop [2509.18542].

| Setting | Average in-distribution score | MedCQA |
|---|---:|---:|
| 0.5B × 4 Symphony-MoE | 31.77 | 29.07 |
| 1.5B × 4 Symphony-MoE | 44.07 | 35.26 |

At \(0.5\)B\(\times 4\), Symphony-MoE achieves an average in-distribution score of \(31.77\), compared with \(29.95\) for Drop, \(28.58\) for BAM, \(26.46\) for BTX, and \(28.46\) for the best dense model. On MedCQA, it reaches \(29.07\), compared with \(27.95\) for Drop and \(26.79\) for BAM. Selected task scores are MMLU \(45.10\), GSM8K \(24.57\), BBH \(29.64\), HumanEval \(28.02\), and TruthfulQA \(31.54\) [2509.18542].

At \(1.5\)B\(\times 4\), Symphony-MoE attains an average in-distribution score of \(44.07\), compared with \(41.10\) for Drop, \(39.55\) for BAM, \(33.94\) for BTX, and \(40.95\) for the best dense model. On MedCQA it reaches \(35.26\), compared with \(32.90\) for Drop and \(28.97\) for BAM. Selected scores are MMLU \(58.91\), GSM8K \(39.12\), BBH \(46.97\), HumanEval \(42.39\), and TruthfulQA \(32.95\) [2509.18542].

The paper stresses that these gains do not arise from simple averaging into a compromise model. On HumanEval at \(1.5\)B scale, the code specialist \(M_3\) remains the strongest single model at \(43.41\), while Symphony-MoE reaches \(42.39\), only about one point lower while remaining stronger on non-code tasks. This is presented as evidence that the MoE preserves specialist competence while improving multi-domain breadth and out-of-distribution generalization [2509.18542].

The ablation study identifies functional alignment as the dominant ingredient. In the \(1.5\)B\(\times 4\) setting, the full method scores ID \(44.07\) and OOD \(35.26\), whereas removing functional alignment yields ID \(33.94\) and OOD \(26.92\). Naive attention merge reduces performance to ID \(37.28\), OOD \(31.77\), and naive embedding merge to ID \(40.19\), OOD \(31.95\). Using biased calibration data degrades results to ID \(39.70\), OOD \(29.44\) [2509.18542].

The paper also uses Centered Kernel Alignment (CKA) to assess specialization. Naive merging without alignment produces high inter-expert CKA around \(0.65\)–\(0.75\), which the paper interprets as representational collapse. Activation-based alignment restores CKA to levels near those of the original unmerged experts. Additional analyses report that the general-purpose expert \(M_1\) is the best anchor choice and that increasing the number of experts from 1 to 4 consistently improves both in-distribution and out-of-distribution performance, with the addition of the math expert \(M_2\) providing the largest gain in the reported progression [2509.18542].

## 6. Scope, limitations, and disambiguation

Symphony-MoE’s scope is sharply defined. It requires source models with identical architecture, and its alignment quality depends on the diversity and representativeness of the calibration dataset. The paper does not fully analyze the exact load-balancing formula, capacity management, scaling to much larger expert counts, behavior when source models are extremely far apart semantically, or exact FLOP and wall-clock savings relative to training from scratch. The paper also notes that future work may need alignment methods that can handle minor architectural differences such as different activation functions or normalization strategies [2509.18542].

The method is distinct from several similarly named systems. It is not the multi-agent planning framework “SYMPHONY: Synergistic Multi-agent Planning with Heterogeneous Language Model Assembly,” which concerns heterogeneous language-model agents and Monte Carlo Tree Search rather than MoE upcycling [2601.22623]. It is also distinct from “From Solo to Symphony,” whose method is Solo-to-Collaborative RL (SoCo); that work contains an MoE-like gating selector inside a policy-fusion module for cooperative MARL, but its “experts” are candidate actions produced from decomposed observations rather than trainable FFN expert blocks in a language model [2511.02762].

It is likewise separate from “SymphonyGen,” a hierarchical symbolic orchestral generation system that uses bar-track-event factorization, harmony-skeleton conditioning, GRPO fine-tuning, and dissonance-averse sampling, but does not use sparse expert routing or any MoE mechanism [2604.25498]. A plausible implication is that the “Symphony” label has been used across several unrelated technical contexts; in the specific case of Symphony-MoE, the term denotes a model-composition method for harmonizing multiple dense pretrained specialists into a coherent sparse MoE, rather than a planning system, a MARL orchestration mechanism, or a music-generation architecture [2509.18542].

Source: https://www.emergentmind.com/topics/symphony-moe