---
title: Stitching of Experts (SoE) Strategies
url: https://www.emergentmind.com/topics/stitching-of-experts-soe
type: topic
---

# Stitching of Experts (SoE) Strategies

Searching arXiv for recent papers and the specific SoE-related works to ground the article.
Stitching of Experts (SoE) denotes a class of methods for combining pre-trained expert models through lightweight learned interfaces rather than full joint retraining. In the literature, the term covers several closely related mechanisms: hidden-state stitching between frozen large language model experts in Branch-Train-Stitch (BTS), bidirectional intermediate-feature translation between independently trained segmentation networks, deep blockwise fusion of audio and video generators, and, in a broader usage, lightweight supervised fusion over expert outputs [2502.00075] [2512.17592] [2509.06155] [2310.01542]. This suggests that SoE is best understood as a family of modular composition strategies whose common objective is to preserve expert specialization while improving generalist or cross-domain behavior.

## 1. Conceptual scope and defining characteristics

Across the cited works, SoE is organized around a recurring pattern: expert models are first trained independently or specialized after branching from a shared initialization, and a comparatively small set of additional parameters is then learned to reconcile their internal representations or outputs. In BTS, a single seed language model is branched into domain-specific experts and later reconnected by stitch layers inserted between frozen experts and the seed LLM [2502.00075]. In medical segmentation, a stitching layer \(S_{A\to B}^{\ell\to m}\) is a small parametric function that translates the representation of one expert into the feature space expected by another, typically using a \(1\times 1\) convolution or a linear projection [2512.17592]. In UniVerse-1, SoE deeply fuses the corresponding blocks of pre-trained video and music generation experts models through cross-modal attention and small projection adapters [2509.06155]. In "Fusing Models with Complementary Expertise," the broader SoE/FoE formulation treats fusion as supervised learning over expert outputs, using a convex combination \(F(x)=\sum_{k=1}^K g_k(x)\cdot E_k(x)\) with a lightweight gating network [2310.01542].

A common source of ambiguity is whether SoE denotes a single architecture. The reported literature does not support that interpretation. Instead, the concrete implementations differ in where fusion occurs—hidden activations, matched intermediate vertices in a directed acyclic graph, paired multimodal Transformer blocks, or expert outputs—while sharing an emphasis on lightweight learned coupling.

## 2. Branch-Train-Stitch and the LLM formulation

In "BTS: Harmonizing Specialized Experts into a Generalist LLM," the SoE mechanism is instantiated as Branch-Train-Stitch over a seed Transformer LLM \(m_0\) of \(2.7\) B parameters with \(L=20\) layers, hidden dimension \(d=3072\), FFN dimension \(4d=12288\), \(24\) attention heads, SwiGLU activation, RoPE positional embeddings, and vocab size \(128\) K [2502.00075]. The branch step clones the seed into \(n=3\) identical copies \(m_1,m_2,m_3\) for code, math, and multilingual specialization; each copy starts with exactly the same weights as \(m_0\), so total parameters \(=4\times 2.7\) B \(\approx 10.8\) B.

The train step performs continual pretraining of each expert on a domain mixture \(\mathcal{D}_i\) of \(200\) B total tokens using standard next-token prediction:
\[
L_i(\theta_i)=-\sum_{(x_1\ldots x_T)\in\mathcal{D}_i}\sum_{t=1}^T \log P_{\theta_i}(x_t\mid x_{<t}).
\]
The optimization uses \(96\) k steps, batch size \(2\) M tokens, and initial LR \(5\times 10^{-6}\) warmed-up then cosine-decayed to \(1\%\) over training. The result is three specialized experts \(m_1,m_2,m_3\) that outperform \(m_0\) in their domain but may forget other domains.

The stitch step keeps \(m_0\ldots m_3\) completely frozen and introduces \(K=4\) stitch layers per model, placed after every \(\lfloor L/K\rfloor =5\) Transformer blocks. At stitch point \(j\), hidden states are collected from the hub \(m_0\) and each expert:
\[
h_0^\ell\in\mathbb{R}^{B\times S\times d},\qquad h_i^\ell\in\mathbb{R}^{B\times S\times d}.
\]
Two alternating stitch types are used. In Experts-into-Hub (E\(\to\)H),
\[
g=\mathrm{softmax}\bigl(\mathrm{Dropout}(w_{\rm gate}(h_0^\ell))\bigr),\qquad
\tilde h_i^\ell=w_{\rm proj,i}(h_i^\ell),\qquad
\tilde h_0^\ell=g_0h_0^\ell+\sum_{i=1}^3 g_i\tilde h_i^\ell.
\]
In Hub-into-Experts (H\(\to\)E),
\[
g=\mathrm{sigmoid}\bigl(\mathrm{Dropout}(w_{\rm gate}(h_0^\ell))\bigr),\qquad
\tilde h_0^\ell=h_0^\ell,\qquad
\tilde h_i^\ell=(1-g_i)h_i^\ell+g_i\,w_{\rm proj,i}(h_0^\ell).
\]
After fusion, the modified states \(\tilde h_i^\ell\) feed into the next Transformer block of each model, and the final output is taken from the hub model’s last layer.

The modular expansion rule is explicit. To add expert \(m_{n+1}\), one branches a fresh copy of \(m_0\), trains it on \(\mathcal{D}_{n+1}\), expands \(w_{\rm proj}\) and \(w_{\rm gate}\) in every stitch layer to handle the new expert, and trains only the expanded \(\Psi\) on a small mix that includes the new domain. To remove expert \(i\), one simply drops its \(w_{{\rm proj},i}\) and removes its hidden track in fusion; no retraining of other experts or seed is needed. The paper identifies this as a “plug-and-play” property yielding a fully modular SoE system.

## 3. Stitch layers, placement, and architectural reconciliation

The most explicit formalization of a stitch appears in the medical segmentation setting. For independently trained expert networks \(A\) and \(B\), with feature maps \(v_A^\ell\in\mathbb{R}^{d_A^\ell\times H_A^\ell\times W_A^\ell}\) and \(v_B^m\in\mathbb{R}^{d_B^m\times H_B^m\times W_B^m}\), a stitching layer \(S_{A\to B}^{\ell\to m}\) is defined as
\[
S_{A\to B}^{\ell\to m}:\mathbb{R}^{d_A^\ell\times H\times W}\to \mathbb{R}^{d_B^m\times H\times W},
\]
and is implemented in most experiments as a \(1\times 1\) convolution or as a linear projection when \(H=W=1\). In either case the total parameter count of one stitch is on the order of \(d_A^\ell\cdot d_B^m\) [2512.17592].

Because nnU-Net-style segmentation models form directed acyclic graphs with skip-connections, stitch placement is handled by representing each network as a DAG, topologically sorting it, and assigning each vertex a normalized depth
\[
p(v)=\frac{\mathrm{dist}(v_{\rm in}\to v)}{\mathrm{dist}(v_{\rm in}\to v)+\mathrm{dist}(v\to v_{\rm out})}.
\]
Vertices are matched only if they operate at the same spatial scale and \(|p(v_A^\ell)-p(v_B^m)|\) is small; a modified Hirschberg-style algorithm is then used to find a maximum-similarity bipartite matching under the acyclicity constraint. Empirically, stitches placed in the mid-to-late decoder \((p\sim 0.6\text{--}0.8)\) yield the best performance gains, whereas early stitches \((p<0.2)\) give negligible benefit and very late stitches \((p>0.9)\) give little room for corrective interaction.

UniVerse-1 addresses a different placement problem: the two experts do not initially have equal depth. If the video expert has \(N_1\) Transformer blocks and the audio expert has \(N_2\) blocks with \(N_1<N_2\), the method inserts \(M=N_2-N_1\) new video blocks at uniform fractional offsets between the original ones and initializes each inserted block by linear interpolation,
\[
W_{\rm new}=\alpha W^{(j)}+(1-\alpha)W^{(j+1)}.
\]
After this step both streams have the same depth \(L=N_2\), block \(\ell\) of video is paired with block \(\ell\) of audio, and each fused block performs unimodal attention together with cross-modal adapter injection and cross-attention [2509.06155].

A plausible implication is that SoE design is governed less by a single canonical stitch operator than by the representational geometry of the experts being coupled: sequential Transformer stacks admit periodic stitch points, DAG segmentation networks require scale- and depth-aware matching, and multimodal generators require layer-depth reconciliation before blockwise fusion.

## 4. Training objectives and optimization regimes

The optimization strategy in SoE depends on the level at which experts are stitched. In BTS, only stitch parameters \(\Psi\) are updated while all constituent LLMs remain frozen. The stitch-layer training uses a data mix of \(15\%\) code, \(15\%\) math, \(15\%\) multilingual, and \(55\%\) remaining seed-domain text, corresponding to \(\approx 15\) B tokens over \(7\) k steps. The objective minimizes the next-token loss of the hub output only:
\[
L(\theta_\Psi)=-\sum_{(x_t)}\log P_{\rm hub}(x_t\mid x_{<t};\theta_\Psi).
\]
Hyperparameters are batch size \(=2\) M tokens, LR warmup \(0\to 5\times 10^{-6}\) over \(2\) k steps, then cosine decay to \(1\%\), with only \(\Psi\) updated [2502.00075].

In the medical setting, SoE proceeds in two phases. Phase A trains \(A\) and \(B\) solitarily on disjoint datasets \(D_A\) and \(D_B\). Phase B freezes \(w_A^\*,w_B^\*\) and optimizes only the stitches \(\phi\). Two alternative losses are studied. The direct matching loss is
\[
L_{\rm match}(\phi)=\mathbb{E}_{x\sim D_C}\sum_{(\ell,m)\in M}\|S_{A\to B}^{\ell\to m}(v_A^\ell(x))-v_B^m(x)\|_2^2
+\|S_{B\to A}^{m\to \ell}(v_B^m(x))-v_A^\ell(x)\|_2^2.
\]
An optional fine-tuning term is
\[
L_{\rm total}(\phi)=L_{\rm match}(\phi)+\lambda\cdot L_{\rm fine}(\phi),\qquad \lambda\ll 1.
\]
The paper emphasizes a double-batched, MSE with downstream awareness scheme in which each batch is doubled so that both “reference” and “stitched” activations can be propagated through downstream layers, allowing all stitches \(\phi\) to be back-propagated in one shot [2512.17592].

UniVerse-1 departs from the frozen-expert pattern. After copying pretrained weights from WAN2.1 into the video stream and from Ace-step into the audio stream, and after inserting interpolated video blocks plus randomly initialized adapter projections and a shared LayerNorm, the model is trained with
\[
\mathcal{L}=\mathcal{L}_{\rm FM\text{-}video}+\mathcal{L}_{\rm FM\text{-}mel}+\lambda_{\rm SSL}\mathcal{L}_{\rm SSL},
\]
where \(\lambda_{\rm SSL}=1.0\). Optimization uses AdamW, \(lr=5\times 10^{-6}\), batch \(=128\), \(50\) K steps, grad-accum \(=4\), and FSDP, with no progressive unfreezing: all parameters are trained from step \(1\) [2509.06155].

The FoE formulation treats stitching or fusion as supervised learning over expert outputs. Given expert outputs concatenated into \(f(x)=[E_1(x);\ldots;E_K(x)]\), the fusion net \(F_\theta\) is trained by minimizing
\[
\theta^\*=\arg\min_\theta \sum_i \ell\bigl(F_\theta([E_1(x_i),\ldots,E_K(x_i)]),y_i\bigr),
\]
with a convex combination
\[
F(x)=\sum_{k=1}^K g_k(x)\cdot E_k(x),
\]
where \(g(x)\) is produced by a small feed-forward MLP followed by softmax [2310.01542].

## 5. Empirical results across domains

The reported results span language modeling, medical image segmentation, multimodal generation, and generic expert fusion. The following table lists representative outcomes exactly as reported.

| Setting | SoE instantiation | Reported result |
|---|---|---|
| Dense \(2.7\) B LLM generalist | BTS with three experts and four stitch layers | avg \(=28.1\), MBPP \(=32.2\), MATH \(=10.6\) |
| Russian-math cross task | BTS with \(2\) B tokens of in-domain Russian math data | Ru-MGSM \(=16.0\) |
| Pelvic MRI | SoE (best-single-stitch) | on A: Dice \(0.83\) / HD95 \(4.9\) mm; on B: Dice \(0.75\) / HD95 \(5.8\) mm |
| Polyp segmentation | SoE | on CVC: Dice \(0.87\) / HD95 \(3.2\) px; on Hyper: Dice \(0.84\) / HD95 \(3.3\) px |
| Verse-Bench joint generation | UniVerse-1 (SoE) | ID \(=0.89\), CS \(=0.16\), AV-A \(=0.23\) |
| CIFAR-100 super-class classification | FoE (full fusion of \(20\) experts) | Final accuracy \(=82.13\%\) |
| MMLU | FoE select best expert | overall accuracy \(49.85\%\) |

In BTS, the main zero-/few-shot benchmark suite includes MMLU (5-shot), BBH (3-shot), MBPP (3-shot), HumanEval (0-shot), FLORES (1-shot, 7 langs, S\(\to\)T & T\(\to\)S), GSM8K (8-shot), and MATH (4-shot). The Seed Model finishes at average \(24.0\); individual experts trade off across domains with average \(\approx 25.4\); expert-upcycling MoE baselines reach average \(\approx 27.4\text{--}27.8\); expert-merging baselines average \(24.8\text{--}27.3\); BTS achieves the best average \(=28.1\). Parameter efficiency is also explicit: total params \(\approx 11\) B and training params \((\Psi)=264\) M, whereas expert-upcycling MoEs train \(>7\) B params and other merging methods train \(\le 1.5\) B but underperform. Ablations show that one stitch layer lags with average \(27.0\) versus \(4\) or \(10\) layers at average \(28.1\), alternating fusion is crucial for cross-capability tasks such as Russian\(\times\)math, and using the seed \(m_0\) as hub outperforms using any expert as hub with average \(28.1\) versus \(26.2\) [2502.00075].

In medical segmentation, SoE is evaluated on pelvic MRI from LUMC vs. AUMC and endoscopy polyp segmentation from HyperKvasir vs. CVC-ClinicDB, with Dice Score and \(95\%\) Hausdorff Distance HD95 as metrics. The pelvic MRI merge-datasets baseline reaches Dice \(0.87\) / HD95 \(3.8\) mm on both, while SoE closes \(\approx 70\%\) of the gap to “Merge” on cross-domain and retains \(>98\%\) of solo expert accuracy. For polyp segmentation, the average relative improvement over solo experts is \(\sim 10\text{--}12\%\) on cross-domain error. Multi-objective plots place SoE close to the Pareto front jointly over performance on A and performance on B, whereas naïve ensembles fall inside the front and solo experts lie at the extreme corners [2512.17592].

In UniVerse-1, the end-to-end comparison on Verse-Bench reports that SVG yields ID \(=0.25\), CS \(=0.08\), AV-A \(=0.09\), whereas UniVerse-1 yields ID \(=0.89\), CS \(=0.16\), AV-A \(=0.23\). The paper reports a \(+256\%\) relative gain in ID, \(+100\%\) in CS, and more than doubling of AV-A consistency. Ablations show that removing LQLS drops ID from \(0.89\to 0.78\), AS from \(0.47\to 0.44\), and worsens AV-A from \(0.23\to 0.28\); removing INSS changes CS from \(0.16\to 0.11\), FD from \(1.25\to 1.43\), and WER from \(0.18\to 0.38\). The reported convergence statement is that SoE converges stably in \(50\) K steps by leveraging the frozen priors of two \(1.3\)B+\(3.5\)B models, while training from scratch on the same data took \(>100\) K steps to reach comparable FM loss but still trailed on ID and CS metrics by \(\sim 30\%\) [2509.06155].

In FoE, the reported task families include CIFAR-100 super-class classification, sentiment analysis, text summarization, MMLU, and automatic summarization evaluation. On CIFAR-100, FoE reaches final accuracy \(82.13\%\) versus ensemble \(76.64\%\), confidence-based fusion \(74.07\%\), average single expert \(50.04\%\), and oracle \(87.63\%\). With only \(37.5\%\) of experts on average, FrugalFoE matches full-fusion accuracy \(82.13\%\). On sentiment analysis, FoE average accuracy is \(91.88\%\) versus best single expert average \(79.44\%\), with expert selection accuracy \(99.1\%\). On MMLU, FoE achieves \(49.85\%\) overall accuracy versus average expert \(41.34\%\pm 7.22\%\) and best single expert \(47.35\%\) [2310.01542].

## 6. Modularity, interpretability, and relation to adjacent paradigms

SoE is often discussed near model merging, mixtures of experts, ensembles, and federated learning, but the cited works draw sharper distinctions. BTS is compared directly to expert-upcycling MoE baselines such as BTX and BAM and to expert-merging baselines such as Model Soup, BTM, Expert Routing, and BAM Adapters; its reported advantage is that it does not alter the constituent LLMs, and experts can be easily removed or new experts can be added with only a small amount of training [2502.00075]. This suggests that SoE, in this formulation, is a modular interface over intact experts rather than a weight-space merger or a sparse-routing MoE.

In the medical report, SoE is positioned against federated learning by emphasizing asynchronous collaboration, where only already trained models are shared. The workflow requires only one exchange of model weights and uses unlabeled data from whichever party is doing the stitching; labels are never shared. The paper characterizes this as asynchronous and privacy-safe, with practical relevance in medical and other privacy-sensitive domains [2512.17592]. A plausible implication is that SoE occupies a distinct point in the design space between post hoc ensembling and synchronous federated optimization.

Interpretability appears in BTS through gate-value visualizations showing that BTS dynamically shifts solver weights token by token, relying on the correct expert when generating code versus math versus translation, and smoothly context-switching across tasks in one prompt [2502.00075]. In the medical setting, the explanation is framed in terms of intermediate feature ensembling, correction of representation symmetries, double-batched downstream awareness, and the absence of catastrophic forgetting because expert weights are frozen during the majority of Phase B [2512.17592]. In FoE, the gating network gives an explicit expert-weight decomposition \(g_k(x)\), while FrugalFoE casts expert selection as a sequential decision problem with a graph-search view [2310.01542].

A recurring misconception is that the only useful fusion point is the final output. The medical study states that ensembling at the final softmax cannot correct mis-detected features early in the network, whereas SoE aligns semantically high-level features through learnable bridges and averages aligned features rather than merely class probabilities [2512.17592]. Conversely, the FoE results show that output-level fusion can still be highly effective when the fusion module is trained rather than naïvely averaged [2310.01542]. Taken together, the literature indicates that SoE is not tied to a single fusion depth; instead, the operative question is which level of representation offers the best trade-off among modularity, trainable parameter count, and cross-domain transfer.

Source: https://www.emergentmind.com/topics/stitching-of-experts-soe