Stitching of Experts (SoE) Strategies

• Stitching of Experts (SoE) is a modular composition strategy that integrates independently trained expert models via lightweight learned interfaces.
• It fuses experts at various levels—hidden states, intermediate features, or outputs—to retain domain-specific strengths while enhancing generalist performance.
• The plug-and-play design of SoE supports scalable, privacy-preserving model fusion by allowing experts to be added or removed without full retraining.

Searching arXiv for papers and the specific SoE-related works to ground the article. Stitching of Experts (SoE) denotes a class of methods for combining pre-trained expert models through lightweight learned interfaces rather than full joint retraining. In the literature, the term covers several closely related mechanisms: hidden-state stitching between frozen LLM experts in Branch-Train-Stitch (BTS), bidirectional intermediate-feature translation between independently trained segmentation networks, deep blockwise fusion of audio and video generators, and, in a broader usage, lightweight supervised fusion over expert outputs (Zhang et al., 31 Jan 2025, Guijt et al., 19 Dec 2025, Wang et al., 7 Sep 2025, Wang et al., 2023). This suggests that SoE is best understood as a family of modular composition strategies whose common objective is to preserve expert specialization while improving generalist or cross-domain behavior.

1. Conceptual scope and defining characteristics

Across the cited works, SoE is organized around a recurring pattern: expert models are first trained independently or specialized after branching from a shared initialization, and a comparatively small set of additional parameters is then learned to reconcile their internal representations or outputs. In BTS, a single seed LLM is branched into domain-specific experts and later reconnected by stitch layers inserted between frozen experts and the seed LLM (Zhang et al., 31 Jan 2025). In medical segmentation, a stitching layer $S_{A\to B}^{\ell\to m}$ is a small parametric function that translates the representation of one expert into the feature space expected by another, typically using a $1\times 1$ convolution or a linear projection (Guijt et al., 19 Dec 2025). In UniVerse-1, SoE deeply fuses the corresponding blocks of pre-trained video and music generation experts models through cross-modal attention and small projection adapters (Wang et al., 7 Sep 2025). In "Fusing Models with Complementary Expertise," the broader SoE/FoE formulation treats fusion as supervised learning over expert outputs, using a convex combination $F(x)=\sum_{k=1}^K g_k(x)\cdot E_k(x)$ with a lightweight gating network (Wang et al., 2023).

A common source of ambiguity is whether SoE denotes a single architecture. The reported literature does not support that interpretation. Instead, the concrete implementations differ in where fusion occurs—hidden activations, matched intermediate vertices in a directed acyclic graph, paired multimodal Transformer blocks, or expert outputs—while sharing an emphasis on lightweight learned coupling.

2. Branch-Train-Stitch and the LLM formulation

In "BTS: Harmonizing Specialized Experts into a Generalist LLM," the SoE mechanism is instantiated as Branch-Train-Stitch over a seed Transformer LLM $m_0$ of $2.7$ B parameters with $L=20$ layers, hidden dimension $d=3072$, FFN dimension $4d=12288$, $24$ attention heads, SwiGLU activation, RoPE positional embeddings, and vocab size $128$ K (Zhang et al., 31 Jan 2025). The branch step clones the seed into $1\times 1$0 identical copies $1\times 1$1 for code, math, and multilingual specialization; each copy starts with exactly the same weights as $1\times 1$2, so total parameters $1\times 1$3 B $1\times 1$4 B.

The train step performs continual pretraining of each expert on a domain mixture $1\times 1$5 of $1\times 1$6 B total tokens using standard next-token prediction: $1\times 1$7 The optimization uses $1\times 1$8 k steps, batch size $1\times 1$9 M tokens, and initial LR $F(x)=\sum_{k=1}^K g_k(x)\cdot E_k(x)$0 warmed-up then cosine-decayed to $F(x)=\sum_{k=1}^K g_k(x)\cdot E_k(x)$1 over training. The result is three specialized experts $F(x)=\sum_{k=1}^K g_k(x)\cdot E_k(x)$2 that outperform $F(x)=\sum_{k=1}^K g_k(x)\cdot E_k(x)$3 in their domain but may forget other domains.

The stitch step keeps $F(x)=\sum_{k=1}^K g_k(x)\cdot E_k(x)$4 completely frozen and introduces $F(x)=\sum_{k=1}^K g_k(x)\cdot E_k(x)$5 stitch layers per model, placed after every $F(x)=\sum_{k=1}^K g_k(x)\cdot E_k(x)$6 Transformer blocks. At stitch point $F(x)=\sum_{k=1}^K g_k(x)\cdot E_k(x)$7, hidden states are collected from the hub $F(x)=\sum_{k=1}^K g_k(x)\cdot E_k(x)$8 and each expert: $F(x)=\sum_{k=1}^K g_k(x)\cdot E_k(x)$9 Two alternating stitch types are used. In Experts-into-Hub (E$m_0$0H),

$m_0$1

In Hub-into-Experts (H$m_0$2E),

$m_0$3

After fusion, the modified states $m_0$4 feed into the next Transformer block of each model, and the final output is taken from the hub model’s last layer.

The modular expansion rule is explicit. To add expert $m_0$5, one branches a fresh copy of $m_0$6, trains it on $m_0$7, expands $m_0$8 and $m_0$9 in every stitch layer to handle the new expert, and trains only the expanded $2.7$0 on a small mix that includes the new domain. To remove expert $2.7$1, one simply drops its $2.7$2 and removes its hidden track in fusion; no retraining of other experts or seed is needed. The paper identifies this as a “plug-and-play” property yielding a fully modular SoE system.

3. Stitch layers, placement, and architectural reconciliation

The most explicit formalization of a stitch appears in the medical segmentation setting. For independently trained expert networks $2.7$3 and $2.7$4, with feature maps $2.7$5 and $2.7$6, a stitching layer $2.7$7 is defined as

$2.7$8

and is implemented in most experiments as a $2.7$9 convolution or as a linear projection when $L=20$0. In either case the total parameter count of one stitch is on the order of $L=20$1 (Guijt et al., 19 Dec 2025).

Because nnU-Net-style segmentation models form directed acyclic graphs with skip-connections, stitch placement is handled by representing each network as a DAG, topologically sorting it, and assigning each vertex a normalized depth

$L=20$2

Vertices are matched only if they operate at the same spatial scale and $L=20$3 is small; a modified Hirschberg-style algorithm is then used to find a maximum-similarity bipartite matching under the acyclicity constraint. Empirically, stitches placed in the mid-to-late decoder $L=20$4 yield the best performance gains, whereas early stitches $L=20$5 give negligible benefit and very late stitches $L=20$6 give little room for corrective interaction.

UniVerse-1 addresses a different placement problem: the two experts do not initially have equal depth. If the video expert has $L=20$7 Transformer blocks and the audio expert has $L=20$8 blocks with $L=20$9, the method inserts $d=3072$0 new video blocks at uniform fractional offsets between the original ones and initializes each inserted block by linear interpolation,

$d=3072$1

After this step both streams have the same depth $d=3072$2, block $d=3072$3 of video is paired with block $d=3072$4 of audio, and each fused block performs unimodal attention together with cross-modal adapter injection and cross-attention (Wang et al., 7 Sep 2025).

A plausible implication is that SoE design is governed less by a single canonical stitch operator than by the representational geometry of the experts being coupled: sequential Transformer stacks admit periodic stitch points, DAG segmentation networks require scale- and depth-aware matching, and multimodal generators require layer-depth reconciliation before blockwise fusion.

4. Training objectives and optimization regimes

The optimization strategy in SoE depends on the level at which experts are stitched. In BTS, only stitch parameters $d=3072$5 are updated while all constituent LLMs remain frozen. The stitch-layer training uses a data mix of $d=3072$6 code, $d=3072$7 math, $d=3072$8 multilingual, and $d=3072$9 remaining seed-domain text, corresponding to $4d=12288$0 B tokens over $4d=12288$1 k steps. The objective minimizes the next-token loss of the hub output only: $4d=12288$2 Hyperparameters are batch size $4d=12288$3 M tokens, LR warmup $4d=12288$4 over $4d=12288$5 k steps, then cosine decay to $4d=12288$6, with only $4d=12288$7 updated (Zhang et al., 31 Jan 2025).

In the medical setting, SoE proceeds in two phases. Phase A trains $4d=12288$8 and $4d=12288$9 solitarily on disjoint datasets $24$0 and $24$1. Phase B freezes $24$2 and optimizes only the stitches $24$3. Two alternative losses are studied. The direct matching loss is

$24$4

An optional fine-tuning term is

$24$5

The paper emphasizes a double-batched, MSE with downstream awareness scheme in which each batch is doubled so that both “reference” and “stitched” activations can be propagated through downstream layers, allowing all stitches $24$6 to be back-propagated in one shot (Guijt et al., 19 Dec 2025).

UniVerse-1 departs from the frozen-expert pattern. After copying pretrained weights from WAN2.1 into the video stream and from Ace-step into the audio stream, and after inserting interpolated video blocks plus randomly initialized adapter projections and a shared LayerNorm, the model is trained with

$24$7

where $24$8. Optimization uses AdamW, $24$9, batch $128$0, $128$1 K steps, grad-accum $128$2, and FSDP, with no progressive unfreezing: all parameters are trained from step $128$3 (Wang et al., 7 Sep 2025).

The FoE formulation treats stitching or fusion as supervised learning over expert outputs. Given expert outputs concatenated into $128$4, the fusion net $128$5 is trained by minimizing

$128$6

with a convex combination

$128$7

where $128$8 is produced by a small feed-forward MLP followed by softmax (Wang et al., 2023).

5. Empirical results across domains

The reported results span language modeling, medical image segmentation, multimodal generation, and generic expert fusion. The following table lists representative outcomes exactly as reported.

Setting	SoE instantiation	Reported result
Dense $128$9 B LLM generalist	BTS with three experts and four stitch layers	avg $1\times 1$00, MBPP $1\times 1$01, MATH $1\times 1$02
Russian-math cross task	BTS with $1\times 1$03 B tokens of in-domain Russian math data	Ru-MGSM $1\times 1$04
Pelvic MRI	SoE (best-single-stitch)	on A: Dice $1\times 1$05 / HD95 $1\times 1$06 mm; on B: Dice $1\times 1$07 / HD95 $1\times 1$08 mm
Polyp segmentation	SoE	on CVC: Dice $1\times 1$09 / HD95 $1\times 1$10 px; on Hyper: Dice $1\times 1$11 / HD95 $1\times 1$12 px
Verse-Bench joint generation	UniVerse-1 (SoE)	ID $1\times 1$13, CS $1\times 1$14, AV-A $1\times 1$15
CIFAR-100 super-class classification	FoE (full fusion of $1\times 1$16 experts)	Final accuracy $1\times 1$17
MMLU	FoE select best expert	overall accuracy $1\times 1$18

In BTS, the main zero-/few-shot benchmark suite includes MMLU (5-shot), BBH (3-shot), MBPP (3-shot), HumanEval (0-shot), FLORES (1-shot, 7 langs, S$1\times 1$19T & T$1\times 1$20S), GSM8K (8-shot), and MATH (4-shot). The Seed Model finishes at average $1\times 1$21; individual experts trade off across domains with average $1\times 1$22; expert-upcycling MoE baselines reach average $1\times 1$23; expert-merging baselines average $1\times 1$24; BTS achieves the best average $1\times 1$25. Parameter efficiency is also explicit: total params $1\times 1$26 B and training params $1\times 1$27 M, whereas expert-upcycling MoEs train $1\times 1$28 B params and other merging methods train $1\times 1$29 B but underperform. Ablations show that one stitch layer lags with average $1\times 1$30 versus $1\times 1$31 or $1\times 1$32 layers at average $1\times 1$33, alternating fusion is crucial for cross-capability tasks such as Russian$1\times 1$34math, and using the seed $1\times 1$35 as hub outperforms using any expert as hub with average $1\times 1$36 versus $1\times 1$37 (Zhang et al., 31 Jan 2025).

In medical segmentation, SoE is evaluated on pelvic MRI from LUMC vs. AUMC and endoscopy polyp segmentation from HyperKvasir vs. CVC-ClinicDB, with Dice Score and $1\times 1$38 Hausdorff Distance HD95 as metrics. The pelvic MRI merge-datasets baseline reaches Dice $1\times 1$39 / HD95 $1\times 1$40 mm on both, while SoE closes $1\times 1$41 of the gap to “Merge” on cross-domain and retains $1\times 1$42 of solo expert accuracy. For polyp segmentation, the average relative improvement over solo experts is $1\times 1$43 on cross-domain error. Multi-objective plots place SoE close to the Pareto front jointly over performance on A and performance on B, whereas naïve ensembles fall inside the front and solo experts lie at the extreme corners (Guijt et al., 19 Dec 2025).

In UniVerse-1, the end-to-end comparison on Verse-Bench reports that SVG yields ID $1\times 1$44, CS $1\times 1$45, AV-A $1\times 1$46, whereas UniVerse-1 yields ID $1\times 1$47, CS $1\times 1$48, AV-A $1\times 1$49. The paper reports a $1\times 1$50 relative gain in ID, $1\times 1$51 in CS, and more than doubling of AV-A consistency. Ablations show that removing LQLS drops ID from $1\times 1$52, AS from $1\times 1$53, and worsens AV-A from $1\times 1$54; removing INSS changes CS from $1\times 1$55, FD from $1\times 1$56, and WER from $1\times 1$57. The reported convergence statement is that SoE converges stably in $1\times 1$58 K steps by leveraging the frozen priors of two $1\times 1$59B+$1\times 1$60B models, while training from scratch on the same data took $1\times 1$61 K steps to reach comparable FM loss but still trailed on ID and CS metrics by $1\times 1$62 (Wang et al., 7 Sep 2025).

In FoE, the reported task families include CIFAR-100 super-class classification, sentiment analysis, text summarization, MMLU, and automatic summarization evaluation. On CIFAR-100, FoE reaches final accuracy $1\times 1$63 versus ensemble $1\times 1$64, confidence-based fusion $1\times 1$65, average single expert $1\times 1$66, and oracle $1\times 1$67. With only $1\times 1$68 of experts on average, FrugalFoE matches full-fusion accuracy $1\times 1$69. On sentiment analysis, FoE average accuracy is $1\times 1$70 versus best single expert average $1\times 1$71, with expert selection accuracy $1\times 1$72. On MMLU, FoE achieves $1\times 1$73 overall accuracy versus average expert $1\times 1$74 and best single expert $1\times 1$75 (Wang et al., 2023).

6. Modularity, interpretability, and relation to adjacent paradigms

SoE is often discussed near model merging, mixtures of experts, ensembles, and federated learning, but the cited works draw sharper distinctions. BTS is compared directly to expert-upcycling MoE baselines such as BTX and BAM and to expert-merging baselines such as Model Soup, BTM, Expert Routing, and BAM Adapters; its reported advantage is that it does not alter the constituent LLMs, and experts can be easily removed or new experts can be added with only a small amount of training (Zhang et al., 31 Jan 2025). This suggests that SoE, in this formulation, is a modular interface over intact experts rather than a weight-space merger or a sparse-routing MoE.

In the medical report, SoE is positioned against federated learning by emphasizing asynchronous collaboration, where only already trained models are shared. The workflow requires only one exchange of model weights and uses unlabeled data from whichever party is doing the stitching; labels are never shared. The paper characterizes this as asynchronous and privacy-safe, with practical relevance in medical and other privacy-sensitive domains (Guijt et al., 19 Dec 2025). A plausible implication is that SoE occupies a distinct point in the design space between post hoc ensembling and synchronous federated optimization.

Interpretability appears in BTS through gate-value visualizations showing that BTS dynamically shifts solver weights token by token, relying on the correct expert when generating code versus math versus translation, and smoothly context-switching across tasks in one prompt (Zhang et al., 31 Jan 2025). In the medical setting, the explanation is framed in terms of intermediate feature ensembling, correction of representation symmetries, double-batched downstream awareness, and the absence of catastrophic forgetting because expert weights are frozen during the majority of Phase B (Guijt et al., 19 Dec 2025). In FoE, the gating network gives an explicit expert-weight decomposition $1\times 1$76, while FrugalFoE casts expert selection as a sequential decision problem with a graph-search view (Wang et al., 2023).

A recurring misconception is that the only useful fusion point is the final output. The medical study states that ensembling at the final softmax cannot correct mis-detected features early in the network, whereas SoE aligns semantically high-level features through learnable bridges and averages aligned features rather than merely class probabilities (Guijt et al., 19 Dec 2025). Conversely, the FoE results show that output-level fusion can still be highly effective when the fusion module is trained rather than naïvely averaged (Wang et al., 2023). Taken together, the literature indicates that SoE is not tied to a single fusion depth; instead, the operative question is which level of representation offers the best trade-off among modularity, trainable parameter count, and cross-domain transfer.