---
title: 'SMEAR-MoE: Soft Merging in MoE Architectures'
url: https://www.emergentmind.com/topics/smear-moe
type: topic
---

# SMEAR-MoE: Soft Merging in MoE Architectures

SMEAR-MoE (Soft Merging of Experts with Adaptive Routing – Mixture-of-Experts) is a fully differentiable, modular neural architecture that addresses classical challenges in sparse expert selection and conditional computation. SMEAR-MoE achieves adaptive routing by constructing a merged expert through a weighted average of all expert parameters, providing dense gradient flow, stability, and efficiency in both vision-language and speech recognition domains. The approach eliminates the need for high-variance gradient estimation techniques and improves empirical performance and expert specialization relative to conventional discrete sparse routing. SMEAR-MoE has been validated in large-scale natural language understanding and multilingual automatic speech recognition tasks, demonstrating significant accuracy improvements and interpretability.

## 1. Principle of Soft Merging in MoE Architectures

Traditional Mixture-of-Experts (MoE) models enable modularity by maintaining a set of specialized expert subnetworks $\{f_i(\cdot;\theta_i)\}_{i=1}^E$ along with a router mechanism that selects the most relevant expert(s) per input. Discrete routing, such as top-$k$ expert selection, introduces non-differentiability, requiring high-variance gradient estimators (e.g., REINFORCE, Gumbel-Softmax) with inherent optimization difficulties and often adverse effects on training stability and capacity utilization.

SMEAR-MoE replaces the discrete expert selection with a parameter-level soft merge. For a given input activation $u$ and a router whose output is a probability vector $R(v)\in\Delta^E$, the merged expert parameter is computed as
\[
\theta_{\mathrm{merge}} = \sum_{i=1}^E R(v)_i \theta_i,
\]
and the block output is
\[
y = f(u; \theta_{\mathrm{merge}}).
\]
This strategy seamlessly integrates expert selection with parameter interpolation, rendering the entire computational pathway $v \mapsto R(v) \mapsto \theta_{\mathrm{merge}} \mapsto y$ fully differentiable and backpropagation-friendly [2306.03745].

## 2. Mathematical Formulation and Training Dynamics

SMEAR-MoE leverages a lightweight adaptive router—typically a linear mapping plus softmax—which yields a fully differentiable, real-valued distribution:
\[
R(v) = \mathrm{softmax}(Wv + b), \quad R(v) \in \Delta^E,
\]
where $v$ is a pooled latent representation.

Gradients propagate through both the merging operation and the router, e.g.,
\[
\frac{\partial L}{\partial \theta_i} = R(v)_i\,\frac{\partial L}{\partial \theta_{\mathrm{merge}}},
\]
guaranteeing that all experts receive nonzero updates. The router parameters $W$, $b$ are updated via the chain rule, circumventing the reliance on policy gradients common in discrete routing [2306.03745].

The framework is directly compatible with standard losses (e.g., cross-entropy, CTC) and requires no additional regularization for routing. In multilingual ASR, SMEAR-MoE can incorporate load-balancing objectives,
\[
L_\mathrm{load} = \lambda M \sum_{m=1}^M (\bar{g}_m)^2,
\]
to encourage uniform expert usage and prevent expert collapse [2601.19451].

## 3. Application Domains and Architectures

### 3.1 Vision and Language Processing

In NLU settings (e.g., T5-GLUE, ResNet-DomainNet), SMEAR-MoE employs adapter-style blocks or ResNet-based experts, where each expert is architecturally identical. The soft parameter merge enables dynamic utilization of several experts per sample. SMEAR-MoE outperforms oracle tag routing, REINFORCE, Gumbel-Softmax, parameter-matched dense adapters, and expert ensembles in both average GLUE score (81.6% ±1.0) and DomainNet accuracy (62.0% ±0.1), with wall-clock throughput matching top-1 routing [2306.03745].

### 3.2 Multilingual Speech Recognition

In multilingual LLM-based ASR, SMEAR-MoE is instantiated as a stabilized multi-expert projector between a frozen speech encoder and LLM decoder. The projector comprises a convolutional downsampler, $M$ expert MLPs, and a gating network that produces per-token expert weights $G\in\mathbb{R}^{T'\times M}$, averaged to form mixture coefficients $\bar{g}$. The merged expert parameters are then computed, and the resultant projection is input to the LLM.

Empirical evaluations on Indic languages confirm that SMEAR-MoE achieves a relative WER reduction of 7.6% compared to a single-projector baseline, maintains near-identical runtime efficiency, and exhibits clear, interpretable expert specialization aligned with linguistic families [2601.19451].

## 4. Computational Considerations and Scalability

The computational complexity of SMEAR-MoE is near that of a single-expert MoE. For $L$ tokens and $E$ experts with $d\to m$ adapters, SMEAR-MoE requires approximately $4dmL + 2Edm$ FLOPs, where the additional merging cost is negligible for practical values $E \ll L$. In ASR, the real-time factor increase is minimal (from 0.196 for a single projector to 0.198 for SMEAR-MoE), whereas static ensembles incur a more significant slowdown [2306.03745][2601.19451].

Unlike sparse top-$k$ routing, where only winning experts receive gradients, SMEAR-MoE’s merging ensures dense update paths, preventing expert collapse—especially important in low-resource and multilingual regimes.

## 5. Expert Specialization, Interpretability, and Sharing

Qualitative analyses show that SMEAR-MoE’s routing distributions are typically sparse and interpretable. In GLUE, per-block router distributions often recover task-style expert usage, facilitating both task-specific specialization and sharing among semantically similar tasks. In DomainNet, certain visual domains share experts while others remain distinct, reflecting latent structure.

In multilingual ASR, per-language average routing heatmaps demonstrate that SMEAR-MoE recapitulates linguistic groupings (e.g., Hindi and Marathi, both Indo-Aryan, share an expert; Tamil, a Dravidian language, focuses on a different expert), with no explicit family supervision—providing insight into model internals and potential for interpretability in cross-lingual scenarios [2306.03745][2601.19451].

## 6. Limitations and Future Directions

SMEAR-MoE is limited to scenarios with homogeneous expert architectures; heterogeneous experts require parameter-alignment techniques such as permutation matching. In settings demanding fine-grained (per-token) routing for very long sequences, the merge cost per token could reach that of full ensemble evaluation.

Open research avenues include scaling to larger expert sets, exploring advanced weight-merging (e.g., Fisher-weighted averages), integrating LoRA or IA³ adapters, further application in massively wide MoEs (as in Switch Transformers), and extension to token-level SMEAR for granular routing and code-switching in multilingual models [2306.03745][2601.19451].

## 7. Comparative Summary and Impact

The following table summarizes SMEAR-MoE’s empirical advantages relative to alternative MoE and ensemble strategies:

| Model/Setting                | Avg. GLUE (%) | Avg. WER (%) | Throughput    |
|------------------------------|:-------------:|:------------:|:-------------:|
| Tag routing (oracle)         | 78.5          | —            | High          |
| Single projector (ASR)       | —             | 30.3         | 0.196 RTF     |
| Static ensemble              | 81.7          | 28.8         | Low (0.243)   |
| Discrete/Top-$k$ MoE         | <78.5         | 30.6–29.7    | High          |
| Dense Adapter                | 80.2          | 60.8 / —     | High          |
| **SMEAR-MoE**                | **81.6**      | **28.0**     | 0.198 RTF     |

In summary, SMEAR-MoE provides a robust, scalable, and interpretable mechanism for adaptive, modular computation in a variety of expert-based architectures. Its fully differentiable soft-merge approach ensures stable training, circumvents gradient estimation pitfalls, and enables both specialization and sharing consistent with underlying task or linguistic structure [2306.03745][2601.19451].

Source: https://www.emergentmind.com/topics/smear-moe