---
title: Convolution-based Mixture of Experts (CMoE)
url: https://www.emergentmind.com/topics/convolution-based-mixture-of-experts-cmoe
type: topic
---

# Convolution-based Mixture of Experts (CMoE)

Convolution-based Mixture of Experts (CMoE) is an architectural paradigm that integrates the Mixture of Experts (MoE) framework with convolutional and MLP-based models, enabling structured, input-adaptive routing and specialization of sub-networks. This approach exploits a collection of independent expert networks and a dynamic routing mechanism, facilitating fine-grained modeling of complex, heterogeneous input spaces. CMoE variants have been proposed for domains such as underwater acoustic target recognition, where conventional approaches struggle with high intra-class diversity and distributional complexity [2402.11919]. Related methods extend the paradigm to per-channel dynamic gating in convolutional networks, yielding computational savings and improvements in accuracy on vision tasks [1806.01531].

## 1. Architectural Framework

At its core, CMoE comprises three key components: a backbone feature extractor, a bank of expert MLPs, and a routing/gating mechanism.

- **Feature Backbone:** For underwater acoustic target recognition, the input $x \in \mathbb{R}^{C \times T \times F}$ (e.g., spectrograms) is mapped to a fixed-dimensional embedding $r = F(x) \in \mathbb{R}^{512}$ by a customized ResNet-AP backbone [2402.11919].
- **Expert Networks:** A set of $m$ experts $\{E_1, \ldots, E_m\}$, each implemented as a two-layer MLP with batch normalization and ReLU activation. Mathematical form:
  $$
  E_j(r) = W_j^{(2)} \, \sigma( BN( W_j^{(1)} r + b_j^{(1)} ) ) + b_j^{(2)}
  $$
  where $W_j^{(1)} \in \mathbb{R}^{128 \times 512}$, $W_j^{(2)} \in \mathbb{R}^{C \times 128}$, $b_j^{(1)}, b_j^{(2)}$ are bias parameters, $\sigma(\cdot) = \operatorname{ReLU}(\cdot)$, and $C$ is the number of classes.
- **Gating/Routing Network:** A single linear layer $G: \mathbb{R}^{512} \to \mathbb{R}^m$ parameterized by $U \in \mathbb{R}^{m \times 512}$, $c \in \mathbb{R}^m$, which generates expert scores $s = G(r) = U r + c$. Selection probabilities are computed as $p_j = \exp(s_j) / \sum_{k=1}^m \exp(s_k)$, with one expert selected by $e_i = \arg\max_j p_i[j]$ for each sample.

An optional residual expert $E_R$ (not gated) can be included, producing logits $\mathrm{logits}_{\mathrm{res}} = E_R(r)$. The final logits are
\[
\begin{cases}
\text{CMoE: } \mathrm{logits} = \mathrm{logits}_{\mathrm{exp}}\\
\text{RCMoE: } \mathrm{logits} = \mathrm{logits}_{\mathrm{exp}} + \mathrm{logits}_{\mathrm{res}}
\end{cases}
\]
This structure provides independent parameter spaces for experts and enables specialization for highly variable input distributions [2402.11919].

## 2. Mathematical Formulation and Learning Objective

The CMoE loss function integrates standard cross-entropy loss with a regularization term enforcing balanced utilization among experts. For a batch of $n$ samples:

- **Routing statistics:**
  - $ef_j = \frac{1}{n} |\{i: e_i = j\}|$ (fraction of samples routed to expert $j$)
  - $ep_j = \frac{1}{n} \sum_{i=1}^n p_i[j]$ (average soft assignment to expert $j$)
- **Balancing regularization:**
  $$
  \mathcal{L}_{\text{balance}} = \alpha m \sum_{j=1}^m (ef_j \cdot ep_j)
  $$
  with a typical value $\alpha = 10^{-2}$.

The overall loss:
\[
\mathcal{L} = \frac{1}{n} \sum_{i=1}^n \mathrm{CE}(\mathrm{logits}_i, y_i) + \alpha m \sum_{j=1}^m ef_j \cdot ep_j
\]
where $\mathrm{CE}(\cdot,\cdot)$ denotes the cross-entropy loss.

Optimization proceeds via AdamW with a constant learning rate (maximum $5 \times 10^{-4}$), weight decay $10^{-5}$, and a training horizon of 200 epochs, with batch size and scheduling tuned on validation splits [2402.11919].

## 3. Computational Structure and Connections to Other MoE Approaches

The convolution-based mixture of experts described above operates at the level of global embeddings and expert networks. In contrast, DeepMoE [1806.01531] applies the MoE principle locally to all convolutional layers within a deep model:

- Every convolutional layer is reinterpreted as a sum of “expert” input channels, with a per-layer gating vector $g^l \in \mathbb{R}^{C^\mathrm{in}_l}$ computing dynamic selection and scaling.
- Gating vectors are derived from a small shared embedding network $M(x)$, with per-layer independent heads.
- The approach enables per-example, per-layer dynamic sparsification, leading to computational thrift while maintaining or expanding representational capacity.
- Sparse execution is encouraged via an $\ell_1$ penalty on the gates.
- Empirical studies confirm accuracy gains and FLOP reductions in vision benchmarks.

A key distinction: the CMoE [2402.11919] approach gates at the level of whole-expert MLPs using high-level embeddings, whereas DeepMoE [1806.01531] gates individual channels or groups at each convolutional layer.

## 4. Experimental Validation and Quantitative Results

In the context of underwater acoustic target recognition [2402.11919], CMoE was evaluated on Shipsear (9 classes), DTIL (2 classes, private), and DeepShip (4 classes) datasets. Acoustic features included STFT, Mel spectrogram, Bark spectrogram, and CQT spectrogram. The evaluation metric was segment-level accuracy, with train/test splits separated by audio track to prevent leakage.

Performance summary (segment-level accuracy):

| Dataset/Feature         | Baseline (ResNet-AP) | CMoE       | CMoE + balance | RCMoE + balance |
|------------------------ |--------------------- |------------|----------------|-----------------|
| Shipsear (STFT)         | 75.24                | 84.91      | 86.21          | 85.34           |
| Shipsear (Mel)          | 77.14                | 83.59      | 85.35          | 84.48           |
| Shipsear (Bark)         | 72.86                | 81.33      | 84.48          | 83.62           |
| Shipsear (CQT)          | 73.33                | 80.48      | 82.76          | 82.76           |
| DTIL (STFT)             | 95.93                | 96.61      | 97.89          | 98.17           |
| DeepShip (CQT)          | 77.82                | 77.09      | 79.62          | 78.76           |

These results indicate absolute accuracy gains of 5–10% on Shipsear and significant improvements on DTIL and DeepShip. In fine-grained analyses, CMoE achieved close to 100% accuracy on under-represented small classes where baselines underperformed (sub-50%) [2402.11919].

Ablation studies confirmed that:
- The balancing regularization prevents under-training or collapse of individual experts.
- The optional residual expert benefits recovery from routings errors but at times reduces sparsity advantages.
- The optimal number of experts depends on dataset size and feature redundancy; higher $m$ risks overfitting in small-data regimes.

## 5. Interpretability and Visualization

Extensive visualization elucidated the internal dynamics of CMoE [2402.11919]:

- **Expert clustering:** Heatmaps revealed that, for $m \in \{4,8\}$, large-vessel classes (e.g., ocean liner, ro-ro) preferentially route to distinct experts versus small boats, reflecting emergent specialization tied to acoustic characteristics.
- **Per-sample routing:** Routings for spectrograms exhibiting different within-class motion states (e.g., motorboat start/stop, arrival, passing) showed shared and unique expert routes, demonstrating disentanglement of intra-class diversity.
- **Inter-class discrimination:** Visually similar spectrograms from different classes (e.g., motorboat vs. passenger-ship) were routed to distinct experts, indicating resilience to inter-class similarity.

This suggests that the gating network learns semantically meaningful and task-aligned partitions, with expert assignment reflecting physically relevant factors such as vessel type or motion.

## 6. Comparative Perspective and Extensions

The convolution-based mixture of experts paradigm enables both increased model expressivity and efficient computation by restricting the activated parameter subspace per input. Within deep vision models [1806.01531], treating channels as experts and leveraging per-layer sparse gating realizes an effective exponential ensemble of subnetworks under dynamic selection. This yields higher accuracy and computational economies compared to both static channel pruning and naive widening strategies.

Empirical evidence indicates that multi-layer application of MoE outperforms single-point gating and, when paired with careful regularization, preserves computational cost at a fixed or improved accuracy.

Proposed future directions for CMoE-type models include:
- Incorporation of physically interpretable routing signals (e.g., propeller blade count relevant for vessel classification).
- Exploration of deeper, multi-stage expert/routing hierarchies.
- Extension to structured operators and attention mechanisms by leveraging learned global or hierarchical embeddings for gating [1806.01531, 2402.11919].

The CMoE approach represents a significant advance in the fine-grained modeling of complex, heterogenous data regimes, demonstrating improved robustness and accuracy in challenging real-world acoustic environments.

Source: https://www.emergentmind.com/topics/convolution-based-mixture-of-experts-cmoe