---
title: Mixture of Chiral Transformer Experts (MoCTE)
url: https://www.emergentmind.com/topics/mixture-of-chiral-transformer-experts-mocte
type: topic
---

# Mixture of Chiral Transformer Experts (MoCTE)

Mixture of Chiral Transformer Experts (MoCTE) is the expert-fusion core of MoCTEFuse, a dynamic multi-level network for infrared and visible image fusion designed to account explicitly for illumination variation, which the underlying work identifies as a source of modality bias when ignored in fusion pipelines [2507.20180]. In this formulation, MoCTE comprises two parallel expert subnetworks for high and low illumination, each built from stacked Chiral Transformer Fusion Blocks (CTFBs), while an illumination-sensitive gating network weights their outputs to preserve texture details and object contrasts in balance [2507.20180].

## 1. System-level formulation within MoCTEFuse

MoCTEFuse is organized into three main parts. The first part consists of two encoders, one for each modality. Each encoder begins with a \(3 \times 3\) convolution and LReLU, followed by cascaded Residual Transformer Blocks (RTBs) and Residual Dense Blocks (RDBs), producing multi-level features \(Z_{vi}^{in}, Z_{ir}^{in} \in \mathbb{R}^{H \times W \times C}\) [2507.20180]. The second part is the Mixture of Chiral Transformer Experts, which contains two parallel expert subnetworks: HI-MoCTE for high illumination, where the visible branch is primary and infrared is auxiliary, and LI-MoCTE for low illumination, where infrared is primary and visible is auxiliary. Each expert stacks \(L\) Chiral Transformer Fusion Blocks to progressively fuse and refine cross-modal features. The third part is an illumination-sensitive gating network implemented with ResNet18, which computes the probabilities \(P_H\) and \(P_L\) of high and low illumination from \(I_{vi}\) and uses them to weight the outputs of HI-MoCTE and LI-MoCTE [2507.20180].

The architecture therefore couples feature extraction, illumination-conditioned expert selection, and multi-stage cross-modal refinement in a single pipeline. The paper positions this design against methods that directly merge source images without modeling illumination changes, arguing that such omission leads to modality bias in the fused results [2507.20180].

## 2. Chiral expert structure and asymmetric cross-attention

The defining mechanism of MoCTE is the Chiral Transformer Fusion Block. In each CTFB, linear projection and shifted window partition with window size \(M \times M\) project \(Z_{vi}^{in}\) and \(Z_{ir}^{in}\) to token sequences, after which an Asymmetric Cross-Attention (ACA) module fuses primary \(\leftarrow\) auxiliary information. The block then applies a feed-forward network composed of two MLP layers and GeLU, together with residual connections and LayerNorm [2507.20180].

The “chiral” property is realized by directional asymmetry in attention. Given a local window of size \(M^2 \times C\) from primary \(X_p\) and auxiliary \(X_a\), the model computes
\[
Q_p = X_p W_p^Q,\quad K_p = X_p W_p^K,\quad V_p = X_p W_p^V
\]
and
\[
Q_a = X_a W_a^Q,\quad K_a = X_a W_a^K,\quad V_a = X_a W_a^V,
\]
where \(W^Q, W^K, W^V \in \mathbb{R}^{C \times C}\) are learnable. The keys and values are concatenated,
\[
K = [K_p; K_a],\quad V = [V_p; V_a],
\]
and ACA is defined as
\[
ACA(X_p, X_a) = \mathrm{softmax}\!\left(\frac{Q_p K^T}{\sqrt{d_k}} + B\right)V,
\]
with learnable relative position bias \(B\) [2507.20180]. In HI-CTFB, \(X_p = Z_{vi}\) and \(X_a = Z_{ir}\); in LI-CTFB, the assignment is reversed. The multi-head extension splits \(Q\), \(K\), and \(V\) into \(h\) heads of dimension \(d_k = C/h\), applies the same operation, and concatenates the results [2507.20180].

This asymmetry is central to the expert definition. Queries originate only from the primary modality, whereas keys and values pool both modalities. As a result, HI-MoCTE and LI-MoCTE are not merely two copies of the same transformer; they differ in which modality drives attention and which modality acts as auxiliary support.

## 3. Illumination-gated expert mixture

The illumination-sensitive sub-network takes \(I_{vi}\) and outputs
\[
P_H = Pr(HighIllum \mid I_{vi}), \quad P_L = Pr(LowIllum \mid I_{vi}), \quad P_H + P_L = 1.
\]
The gating weights are defined as
\[
g_H = P_H, \quad g_L = P_L.
\]
The final fused image is then obtained by weighting the expert outputs \(I_f^H\) and \(I_f^L\):
\[
I_f = g_H \cdot I_f^H + g_L \cdot I_f^L.
\]
At each stage \(\ell\) of the \(L\) stacked CTFBs, each expert produces feature maps \(F_\ell^H\) and \(F_\ell^L\), which may be fused immediately or only at the final stage through the same gating weights,
\[
F_\ell = g_H \cdot F_\ell^H + g_L \cdot F_\ell^L.
\]
In practice, the paper states that the final output after the last CTFB is mixed as \(I_f = g_H \cdot I_f^H + g_L \cdot I_f^L\) [2507.20180].

A common misunderstanding would be to regard the two experts as interchangeable branches. The paper defines them as illumination-specific and modality-asymmetric: under high illumination, the visible branch is primary; under low illumination, the infrared branch is primary [2507.20180]. This suggests that expert specialization is tied both to scene illumination and to the role assignment of the two sensing modalities.

## 4. Multi-level processing pipeline and implementation parameters

The algorithmic flow for one multi-level stage is specified as follows. The input pair \((I_{vi}, I_{ir})\) is first encoded to \(Z_{vi}^{in}\) and \(Z_{ir}^{in}\). For \(\ell = 1 \ldots L\), the model runs HI-CTFB to produce \(Z_{vi}^H, Z_{ir}^H\) and LI-CTFB to produce \(Z_{vi}^L, Z_{ir}^L\), then updates the intermediate representations. Each expert is decoded to \(I_f^H\) and \(I_f^L\), for example through convolutional layers. The gating network computes \(P_H\) and \(P_L\) from \(I_{vi}\), and the final fusion step forms \(I_f = P_H \cdot I_f^H + P_L \cdot I_f^L\) [2507.20180].

The key hyperparameters reported for MoCTEFuse are: window size \(M = 8\), feature dimension \(C = 64\), number of attention heads \(h = 4\) with \(d_k = 16\), and number of CTFB stages \(L = 4\), which is described as empirically chosen. Training uses batch size \(8\), \(60\) epochs, and Adam with a warmup plus cosine learning-rate schedule. The loss weights are \(\alpha = 1\), \(\beta = 5\), and \(\gamma = 10\) [2507.20180].

These choices define the operational scale of the architecture. In particular, the use of stacked CTFBs at multiple stages is presented as a mechanism to progressively aggregate and refine modality-specific and cross-modality information [2507.20180].

## 5. Training objectives and competitive learning

Training separates the illumination-sensitive sub-network from the fusion network. For illumination prediction, the ResNet18 gating module uses Binary Cross-Entropy:
\[
L_{ISSN} = - y \cdot \log(P_H) - (1-y) \cdot \log(P_L),
\]
where \(y = 1\) for high illumination and \(0\) otherwise [2507.20180].

For fusion, the paper introduces a Competitive Learning Loss built from three per-image sub-losses. The intensity term is
\[
L_{int} = \frac{1}{HW}\left(\lVert I_f - I_{ir} \rVert_1 + \lVert I_f - I_{vi} \rVert_1\right).
\]
The gradient term is
\[
L_{grad} = \frac{1}{HW}\left\lVert |\nabla I_f| - \max(|\nabla I_{ir}|, |\nabla I_{vi}|) \right\rVert_1.
\]
The structural term is
\[
L_{ssim} = w_1(1-SSIM(I_f, I_{ir})) + w_2(1-SSIM(I_f, I_{vi})),
\]
with \(w_1 = w_2 = 0.5\). These are combined as
\[
L_{total} = \alpha L_{int} + \beta L_{grad} + \gamma L_{ssim},
\]
where \((\alpha,\beta,\gamma) = (1,5,10)\) [2507.20180].

For each expert \(i \in \{H,L\}\), the output \(o_i\) denotes \(I_f^i\), and
\[
L(o_i) = L_{total}(I_f^i, I_{ir}, I_{vi}).
\]
The final competitive fusion loss is defined as
\[
L_{fusion} = - \ln\!\left(\sum_{i \in \{H,L\}} P_i \exp(-L(o_i))\right).
\]
Its gradient is described as dynamically allocating larger backward weight to the currently better-performing expert [2507.20180].

Within the reported framework, this loss design integrates illumination distributions with three levels of sub-loss terms. A plausible implication is that the optimization objective couples expert specialization to both gating probabilities and instantaneous reconstruction quality, rather than training both experts under a fixed, symmetric penalty.

## 6. Empirical findings, ablations, and downstream detection

The reported experiments cover the DroneVehicle, MSRS, TNO, and RoadScene datasets, where MoCTEFuse is stated to show superior fusion performance [2507.20180]. The paper’s ablation study on MSRS attributes a distinct role to each component. Removing HI-MoCTE, implemented as forcing \(P_H = 0\), reduces EN, SD, MI, and VIF by up to \(0.12/1.0/0.15/0.02\). Removing LI-MoCTE, implemented as \(P_L = 0\), similarly degrades all metrics. Replacing the competitive loss with the simple weighted sum of Eq.10 hurts especially MI, with a drop of approximately \(0.36\), and VIF, with a drop of approximately \(0.10\) [2507.20180].

The paper interprets these results as confirmation that both experts and the competition mechanism are critical for balanced fusion under varying illumination [2507.20180]. More narrowly, the ablations indicate that neither illumination regime can be collapsed into the other without measurable loss, and that the competitive objective contributes beyond the three-term reconstruction loss alone.

For downstream detection, the fused images are reported to achieve the best detection mean Average Precision. On the MFNet general-detection benchmark with \(80\) manually annotated images, MoCTEFuse fused images yield \(AP_{0.5} = 92.80\%\) and \(AP_{0.5:0.95} = 70.93\%\), surpassing all prior fusion baselines by more than \(2\%\) to \(5\%\). On the DroneVehicle oriented detection test, \(AP_{0.5:0.95}\) improves from \(31.32\%\) for infrared alone and \(22.36\%\) for visible alone to \(45.14\%\) after MoCTEFuse [2507.20180]. The code and model are released at the project repository specified in the paper [2507.20180].

Taken together, the reported evidence presents MoCTE as the mechanism through which MoCTEFuse couples twin expert streams, illumination gating, asymmetric cross-attention, and competitive loss to adapt dynamically to lighting variations [2507.20180]. Within the scope of the published results, the method is framed not simply as a fusion backbone but as an illumination-conditioned expert system whose value is evaluated both by image-fusion metrics and by downstream detection accuracy.

Source: https://www.emergentmind.com/topics/mixture-of-chiral-transformer-experts-mocte