Mixture of Chiral Transformer Experts (MoCTE)

• The paper introduces MoCTE as the illumination-conditioned core of MoCTEFuse, employing two expert subnetworks for high and low illumination levels.
• It leverages Chiral Transformer Fusion Blocks with asymmetric cross-attention to dynamically fuse visible and infrared features while reducing modality bias.
• The architecture integrates an illumination-sensitive gating network and competitive learning loss to enhance fusion quality and downstream detection accuracy.

Mixture of Chiral Transformer Experts (MoCTE) is the expert-fusion core of MoCTEFuse, a dynamic multi-level network for infrared and visible image fusion designed to account explicitly for illumination variation, which the underlying work identifies as a source of modality bias when ignored in fusion pipelines (Jinfu et al., 27 Jul 2025). In this formulation, MoCTE comprises two parallel expert subnetworks for high and low illumination, each built from stacked Chiral Transformer Fusion Blocks (CTFBs), while an illumination-sensitive gating network weights their outputs to preserve texture details and object contrasts in balance (Jinfu et al., 27 Jul 2025).

1. System-level formulation within MoCTEFuse

MoCTEFuse is organized into three main parts. The first part consists of two encoders, one for each modality. Each encoder begins with a $3 \times 3$ convolution and LReLU, followed by cascaded Residual Transformer Blocks (RTBs) and Residual Dense Blocks (RDBs), producing multi-level features $$Z_{vi}^{in}, Z_{ir}^{in} \in \mathbb{R}^{H \times W \times C}$$ (Jinfu et al., 27 Jul 2025). The second part is the Mixture of Chiral Transformer Experts, which contains two parallel expert subnetworks: HI-MoCTE for high illumination, where the visible branch is primary and infrared is auxiliary, and LI-MoCTE for low illumination, where infrared is primary and visible is auxiliary. Each expert stacks $L$ Chiral Transformer Fusion Blocks to progressively fuse and refine cross-modal features. The third part is an illumination-sensitive gating network implemented with ResNet18, which computes the probabilities $P_H$ and $P_L$ of high and low illumination from $I_{vi}$ and uses them to weight the outputs of HI-MoCTE and LI-MoCTE (Jinfu et al., 27 Jul 2025).

The architecture therefore couples feature extraction, illumination-conditioned expert selection, and multi-stage cross-modal refinement in a single pipeline. The paper positions this design against methods that directly merge source images without modeling illumination changes, arguing that such omission leads to modality bias in the fused results (Jinfu et al., 27 Jul 2025).

2. Chiral expert structure and asymmetric cross-attention

The defining mechanism of MoCTE is the Chiral Transformer Fusion Block. In each CTFB, linear projection and shifted window partition with window size $M \times M$ project $Z_{vi}^{in}$ and $Z_{ir}^{in}$ to token sequences, after which an Asymmetric Cross-Attention (ACA) module fuses primary $\leftarrow$ auxiliary information. The block then applies a feed-forward network composed of two MLP layers and GeLU, together with residual connections and LayerNorm (Jinfu et al., 27 Jul 2025).

The “chiral” property is realized by directional asymmetry in attention. Given a local window of size $$Z_{vi}^{in}, Z_{ir}^{in} \in \mathbb{R}^{H \times W \times C}$$0 from primary $$Z_{vi}^{in}, Z_{ir}^{in} \in \mathbb{R}^{H \times W \times C}$$1 and auxiliary $$Z_{vi}^{in}, Z_{ir}^{in} \in \mathbb{R}^{H \times W \times C}$$2, the model computes

$$Z_{vi}^{in}, Z_{ir}^{in} \in \mathbb{R}^{H \times W \times C}$$3

and

$$Z_{vi}^{in}, Z_{ir}^{in} \in \mathbb{R}^{H \times W \times C}$$4

where $$Z_{vi}^{in}, Z_{ir}^{in} \in \mathbb{R}^{H \times W \times C}$$5 are learnable. The keys and values are concatenated,

$$Z_{vi}^{in}, Z_{ir}^{in} \in \mathbb{R}^{H \times W \times C}$$6

and ACA is defined as

$$Z_{vi}^{in}, Z_{ir}^{in} \in \mathbb{R}^{H \times W \times C}$$7

with learnable relative position bias $$Z_{vi}^{in}, Z_{ir}^{in} \in \mathbb{R}^{H \times W \times C}$$8 (Jinfu et al., 27 Jul 2025). In HI-CTFB, $$Z_{vi}^{in}, Z_{ir}^{in} \in \mathbb{R}^{H \times W \times C}$$9 and $L$0; in LI-CTFB, the assignment is reversed. The multi-head extension splits $L$1, $L$2, and $L$3 into $L$4 heads of dimension $L$5, applies the same operation, and concatenates the results (Jinfu et al., 27 Jul 2025).

This asymmetry is central to the expert definition. Queries originate only from the primary modality, whereas keys and values pool both modalities. As a result, HI-MoCTE and LI-MoCTE are not merely two copies of the same transformer; they differ in which modality drives attention and which modality acts as auxiliary support.

3. Illumination-gated expert mixture

The illumination-sensitive sub-network takes $L$6 and outputs

$L$7

The gating weights are defined as

$L$8

The final fused image is then obtained by weighting the expert outputs $L$9 and $P_H$0: $P_H$1 At each stage $P_H$2 of the $P_H$3 stacked CTFBs, each expert produces feature maps $P_H$4 and $P_H$5, which may be fused immediately or only at the final stage through the same gating weights,

$P_H$6

In practice, the paper states that the final output after the last CTFB is mixed as $P_H$7 (Jinfu et al., 27 Jul 2025).

A common misunderstanding would be to regard the two experts as interchangeable branches. The paper defines them as illumination-specific and modality-asymmetric: under high illumination, the visible branch is primary; under low illumination, the infrared branch is primary (Jinfu et al., 27 Jul 2025). This suggests that expert specialization is tied both to scene illumination and to the role assignment of the two sensing modalities.

4. Multi-level processing pipeline and implementation parameters

The algorithmic flow for one multi-level stage is specified as follows. The input pair $P_H$8 is first encoded to $P_H$9 and $P_L$0. For $P_L$1, the model runs HI-CTFB to produce $P_L$2 and LI-CTFB to produce $P_L$3, then updates the intermediate representations. Each expert is decoded to $P_L$4 and $P_L$5, for example through convolutional layers. The gating network computes $P_L$6 and $P_L$7 from $P_L$8, and the final fusion step forms $P_L$9 (Jinfu et al., 27 Jul 2025).

The key hyperparameters reported for MoCTEFuse are: window size $I_{vi}$0, feature dimension $I_{vi}$1, number of attention heads $I_{vi}$2 with $I_{vi}$3, and number of CTFB stages $I_{vi}$4, which is described as empirically chosen. Training uses batch size $I_{vi}$5, $I_{vi}$6 epochs, and Adam with a warmup plus cosine learning-rate schedule. The loss weights are $I_{vi}$7, $I_{vi}$8, and $I_{vi}$9 (Jinfu et al., 27 Jul 2025).

These choices define the operational scale of the architecture. In particular, the use of stacked CTFBs at multiple stages is presented as a mechanism to progressively aggregate and refine modality-specific and cross-modality information (Jinfu et al., 27 Jul 2025).

5. Training objectives and competitive learning

Training separates the illumination-sensitive sub-network from the fusion network. For illumination prediction, the ResNet18 gating module uses Binary Cross-Entropy: $M \times M$0 where $M \times M$1 for high illumination and $M \times M$2 otherwise (Jinfu et al., 27 Jul 2025).

For fusion, the paper introduces a Competitive Learning Loss built from three per-image sub-losses. The intensity term is

$M \times M$3

The gradient term is

$M \times M$4

The structural term is

$M \times M$5

with $M \times M$6. These are combined as

$M \times M$7

where $M \times M$8 (Jinfu et al., 27 Jul 2025).

For each expert $M \times M$9, the output $Z_{vi}^{in}$0 denotes $Z_{vi}^{in}$1, and

$Z_{vi}^{in}$2

The final competitive fusion loss is defined as

$Z_{vi}^{in}$3

Its gradient is described as dynamically allocating larger backward weight to the currently better-performing expert (Jinfu et al., 27 Jul 2025).

Within the reported framework, this loss design integrates illumination distributions with three levels of sub-loss terms. A plausible implication is that the optimization objective couples expert specialization to both gating probabilities and instantaneous reconstruction quality, rather than training both experts under a fixed, symmetric penalty.

6. Empirical findings, ablations, and downstream detection

The reported experiments cover the DroneVehicle, MSRS, TNO, and RoadScene datasets, where MoCTEFuse is stated to show superior fusion performance (Jinfu et al., 27 Jul 2025). The paper’s ablation study on MSRS attributes a distinct role to each component. Removing HI-MoCTE, implemented as forcing $Z_{vi}^{in}$4, reduces EN, SD, MI, and VIF by up to $Z_{vi}^{in}$5. Removing LI-MoCTE, implemented as $Z_{vi}^{in}$6, similarly degrades all metrics. Replacing the competitive loss with the simple weighted sum of Eq.10 hurts especially MI, with a drop of approximately $Z_{vi}^{in}$7, and VIF, with a drop of approximately $Z_{vi}^{in}$8 (Jinfu et al., 27 Jul 2025).

The paper interprets these results as confirmation that both experts and the competition mechanism are critical for balanced fusion under varying illumination (Jinfu et al., 27 Jul 2025). More narrowly, the ablations indicate that neither illumination regime can be collapsed into the other without measurable loss, and that the competitive objective contributes beyond the three-term reconstruction loss alone.

For downstream detection, the fused images are reported to achieve the best detection mean Average Precision. On the MFNet general-detection benchmark with $Z_{vi}^{in}$9 manually annotated images, MoCTEFuse fused images yield $Z_{ir}^{in}$0 and $Z_{ir}^{in}$1, surpassing all prior fusion baselines by more than $Z_{ir}^{in}$2 to $Z_{ir}^{in}$3. On the DroneVehicle oriented detection test, $Z_{ir}^{in}$4 improves from $Z_{ir}^{in}$5 for infrared alone and $Z_{ir}^{in}$6 for visible alone to $Z_{ir}^{in}$7 after MoCTEFuse (Jinfu et al., 27 Jul 2025). The code and model are released at the project repository specified in the paper (Jinfu et al., 27 Jul 2025).

Taken together, the reported evidence presents MoCTE as the mechanism through which MoCTEFuse couples twin expert streams, illumination gating, asymmetric cross-attention, and competitive loss to adapt dynamically to lighting variations (Jinfu et al., 27 Jul 2025). Within the scope of the published results, the method is framed not simply as a fusion backbone but as an illumination-conditioned expert system whose value is evaluated both by image-fusion metrics and by downstream detection accuracy.