MoCTEFuse: Illumination-Aware Image Fusion

• MoCTEFuse is a dynamic multi-level image fusion architecture that employs an illumination-sensitive gating network to adaptively route inputs through specialized Chiral Transformer Experts.
• It uses parallel shallow and deep encoders along with expert subnetworks (HI-MoCTE and LI-MoCTE) to optimally balance visible and infrared features under varying lighting conditions.
• The design integrates asymmetric cross-attention through Chiral Transformer Fusion Blocks and a competitive loss formulation to ensure robust and balanced feature fusion for downstream tasks.

MoCTEFuse denotes a dynamic multi-level image fusion architecture that employs an illumination-gated Mixture of Chiral Transformer Experts to adaptively preserve both texture details and object contrast for the task of infrared and visible image fusion. It directly addresses illumination effects—specifically, how the optimal weighting of the visible and infrared modalities depends on ambient lighting—by integrating a learned gating mechanism that routes each sample through expert subnetworks specialized for either high- or low-illumination conditions. MoCTEFuse is constructed from two primary innovations: the Mixture-of-Chiral-Transformer-Experts (MoCTE) module and the Chiral Transformer Fusion Block (CTFB), each designed to enable context-sensitive fusion via asymmetric cross-attention and multi-level feature hierarchies (Jinfu et al., 27 Jul 2025).

1. Architectural Overview

MoCTEFuse takes as input a visible image $I_{vi}$ and an infrared image $I_{ir}$, which pass through parallel shallow and deep encoders to produce feature maps $Z_{vi}^0$ and $Z_{ir}^0$. The structure encompasses:

• Illumination-Sensitive Gating Network: A ResNet-18 classifier analyzes $I_{vi}$ to output probabilities $P_H$ (high-illumination/day) and $P_L = 1-P_H$ (low-illumination/night), functioning as gating coefficients.
• Mixture-of-Experts: Two subnetworks—HI-MoCTE (favoring visible detail in bright scenes) and LI-MoCTE (favoring infrared contrast in dark scenes)—process the encoded features.
• Output Fusion: The final fused image is given by $I_f = P_H \cdot I_f^H + P_L \cdot I_f^L$, where $I_f^H$ and $I_f^L$ are the outputs of HI-MoCTE and LI-MoCTE, respectively.

This structure promotes adaptive fusion: under day/night extremes, a single expert dominates; under intermediate illumination, both contribute proportionally.

2. Mixture of Chiral Transformer Experts (MoCTE)

Gating Signal: The gating probability $I_{ir}$0 is produced by the ResNet-18 gating net, trained with a binary cross-entropy loss

$I_{ir}$1

with $I_{ir}$2 denoting ground-truth illumination.

Expert Subnetwork Roles:

• HI-MoCTE: For $I_{ir}$3, processes features with visible as "primary" and infrared as "auxiliary."
• LI-MoCTE: For $I_{ir}$4, processes with infrared as "primary" and visible as "auxiliary."

Both experts comprise $I_{ir}$5 stacked Chiral Transformer Fusion Blocks, each performing modality-refined feature fusion.

3. Chiral Transformer Fusion Block (CTFB)

Asymmetric Cross-Attention:

In each local window, visible and infrared features $I_{ir}$6 and $I_{ir}$7 generate queries, keys, and values: $I_{ir}$8 (similarly for $I_{ir}$9). The shared key/value bank is formed via channel-concatenation. For primary modality (e.g., visible),

$Z_{vi}^0$0

$Z_{vi}^0$1 is a trainable relative positional bias. The "chiral" property refers to primary/auxiliary modality reversal between experts.

Block Structure:

Each block iteratively applies asymmetric cross-attention, add-skip, layer normalization, and a two-layer GeLU-based feedforward MLP to both modalities.

Modality Weighting:

Although each CTFB designates a primary modality locally, the network globally fuses outputs from both experts via the illumination-weighted sum.

4. Multi-Level Progressive Aggregation

MoCTEFuse organizes CTFBs in a hierarchical fashion. At stage $Z_{vi}^0$2, the block receives features $Z_{vi}^0$3 and outputs refined features for each modality. This multi-stage stacking enables coarse edge alignment in early layers and gradual refinement of detailed texture or contrast in later stages. After $Z_{vi}^0$4 levels, the deepest fused features are upsampled and decoded to spatial images $Z_{vi}^0$5 and $Z_{vi}^0$6.

5. Competitive Loss Formulation

Three sub-losses are computed for each expert output $Z_{vi}^0$7:

• Intensity Loss: $Z_{vi}^0$8
• Gradient Loss: $Z_{vi}^0$9
• Structural Loss: $Z_{ir}^0$0 (with $Z_{ir}^0$1)

The per-expert weighted loss is

$Z_{ir}^0$2

and the overall competitive fusion loss is

$Z_{ir}^0$3

encouraging dominance of the better expert (with lower loss) in gradient backpropagation.

6. Training Protocol and Empirical Results

Training Schedule:

• Gating Net: ResNet-18 is trained for 100 epochs with BCE on annotated day/night data.
• Fusion Network: The gating net is frozen, and MoCTEFuse is trained for 60 epochs (batch size 8, Adam optimizer, warm-up cosine schedule, initial learning rate 1e-4), input patches resized to $Z_{ir}^0$4, loss weights $Z_{ir}^0$5, window size $Z_{ir}^0$6.

Datasets Evaluated:

• DroneVehicle (4,641 train/256 test)
• MSRS (1,083 train/361 test)
• TNO (21 test pairs)
• RoadScene (221 test pairs)
• MFNet (80 test pairs for object detection)

Quantitative Results:

MoCTEFuse achieves top performance on reference-free fusion metrics (Entropy, SD, MI, VIF) compared with twelve prior methods. For downstream detection:

Dataset	Detector	AP₀.₅	AP₀.₅:₀.₉₅
MFNet	YOLOX-s	92.80%	70.93%
DroneVehicle	PSCD+YoloV5s	71.94%	45.14%

Qualitatively, it preserves edge sharpness in low-light, maintains visible texture in daylight, and avoids ghosting or modality dominance.

7. Significance and Context

MoCTEFuse demonstrates a principled approach to illumination-aware modality fusion, systematically routing data through specialized "chiral" experts and fusing their outputs via an adaptive, data-driven gating mechanism. Its architectural design, leveraging cross-attention and hierarchical feature aggregation, contrasts with static or naïvely weighted fusion baselines. The competitive loss formulation further incentives expert specialization and balanced content preservation, directly supporting robust transfer to downstream tasks such as object detection under variable lighting. These features contribute to its state-of-the-art performance profile on both fusion quality metrics and high-level vision benchmarks (Jinfu et al., 27 Jul 2025).