Chiral Transformer Fusion Block in MoCTEFuse

• CTFB is a fusion unit in MoCTEFuse that employs a windowed Transformer with asymmetric cross-attention to combine infrared and visible features.
• It dynamically assigns primary and auxiliary roles to each modality based on illumination, enhancing image fusion under varying lighting conditions.
• CTFB integrates competitive mixture loss and local window strategies to progressively refine feature integration and improve fusion performance.

Searching arXiv for the named papers and closely related context. arxiv_search query: "(Jinfu et al., 27 Jul 2025) MoCTEFuse Chiral Transformer Fusion Block" Chiral Transformer Fusion Block (CTFB) denotes, in MoCTEFuse, the core fusion unit inside each expert subnetwork of an illumination-gated Mixture of Chiral Transformer Experts for multi-level infrared and visible image fusion; it fuses infrared (IR) and visible (VI) features by jointly modeling intra- and inter-modal relationships through a windowed Transformer with an asymmetric cross-attention (ACA) mechanism (Jinfu et al., 27 Jul 2025). In the same data record, an unrelated synthesis of the rational VOA/RCFT paper “Modular Exercises for Four-Point Blocks -- I” also uses “CTFB” to denote a vector of chiral four-point sphere core blocks together with its modular transformation law and fusion-braiding data (Cheng et al., 2020). The acronym therefore has a cross-disciplinary ambiguity, but in computer-vision usage it refers to the transformer fusion block introduced within MoCTEFuse.

1. Definition, Motivation, and Functional Role

Within MoCTEFuse, the CTFB is the core fusion unit in each expert subnetwork of the Mixture of Chiral Transformer Experts (MoCTE). Its stated function is to fuse IR and VI features by jointly modeling intra- and inter-modal relationships, using a windowed Transformer with ACA. “Chiral” denotes asymmetry in the cross-attention direction: the modality deemed primary supplies the queries, while the other modality, termed auxiliary, supplies supplementary keys and values. This design is motivated by the observation that under high illumination, VI carries more reliable textures and details, whereas under low illumination, IR carries stronger target and contrast cues; many methods that directly merge source images are described as suffering modality bias because they fuse modalities uniformly rather than conditionally (Jinfu et al., 27 Jul 2025).

The primary–auxiliary assignment is not fixed globally. MoCTE contains two experts: HI-MoCTE and LI-MoCTE. In HI-MoCTE, visible is primary and infrared is auxiliary, yielding an HI-CTFB; in LI-MoCTE, infrared is primary and visible is auxiliary, yielding an LI-CTFB. An illumination-sensitive gate estimates illumination probability from the VI image and weights the high- and low-illumination experts accordingly. This means that the operational semantics of “chiral” are illumination-dependent rather than merely architectural.

A common misunderstanding is to treat the asymmetry as a hard exclusion of the auxiliary modality. The paper instead defines ACA so that the primary modality supplies queries, while attention is computed over concatenated keys and values from both modalities. The auxiliary modality is therefore not discarded; it is incorporated as supplementary context under a directional attention rule.

2. Architectural Placement and Block Topology

The MoCTEFuse pipeline takes an IR image $I_{ir}$ and a VI image $I_{vi}$. Two modality-specific encoders first extract features. The shallow feature extractor is a $3\times 3$ convolution followed by LReLU. The deep feature extractor uses Residual Transformer Blocks (RTB) and Residual Dense Blocks (RDB), described as being “as in DCTNet.” These stages produce aligned features $Z_{vi}^{in}, Z_{ir}^{in}\in R^{H\times W\times C}$, which serve as CTFB inputs (Jinfu et al., 27 Jul 2025).

Each CTFB partitions feature maps into non-overlapping local windows of size $M\times M$, with $M=8$ by default, producing $HW/M^2$ windows. Each window is reshaped to a sequence of shape $M^2\times C$, and a linear projection (LP) maps inputs to the attention space. The ordering inside the block is explicitly given as

$$\mathrm{LP}\rightarrow \mathrm{LN}\rightarrow \mathrm{ACA}\rightarrow \text{residual},\qquad \mathrm{LN}\rightarrow \mathrm{FFN}\rightarrow \text{residual}.$$

The FFN uses two MLP layers with GeLU activation. Attention is local within each window rather than global, and the paper does not specify multi-head attention; the formulation is single-head. A shifted window strategy is used in conjunction with the windowed LP and ACA.

The forward equations are written for both directional paths. For the path “IR$\rightarrow$VI” with VI as primary and IR auxiliary,

$I_{vi}$0

$I_{vi}$1

For the path “VI$I_{vi}$2IR” with IR as primary and VI auxiliary,

$I_{vi}$3

$I_{vi}$4

The outputs $I_{vi}$5 are then fed into the next CTFB stage. Because CTFBs are stacked across multiple hierarchy levels in each expert, the architecture performs progressive refinement of modality-specific and cross-modal information rather than single-pass fusion.

3. Asymmetric Cross-Attention as the “Chiral” Mechanism

For a given window, the tokens are denoted $I_{vi}$6. The block uses distinct learnable modality-specific projections

$I_{vi}$7

The projected queries, keys, and values are

$I_{vi}$8

$I_{vi}$9

The keys and values are concatenated along the token dimension: $3\times 3$0 This is the formal point at which the auxiliary modality enters the attention computation (Jinfu et al., 27 Jul 2025).

The two directional ACA paths are then defined as

$3\times 3$1

$3\times 3$2

where $3\times 3$3 is the key dimension, equal to $3\times 3$4 in the single-head formulation, and $3\times 3$5 are learnable relative positional encodings. The visible-primary path corresponds to the HI-CTFB direction, and the infrared-primary path corresponds to the LI-CTFB direction.

This mechanism is “chiral” in the precise sense that the query source changes with the designated primary modality, while the attended context remains a concatenated cross-modal token set. The paper further states that there are two parallel directional paths, IR$3\times 3$6VI and VI$3\times 3$7IR, computed per block; expert specialization and illumination gating determine which direction dominates in forming the final fused output. This suggests that the asymmetry is implemented as a directional bias rather than as a one-direction-only information flow.

4. Illumination Gating, Output Formation, and Optimization

The illumination-sensitive gating network is a ResNet18-based classifier that takes the visible image as input and outputs probabilities $3\times 3$8 and $3\times 3$9 for high and low illumination through a final sigmoid activation. Its structure is given as Conv1: $Z_{vi}^{in}, Z_{ir}^{in}\in R^{H\times W\times C}$0, 64, stride 2; MaxPool $Z_{vi}^{in}, Z_{ir}^{in}\in R^{H\times W\times C}$1, stride 2; then ResNet18 stages Conv2–Conv5; Average Pool, Fully-Connected, Sigmoid, producing an output of $Z_{vi}^{in}, Z_{ir}^{in}\in R^{H\times W\times C}$2 interpreted as high- and low-illumination probabilities (Jinfu et al., 27 Jul 2025).

These probabilities modulate the experts and therefore the CTFBs nested inside them. Each expert decodes its fused features into a candidate fused image, denoted $Z_{vi}^{in}, Z_{ir}^{in}\in R^{H\times W\times C}$3 and $Z_{vi}^{in}, Z_{ir}^{in}\in R^{H\times W\times C}$4. The final output is

$Z_{vi}^{in}, Z_{ir}^{in}\in R^{H\times W\times C}$5

The paper states explicitly that the prediction probability is used to assign appropriate weights to HI-MoCTE and LI-MoCTE, with the weighted output being the fused image. Decoder details are not elaborated beyond standard reconstruction from fused features.

Training is divided into two stages. The illumination gate is trained with binary cross-entropy,

$Z_{vi}^{in}, Z_{ir}^{in}\in R^{H\times W\times C}$6

where $Z_{vi}^{in}, Z_{ir}^{in}\in R^{H\times W\times C}$7 is the ground-truth illumination label. The fusion network then uses three sub-losses: $Z_{vi}^{in}, Z_{ir}^{in}\in R^{H\times W\times C}$8

$Z_{vi}^{in}, Z_{ir}^{in}\in R^{H\times W\times C}$9

$M\times M$0

These are combined as

$M\times M$1

with $M\times M$2, $M\times M$3, and $M\times M$4 in experiments.

A distinctive element is the competitive mixture loss. Let the expert outputs be $M\times M$5 with fused images $M\times M$6 and probabilities $M\times M$7 from the gate, with $M\times M$8 experts and $M\times M$9, $M=8$0. The per-expert loss is defined as $M=8$1, and the competitive loss is

$M=8$2

Its derivative is written as

$M=8$3

The paper interprets this as encouraging experts to compete under the illumination distribution, thereby sharpening specialization of the high- and low-illumination branches.

Implementation details report window size $M=8$4, training resolution $M=8$5 normalized to $M=8$6, testing at original resolution, Adam with a warmup cosine annealing learning-rate schedule, 60 epochs, and batch size 8. The gate is trained first and then frozen while the fusion model is trained with the competitive loss.

5. Complexity, Empirical Behavior, and Reported Performance

Let $M=8$7 and let each local window contain $M=8$8 tokens. The paper gives the single-head ACA complexity per window as $M=8$9 in compute and $HW/M^2$0 in attention memory. Across all windows, this becomes

$HW/M^2$1

for compute and

$HW/M^2$2

for memory. With $HW/M^2$3, attention is therefore local and described as more tractable than global $HW/M^2$4. Each CTFB uses modality-specific linear projections for $HW/M^2$5, two learnable relative positional encodings, LayerNorms, and a two-layer MLP FFN with GeLU. The paper does not report parameter counts, and no weight sharing across experts is stated (Jinfu et al., 27 Jul 2025).

Ablation experiments isolate the contributions of HI-MoCTE, LI-MoCTE, and the competitive loss. On MSRS, the full MoCTEFuse reports EN $HW/M^2$6, SD $HW/M^2$7, MI $HW/M^2$8, and VIF $HW/M^2$9; removing HI-MoCTE gives EN $M^2\times C$0, SD $M^2\times C$1, MI $M^2\times C$2, VIF $M^2\times C$3; removing LI-MoCTE gives EN $M^2\times C$4, SD $M^2\times C$5, MI $M^2\times C$6, VIF $M^2\times C$7; replacing the competitive loss with $M^2\times C$8 gives EN $M^2\times C$9, SD $$\mathrm{LP}\rightarrow \mathrm{LN}\rightarrow \mathrm{ACA}\rightarrow \text{residual},\qquad \mathrm{LN}\rightarrow \mathrm{FFN}\rightarrow \text{residual}.$$0, MI $$\mathrm{LP}\rightarrow \mathrm{LN}\rightarrow \mathrm{ACA}\rightarrow \text{residual},\qquad \mathrm{LN}\rightarrow \mathrm{FFN}\rightarrow \text{residual}.$$1, VIF $$\mathrm{LP}\rightarrow \mathrm{LN}\rightarrow \mathrm{ACA}\rightarrow \text{residual},\qquad \mathrm{LN}\rightarrow \mathrm{FFN}\rightarrow \text{residual}.$$2. On DroneVehicle, the full model reports EN $$\mathrm{LP}\rightarrow \mathrm{LN}\rightarrow \mathrm{ACA}\rightarrow \text{residual},\qquad \mathrm{LN}\rightarrow \mathrm{FFN}\rightarrow \text{residual}.$$3, SD $$\mathrm{LP}\rightarrow \mathrm{LN}\rightarrow \mathrm{ACA}\rightarrow \text{residual},\qquad \mathrm{LN}\rightarrow \mathrm{FFN}\rightarrow \text{residual}.$$4, MI $$\mathrm{LP}\rightarrow \mathrm{LN}\rightarrow \mathrm{ACA}\rightarrow \text{residual},\qquad \mathrm{LN}\rightarrow \mathrm{FFN}\rightarrow \text{residual}.$$5, and VIF $$\mathrm{LP}\rightarrow \mathrm{LN}\rightarrow \mathrm{ACA}\rightarrow \text{residual},\qquad \mathrm{LN}\rightarrow \mathrm{FFN}\rightarrow \text{residual}.$$6; the corresponding ablations yield EN $$\mathrm{LP}\rightarrow \mathrm{LN}\rightarrow \mathrm{ACA}\rightarrow \text{residual},\qquad \mathrm{LN}\rightarrow \mathrm{FFN}\rightarrow \text{residual}.$$7, SD $$\mathrm{LP}\rightarrow \mathrm{LN}\rightarrow \mathrm{ACA}\rightarrow \text{residual},\qquad \mathrm{LN}\rightarrow \mathrm{FFN}\rightarrow \text{residual}.$$8, MI $$\mathrm{LP}\rightarrow \mathrm{LN}\rightarrow \mathrm{ACA}\rightarrow \text{residual},\qquad \mathrm{LN}\rightarrow \mathrm{FFN}\rightarrow \text{residual}.$$9, VIF $\rightarrow$0 without HI-MoCTE, EN $\rightarrow$1, SD $\rightarrow$2, MI $\rightarrow$3, VIF $\rightarrow$4 without LI-MoCTE, and EN $\rightarrow$5, SD $\rightarrow$6, MI $\rightarrow$7, VIF $\rightarrow$8 without the competitive loss.

Across DroneVehicle, MSRS, TNO, and RoadScene, the paper states that MoCTEFuse achieves superior fusion performance and state-of-the-art averages on EN, SD, MI, and VIF in most cases. It also reports the best detection mean Average Precision of $\rightarrow$9 on MFNet and $I_{vi}$00 on DroneVehicle. Qualitatively, the model is described as preserving salient IR targets under low light, preserving VI details under high light, and reducing ghosting artifacts under weak misalignment on DroneVehicle. The principal limitation noted is relatively high computational cost, since windowed attention remains tractable but stacked CTFBs at multiple stages in two experts are not lightweight.

6. Cross-Disciplinary Ambiguity: the Unrelated VOA/RCFT Usage

A separate use of “CTFB” appears in a concise synthesis of “Modular Exercises for Four-Point Blocks -- I,” where the acronym denotes the vector of chiral four-point sphere core blocks together with its modular transformation law under the appropriate modular group and the fusion-braiding data of the rational VOA/RCFT (Cheng et al., 2020). In that setting, the construction packages the chiral four-point block decomposition in intermediate channels, the modular map from the sphere cross-ratio to the torus modulus, vector-valued modularity of the block vector, crossing and fusion modular transformations, and the modular-invariant full correlator.

The same synthesis states that the sphere cross-ratio $I_{vi}$01 is identified with the torus modulus $I_{vi}$02 through the modular lambda function,

$I_{vi}$03

and that the block vector becomes a weight-0 vector-valued modular form for $I_{vi}$04, $I_{vi}$05, or $I_{vi}$06, depending on the multiplicities of identical external insertions. In this language, crossing symmetry is recast as modular symmetry, and the full four-point correlator is modular invariant. The synthesis also highlights a sphere–torus correspondence relating certain sphere quantities of a family $I_{vi}$07 to torus quantities of another family $I_{vi}$08.

This usage is unrelated to the transformer block in MoCTEFuse. The shared acronym does not indicate methodological continuity between infrared-visible image fusion and rational VOA/RCFT. A plausible implication is that citations using “CTFB” require disambiguation by field, since the term can refer either to an asymmetric cross-attention fusion block in multimodal vision or to a vector-valued modular-form packaging of chiral four-point blocks in conformal field theory.