FMCAF Architecture for Multimodal Detection

• FMCAF architecture is a multimodal fusion pipeline that combines RGB and IR data using frequency-domain filtering and symmetric cross-attention.
• The Freq-Filter block uses FFT, soft top-k masking, and learnable blending to denoise channel-specific artifacts before fusion.
• The MCAF block applies local cross-attention for context-aware feature integration, yielding significant mAP improvements on benchmark datasets.

The Filtered Multi-Modal Cross Attention Fusion (FMCAF) architecture is a preprocessing and fusion pipeline for robust multimodal object detection, with specific innovations for combining RGB and infrared (IR) image data. FMCAF addresses limitations of naïve fusion techniques by employing frequency-domain adaptive filtering and symmetric cross-attention to selectively enhance complementary aspects of each modality. The architecture is designed for generalizability across datasets and detection scenarios, yielding significant accuracy improvements in empirical evaluations on challenging benchmarks (Berjawi et al., 20 Oct 2025).

1. Motivation and Rationale

Classic mid-level fusion strategies, such as early concatenation of RGB and IR channels into a 4-channel input tensor, fail to properly exploit the complementary statistical properties of each modality. These methods are highly susceptible to differing noise characteristics (e.g., high-frequency RGB artifacts versus diffuse IR noise) and lack adaptive weighting mechanisms that can emphasize the most informative modality per scene instance. Previous approaches based on attention (e.g., transformer-derived cross-channel attention, MEFA) either do not support explicit cross-modal information exchange or require heavy per-dataset tuning and significant computational cost.

FMCAF is designed to overcome these challenges through two architectural principles:

• Modality-wise frequency-domain filtering to suppress redundant and noisy spectral features independently in each channel set;
• Lightweight, symmetric cross-attention to enable context-aware mixing of cues between modalities, enhancing scene understanding in diverse conditions.

2. Frequency-Domain Filtering: Freq-Filter Block

The architecture begins with a frequency-domain filtering stage (Freq-Filter), which processes the raw $4$-channel $\mathrm{RGB} + \mathrm{IR}$ input $X \in \mathbb{R}^{B \times 4 \times H \times W}$ as follows:

1. Channelwise Decomposition and FFT: $X$ is partitioned into $$X^{\rm RGB}\in\mathbb{R}^{B\times3\times H\times W}$$ and $X^{\rm IR}\in\mathbb{R}^{B\times1\times H\times W}$. Each set is transformed with a 2D FFT per channel:

$$\mathcal{F}^m = \mathrm{FFT2D}(X^m), \quad m \in \{\mathrm{RGB},\mathrm{IR}\}$$

2. Saliency Estimation: For each modality $m$, the amplitude spectrum is averaged over channels:

$$A^m_{b,i,j} = \frac{1}{C_m}\sum_{c=1}^{C_m} \left|\mathcal{F}^m_{b,c,i,j}\right|$$

This amplitude map is passed through a compact convolutional encoder to derive $E^m$, a feature map.

1. Soft Top-$\mathrm{RGB} + \mathrm{IR}$0 Masking: A differentiable soft mask $\mathrm{RGB} + \mathrm{IR}$1 selects the top $\mathrm{RGB} + \mathrm{IR}$2 activations in $\mathrm{RGB} + \mathrm{IR}$3 (rank-based), with other values set to zero. The mask is applied multiplicatively to form a sparsified frequency spectrum.
2. Inverse FFT and Learnable Blending: The filtered spectrum is returned to the spatial domain via $\mathrm{RGB} + \mathrm{IR}$4, producing $\mathrm{RGB} + \mathrm{IR}$5. Results are combined with the original modalities using a learned parameter $\mathrm{RGB} + \mathrm{IR}$6:

$\mathrm{RGB} + \mathrm{IR}$7

The Freq-Filter thus “cleans” each channel group, explicitly reducing redundant spectral content that can otherwise confound fusion.

3. Cross-Attention Fusion Block: MCAF

FMCAF’s second stage leverages a multimodal cross-attention fusion (MCAF) block to exchange discriminative features between modalities at a local scale:

1. Embedding and Q/K/V Projection: Blended RGB and IR features $\mathrm{RGB} + \mathrm{IR}$8 are passed through convolutional “Inception” stems, flattened to produce query ($\mathrm{RGB} + \mathrm{IR}$9), key ($X \in \mathbb{R}^{B \times 4 \times H \times W}$0), and value ($X \in \mathbb{R}^{B \times 4 \times H \times W}$1) matrices for each modality.
2. Local Symmetric Cross-Attention: For each non-overlapping $X \in \mathbb{R}^{B \times 4 \times H \times W}$2 patch, cross-attention is computed between $X \in \mathbb{R}^{B \times 4 \times H \times W}$3 and $X \in \mathbb{R}^{B \times 4 \times H \times W}$4, $X \in \mathbb{R}^{B \times 4 \times H \times W}$5 ($X \in \mathbb{R}^{B \times 4 \times H \times W}$6):

$X \in \mathbb{R}^{B \times 4 \times H \times W}$7

This operation allows, for example, the IR stream to inject heat-based saliency into the RGB path, and vice versa.

1. Self-Attention and Channel Fusion: Each $X \in \mathbb{R}^{B \times 4 \times H \times W}$8 undergoes local self-attention and normalization. The modalities are concatenated and processed via an Inception block, yielding $X \in \mathbb{R}^{B \times 4 \times H \times W}$9.
2. Sigmoid-Gated Global Attention: $X$0 is partitioned into large spatial patches and pooled. Per-patch descriptors are passed through a shallow MLP with a sigmoid gate, and these activations modulate $X$1 via residual addition:

$X$2

1. Projection to Output: A $X$3 convolution projects $X$4 to a 3-channel map, which serves as the fused image input to the detection backbone.

4. End-to-End Detection Pipeline

The fusion output, a $X$5 image, is passed downstream to a YOLOv11 backbone (DarkNet-style with CSP), producing multi-scale features (P3: $X$6, P4: $X$7, P5: $X$8) suitable for object detection. The YOLO detection head operates on each scale, outputting objectness, bounding-box regression, and class probabilities.

The full processing flow is summarized as follows:

Stage	Input/Output Shape	Description
Freq-Filter	$X$9	Spectral masking + blending
MCAF	$$X^{\rm RGB}\in\mathbb{R}^{B\times3\times H\times W}$$0	Cross-attention, channel fusion
YOLOv11 Backbone	$$X^{\rm RGB}\in\mathbb{R}^{B\times3\times H\times W}$$1	FPN feature extraction
Detection Head	Multi-scale	Object detection outputs

5. Training Regime and Losses

FMCAF is optimized using a multi-term YOLOv11 objective:

$$X^{\rm RGB}\in\mathbb{R}^{B\times3\times H\times W}$$2

where $$X^{\rm RGB}\in\mathbb{R}^{B\times3\times H\times W}$$3 is binary cross-entropy for objectness, $$X^{\rm RGB}\in\mathbb{R}^{B\times3\times H\times W}$$4 is cross-entropy or BCE for class prediction, and $$X^{\rm RGB}\in\mathbb{R}^{B\times3\times H\times W}$$5 is GIoU or CIoU regression for bounding boxes. Training uses AdamW, cosine learning rate schedules, and extensive geometric/photometric augmentations (flips, scaling, rotation, HSV jitter, mosaic, mixup). Batch sizes varied to utilize full NVIDIA A100 GPU memory.

Distinct hyperparameters were set per dataset:

• VEDAI: 250 epochs, initial LR $$X^{\rm RGB}\in\mathbb{R}^{B\times3\times H\times W}$$6 with weight decay $$X^{\rm RGB}\in\mathbb{R}^{B\times3\times H\times W}$$7, various augmentation strategies.
• LLVIP: 20 epochs, initial LR $$X^{\rm RGB}\in\mathbb{R}^{B\times3\times H\times W}$$8, weight decay $$X^{\rm RGB}\in\mathbb{R}^{B\times3\times H\times W}$$9, streamlined augmentations.

6. Empirical Performance and Ablation Results

FMCAF achieves substantial gains over simple concatenation and its individual submodules:

Method	VEDAI (mAP@50)	LLVIP (mAP@50)
Concat (RGB+IR)	62.6%	94.3%
Freq-Filter only	52.5%	91.6%
MCAF only	71.8%	94.8%
FMCAF (both)	76.5%	95.4%

Component ablation demonstrates that while MCAF contributes the majority of the performance improvement, the combination with Freq-Filter (removing redundant spectral content before attention) yields synergistic benefits. On a per-class basis, VEDAI categories such as van (56.6% $X^{\rm IR}\in\mathbb{R}^{B\times1\times H\times W}$0 92.7%), car (84.3% $X^{\rm IR}\in\mathbb{R}^{B\times1\times H\times W}$1 93.6%), and truck (61.1% $X^{\rm IR}\in\mathbb{R}^{B\times1\times H\times W}$2 93.7%) exhibit substantial improvements.

Inference latency is approximately 50 ms per $X^{\rm IR}\in\mathbb{R}^{B\times1\times H\times W}$3 image (20 FPS) on unoptimized FP32 code on A100. Quantized/pruned variants are projected to achieve real-time throughput ($X^{\rm IR}\in\mathbb{R}^{B\times1\times H\times W}$430 FPS).

7. Significance, Extensions, and Limitations

FMCAF provides a robust, generalizable fusion frontend for multimodal object detection without the need for dataset-specific tuning. The blending coefficient $X^{\rm IR}\in\mathbb{R}^{B\times1\times H\times W}$5 enables scene-adaptive denoising, and early cross-attention in the processing pipeline outperforms late fusion approaches. The architecture is readily extensible to additional modalities (radar, LiDAR) and tasks (segmentation, depth estimation) and is suitable for deployment on edge MCUs given further optimization. A plausible implication is that spectral filtering, when combined with cross-modal attention, can be a foundational pattern for multimodal preprocessing across vision tasks (Berjawi et al., 20 Oct 2025).