MCAF: Cross-Attention Fusion Module

• MCAF is a neural network module that fuses heterogeneous sensor features using cross-attention to explicitly learn intermodal relationships.
• It integrates a frequency-filtering step that denoises inputs via Fourier transforms and learnable soft-masking, ensuring robust feature blending.
• MCAF enhances multimodal object detection performance in challenging scenarios like low-light, aerial surveillance, and autonomous driving through adaptive feature integration.

A Cross-Attention-Based Fusion Module (MCAF) is a neural network component designed to fuse heterogeneous feature representations from different sensor modalities or data sources by explicitly modeling intermodal relationships through cross-attention mechanisms. In the context of multimodal object detection and perception, as exemplified by the FMCAF architecture, MCAF facilitates feature exchange and integration between channels such as RGB and infrared (IR) imagery by allowing the network to attend selectively to salient and complementary features across inputs. This architectural principle underlies robust and generalizable multimodal fusion pipelines for tasks in low-light, aerial, and complex sensing scenarios (Berjawi et al., 20 Oct 2025).

1. FMCAF Architecture: Modular Preprocessing and Fusion Design

FMCAF is a two-stage multimodal data preprocessing and fusion architecture that precedes conventional object detection backbones. It consists of two principal modules in sequence:

• Freq-Filter Module: Operates on raw RGB and IR inputs by applying channel-wise 2D Fourier transforms. For each modality, a learnable mask based on a soft top-k% selection is applied to suppress redundant or noisy frequency components in the amplitude spectrum. The filtered signal is then blended, using a learnable weight $\alpha$, with the unfiltered input in the spatial domain:

$$X_{\text{blend}} = \alpha \cdot X_{\text{filtered}} + (1 - \alpha) \cdot X_{\text{raw}}$$

This adaptive blending enables the network to control the trade-off between denoised and raw feature contributions.

• MCAF (Multimodal Cross Attention Fusion Module): Takes the frequency-filtered and adaptively blended outputs from both modalities and fuses them using spatially windowed cross-attention and hierarchical (local/global) attention mechanisms. The feature pipeline is:

1. Local Cross-Attention: Features are partitioned into local non-overlapping windows. Within each window, the cross-attention is formulated as:

   $$F_m' = \text{Softmax}\left(\frac{Q_m K_{m'}^\top}{\sqrt{d}}\right) V_{m'}$$

   where $m$ and $m'$ index modalities (e.g., $m=$ RGB, $m'=$ IR), $Q_m$ is the query projection for modality $m$, and $K_{m'}, V_{m'}$ are respectively key and value projections for modality $m'$. $d$ is the per-head dimensionality for $h$ heads.

2. Local Attention: An additional local attention block further aggregates spatial context within windows.
3. Global Sigmoid-Gated Attention: A global attention block with sigmoid gating serves as a residual path to refine salient global semantic features before projecting the output to a unified 3-channel representation.

This arrangement ensures that intermodal feature sharing through explicit cross-attention is prioritized and then consolidated using spatial and global attention cues.

2. Cross-Attention Mechanism for Feature Fusion

The core functionality of MCAF is realized via a cross-attention mechanism that explicitly learns dependencies between the features from the two modalities. For each spatial location or local window, the module computes similarity scores between a query vector from one modality and key vectors from the other modality, and then aggregates value vectors from the latter, weighted by the similarity:

• Mathematical Formulation:
  • For a local window,

  $$Q_m = X_m W^Q,\quad K_{m'} = X_{m'} W^K,\quad V_{m'} = X_{m'} W^V$$

  $$\text{CrossAttn}(X_m, X_{m'}) = \text{Softmax}\left(\frac{Q_m K_{m'}^\top}{\sqrt{d}}\right) V_{m'}$$ - This operation replaces naive fusion (e.g., concatenation) and allows the network to reweight or select features that are most informative across the modality boundary, ensuring that complementary cues from RGB and IR can guide the aggregation step-by-step.

Following cross-attention, hierarchical local and global attention further reinforce spatial alignment and semantic coherence.

3. Frequency-Domain Filtering and Modality-Specific Denoising

The Freq-Filter’s integration significantly affects the fusion, especially at high spatial resolutions where high-frequency sensor noise in modalities like IR is prevalent.

• Fourier-based Analysis: Each channel of the input is subjected to a 2D Fourier transform to obtain the amplitude spectrum.
• Soft Masking: A learnable encoder generates a binary mask (using a soft top-k% threshold) to retain only salient frequencies.
• Inverse Reconstruction and Blending: Filtered features are reconstructed via inverse transform and blended as per the learned $\alpha$ weight.

This denoising pre-processing ensures that only salient (and not spurious) modality-specific features propagate into the cross-attention fusion phase, thereby stabilizing generalization across datasets and sensing domains.

4. Generalizability and Empirical Performance

FMCAF is designed to avoid reliance on dataset-specific tuning by using generalized, learnable, and modular pre-fusion operations. This quality is empirically validated on distinct benchmark datasets:

• VEDAI: Aerial vehicle detection benefits from robust small-object and occlusion-aware fusion, with FMCAF improving mAP@50 by +13.9% over early concatenation-based fusion.
• LLVIP: For low-light pedestrian detection, FMCAF achieves a +1.1% mAP@50 improvement, demonstrating resilience to variable lighting and noise.

Adaptive blending, modular Freq-Filter design, and cross-attention-based fusion collectively enable strong transferability and robustness across challenging multimodal detection tasks.

5. Applications and Broader Implications

MCAF’s cross-attention fusion paradigm is directly applicable to:

• Autonomous Driving and ADAS: Combining RGB (context, color) and IR (thermal, structural) information for robust obstacle and pedestrian recognition, especially under adverse weather or low-visibility conditions.
• Aerial and Remote Sensing: Enhanced detection of small or occluded objects in UAV-based surveillance, exploiting complementary visible and IR spectra.
• Security and Surveillance: Improved reliability in cluttered or low-light environments where cross-modal information is vital to reduce false positives and negatives.
• Multi-Sensor Perception: The principles demonstrated here can extend to LiDAR, hyperspectral, or medical modalities, utilizing cross-attention fusion to bridge cultural and physical differences in data distributions.

Potential challenges include computation overhead from local windowed attention and the need to balance the fidelity of fine frequency details against noise suppression.

6. Comparative Perspective and Future Directions

Comparisons with related architectures reveal that MCAF’s principled cross-attention yields more robust intermodal feature sharing than vanilla concatenation or even simple attention fusion, especially in cross-domain generalization (Berjawi et al., 20 Oct 2025). Its flexibility allows for adaptation to further tasks such as segmentation or multimodal classification. Future research may explore scaling the architecture to three or more modalities, real-time deployment constraints, or integration with more advanced backbone detectors and domain adaptation frameworks.

In conclusion, MCAF as implemented in FMCAF demonstrates how cross-attention-based fusion, complemented by adaptive frequency-domain filtering, can serve as a generalizable foundation for multimodal object detection pipelines, moving beyond dataset-specific tuning and advancing robust perception in real-world environments.