MAVFusion: Efficient Infrared and Visible Video Fusion via Motion-Aware Sparse Interaction

Abstract: Infrared and visible video fusion combines the object saliency from infrared images with the texture details from visible images to produce semantically rich fusion results. However, most existing methods are designed for static image fusion and cannot effectively handle frame-to-frame motion in videos. Current video fusion methods improve temporal consistency by introducing interactions across frames, but they often require high computational cost. To mitigate these challenges, we propose MAVFusion, an end-to-end video fusion framework featuring a motion-aware sparse interaction mechanism that enhances efficiency while maintaining superior fusion quality. Specifically, we leverage optical flow to identify dynamic regions in multi-modal sequences, adaptively allocating computationally intensive cross-modal attention to these sparse areas to capture salient transitions and facilitate inter-modal information exchange. For static background regions, a lightweight weak interaction module is employed to maintain structural and appearance integrity. By decoupling the processing of dynamic and static regions, MAVFusion simultaneously preserves temporal consistency and fine-grained details while significantly accelerating inference. Extensive experiments demonstrate that MAVFusion achieves state-of-the-art performance on multiple infrared and visible video benchmarks, achieving a speed of 14.16\,FPS at $640 \times 480$ resolution. The source code will be available at https://github.com/ixilai/MAVFusion.

• The paper introduces MAVFusion, which decouples dynamic and static regions using optical flow to achieve accurate video fusion with reduced computational cost.
• It employs a motion-aware feature alignment module and a dual-interaction module to apply intensive processing only on salient motion regions while preserving static backgrounds.
• Experimental results demonstrate that MAVFusion outperforms dense-attention baselines in fusion quality and downstream tasks like detection and segmentation, with over 15× efficiency gains.

MAVFusion: Efficient Infrared and Visible Video Fusion via Motion-Aware Sparse Interaction

Introduction and Motivation

Infrared and visible image fusion is essential for robust visual perception in complex environments as it combines the salient targets from infrared imagery with the rich details from visible images. Most prevailing image fusion approaches, however, are static in nature and do not address temporal consistency or motion-related artifacts in video scenarios. Dense temporal attention methods have been proposed for video fusion, but these methods incur high computational costs and often waste resources on static background regions, failing to exploit motion information for selective processing.

MAVFusion directly addresses these deficiencies by explicitly decoupling the processing of dynamic and static regions, using optical flow to focus intensive spatial-temporal reasoning only where required. This yields a model that offers high-fidelity, temporally consistent fusion at a significantly reduced computational burden compared to dense-attention or global-transformer-based video fusion solutions.

Methodology Overview

MAVFusion is organized around a motion-aware and region-decoupled processing paradigm.

Figure 1: The overall framework of the proposed algorithm.

Motion-Aware Feature Alignment Module (MAFM)

To suppress cross-modal misalignment and motion-induced artifacts (e.g., ghosting), MAVFusion employs a lightweight MAFM. Optical flow—computed using a frozen SEA-RAFT network—is used to align features across consecutive frames. Initial coarse alignment is followed by cross-modal residual refinement, where the alignment is further corrected in feature space by leveraging spatial anchors from both modalities. Only motion regions undergo multi-frame temporal aggregation and fusion; static regions are simply propagated, avoiding unnecessary noise and computation.

Motion-Guided Dual-Interaction Module (MDIM)

MAVFusion segregates the frame into dynamic (motion) and static (background) regions. Sparse, computationally expensive cross-modal attention is deployed only to top-K salient dynamic patches, as determined by pooled optical flow magnitudes. For static backgrounds, lightweight local convolutions suffice, preserving details in a cost-effective manner. The integration of these branches is spatially masked and smoothed to eliminate blocky artifacts and ensure seamless fusion.

Figure 2: Effect of Global Strong Interaction on Motion Clues and Structural Integrity.

This design strictly limits quadratic complexity attention to a minimal spatial extent, making the approach tractable even for long sequences and high resolutions.

Loss Formulation

The loss is composed of a spatial fidelity term (combining pixel-wise and structural similarity) and a temporal consistency loss, which penalizes misalignment of adjacent frames after accounting for optical flow and occlusion masks. This ensures that the network learns both per-frame quality and smooth transitions.

Experimental Evaluation

Qualitative and Quantitative Performance

MAVFusion is evaluated on three video fusion benchmarks: M3SVD, HDO, and VTMOT. In qualitative comparisons, static background preservation and sharper, ghost-free moving targets are observed with MAVFusion relative to recent image and video fusion baselines.

Figure 3: Qualitative comparison of all methods on the M3SVD dataset.

Figure 4: Qualitative comparison of all methods on the VTMOT dataset.

On all datasets, MAVFusion yields the best or second-best results in all core image fusion metrics, including $Q_G$, $Q_M$, $Q_S$, $VIF$, and MS2R. Compared with dense global attention methods, it not only maintains or improves spatial and temporal quality but does so with radically reduced computation.

Downstream Task Transfer: Object Detection

Fusion quality is validated on object detection with YOLO26 as the downstream model. Particularly in low-illumination scenarios, MAVFusion delivers higher detection confidence and target–background contrast, outright outperforming static and video fusion baselines.

Figure 5: Comparison of detection accuracy of different fusion results in the object detection task.

Robustness and Segmentation Transferability

Further, MAVFusion enables improved downstream segmentation quality, as evidenced by superior region and boundary recall for small and occluded objects, compared to prominent video fusion baselines.

Figure 6: Comparison of segmentation accuracy of different fusion results in the video segmentation task.

Ablation Analysis

Ablation studies unambiguously confirm that the dynamic-static region decoupling is essential: applying strong attention globally (Full-DB) is both computationally prohibitive and yields inferior results due to semantic feature dilution. Conversely, static-only (Full-SB) designs lose motion saliency. Motion-guided sparse attention and the use of MAFM are both vital for minimizing ghosting and maximizing fusion quality.

Computational Efficiency

MAVFusion achieves an inference rate of 14.16 FPS at $640\times480$, requiring only 123.37 GFLOPs—over 15× faster than dense-attention video fusion methods at a fraction of the compute. As resolution increases, its computational complexity scales sub-linearly with the number of pixels, owing to the localized deployment of high-cost modules.

Implications and Future Directions

MAVFusion demonstrates that motion-guided, region-adaptive fusion architectures can dramatically improve the tractability and quality of infrared–visible video fusion. By differentiating processing by spatial context, such models make real-time deployment in embedded or mobile settings plausible—an essential advance for surveillance, autonomous vehicles, and multisensor robotics.

Theoretically, the results clarify that dense spatial-temporal modeling is not necessary everywhere—a paradigm that could influence the design of efficient transformers and hybrids in both multi-modal and single-modal video tasks. MAVFusion’s strong performance despite adversarial optical flow (as demonstrated on image-only datasets) suggests surprising robustness for region decoupling and sparse attention principles.

Future research may further reduce dependence on external optical flow, co-learn modality alignment, or embed more task awareness (e.g., detection/segmentation labels) directly into the fusion architecture.

Conclusion

MAVFusion sets a new standard for efficient video fusion of infrared and visible modalities, coupling region-aware interaction with optical-flow-guided attention. It outperforms previous baselines in spatial and temporal metrics, yields improved results on detection and segmentation, and notably cuts computational demands. The proposed decoupled and sparsified processing pipeline provides a framework for high-performance, real-time, and deployment-ready multi-modal fusion in dynamic environments.

Reference: "MAVFusion: Efficient Infrared and Visible Video Fusion via Motion-Aware Sparse Interaction" (2604.01958)