FTPFusion: Frequency-Aware Infrared and Visible Video Fusion with Temporal Perturbation

Abstract: Infrared and visible video fusion plays a critical role in intelligent surveillance and low-light monitoring. However, maintaining temporal stability while preserving spatial detail remains a fundamental challenge. Existing methods either focus on frame-wise enhancement with limited temporal modeling or rely on heavy spatio-temporal aggregation that often sacrifices high-frequency details. In this paper, we propose FTPFusion, a frequency-aware infrared and visible video fusion method based on temporal perturbation and sparse cross-modal interaction. Specifically, FTPFusion decomposes the feature representations into high-frequency and low-frequency components for collaborative modeling. The high-frequency branch performs sparse cross-modal spatio-temporal interaction to capture motion-related context and complementary details. The low-frequency branch introduces a temporal perturbation strategy to enhance robustness against complex video variations, such as flickering, jitter, and local misalignment. Furthermore, we design an offset-aware temporal consistency constraint to explicitly stabilize cross-frame representations under temporal disturbances. Extensive experiments on multiple public benchmarks demonstrate that FTPFusion consistently outperforms state-of-the-art methods across multiple metrics in both spatial fidelity and temporal consistency. The source code will be available at https://github.com/ixilai/FTPFusion.

• The paper introduces a frequency-aware dual-branch fusion method that separately processes low- and high-frequency components to balance temporal stability and spatial detail.
• It employs a temporal perturbation module and an offset-aware temporal consistency loss to robustly mitigate flicker, jitter, and misalignment artifacts in video sequences.
• The approach outperforms state-of-the-art methods on both spatial and temporal metrics while maintaining a compact and efficient model architecture.

FTPFusion: Frequency-Aware Infrared and Visible Video Fusion with Temporal Perturbation

Introduction

The paper addresses the persistent challenge in infrared and visible video fusion (IVVF): achieving an optimal trade-off between spatial fidelity within individual frames and temporal consistency across consecutive frames. Existing IVVF solutions exhibit limitations—frame-wise methods neglect temporal dependencies, resulting in flickering and motion discontinuity, while holistic spatio-temporal models mitigate flicker but compromise high-frequency details and are vulnerable to misalignment-induced ghosting artifacts. FTPFusion introduces a frequency-aware architecture to differentially model low- and high-frequency components, explicitly separating temporal stability and spatial detail preservation through a specialized dual-branch fusion design augmented with a temporal perturbation mechanism.

Methodology

Frequency-Aware Dual-Branch Fusion

FTPFusion relies on the premise that low-frequency (LF) and high-frequency (HF) components in video sequences encode distinct, complementary information relevant to the IVVF task. Specifically:

• Low-frequency Branch: Primarily responsible for temporal stability and structural continuity. It integrates a Low-Frequency Temporal Perturbation Module (LFPM) that augments robustness by introducing temporal perturbations (shifts) during training, exposing the network to realistic video degradations such as frame-wise flicker, jitter, and misalignment. Further, a Low-Frequency Shared Temporal Context Module (LTCM) encodes shared contextual information from both modalities, generating cross-modal modulation gates that enhance temporal coherence in the fused representation.
• High-frequency Branch: Dedicated to capturing fine-grained spatial details and complementary information. A key innovation here is the Sparse Cross-Modal Attention Module (SCAM), which restricts expensive cross-modal spatio-temporal attention to selectively identified saliency blocks, thereby enabling faithful motion context fusion without incurring excessive computational overhead or global over-smoothing.

These branches independently process frequency-decomposed features, which are subsequently merged via a learnable gain-weighted integration.

Offset-Aware Temporal Consistency Constraint

To directly supervise the model with respect to temporal coherence, the authors introduce an Offset-aware Temporal Consistency (TC) loss. This constraint leverages the low-frequency content, conducting local offset estimation and alignment between adjacent frames and penalizing discrepancies in residual and residual gradients post-alignment—explicitly enforcing robustness to local disturbances (jitter, misalignment) and promoting strong cross-frame consistency in the fused sequence.

Loss Design

A composite loss integrates: pixel-level intensity, gradient-based, color, and the aforementioned offset-aware TC terms. The final loss optimizes for both spatial content preservation and temporal continuity, with adjustable weighting for each component to enable a flexible trade-off during training.

Experimental Analysis

Benchmarks and Evaluation Metrics

FTPFusion is extensively validated against state-of-the-art image and video fusion methods on three public IVVF benchmarks—M3SVD, HDO, and VTMOT—covering diverse scenes and dynamic perturbations. The evaluation suite incorporates eight metrics, including:

• Spatial metrics: $Q_{MI}$, $Q_{TE}$, entropy ($EN$), and source complementary detail ($SCD$).
• Temporal metrics: BiSwE, MS2R, and two task-specific metrics—Modal Mixing Continuity Index (MMCI) and Temporal Correlation Preservation Error (TCPE), both specifically proposed in this work to quantitatively isolate modal mixing stability and temporal correlation retention, respectively. Stress benchmarks further substantiate their discriminative power relative to existing indicators under controlled perturbation regimes.

Main Results

FTPFusion consistently surpasses competitive methods across all datasets, yielding leading scores on both spatial and temporal metrics. Notably:

• On M3SVD, FTPFusion attains $Q_{MI}=0.5816$ (best), $Q_{TE}=0.4088$ (best), and $MMCI=0.1834$ (best), asserting its superior detail preservation and temporal smoothness.
• On HDO, FTPFusion achieves highest performance on $Q_{MI}$, $Q_{TE}$, $EN$, and MMCI.
• On VTMOT, a particularly dynamic dataset, FTPFusion remains robust, with highest or second-highest performance among all methods on both spatial and temporal quality indicators.

Qualitative analysis affirms the preservation of crucial details (e.g., object contours, background textures) and notably reduced flicker and misalignment artifacts compared to both image fusion and video fusion competitors. Downstream video segmentation on fused outputs further demonstrates the method’s ability to maintain semantic and structural integrity, leading to superior segmentation accuracy and temporal stability.

Ablation and Efficiency Studies

Ablations confirm the necessity of each core component:

• Removal of the dual-branch frequency-aware module results in pronounced degradation of both spatial and temporal metrics.
• Isolating the low-frequency branch yields optimal temporal quality but insufficient detail retention, whereas using only the high-frequency branch enhances spatial metrics at the expense of temporal stability.
• Exclusion of the offset-aware TC loss particularly affects temporal quality.

On compute efficiency, FTPFusion is notably compact ($Q_{TE}$01.1M parameters, 132G FLOPs for $Q_{TE}$1P video)—substantially leaner (about one-sixth to one-eighth the size) than baselines such as VideoFusion or UniVF.

Implications and Future Directions

The separation of low- and high-frequency modeling in FTPFusion challenges prior unified frameworks, demonstrating that frequency-aware decoupling is advantageous for balancing temporal and spatial objectives in video fusion. The frequency-specific temporal perturbation strategy effectively regularizes the model against a spectrum of real-world video degradations, indicating broader applicability to other multimodal video understanding problems characterized by dynamic temporal inconsistencies and non-uniform spatio-temporal saliency.

The proposed evaluation metrics, MMCI and TCPE, establish new standards for fine-grained assessment of fused video quality, particularly in robustness-critical applications such as surveillance, autonomy, and video-based perception under degraded visual conditions.

Potential future avenues include extending frequency-aware temporal perturbation to more adverse or non-cooperative sensing regimes, integrating with language-driven guidance to adaptively adjust fusion based on high-level intent, and leveraging more efficient state-space or transformer variants for further reductions in computational overhead.

Conclusion

FTPFusion delivers a compact, computationally efficient framework that explicitly disentangles and optimally models low-frequency (temporal stability) and high-frequency (spatial detail) signals in infrared and visible video fusion. The integration of temporally perturbed learning and offset-aware supervision leads to significant performance improvements on both spatial and temporal axes, with strong evidence provided through standard and purpose-designed evaluation metrics. The methodological innovations of FTPFusion establish clear directions for future research in frequency-aware video fusion, temporal robustness, and multimodal perception pipelines.