LDFE: Laplacian Decoupled Feature Enhancement Block for Dual-Stream CNN-based RGB-IR Object Detection

Abstract: The complementary information between RGB and IR images can significantly enhance object detection performance under extreme conditions. Existing methods prefer dual-stream CNN backbones built upon YOLO for feature extraction and focus on the design of feature fusion. In this paper, we introduce the Laplacian Decoupled Feature Enhancement block (LDFE) to fuse features from different stages of the dual-stream CNN backbone. By design, LDFE simultaneously considers the characteristics of modalities and structures for feature fusion by employing global-local decomposition, denoising, fusion, and reconstruction, sequentially. The LDFE first separates features into global and local components based on Laplacian Pyramid, and then performs denoising and fusion based on Global State Space Enhancement module (GS2E) and Local Convolutional Correlation Enhancement module (LC2E) separately. Specifically, the GS2E conducts a two-branch architecture for the main and auxiliary modalities. It dynamically suppresses noise in the main modality through cross-modal attention derived from the auxiliary modality, while employing a State Space Model to capture long-range dependencies within the global feature representations of the main modality. To obtain bidirectional interaction, the two modalities systematically alternate their main/auxiliary roles. Moreover, the LC2E suppresses noise in local features and leverages spatial and channel dimension along with triple convolution to extract fine-grained details for fusion. These innovative designs achieve a significant performance improvement, with mAP surpassing the SOTA methods 6.2%, 3.7%, 4.7%, 2.3%, 4.1% and 2.0% on M3FD, DroneVehicle, LLVIP, FLIR-Aligned, KAIST and VEDAI datasets,respectively.

• The paper introduces a novel Laplacian decoupled feature enhancement block that separates global and local features for improved RGB-IR fusion.
• The paper incorporates GS²E and LC²E modules with targeted denoising and fine-detail extraction, achieving significant improvements in mAP and efficiency.
• The paper demonstrates enhanced robustness and reduced latency across six benchmarks, underpinning its effectiveness in real-time, dual-stream object detection.

LDFE: Laplacian Decoupled Feature Enhancement Block for Dual-Stream CNN-based RGB-IR Object Detection

Introduction

The paper presents LDFE, a Laplacian Decoupled Feature Enhancement block, integrated within dual-stream CNN architectures for RGB-IR object detection. Identifying the complementary nature of RGB (detail, texture) and IR (contour, robustness to illumination) modalities, the paper highlights critical fusion limitations in existing architectures: CNNs excel in local feature extraction but lack global context, whereas Transformers and Mamba achieve global modeling with inferior fine-grained detail sensitivity. The proposed LDFE explicitly decomposes feature maps into global and local components using a Laplacian Pyramid, then applies modality-aware denoising and fusion strategies via tailored modules – Global State Space Enhancement (GS$^2$E) and Local Convolutional Correlation Enhancement (LC$^2$E) – followed by reconstruction. This design achieves significant gains in detection accuracy, runtime efficiency, and parameter compactness across six benchmark datasets.

Figure 1: Deployment efficacy of LDFE versus competing multimodal detection models on FLIR-Aligned, showing superior mAP and resource efficiency.

Architecture: Laplacian Feature Decomposition and Fusion Paradigm

The backbone follows a dual-stream CNN, where paired RGB and IR images are processed independently. Outputs from selected backbone stages (2, 3, 5) are routed through the LDFE block:

• Laplacian Pyramid Decomposition: Features at each stage are separated into global (low-frequency, Gaussian-smoothed) and local (high-frequency, difference) components.
• Global State Space Enhancement (GS$^2$E): Swapped channels from both modalities are utilized to generate cross-modal attention for dynamic noise suppression. Subsequently, state space modeling (Mamba) encodes long-range contextual dependencies, alternating the main/auxiliary modality roles.
• Local Convolutional Correlation Enhancement (LC$^2$E): $L_1$ normalization removes modal-specific local noise. Softmax Fusion integrates modality information. Spatial and channel attention combined with convolutional correlation extraction focus on fine-grained details.

Figure 2: Schematic overview of the dual-stream CNN with LDFE blocks at backbone layers 2, 3, and 5.

Figure 3: Internal structure of GS$^2$E module, highlighting branch alternation and cross-modal noise suppression.

Figure 4: LC$^2$E, demonstrating denoising, attention mechanisms, and correlated convolution for local feature enhancement.

The reconstructed features (global + local fusion) at each stage are aggregated and forwarded to the YOLOv8 neck and head.

Experimental Evaluation

LDFE is extensively evaluated on M$^3$FD, DroneVehicle, LLVIP, FLIR-Aligned, KAIST, and VEDAI datasets, utilizing three standard CNN backbones (ResNet50, CSPDarknet53v5/v8). Results consistently demonstrate:

• M$^3$FD: LDFE achieves mAP improvements of 6.2% over SOTA with CSPDarknet53v8.
• DroneVehicle: Superior classification in categories with high confusion (Truck, Freight-car, Van), ascribed to robust local fusion.
• LLVIP: Gains up to 4.7% mAP, outperforming methods relying solely on pixel-level or global fusion.
• FLIR-Aligned: LDFE not only delivers highest mAP, precision, recall, and F1, but also minimizes parameters and inference latency for real-time deployment.

Figure 5: Grad-CAM heatmaps showing LDFE's focus on target regions versus competitor methods on M$^3$FD, DroneVehicle, LLVIP, FLIR-Aligned.

Figure 6: Detection outputs across several RGB-IR benchmarks; LDFE reduces missed and false detections under occlusion and adverse scenarios.

Comprehensive ablations reveal the necessity of channel swapping, GS$^2$0E/LC$^2$1E modules, optimal fusion block placement, and denoising. Substituting Mamba-like SSM with Transformer for global features reduces accuracy and increases computational burden.

Robustness and Generalization

Tests on M$^2$2FD partitioned by scene (day, night, overcast, challenge) and FLIR-Aligned with synthetic overexposure show LDFE's resilience. Performance degrades minimally compared to non-adaptive architectures, indicating effective suppression of modality-specific noise and adaptability across diverse environments and illumination shifts.

Practical and Theoretical Implications

LDFE advances explicit separation and targeted enhancement of modality-specific global and local features within multimodal detection pipelines. The approach demonstrates:

• Deployment Scalability: Parameter and runtime reductions make LDFE noteworthy for embedded/mobile inference.
• Fusion Algorithm Design: Using Laplacian decomposition enables modular global/local fusion, adaptable to other vision tasks (e.g., segmentation, image fusion).
• State Space Modeling: Integrating dynamic state space modeling (Mamba) in global fusion surpasses Transformer approaches for multimodal context aggregation.
• Noise Robustness: Alternating cross-modal attention for denoising enhances stability against input perturbations and environmental variation.

Future work should explore domain adaptation for dense/small object scenarios and leverage LDFE in broader spectrum fusion applications.

Conclusion

The LDFE block delivers modality-aware, structurally decoupled feature fusion for dual-stream CNN-based RGB-IR object detection. Utilizing Laplacian decomposition, tailored denoising, state space global fusion, and local convolutional attention, LDFE achieves state-of-the-art accuracy and efficiency on multiple benchmarks. Its explicit global/local modeling paradigm sets a foundation for scalable multimodal fusion, with promising extension to other vision tasks and architectures.