Efficient RGB-T Object Detection via Sparse Cross-Modality Fusion

Abstract: RGB-T detectors leverage the complementary strengths of visible and thermal infrared modalities, achieving robust performance under challenging conditions. Many of them resort to heavy dual backbones and exhaustive cross-modality fusion across the entire image, leading to impractically high computational costs. We observe that most image regions are smooth backgrounds (e.g., sky, ground) that can be easily handled by lightweight single-modality models. In light of this observation, we propose a sparse fusion mechanism for efficient RGB-T detection: first rapidly scanning the image to identify the proposals and then carefully examining the remaining sparse proposals via feature fusion. We propose a two-stage framework to instantiate this mechanism, which performs detection in two stages: 1) a lightweight and modality-specific detection stage that produces high-recall RoIs, and 2) a fusion-driven examination and refinement stage that filters out the false positives and refines the bounding boxes. This design enables the detector to adaptively allocate more computational resources to the potential foregrounds, improving the efficiency while ensuring detection accuracy. Extensive experiments show that our method achieves competitive performance with substantially fewer parameters and lower cost, while maintaining strong scalability to high-resolution images.

• The paper introduces SFEDet, a sparse two-stage detector that uses lightweight RGB and thermal proposal networks before applying cross-modality fusion only to candidate regions.
• SFEDet reaches 61.0 mAP and 89.8 AP50 on M3FD with 24.5M parameters and 112.6G FLOPs, substantially reducing computation compared with dense-fusion models such as Fu-Mamba.
• Differential fusion, rolling convolutions, denoising training, and step-wise box refinement improve localization and robustness, while compute scales mainly with proposal count rather than image area.

Motivation and core idea

Feature-fusion-based RGB-T detectors dominate the accuracy leaderboards on multispectral benchmarks, but they uniformly apply heavy dual backbones and dense cross-modality fusion over every pixel, with computational costs reaching hundreds of GFLOPs (e.g., 1133G for Fusion-Mamba at M3FD resolution). The paper's central observation is that most image area consists of smooth backgrounds—sky, ground—that a lightweight single-modality model can already classify correctly. Spending dual-backbone capacity and sophisticated fusion on these regions is wasteful. The authors therefore propose a sparse fusion mechanism: rapidly scan the full image with lightweight modality-specific detectors to obtain high-recall proposals, then apply expensive cross-modality fusion only to the resulting sparse RoIs.

The feasibility of this design rests on an empirical claim the authors verify directly: on FLIR, YOLOv8-Small achieves cut-off recall comparable to YOLOv8-Large (both near 100% at IoU=0.5), despite lower precision and sharply degraded recall at IoU=0.75. This means lightweight RPNs rarely miss objects but produce many false positives and poorly localized boxes—exactly the failure modes that cross-modality fusion can repair. The two limitations of lightweight proposals thus motivate, rather than obstruct, the sparse architecture.

The SFEDet framework

SFEDet instantiates the mechanism as a two-stage detector:

• Stage 1 — proposal extraction: two independent YOLOv8-Small detectors act as modality-specific RPNs. Their proposal sets are unioned and NMS-deduplicated, ensuring high recall since an object missed by one modality may be caught by the other.
• Stage 2 — fusion-driven examination and refinement (FER): RoI features are extracted via RoI Align from three FPN levels per modality and reduced to a $C \times 5 \times 5$ representation. A differential-enhanced group-wise fusion (DEGF) module computes softmax-normalized group weights ($N_{gw}=4$) between modalities, then adaptively supplements each modality using feature differences between fused and original features—an operation the ablation shows is cheaper than concatenation yet outperforms addition. Low-level backbone features are re-captured as auxiliary inputs, which the ablation shows beat high-level FPN features for this purpose.

Two further components target efficiency and robustness. Rolling Convolution replaces all conventional convolutions in FER: it runs two group convolutions in parallel, rolling channels by half a group width in one branch before convolution, then merges with a pointwise layer. This restores inter-group information flow that plain cascaded group convolutions lack; against Conv2d it cuts parameters from 29.0M to 24.5M while matching or exceeding mAP (61.0 vs. 61.1 on M3FD at roughly 22% fewer FLOPs). Denoising training injects $N_{de}=10$ groups of GT boxes perturbed by uniform noise up to ±40% of box dimensions, so the FER module learns from hard, poorly localized proposals that early-training RPNs would otherwise discard.

Box refinement is step-wise: two sequential prediction heads decode residuals conditioned on previous predictions, with auxiliary features re-captured after each box update. Because RoI features lose stride-scale information, the authors normalize distances by the proposal dimensions, unifying the loss format across RPNs and FER heads under standard YOLOv8 losses (BCE + CIoU + DFL). Training proceeds in two stages: RPNs are pretrained separately, then the full detector is trained end-to-end with RPN learning rates decayed by $\lambda = 0.005$.

Empirical results

On M3FD (1024×768), SFEDet reaches 61.0 mAP / 89.8 AP50 with only 24.5M parameters and 112.6G FLOPs—roughly 30% of the compute of comparable detectors. Fu-Mamba attains higher mAP (61.9) but at 288M parameters and 1133G FLOPs, nearly 10× the cost. On FLIR, SFEDet achieves 43.0 mAP / 81.7 AP50 at 69.0G FLOPs versus COFNet's 44.6/83.6 at 197G; on LLVIP it matches COFNet's 65.9 mAP at 163G versus 628G. A sparser SFEDet-Lite variant trades ≤0.1 AP50 for a further ~10G reduction.

Method	Params	FLOPs	mAP	AP50
ICAFusion	120M	370G	–	88.2
EI²Det	116M	391G	55.5	86.2
Fu-Mamba	288M	1133G	61.9	88.0
SFEDet	24.5M	112.6G	61.0	89.8

(M3FD, 1024×768; FLOPs re-evaluated by the authors where marked.)

A controlled comparison against dense-fusion prototypes built on identical backbones isolates the contribution of sparsity itself: on FLIR, sparse SFEDet (69.0G) beats its own dense DEGF variant (75.2G) by 1.6 mAP, indicating that restricting fusion to foreground RoIs improves rather than merely preserves accuracy—consistent with the FER module suppressing false positives (from 26.5 to 2.1 per image on M3FD at matched recall).

Scalability results support the central efficiency claim. When LLVIP image area grows 400% relative to FLIR, SFEDet's FLOPs grow only 2.4×, versus 3.8× for EI²Det—the sub-linear growth follows analytically because FER cost depends on RoI count, not image area. FPS likewise degrades gracefully with object density: SFEDet matches large dense detectors on crowded scenes and substantially exceeds them on sparse ones. Robustness tests show AP50 remains essentially unchanged when proposals are noised down to IoU upper bound 0.7 without retraining (81.5 on FLIR, 89.5 on M3FD), attributable to denoising training and a 1.1× proposal enlargement before pooling. Spatial misalignment between modalities degrades performance as it does for dense detectors, but shift augmentation mitigates it.

Limitations and open questions

Several constraints qualify these results. First, the framework inherits the recall ceiling of its RPNs: any object both single-modality detectors miss is unrecoverable, and the paper does not report end-to-end recall loss on the final benchmarks, leaving the tail-risk of missed hard objects quantified only indirectly through the PR-curve analysis. Second, the reported FLOPs measure computation, not wall-clock latency; the two-stage pipeline with NMS and RoI Align involves data-dependent control flow whose deployment cost on edge hardware is not characterized. Third, the method was evaluated on well-aligned or shift-augmented pairs; behavior under severe unregistration, beyond the shift augmentation studied, remains open. Fourth, the comparison set includes re-evaluated FLOPs computed by the authors rather than universally standardized measurements, though they use official codebases for fairness. Finally, whether the sparse mechanism generalizes to more than two modalities, or to instance-level tasks beyond detection, is not addressed.

Conclusion

SFEDet demonstrates that RGB-T detection need not pay dense-fusion costs across the entire image. By confining expensive cross-modality fusion to high-recall proposals from lightweight modality-specific RPNs, and equipping that stage with group-wise differential fusion, rolling convolutions, step-wise refinement, and denoising training, the framework achieves near state-of-the-art accuracy at roughly a quarter to a tenth of the compute of competing detectors, with sub-linear scaling in image area. The main open question is how much recall headroom the lightweight first stage sacrifices on genuinely hard objects, and whether latency on embedded platforms mirrors the favorable FLOPs profile.