Bridging the RGB-IR Gap: Consensus and Discrepancy Modeling for Text-Guided Multispectral Detection

Abstract: Text-guided multispectral object detection uses text semantics to guide semantic-aware cross-modal interaction between RGB and IR for more robust perception. However, notable limitations remain: (1) existing methods often use text only as an auxiliary semantic enhancement signal, without exploiting its guiding role to bridge the inherent granularity asymmetry between RGB and IR; and (2) conventional data-driven attention-based fusion tends to emphasize stable consensus while overlooking potentially valuable cross-modal discrepancies. To address these issues, we propose a semantic bridge fusion framework with bi-support modeling for multispectral object detection. Specifically, text is used as a shared semantic bridge to align RGB and IR responses under a unified category condition, while the recalibrated thermal semantic prior is projected onto the RGB branch for semantic-level mapping fusion. We further formulate RGB-IR interaction evidence into the regular consensus support and the complementary discrepancy support that contains potentially discriminative cues, and introduce them into fusion via dynamic recalibration as a structured inductive bias. In addition, we design a bidirectional semantic alignment module for closed-loop vision-text guidance enhancement. Extensive experiments demonstrate the effectiveness of the proposed fusion framework and its superior detection performance on multispectral benchmarks. Code is available at https://github.com/zhenwang5372/Bridging-RGB-IR-Gap.

• The paper introduces a novel semantic-bridge framework that treats text as a central anchor to guide dynamic fusion of RGB and IR features.
• It decomposes multispectral evidence into consensus and discrepancy supports, achieving significant mAP gains and reducing computational complexity.
• Empirical evaluations demonstrate enhanced detection robustness and efficiency under adverse conditions and modality degradation.

Semantic Bridge Fusion for Text-Guided Multispectral Object Detection

Introduction and Motivation

Multispectral object detection leveraging RGB and IR imagery is fundamental for achieving robust perception in challenging environments, such as low illumination and adverse weather. Prevailing fusion approaches have treated textual input as an auxiliary, failing to explicitly leverage language for reconciling the highly asymmetric representations produced by RGB (fine-grained textures and color) and IR (spatially coarse, thermally salient responses). The paper "Bridging the RGB-IR Gap: Consensus and Discrepancy Modeling for Text-Guided Multispectral Detection" (2604.11234) introduces a semantic-bridge-guided fusion framework that treats text as a first-class semantic anchor, structuring both cross-modal alignment and interaction. This approach decomposes RGB-IR fusion into consensus (jointly supported evidence) and discrepancy (modality-specific supplementary cues) and fuses them through learnable, dynamic recalibration, guided by text semantics.

Semantic-Bridge-Guided Framework

The proposed framework structurally reframes the RGB-IR fusion problem at the semantic level, leveraging language as the semantic bridge:

• Shared Semantic Bridge: Text representations provide unifying category-level anchors, ensuring that both RGB and IR features are conditioned on consistent semantics during interaction.
• Consensus and Discrepancy Support Modeling: The framework disentangles multimodal evidence into consensus (joint activations) and discrepancy (modality-unique responses). This decomposition is essential, as valuable discriminative cues—especially under severe modality degradation—may manifest more strongly as discrepancies rather than consensus.
• Dynamic Recalibration Mechanism: Both consensus and discrepancy supports are injected into the fused representation via learnable coefficients, allowing the model to adaptively select and integrate information from both support types, suppressing spurious discrepancies while preserving critical evidence.

  Figure 1: Overview of the framework, showing the main pipeline: semantic-bridge-guided dynamic fusion, bidirectional semantic alignment, and detection.

  

  Figure 2: Illustration of the bi-support modeling, where consensus and discrepancy supports are constructed and dynamically integrated.

Bidirectional Semantic Alignment

To ensure tight coupling between visual features and language semantics, the detection architecture introduces a bidirectional semantic alignment module:

• Text-to-Vision Attention: Text queries retrieve spatial evidence from vision features, allowing language to selectively focus on semantically relevant regions.
• Vision-to-Text Update: Fused visual cues reciprocally update the text embedding, evolving raw semantic anchors into context-aware, scene-adaptive semantics.

This closed-loop strategy maximizes cross-modal consistency, improves discriminative capacity, and ensures updated text embeddings reflect domain-specific or open-vocabulary detection targets.

Lightweight, Frequency-Aware IR Backbone

IR feature extraction is implemented via a dual-branch, frequency-aware backbone, which separates IR signals into low-frequency (thermal/statistical) and high-frequency (structural) components with a learnable spectral mask. This design ensures that stable IR cues are efficiently harnessed, compensating for IR’s inherent lack of fine-grained texture without incurring significant computational overhead.

Empirical Results and Analysis

The evaluation on four representative RGB-IR benchmarks—FLIR, LLVIP, DroneVehicle, and M3FD—demonstrates pronounced advantages over both unimodal and leading multimodal baselines:

• Performance: On FLIR and LLVIP, the framework achieves a notable mAP improvement over state-of-the-art fusion networks, e.g., a 12.9% mAP gain versus MS2Fusion with 88.1% fewer parameters. On DroneVehicle and M3FD, it achieves further mAP improvements of 3.8% to 19% over top-performing baselines.
• Efficiency: The semantic-bridge design reduces computational complexity by replacing dense visual-to-visual cross-attention with sparse, text-mediated interactions. Concretely, FLOPs are reduced by 70.1% compared to conditional prompt or direct fusion paradigms.
• Robustness to Degradation: Under progressive RGB degradation, population-level and instance-level studies show a systematic migration of discriminative evidence from consensus to discrepancy support. The system maintains detection capability as conventional consensus cues vanish.

  Figure 3: FLIR qualitative comparison—reliable detection in varied conditions: day, night, occlusion, and crowding.

  

  Figure 4: Performance comparison on FLIR shows the superiority of semantic-bridge fusion over direct and conditional prompt fusion.

  

Figure 5: Instance-level degradation analysis—the consensus response diminishes as RGB is degraded, while discrepancy support grows in importance.

Figure 6: Population-level LLVIP trend—consensus support in GT regions falls with RGB degradation, discrepancy support reciprocally rises, validating support pattern complementarity.

Component-Level and Analytical Insights

Ablation studies demonstrate that:

• Both consensus and discrepancy supports provide measurable performance boosts when included; dynamic recalibration further amplifies this benefit.
• Removing bidirectional semantic updates or anchoring text with fixed random embeddings consistently degrades detection accuracy, confirming that tight vision-text coupling is essential.
• The lightweight IR backbone yields detection performance on par with, or superior to, much larger backbones (e.g., ResNet50), while incurring significantly less computational cost.

Figure 7: Visualization under challenging dark conditions—discrepancy support in GT boxes adds unique discriminative evidence beyond consensus.

Figure 8: Trends in consensus/discrepancy activation across dataset degradation levels, quantifying the adaptive complementarity.

Practical and Theoretical Implications

This work establishes textual semantics as a foundational modality, not merely an auxiliary cue, in multispectral detection. Treating text as a semantic bridge organizes RGB and IR evidence extraction at a conceptual, category-aligned level, facilitating robust detection under adverse and open-vocabulary conditions. The explicit modeling of consensus and discrepancy, with adaptive gating, generalizes to any scenario characterized by modal asymmetry and conditional evidence migration. The proposed efficiency improvements also enable deployment on real-time embedded systems.

Future directions include extending semantic bridge and bi-support fusion to more diverse sensory modalities (e.g., LiDAR, radar) and investigating self-supervised or continual adaptation for evolving open-vocabulary regimes.

Conclusion

The semantic-bridge-guided fusion framework advances multispectral object detection by raising the granularity of RGB-IR interaction from static, pixel-level aggregation to language-guided, category-consistent support modeling. The introduction of consensus-discrepancy bi-support, dynamically recalibrated under textual semantic anchors, is shown to yield robust accuracy, efficiency, and generalization across diverse, challenging environments. The approach has substantial implications for cross-modal perception and motivates future research into higher-order semantic fusion architectures and open-set detection scenarios.