Thermal-Det: Language-Guided Cross-Modal Distillation for Open-Vocabulary Thermal Object Detection

Abstract: Existing open-vocabulary detectors focus on RGB images and fail to generalize to thermal imagery, where low texture and emissivity variations challenge RGB-based semantics. We present Thermal-Det, the first LLM supervised open-vocabulary detector tailored for thermal images. To enable large-scale training, we develop a synthetic dataset by converting GroundingCap-1M into the thermal domain and filtering captions to remove RGB-specific terms, yielding over one million thermally aligned samples with bounding boxes, grounding texts, and detailed captions. Thermal-Det jointly optimizes detection, captioning, and cross-modal distillation objectives. A frozen RGB teacher provides geometric and semantic pseudo-supervision for paired but unlabeled RGB-thermal data, transferring open-vocabulary knowledge without manual annotation. The model further employs a Thermal-Text Alignment Head for text calibration and a Modality-Fused Cross-Attention module for dual-modality reasoning. Unlike prior domain-adaptation methods, the detector is fully fine-tuned to internalize thermal contrast patterns while preserving language alignment. Experiments on public benchmarks show consistent 2-4% AP gains over existing open-vocabulary detectors, establishing a strong foundation for scalable, language-driven thermal perception.

• The paper introduces an annotation-free framework that distills knowledge from a frozen RGB detector with synthetic thermal data to achieve zero-shot open-vocabulary detection.
• It leverages an RGB-to-thermal translation model and multi-scale distillation losses—including spatial, semantic, and confidence distillation—to align text and thermal features.
• Empirical evaluations show notable AP improvements over RGB-based detectors on multiple benchmarks, paving the way for scalable thermal perception in critical applications.

Thermal-Det: Language-Guided Cross-Modal Distillation for Open-Vocabulary Thermal Object Detection

Introduction and Motivation

Thermal-Det targets the problem of open-vocabulary object detection (OVD) in the thermal domain, a setting where detectors must identify arbitrary classes specified by text queries, yet where annotation resources are exceptionally limited. While modern RGB-based OVDs have leveraged large-scale language supervision and vision-language foundation models, transferring these capabilities to thermal imagery leads to significant performance degradation due to the substantial modality shift — low texture, differing emissivity, and weak visual cues inherent in thermal data fundamentally challenge RGB-trained semantics. Thermal-Det addresses these deficiencies by proposing an annotation-free, language-driven detection framework capable of robust zero-shot OVD on thermal images.

Methodological Framework

Synthetic Data Generation

Thermal-Det's foundation is a large-scale synthetic thermal dataset, produced by converting the one-million-image GroundingCap-1M RGB corpus into the thermal domain using F-ViTA, an RGB-to-infrared cross-domain translation model. The process yields thermally aligned data with preserved bounding boxes, object-level grounding texts, and detailed scene-level captions. Caption adaptation removes all RGB-specific descriptors, ensuring that textual supervision remains semantically congruent with thermal visual evidence. This strategy enables exposure to high label diversity without the cost of manually annotating real thermal imagery.

Cross-Modal Knowledge Distillation

A frozen RGB open-vocabulary detector acts as a teacher, guiding the training of a thermal student model using a teacher-student setup. Cross-modal distillation objectives are employed at multiple levels:

• Spatial Distillation via GIoU loss enforces geometric alignment of predicted bounding boxes between teacher and student.
• Semantic Distillation through cosine-based InfoNCE aligns feature representations cross-modally.
• Confidence Distillation with KL divergence ensures calibration of class probability outputs.

This enables transfer of both localization and high-level semantics from RGB to thermal, without direct use of thermal annotations.

Language Alignment and Adaptation

To overcome the semantic bias in language representations tied to RGB pretraining, Thermal-Det introduces:

• Thermal-Text Alignment Head (TTAH): This module recalibrates CLIP text embeddings using a learnable bank of radiometric attribute vectors, such as “hot”, “silhouette”, or “reflective”. The TTAH augments each text token with a relevant attribute vector and transforms it via a two-layer MLP with layer normalization. During inference, multiple subclass embeddings per label are scored, promoting high correspondence between region features and their most semantically and thermally plausible text embedding.
• LLM-Based Captioning: A LLM is attached with thermal adapters (LoRA-style residuals in every transformer block), enabling it to contextualize and generate natural language grounded in thermal specificities. Object-level and scene-level captions are jointly trained using thermal projections of detector features, with modulation by both synthetic data and RGB-to-thermal paired imagery.

End-to-End Training Objective

Thermal-Det jointly optimizes detection, distillation, text alignment, and captioning losses, forming an end-to-end system that internalizes both visual and linguistic knowledge for robust open-vocabulary thermal perception.

Figure 1: The Thermal-Det framework employs dual-stream RGB–thermal detection, guided by a frozen RGB teacher and distillation losses at multiple semantic scales.

Experimental Evaluation

Thermal-Det is comprehensively evaluated on a suite of public thermal benchmarks, including FLIR-Aligned, FLIR-V2, CAMEL, SMOD, Utokyo, MFAD, and LLVIP, all under the zero-shot transfer setting (no thermal annotation during training). Across these diverse conditions and datasets, the model consistently delivers strong zero-shot AP and AP$_{50}$/AP$_{75}$ gains over RGB-trained OVDs (GLIP, G-DINO, MM-GDINO, LLMDet, etc.) when using the same Swin-T backbone.

For instance, on FLIR-Aligned, Thermal-Det achieves 0.372 AP and 0.664 AP$_{50}$, outperforming prior bests by 1–3 absolute AP points. On challenging benchmarks such as FLIR-V2 and CAMEL, improvements are even more pronounced, with 0.096 AP and 0.511 AP, respectively, setting new state-of-the-art for open-vocabulary thermal detection without thermal annotations.

Figure 2: Zero-shot detection AP comparison on major benchmarks, demonstrating Thermal-Det's consistent improvement over existing OVDs.

Further class- and scale-wise analyses reveal strong performance for dominant thermal categories (e.g., “person”, “car”) and competitive large-object accuracy, with residual failures linked mainly to small, rare, or extremely ambiguous thermal targets — a pattern that persists even under full supervision, indicating fundamental sensor and resolution challenges, not architectural shortcomings.

Ablation and backbone sensitivity studies confirm:

• Each component (synthetic captioning, object-level grounding, cross-modal distillation, TTAH) yields additive improvements;
• The confidence-gated TTAH surpasses naive averaging/randomization for sublabel selection, emphasizing its efficacy for robust text-thermal alignment;
• Improvements generalize across multiple OVD teacher architectures, affirming the backbone-agnostic nature of the method.

Theoretical and Practical Implications

Thermal-Det establishes the feasibility of fully annotation-free, open-set object detection in the thermal domain, demonstrating that synthetic supervision, language-grounded adaptation, and cross-modal distillation collectively bridge the RGB–thermal modality gap. The approach decouples OVD performance from labeled thermal data, facilitating rapid deployment of perception systems for safety-critical and dynamically evolving scenarios (autonomous vehicles, surveillance, search-and-rescue) where annotated data acquisition is impractical or impossible.

Theoretically, the modular alignment of language and thermal vision, via attribute-based text calibration and direct knowledge transfer, provides a template for extending large-scale vision-LLMs to new sensor domains. The TTAH, in particular, offers a scalable route for semantic adaptation in the presence of nontrivial visual domain shifts.

Future Research Directions

Building on these results, several future directions emerge:

• Improved Synthetic Generation: Reducing the domain shift between synthetic and real thermal images via more faithful sensor simulation or learned augmentation pipelines.
• Adaptive Cross-Modal Teachers: Leveraging ensembles or autoregressive feedback to improve the reliability and diversity of transferred detection supervision.
• Real-World Robustness: Extending to multi-modal temporal reasoning, handling occlusions, adversarial thermal environments, and integrating active inquiry with embodied agents.
• Language-Driven Scene Understanding: Generalizing beyond detection to dense captioning, referring expression comprehension, and compositional reasoning within thermal domains.

Conclusion

Thermal-Det represents a significant technical advance for open-vocabulary thermal perception, fusing language guidance, synthetic supervision, and cross-modal transfer with no dependence on thermal annotations. Empirical results reflect robust generalization, strong AP improvements, and successful adaptation of language-conditioned models to fundamentally texture-poor and visually ambiguous thermal scenes. These results offer a modular blueprint for annotation-light learning in new modalities, accelerating progress in scalable AI systems for non-visible spectra.

(2605.10130)