Thermal-Det: Open-Vocabulary Thermal Detector

Updated 5 July 2026

Thermal-Det is an open-vocabulary detector that applies synthetic thermal pretraining and cross-modal distillation to bridge the RGB and thermal modality gap.
It adapts a Grounding DINO-style framework with a frozen CLIP text encoder and a Thermal-Text Alignment Head to recalibrate language semantics for thermal imagery.
Zero-shot, language-guided detection performance improves by 2–4% AP over RGB-trained baselines, enabling scalable thermal object detection.

Thermal-Det is an open-vocabulary object detector for thermal imagery that is designed to operate directly on infrared data without requiring manual thermal annotations. It is presented as the first LLM supervised open-vocabulary detector tailored for thermal images, addressing the modality gap between RGB-trained vision-language detectors and thermal scenes, where low texture, emissivity variation, temperature ambiguity, sensor noise, and weak boundaries make RGB-based semantics unreliable. The framework combines synthetic thermal pretraining, language-guided thermal adaptation, and RGB-to-thermal cross-modal distillation, and is instantiated on a Grounding DINO-style detector with a frozen CLIP text encoder, a Thermal-Text Alignment Head, a captioning branch, and a distillation stage driven by a frozen RGB teacher (Ranasinghe et al., 11 May 2026).

1. Problem setting and conceptual scope

Thermal-Det is situated in the open-vocabulary detection literature rather than in thermal metrology or cryogenic detector physics. Its target problem is zero-shot, language-guided object detection in thermal images, where detectors trained on RGB corpora such as GLIP, Grounding DINO, YOLO-World, T-Rex2, and LLMDet do not transfer well because their semantics are anchored by texture, color, and appearance statistics from visible light (Ranasinghe et al., 11 May 2026).

The central claim is that thermal imagery is not merely a domain-shifted version of RGB imagery. In thermal scenes, objects are represented by radiometric emission patterns rather than color or fine texture, and many categories collapse into similar low-detail silhouettes. The paper therefore treats the mismatch as a cross-modal alignment problem involving both visual representation and language grounding. This is why the method does not rely on lightweight adapter-only transfer; instead, it fully fine-tunes the detector so that visual layers can internalize thermal contrast patterns while the frozen text space is recalibrated through an auxiliary alignment head (Ranasinghe et al., 11 May 2026).

A common ambiguity arises from the phrase “thermal detector.” In other literatures, this phrase refers to cryogenic bolometers, calorimeters, or related sensing devices, including the superconductor-ferromagnet thermoelectric detector (Geng et al., 2019) and millikelvin thermal detectors in rare-event experiments (Biassoni et al., 2021, Bucci et al., 2011). It has also been used in a distinct metrology context for nanoscale thermometry via Kikuchi diffraction thermometry (Chen et al., 16 Oct 2025). Thermal-Det, by contrast, is a computer-vision detector for thermal images rather than a physical temperature sensor or thermometric instrument (Ranasinghe et al., 11 May 2026).

2. Synthetic thermal pretraining data

A defining component of Thermal-Det is the construction of a large synthetic thermal dataset from GroundingCap-1M. GroundingCap-1M contains over one million samples with an RGB image $I_{\text{rgb}}$ , grounding text $T_g$ tied to object-level bounding boxes $B$ , and a detailed scene-level caption $T_c$ . Thermal-Det converts these RGB samples into synthetic thermal samples using F-ViTA, a visible-to-thermal translation model, while preserving geometric supervision because the scene layout is retained. The resulting supervision tuple is

$(I_{\text{th}}^{\text{syn}}, T_g, B, T_c).$

This yields over one million thermally aligned samples with bounding boxes, grounding texts, and detailed captions (Ranasinghe et al., 11 May 2026).

Caption adaptation is an essential part of the dataset construction. Because the original captions were written for RGB scenes, the method removes RGB-specific descriptors such as color names and terms like bright, shadowed, or sunlit. The grounding phrases $T_g$ and bounding boxes $B$ are reused as-is. This preserves object identities, spatial relations, and contextual content while making the language more appropriate to thermal imagery (Ranasinghe et al., 11 May 2026).

The synthetic corpus is intended to provide a category vocabulary far broader than conventional thermal benchmarks such as KAIST, FLIR, or LLVIP. This suggests that the pretraining stage is less about reproducing sensor physics exactly and more about preserving large-scale grounding supervision while moving the visual appearance into the thermal domain. The paper also notes, however, that synthetic thermal images do not fully reproduce real sensor effects such as emissivity variation, temperature gradients, and noise, making the synthetic data a scalability mechanism rather than a full substitute for real thermal statistics (Ranasinghe et al., 11 May 2026).

3. Architecture and optimization objectives

Thermal-Det is built on a Grounding DINO-style open-vocabulary detector with a frozen CLIP text encoder. Unlike parameter-efficient thermal adaptation methods, it fully fine-tunes the detector so that both convolutional and attention layers can adapt to thermal patterns. The implementation uses Grounding DINO with a Swin-T backbone; the frozen RGB teacher is also Grounding DINO with a Swin-T backbone (Ranasinghe et al., 11 May 2026).

The total training objective is

$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{det}} + \mathcal{L}_{\text{KD}} + \mathcal{L}_{\text{TTAH}} + \mathcal{L}_{\text{cap}}.$

The detection term is

$\mathcal{L}_{\text{det}} = \mathcal{L}_{\text{cls}} + \mathcal{L}_{\text{box}},$

where $\mathcal{L}_{\text{cls}}$ aligns region features with text embeddings using cosine-similarity contrastive learning, and $T_g$ 0 is a localization loss combining $T_g$ 1 and GIoU/CIoU-style terms (Ranasinghe et al., 11 May 2026).

The captioning term is

$T_g$ 2

Scene-level captioning uses thermal feature maps to generate full descriptive captions for synthetic thermal images. Object-level captioning is supervised by short region phrases such as “man walking” or “car headlights.” For paired real datasets lacking captions, pseudo-phrases are created from the RGB teacher’s open-vocabulary predictions or from available class labels. The paper explicitly states that object-level captioning is especially important because it creates direct semantic grounding between localized thermal regions and linguistic concepts (Ranasinghe et al., 11 May 2026).

Training uses MMDetection v3.2, mixed precision, and gradient checkpointing. The reported settings include input resolution $T_g$ 3, batch size 16, 150K iterations, AdamW with learning rate $T_g$ 4, weight decay 0.05, a cosine schedule with 2K warmup, and a KD:synthetic batch ratio of 3:2. The CLIP text encoder is frozen, while adapters, detection head, and TTAH are trained (Ranasinghe et al., 11 May 2026).

The distillation stage is designed for paired but unlabeled RGB-thermal data. For aligned inputs $T_g$ 5, a frozen RGB teacher produces pseudo-labels and similarity structure, and the trainable thermal student matches them through

$T_g$ 6

The geometric term is

$T_g$ 7

the semantic term is an InfoNCE-style contrastive objective,

$T_g$ 8

and the confidence term is

$T_g$ 9

Distillation is only applied to paired RGB-thermal batches; synthetic thermal batches do not use it (Ranasinghe et al., 11 May 2026).

Thermal-Det supplements distillation with the Thermal-Text Alignment Head (TTAH), motivated by the observation that a frozen CLIP text encoder remains biased toward RGB semantics even after thermal visual adaptation. For each text token embedding $B$ 0, TTAH concatenates it with a learned thermal attribute vector $B$ 1 from a bank containing descriptors such as “hot,” “silhouette,” “reflective,” and “high-emissivity”: $B$ 2 The alignment loss includes a contrastive term,

$B$ 3

and a drift regularizer,

$B$ 4

with

$B$ 5

TTAH also expands each class into multiple thermal sublabels,

$B$ 6

scores them against a thermal region feature $B$ 7,

$B$ 8

and selects the best-matching variant,

$B$ 9

using $T_c$ 0 as the class score (Ranasinghe et al., 11 May 2026).

The caption head employs Modality-Fused Cross-Attention (MFCA), in which LLM queries attend jointly to thermal features and RGB teacher features: $T_c$ 1 with learnable gates $T_c$ 2 and $T_c$ 3. At inference, RGB is absent and the mechanism reduces to thermal-only by setting $T_c$ 4 (Ranasinghe et al., 11 May 2026).

5. Benchmarks, reported performance, and ablations

Thermal-Det is evaluated in true zero-shot thermal detection settings on seven public benchmarks: FLIR-Aligned, FLIR-V2, CAMEL, SMOD, Utokyo, MFAD, and LLVIP (Ranasinghe et al., 11 May 2026).

The main reported benchmark results are summarized below.

Benchmark	Thermal-Det	Comparator values reported
FLIR-Aligned	AP 0.372	LLMDet 0.359; G-DINO 0.337
FLIR-V2	AP 0.096	G-DINO 0.081; LLMDet 0.048
CAMEL	AP 0.511	G-DINO 0.482; LLMDet 0.383
Utokyo	AP 0.065	G-DINO 0.050
SMOD	AP 0.152	G-DINO 0.135
MFAD	AP 0.100	LLMDet 0.102
LLVIP	AP 0.566, AP $T_c$ 5 0.856	Outperforming all zero-shot baselines

The paper characterizes the aggregate improvement as consistent 2–4% AP gains over existing open-vocabulary detectors (Ranasinghe et al., 11 May 2026). At the same time, it compares against fully supervised Grounding DINO fine-tuning as an upper bound and notes that this supervised model still performs substantially better in many cases. This establishes a remaining gap between annotation-free thermal transfer and full thermal supervision rather than claiming parity with supervised thermal detection (Ranasinghe et al., 11 May 2026).

The ablation study attributes gains to several distinct components. Starting from a zero-shot baseline, scene-level captioning provides a small gain, object-level captioning provides a larger gain, cross-modal distillation adds another noticeable boost, and removing $T_c$ 6 reduces performance. The paper also studies TTAH subclass selection strategies: average pooling over thermal sublabels performs worst, random selection is better, and confidence gating is best. On FLIR-Aligned, confidence gating reaches 0.261 AP, compared with 0.234 for average pooling and 0.248 for random selection (Ranasinghe et al., 11 May 2026).

A plausible implication is that the method’s improvements are not dominated by any single module. The reported evidence instead supports a layered view in which synthetic thermal pretraining, region-level caption grounding, RGB-thermal distillation, and text-space recalibration each address a different part of the thermal open-vocabulary problem.

6. Limitations, interpretation, and relation to adjacent thermal-imaging research

Thermal-Det is explicitly limited by the realism of its synthetic data and by the quality of its teacher supervision. The paper states that synthetic thermal images do not fully reproduce real sensor effects such as emissivity variation, temperature gradients, and noise, and that distillation quality depends on the RGB teacher, so teacher errors can propagate. It also notes that the synthetic dataset is richer in region-level phrases than in long image-level descriptions, which can constrain the informativeness of generated captions (Ranasinghe et al., 11 May 2026).

These limitations distinguish Thermal-Det from thermal imaging methods whose outputs are direct temperature or heat-flow measurements. For example, Kikuchi diffraction thermometry maps temperature and pressure by fitting Kikuchi patterns and converting lattice expansion to temperature via the coefficient of thermal expansion (Chen et al., 16 Oct 2025). Electron thermal microscopy detects the liquid-solid transition of nanoscale indium islands to obtain real-time thermal maps in a transmission electron microscope (0708.1522). Lock-in thermography using diamond quantum sensors measures periodic surface temperature through NV-center spin-resonance shifts (Ogawa et al., 2022). Those methods infer thermodynamic quantities. Thermal-Det, by contrast, performs semantic detection in thermal imagery and does not estimate temperature fields (Ranasinghe et al., 11 May 2026).

Within computer vision, its significance lies in making thermal imagery compatible with open-vocabulary language grounding without manual thermal box annotations. This suggests a shift from closed-set thermal detection toward scalable thermal-language perception. The paper identifies likely future directions as better thermal synthesis with more realistic sensor physics, stronger image-level and long-form language supervision, improved teacher robustness or multi-teacher distillation, and larger real paired thermal corpora for more accurate alignment (Ranasinghe et al., 11 May 2026).