IR-TD: Infrared-Text Dataset for Multimodal Learning

Updated 6 July 2026

IR-TD is a large-scale authentic infrared image-text corpus comprising over 260K real image-text pairs designed to advance infrared multimodal learning.
It is structured into pretraining, instruction, and benchmarking subsets, enabling robust evaluation across nine specific infrared understanding tasks.
The dataset employs innovative tri-modal data translation and adaptive curriculum learning to optimize infrared-text alignment and boost model performance.

InfraRed-Text Dataset (IR-TD) is a large-scale real-world infrared vision-language resource introduced with IRGPT and described as comprising over 260K authentic image-text pairs built from real infrared imagery rather than style-transferred visible images (Cao et al., 19 Jul 2025). It is designed to address a structural limitation in infrared multimodal learning: visible-domain vision-LLMs benefit from web-scale image-text corpora, whereas infrared imagery has very little aligned text and is intrinsically harder to describe because it is semantically sparse and visually different from natural RGB imagery. In that framing, IR-TD functions simultaneously as pretraining data, instruction-tuning data, and a benchmark substrate for nine infrared understanding tasks (Cao et al., 19 Jul 2025).

1. Historical setting and motivation

IR-TD is motivated by the claim that the lack of aligned infrared-text supervision is a primary reason existing multimodal LLMs hallucinate on infrared inputs (Cao et al., 19 Jul 2025). The associated paper contrasts IR-TD with prior synthetic infrared-text resources constructed by style-transferring visible images into infrared-like images, arguing that such data are bounded by the generative model’s training distribution and therefore “fail to preserve the authenticity” of infrared imagery. IR-TD instead emphasizes authentic infrared images and treats real thermal patterns, spectral characteristics, and application-relevant semantics as indispensable training signals (Cao et al., 19 Jul 2025).

This positioning is also historically significant because earlier broad surveys of infrared datasets did not identify any standard public benchmark category centered on infrared images paired with text, captions, OCR strings, or scene-text annotations (Danaci et al., 2022). That absence is consistent with the paper’s diagnosis that infrared-language alignment had been bottlenecked by a missing data layer rather than only by missing model architecture.

The resource is presented as open-source, authentic, and curated to reduce redundancy. The paper frames it not as a narrow benchmark for a single downstream task, but as an enabling corpus for infrared multimodal learning at multiple stages, from incremental pretraining to supervised instruction tuning and held-out evaluation (Cao et al., 19 Jul 2025).

2. Corpus structure, scale, and coverage

IR-TD contains more than 260K total samples and is divided into three parts: 190K pre-training image samples, 33K instruction samples, and 37K benchmark samples (Cao et al., 19 Jul 2025).

Component	Size	Role
Pre-training image samples	190k	Infrared-text alignment during incremental pretraining
Instruction samples	33k	Visual instruction tuning via Q/A pairs
Benchmark samples	37k	Evaluation across 9 tasks

The images are aggregated from 63 publicly available datasets. The paper names example sources such as UVT2000, UVT20K, and VTUAV, and indicates that the corpus spans multiple infrared sensing regimes, explicitly mentioning NIR, SWIR, and TIR in the curriculum figure (Cao et al., 19 Jul 2025). The sample-distance distribution is described as bimodal because infrared cameras are “predominantly” used in two spectral bands—thermal imaging and near-infrared—and because image clarity varies strongly across datasets.

The pretraining subset is formed from a final curated collection of 84,284 RGBT pairs plus additional rule-generated infrared-text samples. The paper states that visible images were processed by LLMs to generate descriptive texts, while rule-based methods created 106k infrared-text pairs using annotations. It does not provide an exact per-source decomposition summing cleanly to the reported 190K pretraining samples, and it does not explicitly state whether a single infrared image can appear with multiple texts (Cao et al., 19 Jul 2025).

Task coverage reveals the semantic breadth of the benchmark subset. The nine tasks are recognition, grounding, location prediction, relationship judgment, person re-identification, security screening, aerial vehicle counting, pedestrian counting, and scene understanding (Cao et al., 19 Jul 2025). This suggests that IR-TD is not merely a caption corpus; it is a mixed infrared-text resource spanning caption-like descriptions, instruction-following, localization language, relation reasoning, and counting-oriented supervision.

3. Construction pipeline and annotation methodology

The paper frames IR-TD construction as “tri-modal data translation” among infrared images, visible images, and text (Cao et al., 19 Jul 2025). It considers several possible routes for creating infrared-text pairs and retains two: generation with aligned RGBT pairs, and rule-based generation from annotations.

In the first route, real visible-infrared paired images are collected and manually chosen. Because many datasets exhibit field-of-view mismatch between visible and infrared cameras, the semantic content is aligned by cropping images based on object locations derived from labels. For nighttime visible images, the pipeline uses Retinexformer “to improve observability.” The visible images are then sent to an LLM, which generates descriptive texts, and those drafts are “carefully adapted” to annotate the corresponding infrared images and create QA pairs (Cao et al., 19 Jul 2025). The abstract describes the resulting texts as “meticulously handcrafted,” but the paper does not document the exact annotation interface, number of annotators, inter-annotator agreement, or editing protocol.

In the second route, rule-based generation is used when visible imagery is inadequate, explicitly including nighttime, small targets, and camouflaged objects. This branch creates infrared-text pairs directly from available labels and is also used to craft task-oriented QA pairs in the instruction subset (Cao et al., 19 Jul 2025).

Two quality-control procedures are concrete. First, semantic alignment is enforced by object-based cropping between visible and infrared views. Second, redundancy is reduced by rigorous resampling of video-derived datasets; the paper gives the explicit example of keeping only 1% of VTUAV’s 1.7M images (Cao et al., 19 Jul 2025). The paper does not mention near-duplicate hashing, textual deduplication, or formal human verification metrics.

Textually, IR-TD supports at least two annotation styles. The pretraining portion contains descriptive image-text pairs derived from visible-image descriptions and adapted for infrared images. The instruction portion contains Q/A pairs, and the LLM-based component uses “the same prompts as LLaVA” to transform descriptive pairs into instruction data (Cao et al., 19 Jul 2025). The exact prompt templates are not reproduced in the main text.

4. Benchmark design and use in IRGPT

IR-TD serves as the substrate for IRGPT’s incremental pretraining, supervised instruction tuning, and benchmark evaluation (Cao et al., 19 Jul 2025). The benchmark uses Accuracy for Scene, Recognition, Relationship, ReID, and Security; [email protected] for Grounding; and MAE for Location, Aerial Counting, and Pedestrian Counting. To summarize mixed metric directions, the paper aggregates six “positive” metrics into psum and three error metrics into nsum (Cao et al., 19 Jul 2025).

A technical feature of IR-TD is that its pretraining subset is ordered by a bi-cross-modal curriculum using two sample-level difficulty notions: IR-VIS difficulty and IR-T difficulty. The exact IR-VIS equations are not reproduced in the provided excerpt, but the IR-T component is explicit. A dynamic metric called loss variation rate is defined as

$\alpha = \frac{l'-l}{l},$

where $l$ is the pre-warm-up loss and $l'$ is the post-warm-up loss for a sample (Cao et al., 19 Jul 2025). The adaptive sample weight is then

$w_i= \begin{cases}1-\sigma\left(\frac{\alpha_i}{\operatorname{Median}\left(\left\{\alpha_j \mid \alpha_j>0\right\}\right)}\right) & \alpha_i>0, \ 1+\sigma\left(\frac{-\alpha_i}{\operatorname{Median}\left(\left\{-\alpha_k \mid \alpha_k \leq 0\right\}\right)}\right) & \alpha_i \leq 0,\end{cases}$

followed by the weighted loss

$\mathcal{L} = \frac{1}{N} \sum_{i=1}^N w_i \cdot \left[ -\sum_{c=1}^C y_{i,c} \log p_{i,c}\right].$

This curriculum is integrated into incremental pretraining on IR-TD and is coupled with an ascending-stratified random schedule over difficulty tiers (Cao et al., 19 Jul 2025).

End-to-end results are presented as evidence that IR-TD is practically useful. In zero-shot evaluation, curriculum-trained IRGPT reaches 328.65 psum and 93.87 nsum, compared with 252.30 psum and 125.43 nsum for InternVL2-8B. After fine-tuning, IRGPT reaches 485.79 psum and 4.39 nsum, outperforming InternVL2-26B at 473.78 psum and 19.57 nsum (Cao et al., 19 Jul 2025). The paper also reports that using both curriculum lessons plus the $\alpha$ -based weighting yields the best fine-tuned result.

5. Relation to adjacent infrared-text resources

Several later or adjacent resources are relevant to IR-TD, but they differ substantially in scope, authenticity, or task structure.

Resource	What it provides	Relation to IR-TD
MM-RIS (Ma et al., 16 Sep 2025)	12.5k training and 3.5k testing triplets of IR-VIS pair, referring expression, and binary mask	A task-specific multimodal referring segmentation benchmark, not a general infrared-text corpus
FZDT (Huang et al., 10 Mar 2025)	2,755 infrared images with fuzzy semantic textual annotations for IR small target detection	A text-image bimodal benchmark for IRSTD, narrower and detection-centric
LangIR (Singh et al., 17 Jul 2025)	Text descriptions added to IRSTD-1k and NUAA-SIRST, with 427 and 1,001 text descriptions respectively	A multimodal IRSTD extension using GPT-4V-generated image-level language prior
FusionRS (Han et al., 15 Jun 2026)	600,000 RGB–infrared-style–text triplets in remote sensing	Large-scale and text-rich, but the IR modality is translated and explicitly not sensor-captured

A distinct line of work uses text with infrared data without yielding a reusable IR-TD-style corpus. DiffV2IR introduces IR-500K, a 500,000-image infrared collection plus about 70,000 visible-infrared pairs, but the only explicit text tied directly to IR-500K is the generic prompt “an infrared image,” rather than image-specific captions or rich natural-language supervision (Ran et al., 24 Mar 2025). The text-guided IVIF work on M $^3$ FD, TNO, and RoadScene claims “the first dataset of paired infrared and visible images accompanied by text prompts,” but does not assign a stable formal dataset name, and its prompt-construction protocol remains under-described (Li et al., 2023). A further methodological paper on inference-time scaling for infrared generation uses KAIST and 1,000 infrared image-caption pairs but explicitly does not introduce a new infrared-text dataset; its main relevance lies in prompt formatting and verifier design for low-data generation settings (Horstmann et al., 10 Nov 2025).

Taken together, these resources show that “infrared-text dataset” now denotes several distinct regimes: authentic image-text corpora for VLMs, referring-expression datasets, text-guided detection datasets, translated RGB-to-IR remote-sensing corpora, and method-specific small paired subsets. IR-TD, in the narrow sense established by IRGPT, refers to the large authentic real-world corpus rather than to this broader family (Cao et al., 19 Jul 2025).

6. Limitations, ambiguities, and nomenclature

IR-TD is substantial, but the main paper leaves several practical details under-specified. The exact prompt templates, detailed annotation protocol, number of annotations per image, per-task sample counts, text-length statistics, dataset-by-dataset composition, and explicit train/validation/test split definitions are omitted or deferred to appendices not included in the provided text (Cao et al., 19 Jul 2025). The paper also does not discuss formal licensing terms, consent procedures, or privacy processing in the main text.

Another limitation is evaluative rather than curatorial: the paper argues strongly that authentic infrared-text pairs are preferable to synthetic pairs, but it does not provide a direct controlled synthetic-versus-real ablation. The superiority of real infrared-text supervision is therefore argued conceptually and supported indirectly by end-to-end performance rather than isolated experimentally (Cao et al., 19 Jul 2025).

The term “IR-TD” also coexists with an unrelated “TeX” nomenclature in thermal imaging. In “TeX-NeRF” and “TeX-1500,” “TeX” means temperature, emissivity, and texture, not textual annotations; those datasets are relevant to infrared-to-texture or physical-property decomposition, not to infrared-plus-language learning (Zhong et al., 2024, Dai et al., 2 Jun 2026). This distinction is essential because the lexical similarity between “text” and “TeX” can obscure a categorical divide between language supervision and physical decomposition targets.

A plausible implication is that IR-TD’s most durable contribution lies not only in raw scale, but in establishing authentic infrared-text alignment as a first-class data problem. The surrounding literature increasingly adopts text prompts, referring expressions, QA supervision, or synthetic IR-style captions, yet much of it remains task-specific or sensor-inauthentic. IR-TD’s contribution is to consolidate real infrared imagery, descriptive language, instruction data, and benchmark evaluation within one infrared-native vision-language framework (Cao et al., 19 Jul 2025).