IRGPT: Multimodal Infrared Vision-Language Model

• IRGPT is a multimodal infrared vision-language model that leverages authentic infrared images and a bi-cross-modal curriculum to overcome the limitations of synthetic data.
• The architecture builds on InternVL2-8B and employs a two-stage training process with incremental pre-training of the vision encoder followed by LoRA-based fine-tuning.
• IRGPT achieves state-of-the-art zero-shot and fine-tuned performance across nine infrared tasks by using an ascending-stratified random curriculum and bi-cross-modal weighting.

IRGPT is a multimodal LLM specialized for understanding real-world infrared imagery. It was introduced together with IR-TD, a large-scale infrared–text corpus comprising over 260K authentic image–text pairs, and a benchmark spanning nine tasks such as recognition and grounding. The system addresses a central limitation of prior infrared vision-language work: most earlier pipelines depended on synthetic thermal images generated from visible imagery, whereas IRGPT is trained on real infrared data and adapted through a bi-cross-modal curriculum that ranks difficulty along both infrared–visible and infrared–text axes. In reported zero-shot and fine-tuned evaluations, IRGPT achieves state-of-the-art performance even against larger general-purpose multimodal models (Cao et al., 19 Jul 2025).

1. Infrared vision-language understanding as a distinct modality problem

Real-world infrared imagery differs from visible imagery at both the sensing and semantic levels. Infrared sensors measure thermal or near-infrared energy rather than reflected visible light, so saliency is often dominated by thermal signatures, fine texture is suppressed, object boundaries can be weak, and noise patterns, dynamic range, and contrast vary with sensor, wavelength band, and scene conditions (Cao et al., 19 Jul 2025). The paper explicitly identifies Near-Infrared (NIR), Short-Wave Infrared (SWIR), and Thermal Infrared (TIR), and notes the typical ranges used in remote sensing and vision practice: NIR is approximately $0.75\text{–}1.0\,\mu\mathrm{m}$, SWIR approximately $1.0\text{–}2.5\,\mu\mathrm{m}$, mid-wave IR approximately $3\text{–}5\,\mu\mathrm{m}$, and long-wave IR approximately $8\text{–}14\,\mu\mathrm{m}$. Thermal IR in the LWIR regime is the most visually dissimilar to visible imagery.

This modality gap has direct consequences for vision-LLMs. Infrared images are not accompanied by web-scale caption corpora, and they are difficult to describe because their semantics depend on thermal radiation and emissivity rather than reflectance and appearance. The paper also emphasizes label sparsity and semantic ambiguity: in infrared scenes, material and contextual distinctions can be confounded by temperature and emissivity, and small or camouflaged targets often present weak textual cues.

A recurring misconception in the literature is that visible-to-thermal style transfer can supply adequate pretraining data. IRGPT rejects that premise. According to the paper, synthetic IR produced by RGB-to-TIR translation adheres to the generator’s training distribution, inherits generator artifacts, and fails to capture sensor physics, noise statistics, and out-of-distribution conditions such as nighttime imagery, adverse weather, and small-target scenes. The resulting modality gap leads mainstream multimodal LLMs to hallucinate when applied directly to infrared data.

2. IR-TD: dataset construction, alignment, and benchmark design

IRGPT is built on IR-TD, a large-scale infrared–text dataset assembled from 63 publicly available infrared and RGB-T corpora (Cao et al., 19 Jul 2025). The dataset is divided into three components: a pre-training subset with 190k image–text samples, an instruction subset with 33k samples for supervised instruction tuning, and a benchmark with 37k samples across nine evaluation tasks. After alignment and filtering, 84,284 aligned RGB-T pairs are retained, and additional rule-based pairs raise the total to more than 260K infrared–text pairs.

Component	Size	Role
Pre-training subset	190k	Incremental pre-training
Instruction subset	33k	Supervised instruction tuning
Benchmark	37k	Nine-task evaluation

The data pool spans day and night scenes, indoor and outdoor environments, aerial and ground viewpoints, varied weather, and multiple sensor types and spectral bands. To address field-of-view and camera-parameter mismatches in visible–infrared pairs, the pipeline crops images using labeled object locations so that the semantics align across modalities. Video redundancy is explicitly reduced; for example, only $1\%$ of VTUAV’s $1.7$M frames are retained.

IR-TD uses two complementary text-generation pathways. First, for paired visible and infrared images, a LLM following LLaVA prompt templates generates detailed descriptions from the visible image, and the resulting text is transferred to the corresponding infrared image with adaptations when needed. If the visible image is too dark, as in nighttime conditions, Retinexformer is used before description generation. Second, datasets with structured annotations are converted into captions and question-answer pairs through deterministic templates designed for tasks such as recognition, grounding, counting, and security. This second path is particularly important when visible imagery is not sufficiently informative, such as for small or camouflaged targets.

The paper reports semantic alignment via cropping and label-driven correspondence, redundancy mitigation through subsampling, and later curriculum-based filtering through difficulty metrics. It also notes what is not reported: no inter-annotator agreement or human validation figures are given, and caption length distributions and vocabulary size are not specified. IR-TD is presented as openly released, with the manuscript indicating an open-source repository at https://github.com/WheatCao/ICCV2025-IRGPT.

3. Model architecture and training procedure

IRGPT is an infrared-specialized multimodal LLM built on InternVL2-8B (Cao et al., 19 Jul 2025). The core LLM is the InternVL2-8B decoder, while the visual side uses the visual backbone from InternVL2 and adapts it to infrared through incremental pre-training. A trainable vision projector maps image features into the LLM’s token embedding space, and the multimodal connector is a learned direct projector rather than a Q-Former. Cross-attention is therefore handled by the standard LLM architecture after visual-token projection.

The architecture is trained in two stages. During incremental pre-training, only the vision encoder and projector are updated, while the LLM remains frozen. This stage aligns infrared image features to text using the curriculum mechanism. During supervised instruction fine-tuning, the projector and the LLM are tuned jointly, with the LLM adapted through LoRA on task-oriented question-answer data.

The reported optimization setup is specific. Incremental pre-training uses AdamW with learning rate $10^{-3}$, batch size $32$, one epoch, and a cosine learning-rate schedule. Supervised instruction fine-tuning uses AdamW with learning rate $2\times 10^{-5}$, batch size $32$, five epochs, and a cosine schedule. Compute resources and mixed-precision settings are not reported in the manuscript.

Conceptually, the architecture is conservative: IRGPT does not introduce a radically new multimodal connector, but instead combines an established general-purpose MLLM backbone with IR-specific data, IR-specific curriculum transfer, and a two-stage adaptation path. A plausible implication is that its reported gains derive less from architectural novelty in the connector and more from modality-authentic supervision and sample scheduling.

4. Bi-cross-modal curriculum transfer learning

A defining feature of IRGPT is its bi-cross-modal curriculum, which organizes training examples by difficulty from two complementary perspectives and then schedules them in ascending, stratified-random order (Cao et al., 19 Jul 2025). The curriculum has two “lessons.”

The first lesson measures infrared–visible transfer difficulty. Its premise is that some infrared samples are closer to the visible domain and are therefore easier for a visible-trained encoder to adapt to, whereas others, such as LWIR thermal scenes, lie farther away. To estimate this, the method retrains a feature extractor from InfMAE on both infrared and visible grayscale images, computes domain discrepancy with Maximum Mean Discrepancy under a Gaussian kernel, and uses the infrared and visible domain centers to define an inter-domain direction. Sample difficulty is then derived from the projection of an infrared sample’s displacement from the infrared center onto that inter-domain direction, together with the domain discrepancy term.

The second lesson measures infrared–text alignment difficulty. A pre-warmed CLIP model provides an initial loss $1.0\text{–}2.5\,\mu\mathrm{m}$0 for each sample, and a post-warm stage provides $1.0\text{–}2.5\,\mu\mathrm{m}$1. Their relative change is summarized by

$1.0\text{–}2.5\,\mu\mathrm{m}$2

If $1.0\text{–}2.5\,\mu\mathrm{m}$3, the loss worsens, which the paper interprets as a sign of misalignment or noisy supervision and therefore a case for down-weighting. If $1.0\text{–}2.5\,\mu\mathrm{m}$4, the loss improves, which indicates learnable alignment and motivates a higher weight.

The two rankings are integrated into a comprehensive difficulty ordering. Training data are partitioned into $1.0\text{–}2.5\,\mu\mathrm{m}$5 tiers and presented from easy to hard, while samples are drawn randomly within each tier. This “ascending-stratified random” schedule is paired with a piecewise dynamic weighting rule and a weighted cross-entropy objective,

$1.0\text{–}2.5\,\mu\mathrm{m}$6

The intent is to emphasize hard but well-aligned samples while suppressing misaligned examples.

The paper further reports that the infrared–visible difficulty distribution is bimodal, reflecting the predominance of two bands and variable dataset clarity, whereas the infrared–text CLIP-loss distribution is Gaussian-like. It also finds that Lesson 2, the infrared–text alignment component, is more sensitive for improving the aggregate positive-task score $1.0\text{–}2.5\,\mu\mathrm{m}$7, indicating that semantic correctness is especially vulnerable to noisy cross-modal pairing.

5. Benchmark tasks, metrics, and empirical performance

The IR-TD benchmark evaluates nine tasks over 37k+ samples (Cao et al., 19 Jul 2025).

Task	Output type	Metric
Scene classification	Class label	Acc.
Recognition	Multiple-choice category	Acc.
Grounding	Bounding box	[email protected]
Relationship	Spatial statement validation	Acc.
Re-ID	Identity match in $1.0\text{–}2.5\,\mu\mathrm{m}$8 grid	Acc.
Security	Absent-option selection	Acc.
Aerial Counting	Vehicle count	MAE
Pedestrian Counting	Crowd count	MAE
Location	Coordinates of all instances	MAE

Aggregate reporting uses $1.0\text{–}2.5\,\mu\mathrm{m}$9, the sum of the six positive metrics where higher is better, and $3\text{–}5\,\mu\mathrm{m}$0, the sum of the three MAEs where lower is better.

In the zero-shot setting, IRGPT with the proposed curriculum achieves $3\text{–}5\,\mu\mathrm{m}$1 and $3\text{–}5\,\mu\mathrm{m}$2, reaches state of the art on seven tasks, and exceeds the InternVL2-8B baseline by $3\text{–}5\,\mu\mathrm{m}$3 points on $3\text{–}5\,\mu\mathrm{m}$4. The paper specifically highlights a $3\text{–}5\,\mu\mathrm{m}$5 gain on Re-ID over baseline. Curriculum variants matter: IRGPT trained with random scheduling yields $3\text{–}5\,\mu\mathrm{m}$6 and $3\text{–}5\,\mu\mathrm{m}$7, while anti-curriculum, which presents hard samples first, drops to $3\text{–}5\,\mu\mathrm{m}$8 and $3\text{–}5\,\mu\mathrm{m}$9. Even InternVL2-26B, a larger baseline, reports only $8\text{–}14\,\mu\mathrm{m}$0 and $8\text{–}14\,\mu\mathrm{m}$1.

In the fine-tuned regime, IRGPT with the proposed curriculum reaches $8\text{–}14\,\mu\mathrm{m}$2 and $8\text{–}14\,\mu\mathrm{m}$3, outperforming InternVL2-26B at $8\text{–}14\,\mu\mathrm{m}$4 and $8\text{–}14\,\mu\mathrm{m}$5. The reported margin is $8\text{–}14\,\mu\mathrm{m}$6 on $8\text{–}14\,\mu\mathrm{m}$7 and $8\text{–}14\,\mu\mathrm{m}$8 on $8\text{–}14\,\mu\mathrm{m}$9. Random scheduling remains competitive at $1\%$0 and $1\%$1, but still trails the full curriculum. Anti-curriculum degrades to $1\%$2 and $1\%$3, and the paper notes failures in Re-ID, including collapse to repeated outputs.

Ablations reinforce the curriculum claim. Removing the curriculum or reversing it reduces performance. Combining both lessons with dynamic $1\%$4-based weighting gives the best fine-tuned result, again $1\%$5 and $1\%$6. The proposed ascending-stratified random schedule also outperforms purely deterministic ascending order, descending order, and bidirectional schedules.

6. Significance, limitations, and boundaries of the term

IRGPT’s reported contribution is threefold: authentic infrared–text supervision at scale, a bi-cross-modal curriculum that jointly models domain transfer and text alignment, and a two-stage adaptation recipe for a strong general-purpose multimodal backbone (Cao et al., 19 Jul 2025). Relative to mainstream MLLMs such as LLaVA, Qwen2-VL, and InternVL2, the paper’s interpretation is that visible-domain competence does not transfer reliably to infrared imagery without explicit IR-specific pretraining. Relative to prior synthetic-infrared pipelines, the central claim is that authenticity of the infrared modality is indispensable for reducing hallucinations and improving robustness.

The paper also states several limitations. Domain gaps remain under extreme conditions such as very low resolution, severe weather, heavy occlusion, and high sensor noise. Deeper thermal-physics understanding is still limited; the model does not explicitly estimate thermal radiation, emissivity, or material-specific thermal behavior. Although IR-TD aggregates 63 datasets, the coverage of sensor types, spectral ranges, and application contexts could be broadened further. Evaluation is centered on recognition, grounding, and counting, and the absence of specialized thermal-semantics metrics is presented as an open problem. The manuscript also raises bias and ethics issues, including potential biases in scene types, geography, sensor manufacturers, and annotations, as well as safe deployment concerns in surveillance settings.

The term “IRGPT” should therefore be understood narrowly: it denotes an infrared-specialized multimodal LLM and its associated data and curriculum pipeline, not a generic “GPT for IR” label. It is distinct from inverse-rendering work whose acronym “IR” denotes inverse rendering rather than infrared imagery (Sun et al., 9 Apr 2025), and it is also distinct from “ITGPT,” which addresses irregular multimodal time series rather than infrared vision-language understanding (Honoré et al., 15 May 2026). Within infrared multimodal research itself, IRGPT is characterized by its use of real authenticated IR images, curated cross-modal alignment, and curriculum-based transfer from visible to infrared domains.