ThermEval-B: Thermal VLM Benchmark

• ThermEval-B is a structured benchmark that evaluates vision-language models on thermal imagery by focusing on temperature-grounded perception.
• It organizes evaluation into 7 tasks—including modality recognition, human counting, colorbar interpretation, and temperature estimation—with over 55,000 samples.
• The benchmark combines public datasets with the new ThermEval-D to test models' thermal reasoning, addressing limitations of RGB-trained systems.

ThermEval-B is a structured benchmark for evaluating vision-LLMs on thermal imagery, introduced as the benchmark component of the ThermEval framework. It was designed to assess whether models can perform temperature-grounded perception and reasoning rather than merely transferring RGB-centric visual priors to infrared data. The benchmark contains approximately 55,000 thermal visual question answering pairs, with a reported total of 55,895 samples, and organizes evaluation into seven tasks spanning modality identification, colormap robustness, human counting, colorbar interpretation, thermal reasoning, absolute temperature estimation, and temperature estimation under varying distance. Its data model combines public thermal datasets with ThermEval-D, a newly collected dataset providing dense per-pixel temperature maps and semantic body-part annotations across indoor and outdoor environments (Shrivastava et al., 16 Feb 2026).

1. Conceptual basis and scope

ThermEval-B was introduced in response to a specific limitation of existing vision-language evaluation practice: conventional benchmarks are overwhelmingly RGB-centric and therefore do not test whether a model can interpret the core signal in thermal images, namely temperature. In thermal imagery, pixels encode emitted infrared radiation and, in practice, temperature values or values proportional to heat. Consequently, the benchmark treats thermal understanding as a problem of physically grounded interpretation rather than ordinary semantic recognition (Shrivastava et al., 16 Feb 2026).

The benchmark is structured around seven foundational primitives for thermal vision-language understanding. These range from simple identification of whether an image is thermal or RGB to numerically estimating temperatures in degrees Celsius and reasoning about how thermal measurements change with camera distance. This task progression encodes the paper’s central position that thermal understanding is not reducible to generic multimodal competence.

A common misconception addressed by the benchmark is that strong RGB performance should transfer straightforwardly to thermal imagery. ThermEval-B was constructed precisely to test that assumption under controlled conditions. Its task design emphasizes modality-specific robustness, numerical grounding, and the interpretation of thermal-specific visual structures such as embedded colorbars.

2. Task organization and benchmark composition

ThermEval-B is divided into seven tasks with increasing difficulty. Tasks T1–T3 focus on modality recognition and semantic extraction from thermal scenes, while T4–T7 shift to explicitly temperature-grounded interpretation and reasoning (Shrivastava et al., 16 Feb 2026).

Task	Function	Sample count
T1	Modality identification	10,000
T2	Modality identification under colormap transformations	10,000
T3	Human counting	20,000
T4	Colorbar interpretation	8,392
T5	Thermal reasoning	536
T6	Absolute temperature estimation	5,513
T7	Temperature estimation at varying distance	1,454

T1 is a binary classification task asking whether an image is thermal or RGB. T2 repeats that task after thermal images are rendered with different colormaps, including grayscale, Type I sequential colormaps such as Magma and Viridis, and more complex Type II colormaps such as Spring and Summer. T3 asks models to count people in thermal road-scene images and uses mean absolute error as the primary metric.

T4 evaluates colorbar interpretation, a prerequisite for temperature-grounded reasoning in many thermal images. It is subdivided into colorbar detection, localization, and extraction of minimum and maximum temperatures. T5 targets relative thermal reasoning with two subtasks: deciding whether the left or right person has the hotter specified body part, and ranking body parts within a single person from hottest to coolest. T6 shifts from relative to absolute estimation through coordinate-based, arrow-based, and region-based temperature queries. T7 then asks for semantic body-part temperature estimation at distances of 2 ft, 6 ft, and 10 ft.

The reported sample counts sum to 55,895. This scale is large for a benchmark centered on thermal reasoning, particularly because the higher-order tasks require dense temperature supervision rather than ordinary categorical labels.

3. Data sources and the role of ThermEval-D

ThermEval-B combines public datasets with a newly collected dataset, ThermEval-D. For T1–T3, it uses FLIR-ADAS and LLVIP, which provide paired thermal and visible-light imagery and, for counting, pedestrian annotations. These datasets are suitable for modality recognition and human counting but do not provide dense temperature ground truth (Shrivastava et al., 16 Feb 2026).

For T4–T7, the benchmark relies on ThermEval-D. The paper describes ThermEval-D as the first dataset in this setting to provide both dense per-pixel temperature maps and semantic body-part annotations. The thermal images are 256 × 192, matching the temperature matrix resolution. Body-part labels include Person, Forehead, Nose, and Chest, with polygonal segmentations and automatically derived bounding boxes.

ThermEval-D includes over 1,000 images from 35 adult participants aged 18–47 with weight range 64–108 kg and varied skin tones. Data collection was conducted with Institutional Ethics Committee approval, informed consent, and anonymization. Participants performed natural actions including standing, sitting, walking, and navigating stairs. The scenes span indoor and outdoor environments, including offices, laboratories, workspaces, parks, and open grounds. The dataset was acquired using the TOPDON TC001 Plus thermal camera, which has a 256 × 192 infrared sensor, sub-40 mK thermal sensitivity, 25 Hz frame rate, temperature range from –20°C to 550°C, and accuracy of ±1°C (Shrivastava et al., 16 Feb 2026).

Annotation quality is quantified explicitly. Across three expert annotators, the appendix reports BBox IoU of 0.77, Segm. IoU of 0.72, BBox Dice of 0.87, and Segm. Dice of 0.84. The paper also reports very low temperature disagreement among annotators, including an illustrative example with temperatures 32.26°C, 32.15°C, and 32.18°C, majority-vote temperature 32.17°C, standard deviation 0.04°C, and mean per-label standard deviation across the dataset of 0.18°C. This annotation regime is central to ThermEval-B because it enables evaluation of coordinate-, region-, and distance-dependent thermal reasoning rather than only coarse semantic inference.

4. Evaluation protocol and measurement methodology

The benchmark evaluates 25 models spanning open-source, closed-source, chart-specialized, and fine-tuned systems. The open-source set includes model families such as InternVL 3, LLaVA-1.5, LLaMA-3.2, MiniCPM-V 2.6, Phi-3, Phi-3.5, Qwen-VL 2, Qwen-VL 2.5, Qwen A22, PaliGemma-2, IDEFICS-3, SmolVLM, Jina-VLM, and BLIP-2. Closed-source models include Gemini 2.5 Flash, Gemini 2.5 Pro, Gemini 3 Flash, and Gemini 3 Pro. Because thermal tasks frequently require reading legends and numeric scales, chart-focused systems such as ChartGemma, ChartInstruct, and TinyCharts are אויך included. The paper also evaluates a supervised fine-tuned Qwen-VL 2.5 (8B) variant (Shrivastava et al., 16 Feb 2026).

The default protocol is zero-shot prompting, with no thermal-specific training unless explicitly noted. All evaluations are run on a single NVIDIA A100 80GB, using one forward pass without ensembling or repeated sampling. Accuracy is the principal metric for classification tasks, while mean absolute error is used for numeric estimation and counting; the appendices additionally report RMSE, Bias, and STD for some tasks.

A distinctive methodological feature is the use of a language-only LLM parser to normalize heterogeneous model outputs into class labels or numerical values. The parser, from the Gemini 2.5 family, had no access to the image and was used only for extraction rather than scoring. It was validated on a stratified gold set of roughly 1,200 outputs sampled from a population of 700,000 outputs. Reported agreement for structured judging was 99.01% for Gemini 2.5 Pro, 99.07% for Gemini 2.5 Flash, and 98.24% for Gemini 2.5 Flash Lite (Shrivastava et al., 16 Feb 2026).

The paper also specifies a sample-size formula for the parser validation study,

$$n = \frac{Z^2 \, p \, (1-p)}{e^2} \cdot \frac{N}{N-1 + \frac{Z^2 \, p \, (1-p)}{e^2}},$$

with $N = 700{,}000$, $p = 0.5$, $e = 0.03$, and $Z = 1.96$, yielding approximately 1,067 required samples. This formalization is relevant because ThermEval-B evaluates models through free-form textual outputs that must be rendered comparable across architectures.

5. Reported empirical behavior and failure modes

The evaluation shows a consistent pattern: many models can recognize thermal imagery as a modality, but substantially fewer can reason reliably about temperature. Modality identification is described as relatively easy for many modern VLMs, especially strong InternVL 3 and Qwen-VL variants, whereas BLIP-2 is reported as an outlier that often predicts everything as thermal. Under colormap perturbation, sequential colormaps such as Magma and Viridis are easier than more complex mappings such as Spring and Summer, indicating that some systems rely on superficial color statistics rather than modality-invariant thermal structure (Shrivastava et al., 16 Feb 2026).

Human counting remains difficult, especially in cluttered FLIR scenes. Reported examples include human MAE of 1.73 on FLIR and 0.30 on LLVIP, compared with 2.93 and 0.48 respectively for InternVL 3 (38B), and 4.69 and 2.99 for BLIP-2. The paper notes that LLVIP is easier than FLIR, but the residual gap indicates that semantic extraction from thermal imagery is not trivial even before temperature reasoning begins.

Colorbar interpretation is treated as a prerequisite skill, and the reported failures are often systematic rather than random. LLaVA is said to hallucinate very large values, InternVL 3 (8B) makes decimal-shift mistakes such as 33.5 → 335, and MiniCPM sometimes adds a constant offset of 200. PaliGemma 2, LLaVA-1.5, and BLIP-2 struggle with colorbar localization. These errors are consequential because T6 and T7 depend on accurate mapping between pseudocolor and temperature.

Thermal reasoning in T5 remains weak. In the double-person comparison task, open-source models typically fall in the range 0.37–0.61 accuracy, with LLaMA-3.2 reported at 0.61, Gemini 3 Pro at 0.74, and humans at 0.84. The within-person ranking task is harder: Qwen-VL 2.5 (8B) reaches 0.51, InternVL 3 (8B) 0.53, and the human reference is 0.54, while PaliGemma 2 and BLIP-2 collapse. The paper explicitly notes that scaling alone does not reliably improve this behavior.

Absolute temperature estimation is where the benchmark records the clearest failures. For coordinate-based estimation, reported MAE values include 58.42°C for PaliGemma 2, 31.33°C for BLIP-2, 31.92°C for InternVL 3 (8B), and 3.21°C for Qwen-VL 2.5 (8B). The human reference on the arrow task is 2.73°C MAE. Region-based estimation is easier, but still imperfect: InternVL 3 (38B) achieves 1.51°C MAE, Gemini 3 Pro 1.47°C, Qwen-VL 2.5 with supervised fine-tuning 1.03°C, and humans 2.04°C. The paper further reports that some models, especially MiniCPM and InternVL 8B, sometimes output fixed values such as 37°C, suggesting reliance on language priors, particularly canonical body-temperature priors, rather than image-grounded inference (Shrivastava et al., 16 Feb 2026).

Distance variation in T7 is manageable for stronger systems but remains nontrivial. The best fine-tuned Qwen-VL 2.5 results are reported at approximately 0.53 MAE for 2 ft, 0.49 for 6 ft, and 0.61 for 10 ft. Strong zero-shot closed models such as Gemini 3 Pro are reported around 0.74–1.00 MAE depending on distance.

6. Prompting, fine-tuning, and broader significance

ThermEval-B includes two important intervention studies: prompt ablation and supervised fine-tuning. Richer prompting substantially helps on easy modality tasks, with examples such as Qwen-VL 2.5 improving from 0.71 to 0.96 on T1 FLIR and from 0.61 to 0.98 on T2 FLIR, and InternVL-14B improving from 0.86 to 0.99 on T2 FLIR. By contrast, the paper reports that prompting yields only small, inconsistent, or negative changes on T5–T7, indicating that better textual framing does not create missing thermal grounding (Shrivastava et al., 16 Feb 2026).

Supervised fine-tuning is more effective. The reported fine-tuned model is Qwen-VL 2.5 (8B) trained with LoRA for 5 epochs, rank $r=16$, scaling $\alpha=16$, dropout 0.1, learning rate $5 \times 10^{-6}$, and batch size 4 per device. This setting produces the strongest overall benchmark performance in the paper. Reported improvements include T3 FLIR from 3.55 to 1.85 MAE, T5 double from 0.44 to 0.58 accuracy, T5 single from 0.32 to 0.56 accuracy, T6 coordinates from 3.21 to 1.58 MAE, T6 arrow from 2.88 to 1.55 MAE, T6 region from 2.14 to 1.03 MAE, and T7 from 1.27/0.94/0.79 to 0.53/0.49/0.61 MAE across 2 ft, 6 ft, and 10 ft (Shrivastava et al., 16 Feb 2026).

Even so, the paper does not treat fine-tuning as a complete solution. It states that some tasks remain below human level and that temperature estimation errors of 1–2°C persist. A plausible implication is that ThermEval-B is diagnosing a deficit in physical grounding rather than merely insufficient task instruction. This interpretation is reinforced by the counterfactual appendix experiment in which forehead, chest, and nose temperatures are perturbed by $\pm 1$, $\pm 2$, and $N = 700{,}000$0C. Without fine-tuning, Qwen2.5-VL shows bias toward higher temperatures, weak sensitivity to perturbation direction, and collapse toward preferred values; after fine-tuning, causal consistency improves, but residual bias remains.

ThermEval-B therefore functions as more than a benchmark leaderboard. It formalizes a set of measurement primitives for thermal vision-language understanding and demonstrates that RGB-trained VLMs can detect thermal modality without achieving reliable temperature-grounded reasoning. Within the framework described in the paper, its larger significance lies in motivating thermal-aware pretraining, stronger physical grounding, improved robustness to colormap transformations, and explicit treatment of non-RGB sensor modalities in multimodal model development (Shrivastava et al., 16 Feb 2026).