ThermEval-D: Thermal Imagery & Climate Diagnostics

• ThermEval-D is an ambiguous term referring to both a thermal imagery dataset with dense per-pixel temperature maps and a climate-model diagnostic package computing energy, water, and entropy metrics.
• In the thermal-imagery context, the dataset supports visual question answering by providing semantic body-part annotations and rigorous benchmark tasks for thermal reasoning.
• In the climate-model context, ThermEval-D offers modular diagnostic tools within ESMValTool to evaluate energy budgets, hydrological cycles, and non-equilibrium thermodynamics.

ThermEval-D is an ambiguous designation in the arXiv literature. In "ThermEval: A Structured Benchmark for Evaluation of Vision-LLMs on Thermal Imagery," it denotes a newly collected thermal-imagery dataset that provides dense per-pixel temperature maps with semantic body-part annotations and is integrated into the broader ThermEval-B benchmark for thermal visual question answering (Shrivastava et al., 16 Feb 2026). In "A new diagnostic tool for water, energy and entropy budgets in climate models," it denotes a modular diagnostic package developed within the ESMValTool framework to compute energy, water and entropy metrics from climate-model output (Lembo et al., 2019). The two usages are unrelated in domain, methodology, and intended application, and precise citation context is therefore necessary.

1. Name disambiguation

The cited literature uses the same label for two distinct research artifacts.

Usage of “ThermEval-D”	Domain	Core description
ThermEval-D in ThermEval-B	Thermal vision-language evaluation	The first dataset to provide dense per-pixel temperature maps with semantic body-part annotations across diverse indoor and outdoor environments
ThermEval-D in ESMValTool	Climate-model diagnostics	A modular diagnostic package developed within the ESMValTool framework to compute energy, water and entropy metrics from climate-model output

The thermal-imagery ThermEval-D is embedded in a benchmark motivated by the observation that vision LLMs achieve strong performance on RGB imagery, but they do not generalize to thermal images (Shrivastava et al., 16 Feb 2026). The climate-model ThermEval-D is embedded in a process-oriented evaluation workflow whose primary objectives are to quantify imbalances and transports in the top-of-atmosphere, atmospheric and surface energy budgets; close the global hydrological budgets; diagnose the Lorenz Energy Cycle; and estimate the climate system’s material entropy production by two complementary routes (Lembo et al., 2019).

A common misconception is to treat ThermEval-D as a single framework. The cited sources do not support that interpretation. Instead, they document homonymous artifacts serving different communities: one centered on thermal vision-language understanding, the other on non-equilibrium thermodynamics and climate-model evaluation.

2. ThermEval-D as a thermal-imagery dataset

Within the thermal-imagery literature, ThermEval-D is introduced as part of ThermEval-B, a structured benchmark of approximately 55,000 thermal visual question answering pairs designed to assess the foundational primitives required for thermal vision language understanding (Shrivastava et al., 16 Feb 2026). ThermEval-D itself contributes newly collected data with dense temperature supervision and semantic annotations. The stated sensor is a TOPDON TC001 Plus handheld thermal imager with native resolution 256 × 192 pixels, thermal sensitivity less than 40 mK, frame rate 25 Hz, and measurement range –20 °C to 550 °C with accuracy ±1 °C.

The data-acquisition protocol covers diverse indoor scenes such as offices, labs, and workspaces, and outdoor scenes such as parks and open grounds on a university campus. The subjects are 35 adult volunteers, age 18–47 yrs and weight 64–108 kg, with mixed skin tones, performing natural activities including standing, sitting, walking, and stair climbing. The collection procedure includes Institutional Ethics Committee approval, informed consent, anonymization, and on-site medical support. The dataset contains 1,000 thermal images, with approximately 40% outdoor and approximately 60% indoor scenes, and body-pose variation distributed between single-person scenes at approximately 70% and multi-person scenes at approximately 30% (Shrivastava et al., 16 Feb 2026).

The ground truth consists of dense 256 × 192 temperature matrices aligned with false-color images. Raw sensor radiance $R(x,y)$ is internally converted by camera firmware to temperature $T(x,y)$ via factory calibration, and an empirical first-order approximation is given for reference as

$T(x,y) = a \cdot R(x,y) + b,$

where $a,b$ are per-camera calibration constants determined by blackbody reference. Semantic body-part annotations cover four regions of interest: person, forehead, nose, and chest. Three expert annotators independently drew polygon masks for each region per image, periodic consensus meetings resolved edge cases such as glasses and occlusions, and axis-aligned bounding boxes were automatically derived from polygons (Shrivastava et al., 16 Feb 2026).

Inter-annotator agreement is reported as mean across all 1,000 images: bounding-box IoU 0.76, segmentation IoU 0.72, bounding-box Dice 0.86, segmentation Dice 0.84, and temperature SD within region 0.18 °C. The released structure is organized as images/, temps/, and annotations/, with <img_id>.png false-color thermal images, <img_id>.npy NumPy arrays of per-pixel temperature in °C, and <img_id>.json COCO-style JSON entries containing image_id, file_name, width, height, and annotation objects with category_id, segmentation, bbox, and area. The release is described as a Kaggle release under CC BY-NC 4.0 (Shrivastava et al., 16 Feb 2026).

3. Benchmark role, evaluation tasks, and empirical findings

ThermEval-D supports four key tasks in ThermEval-B: T4 colorbar inference, including detection, localization, and min/max extraction; T5 thermal reasoning, including comparative and within-person ranking; T6 absolute temperature estimation, including coordinate, marker, and region; and T7 temperature estimation at varying distance, specifically 2 ft, 6 ft, and 10 ft (Shrivastava et al., 16 Feb 2026). The benchmark metrics are classification accuracy,

$$\mathrm{Acc} = \frac{1}{N}\sum_{i=1}^{N} [\hat{y}_i = y_i],$$

mean absolute error,

$$\mathrm{MAE} = \frac{1}{N}\sum_{i=1}^{N} |\hat{y}_i - y_i|,$$

root mean squared error,

$$\mathrm{RMSE} = \sqrt{\frac{1}{N}\sum_{i=1}^{N} (\hat{y}_i - y_i)^2},$$

and Intersection over Union for segmentation masks,

$$\mathrm{IoU} = \frac{|\hat{M} \cap M|}{|\hat{M} \cup M|}.$$

The reported baseline results emphasize asymmetry across task types. For T4, top models such as Qwen-VL 2.5 and Intern-VL achieve nearly 100% detection and localization accuracy, with MAE approximately 0 °C in min/max extraction, whereas some models such as LLaVA and BLIP-2 exhibit OCR failures including hallucinated scales and decimal-shift errors (Shrivastava et al., 16 Feb 2026). For T5, zero-shot open-source VLMs achieve accuracy approximately 0.35–0.55 on double-person reasoning and 0.18–0.53 on within-person reasoning, compared with human 0.84/0.54; Gemini 3 Pro reaches 0.74/0.61 but remains below full human reliability. The listed failure modes are reliance on canonical body temperature priors and ignoring subtle thermal cues.

For T6 and T7, zero-shot MAE on T6-region for the best open-source model, Intern-VL 38B, is approximately 1.51 °C, compared with human approximately 2.04 °C. Coordinate and marker tasks are harder, with MAE approximately 3–6 °C, and many models default to fixed values such as 37 °C. For T7, top closed-source models achieve MAE 0.74–1.23 °C, compared with human approximately 1.2 °C. Colormap transformations are also diagnostically important: complex maps such as Spring and Summer degrade modality awareness (T2) by up to 30 percentage points in some models. A supervised fine-tuned upper bound, Qwen-VL 2.5 SFT, achieves or exceeds human performance on most tasks, including T5 at 0.58/0.56, T6 at MAE approximately 1.0 °C, and T7 at MAE approximately 0.5–0.6 °C. The paper interprets this as evidence that latent capacity exists, but zero-shot models lack thermal-domain grounding (Shrivastava et al., 16 Feb 2026).

The stated applications are night-time surveillance and perimeter security, search and rescue via aerial drones, autonomous driving under low visibility, industrial inspection, and non-contact medical screening. The stated limitations are dataset size of 1,000 images, limited demographic and environmental diversity, availability of only false-color images and no raw radiometric data, annotation scope restricted to four semantic regions, and benchmark complexity limited to foundational tasks. Future work is framed as expansion to more subjects, longer time spans, varied climates, inclusion of sensor radiance, extension to more anatomical landmarks and non-person objects, and incorporation of higher-level reasoning such as anomaly detection and temporal dynamics (Shrivastava et al., 16 Feb 2026).

4. ThermEval-D as a climate-model diagnostic package

In climate science, ThermEval-D is described as a modular diagnostic package developed within the ESMValTool framework to compute energy, water and entropy metrics from climate-model output, including CMIP5, CMIP6, and other gridded datasets (Lembo et al., 2019). Its primary objectives are explicitly process-oriented: quantify imbalances and transports in top-of-atmosphere, atmospheric and surface energy budgets; close the global hydrological budgets; diagnose the Lorenz Energy Cycle of the atmosphere, including available potential energy reservoirs, kinetic-energy reservoirs, conversion rates and dissipation; and estimate material entropy production by indirect and direct methods. By embedding these metrics in ESMValTool v2 and later, the package is intended to allow systematic intercomparison of many models and scenarios under CMIP6 standards.

The architecture consists of four independent modules that can be switched on or off in an ESMValTool recipe or in a stand-alone bash/Makefile version. The energy-budgets and meridional enthalpy-transports module requires monthly means on a full-globe lon–lat grid for rsdt, rsut, rlut, rsds, rsus, rlds, rlus, hfls, and hfss, with an optional land-sea mask to compute separate ocean and land budgets. The hydrological-cycle module requires monthly means of hfls, pr, and prsn, with optional prw. The Lorenz Energy Cycle module requires daily or higher resolution three-dimensional fields on pressure levels over 1000–1 hPa for ua, va, wap, and ta, and may additionally use tas, uas, and vas when model output omits data below certain pressure levels. The material entropy production module uses the same radiative fields as the energy-budget module for the indirect method, and for the direct method additionally requires hfls, hfss, pr, prsn, tas, 2 m winds, ps, and near-surface humidity hus or huss (Lembo et al., 2019).

The implementation section specifies ESMValTool v2.0 or later, Python 3.x, iris, optional cdo, optional nco, and esmpy for interpolation. A stand-alone version requires CDO, NCO, NCL, MATLAB or gfortran for the Fortran binaries, and uses a master bash script run_ThermEvalD.sh to call preprocessing, Fortran LEC, MATLAB MEP routines, and NCL plotting. The prescribed outputs include global 1D time series, 2D maps of energy budgets, water budgets, and MEP components, diagnostic plots such as meridional-transport profiles and LEC diagrams, and NetCDF output files following CF-1.6/CMOR tables for variables and metadata (Lembo et al., 2019).

5. Mathematical formulations and numerical methods in the climate usage

The climate-model ThermEval-D is built around explicit conservation-law and non-equilibrium thermodynamic formulations (Lembo et al., 2019). For vertically integrated total energy, neglecting kinetic-energy storage in the atmosphere, the tendency is

$$\frac{\partial E}{\partial t} = R_t - \nabla \cdot J_t,$$

with

$$R_t = S_t^\downarrow - S_t^\uparrow - L_t^\uparrow,$$

where $T(x,y)$0 is meridional enthalpy transport. The zonal-mean decomposition into total, atmosphere, and surface/ocean sub-systems is written as

$T(x,y)$1

$T(x,y)$2

$T(x,y)$3

with

$T(x,y)$4

and

$T(x,y)$5

Under steady state, northward transport is computed by integrating the zonal-mean net flux poleward of latitude $T(x,y)$6:

$T(x,y)$7

where $T(x,y)$8, $T(x,y)$9, and $T(x,y) = a \cdot R(x,y) + b,$0. When the model has a small global imbalance $T(x,y) = a \cdot R(x,y) + b,$1, a corrected budget

$T(x,y) = a \cdot R(x,y) + b,$2

is used to avoid spurious cross-polar transports.

For the hydrological cycle, evaporation is derived from latent heat flux as

$T(x,y) = a \cdot R(x,y) + b,$3

with $T(x,y) = a \cdot R(x,y) + b,$4. Rainfall and snowfall are related by $T(x,y) = a \cdot R(x,y) + b,$5, and the stationary mass-budget condition is $T(x,y) = a \cdot R(x,y) + b,$6. The latent energy budget is given as

$T(x,y) = a \cdot R(x,y) + b,$7

with $T(x,y) = a \cdot R(x,y) + b,$8 (Lembo et al., 2019).

For the Lorenz Energy Cycle, the package defines zonal mean APE, eddy APE, zonal mean KE, and eddy KE reservoirs per unit area:

$T(x,y) = a \cdot R(x,y) + b,$9

$a,b$0

$a,b$1

$a,b$2

where $a,b$3 is the static stability parameter, $a,b$4 denotes zonal mean, $a,b$5 global mean, prime denotes time deviation, and star denotes zonal deviation. Conversion terms $a,b$6, $a,b$7, $a,b$8, and $a,b$9 are computed by spectral decomposition in zonal-wavenumber space. Under steady state, the total mechanical work equals the KE dissipation,

$$\mathrm{Acc} = \frac{1}{N}\sum_{i=1}^{N} [\hat{y}_i = y_i],$$0

and $$\mathrm{Acc} = \frac{1}{N}\sum_{i=1}^{N} [\hat{y}_i = y_i],$$1 is obtained as the residual of the KE tendency balance (Lembo et al., 2019).

Material entropy production is implemented by two complementary routes. The indirect method begins from the steady-state entropy-budget equation

$$\mathrm{Acc} = \frac{1}{N}\sum_{i=1}^{N} [\hat{y}_i = y_i],$$2

and yields

$$\mathrm{Acc} = \frac{1}{N}\sum_{i=1}^{N} [\hat{y}_i = y_i],$$3

The first term is the vertical MEP and the second the horizontal MEP. The direct method starts from

$$\mathrm{Acc} = \frac{1}{N}\sum_{i=1}^{N} [\hat{y}_i = y_i],$$4

and, after the approximations stated in Sect. 3.4.1 of the paper, arrives at

$$\mathrm{Acc} = \frac{1}{N}\sum_{i=1}^{N} [\hat{y}_i = y_i],$$5

Here $$\mathrm{Acc} = \frac{1}{N}\sum_{i=1}^{N} [\hat{y}_i = y_i],$$6, $$\mathrm{Acc} = \frac{1}{N}\sum_{i=1}^{N} [\hat{y}_i = y_i],$$7 is the boundary-layer top temperature, $$\mathrm{Acc} = \frac{1}{N}\sum_{i=1}^{N} [\hat{y}_i = y_i],$$8 the cloud characteristic temperature, $$\mathrm{Acc} = \frac{1}{N}\sum_{i=1}^{N} [\hat{y}_i = y_i],$$9 the mean of $$\mathrm{MAE} = \frac{1}{N}\sum_{i=1}^{N} |\hat{y}_i - y_i|,$$0 and $$\mathrm{MAE} = \frac{1}{N}\sum_{i=1}^{N} |\hat{y}_i - y_i|,$$1, and $$\mathrm{MAE} = \frac{1}{N}\sum_{i=1}^{N} |\hat{y}_i - y_i|,$$2 the cloud-top height. The kinetic-dissipation term $$\mathrm{MAE} = \frac{1}{N}\sum_{i=1}^{N} |\hat{y}_i - y_i|,$$3 is obtained by setting $$\mathrm{MAE} = \frac{1}{N}\sum_{i=1}^{N} |\hat{y}_i - y_i|,$$4 (Lembo et al., 2019).

The numerical implementation uses monthly means for energy and hydrological modules, and daily or better temporal resolution for the LEC and direct-MEP modules. Vertically, it uses pressure-level fields, with missing near-surface levels filled by interpolation from surface data, and it has been tested on standard CMIP 17-level grids over 1000–1 hPa. Horizontally, it uses Fourier series in longitude and Legendre polynomials in latitude to separate zonal versus eddy and planetary versus synoptic scales. Meridional transports are computed from zonal mean flux divergences and then integrated poleward, while kinetic-energy dissipation is computed as the residual of the KE-tendency budget. The code is parallelized over latitudinal bands and time slices for speed (Lembo et al., 2019).

6. Validation, applications, and interpretive significance

The thermal-imagery ThermEval-D is validated primarily through benchmark behavior. Its associated ThermEval-B evaluation shows that current vision-LLMs can identify thermal modalities but systematically fail at temperature-grounded reasoning and estimation, while degrading under colormap transformations and often defaulting to language priors or fixed responses (Shrivastava et al., 16 Feb 2026). The stated interpretation is that thermal understanding requires dedicated evaluation beyond RGB-centric assumptions and, more specifically, thermal-aware pretraining and specialized model architectures. A plausible implication is that ThermEval-D functions less as a generic image collection than as an instrument for isolating failures in modality grounding, OCR-based scale parsing, comparative reasoning, and absolute temperature estimation.

The climate-model ThermEval-D is validated through demonstrations on a 20-year subset of the CMIP5 CanESM2 pre-industrial run and on a seven-model ensemble under piControl, historical, and RCP8.5 conditions (Lembo et al., 2019). The reported analyses include spatial maps of TOA, atmosphere, and surface budgets; meridional transports; water-mass convergence; LEC diagrams; and MEP maps. The ensemble study reports biases in energy budgets, changes in meridional transports, partitioning of APE and KE, stability of LEC intensity, increases in MEP components, baroclinic efficiency decline, and irreversibility rise. The applications explicitly listed are multi-model intercomparison of climate change response, model tuning via process-oriented diagnostics, paleoclimate scenario analysis, and potentially exoplanet climate studies. Ongoing work includes sensitivity to vertical and horizontal resolution, inclusion of full 3D radiative-flux MEP, and isentropic mass-streamfunction diagnostics.

Taken together, the two usages show that “ThermEval-D” does not identify a single methodological lineage. In one case it is a dense, semantically annotated thermal dataset designed to probe temperature-grounded visual reasoning; in the other it is a unified, process-oriented suite of diagnostics for energy, water, and entropy in climate models (Shrivastava et al., 16 Feb 2026, Lembo et al., 2019). The shared naming is incidental rather than conceptual.