FlameVQA: Wildfire VQA Benchmark

Updated 5 July 2026

FlameVQA is a wildfire-focused visual question answering benchmark that integrates paired RGB and radiometric thermal UAV data for accurate, safety-critical wildfire intelligence.
It employs physically grounded supervision with radiometric TIFFs to verify temperature thresholds and hotspot detection, enhancing reliability over RGB-only methods.
The benchmark features 34 multiple-choice questions across six operational capability groups, assessing tasks from detection and localization to cross-modal reasoning and flight planning.

Searching arXiv for the FlameVQA paper and closely related VQA benchmark/retrieval/clarification work to ground the article with current sources. FlameVQA is a multiple-choice visual question answering benchmark for UAV-based wildfire intelligence built on FLAME 3, leveraging paired RGB imagery and radiometric thermal TIFFs for temperature-grounded, safety-critical reasoning (Habibpour et al., 25 Jun 2026). It was introduced for wildfire monitoring scenarios in which smoke, scale variation, and occlusions limit RGB-only interpretation, and it evaluates whether multimodal LLMs can reason about wildfire scenes in ways that support operational tasks such as detection, localization, distribution or coverage estimation, cross-modal reasoning, and flight planning (Habibpour et al., 25 Jun 2026).

1. Definition and motivating problem

FlameVQA is a wildfire-focused VQA benchmark designed specifically for UAV imagery (Habibpour et al., 25 Jun 2026). The benchmark is positioned as the first dedicated VQA benchmark for wildfire monitoring, and its central aim is to move beyond generic “what is in the image?” style VQA toward physically grounded, operational reasoning for disaster response (Habibpour et al., 25 Jun 2026).

The motivating premise is that wildfire scenes are not reliably interpretable from RGB alone. Flames may be hidden by smoke, residual heat may persist after visible fire weakens, and hotspots may be invisible in the RGB view but obvious in thermal measurements (Habibpour et al., 25 Jun 2026). FlameVQA therefore targets hazard-centric tasks in which thermal evidence, visibility limitations, and safety constraints are essential, rather than optional auxiliary cues (Habibpour et al., 25 Jun 2026).

The benchmark was created in response to a gap in existing remote sensing VQA resources. Existing datasets such as RSVQA and HRVQA mostly emphasize general scene understanding and typically assume RGB-only imagery, whereas wildfire intelligence requires reasoning about fire and smoke, hotspot visibility, affected coverage, and UAV flight-related judgments (Habibpour et al., 25 Jun 2026). This suggests that FlameVQA is intended not merely as a dataset of image-question pairs, but as an evaluation substrate for operational wildfire scene understanding.

2. Data foundation and physically grounded supervision

FlameVQA is derived from the public FLAME 3 computer vision subset, which provides synchronized aerial RGB imagery and radiometric thermal data collected by UAVs over prescribed burns (Habibpour et al., 25 Jun 2026). The benchmark uses paired RGB and IR images from three burn sites: Sycan Marsh, Willamette Valley, and Shoetank (Habibpour et al., 25 Jun 2026). In total, it uses roughly 6,100 paired RGB–thermal frames (Habibpour et al., 25 Jun 2026).

For each frame, FlameVQA provides three aligned representations: an RGB image, a color-mapped thermal JPEG for visual context, and the underlying radiometric thermal TIFF for exact temperature grounding (Habibpour et al., 25 Jun 2026). The radiometric TIFFs contain actual per-pixel temperature values, enabling precise thermal reasoning and deterministic verification of labels rather than dependence on palette-colored thermal renderings alone (Habibpour et al., 25 Jun 2026).

This physically grounded design is a defining property of the benchmark. The paper emphasizes that answers can be verified against thermal physics using radiometric TIFFs and metadata, not just subjective annotation (Habibpour et al., 25 Jun 2026). A canonical thermal hotspot rule is described in which per-pixel temperature in °C is thresholded by a fixed fire threshold, and connected components of the resulting binary hot mask are used as candidate hotspots (Habibpour et al., 25 Jun 2026). The paper also states that altitude-based pixel area correction can be applied using a field-of-view–based GSD approximation to suppress spurious detections and keep hotspot estimates consistent across UAV heights (Habibpour et al., 25 Jun 2026).

A plausible implication is that FlameVQA treats thermal sensing not as a visualization aid but as a supervisory signal that constrains the semantics of the benchmark.

3. Benchmark composition and operational capability groups

FlameVQA contains 34 multiple-choice questions per image, organized into six operational capability groups (Habibpour et al., 25 Jun 2026). The benchmark uses curated distractors and standardized answer options to reduce ambiguity and make evaluation reproducible (Habibpour et al., 25 Jun 2026). Some questions are intentionally impossible to answer from RGB alone and require cross-modal inference (Habibpour et al., 25 Jun 2026).

Capability group	Primary focus	Representative scope
Presence/Detection	Whether wildfire cues or operationally relevant elements are present	Fire or smoke detection, thermal activity, assets, safety-relevant context
Classification	Discrete scene-state labels	Fire behavior, vegetation, fuel condition, accessibility, visible assets
Distribution/Segmentation	Scene-wide spread and percentage estimates	Fire, smoke, fuels, vegetation, heat distribution
Localization/Direction	Coarse spatial grounding and directional interpretation	Hotspot location, dominant fire activity, smoke origin, man-made elements
Cross-Modal Reasoning	Comparison across RGB and thermal imagery	Modality gaps, occlusions, thermal-only fire evidence
Flight Planning	UAV-centric decision support	Viewpoint, altitude, constraints, risk near flames or smoke

The Presence/Detection group determines whether wildfire cues or operationally relevant elements are present, including cases such as active fire despite heavy smoke or occlusion (Habibpour et al., 25 Jun 2026). Classification assigns discrete labels describing scene state, including fire behavior, vegetation, fuel condition, accessibility, and visible assets (Habibpour et al., 25 Jun 2026). Distribution/Segmentation focuses on how fire, smoke, fuels, vegetation, and heat are spread over the scene, including pattern descriptors and coverage estimates above thermal thresholds; the paper identifies this as one of the most difficult groups because it requires spatial aggregation rather than only object detection (Habibpour et al., 25 Jun 2026).

Localization/Direction tests coarse spatial grounding and directional interpretation, including where hotspots or dominant fire activity are located (Habibpour et al., 25 Jun 2026). Cross-Modal Reasoning requires comparing RGB and thermal imagery and probes modality gaps, occlusions, and whether fire is visible in one modality but not the other (Habibpour et al., 25 Jun 2026). Flight Planning evaluates UAV-centric decision support, including viewpoint or altitude considerations, environmental constraints, risk near flames or smoke, and flight safety (Habibpour et al., 25 Jun 2026).

The benchmark structure is intentionally broader than standard VQA. The paper explicitly frames it as mirroring what wildfire operators actually need, rather than serving as a generic multimodal QA suite (Habibpour et al., 25 Jun 2026).

4. Annotation pipeline and consistency control

FlameVQA uses a hybrid annotation pipeline that combines MLLM assistance, deterministic thermal rules, cross-question consistency checks, and human auditing (Habibpour et al., 25 Jun 2026). This pipeline is designed to improve label reliability in a setting where wildfire interpretation can be subjective if based only on visible imagery (Habibpour et al., 25 Jun 2026).

The MLLM-assisted stage uses Gemini 2.5 Pro with representative FLAME 3 samples to generate an initial pool of about 50 candidate questions, after which the questions are curated by removing ones that are unanswerable, unsupported by the data, or not operationally useful (Habibpour et al., 25 Jun 2026). The answer options are also refined so that distractors are informative and non-trivial, and expert-designed questions are added to fill gaps (Habibpour et al., 25 Jun 2026). For dataset-wide labeling, Gemini 2.5 Pro is used again to generate initial answers across all samples, conditioned on the RGB image, the color-mapped thermal visualization, and a compact thermal summary derived from the radiometric TIFF, such as minimum or maximum temperature and threshold exceedance rates (Habibpour et al., 25 Jun 2026).

Deterministic thermal verification is then applied. For temperature-critical labels, the radiometric TIFF is directly read using NumPy to inspect per-pixel temperatures and validate answers (Habibpour et al., 25 Jun 2026). The paper also uses metadata-based logic, including computation of UAV altitude from EXIF GPS or altitude metadata and a terrain elevation model, so altitude categories are not based on subjective visual inference (Habibpour et al., 25 Jun 2026).

Cross-question consistency checks impose logical constraints between related questions (Habibpour et al., 25 Jun 2026). The paper gives several explicit examples: if No fire is selected, hotspot-related questions must be No hotspots; if No smoke is selected, smoke coverage must be None; and if No structures are detected, structure localization must be No structures visible (Habibpour et al., 25 Jun 2026). Violations are flagged for manual expert review and correction (Habibpour et al., 25 Jun 2026).

The final stage is human auditing. Human review is used to audit difficult cases and assess reliability, including a targeted human evaluation on seven especially challenging questions in the Willamette Valley subset that stress subjectivity, aggregation, and cross-modal interpretation (Habibpour et al., 25 Jun 2026).

5. Reliability assessment and label quality

The benchmark reports a targeted human evaluation on the challenging Willamette Valley subset, where the automated labels matched a human expert in 70.78% of cases overall (Habibpour et al., 25 Jun 2026). Performance varies substantially by task, indicating that label reliability is not uniform across capability groups (Habibpour et al., 25 Jun 2026).

The hardest case is PD5 (active fire despite heavy smoke), with 36.12% agreement (Habibpour et al., 25 Jun 2026). DS5 (smoke coverage) reaches 45.96%, while LD3 (dominant fire location) reaches 75.48% and DS4 (fire coverage) 77.68% (Habibpour et al., 25 Jun 2026). Agreement is higher for explicitly cross-modal tasks: CMR1 (fire observable in RGB or only via thermal) reaches 82.68%, DS6 (hotspot coverage) 86.53%, and CMR3 (RGB/thermal consistency) 90.98% (Habibpour et al., 25 Jun 2026).

The paper interprets most mismatches as near-boundary errors, such as adjacent percentage bins or partial-versus-full occlusion disagreements, rather than major semantic failures (Habibpour et al., 25 Jun 2026). This suggests that the quality-control pipeline is strongest when the decision boundary is tightly tied to thermal evidence or modality comparison, and weaker when the task depends on coarse aggregation or severe smoke-induced ambiguity.

6. Baseline MLLM performance and observed failure modes

FlameVQA evaluates two representative open-source multimodal LLMs on the Willamette human-evaluated subset: LLaVA-1.6-7B and Qwen3-VL-8B-Instruct (Habibpour et al., 25 Jun 2026). The reported results are bucketed by capability type and expose a clear asymmetry between explicit cross-modal comparison and spatial aggregation (Habibpour et al., 25 Jun 2026).

Model	Cross-modal reasoning	Distribution/segmentation	Combined PD+LD	Overall
LLaVA-1.6-7B	85.51%	18.07%	14.18%	36.23%
Qwen3-VL-8B-Instruct	87.19%	25.18%	45.33%	48.65%

Both models perform relatively strongly on cross-modal reasoning, with 85.51% for LLaVA and 87.19% for Qwen-VL (Habibpour et al., 25 Jun 2026). By contrast, distribution or segmentation is very weak, at 18.07% for LLaVA and 25.18% for Qwen-VL (Habibpour et al., 25 Jun 2026). Presence or localization under smoke is also difficult, especially for LLaVA, whose combined Presence/Detection plus Localization/Direction accuracy is 14.18%, compared with 45.33% for Qwen-VL (Habibpour et al., 25 Jun 2026). Overall accuracy across all seven challenging questions is 36.23% for LLaVA and 48.65% for Qwen-VL (Habibpour et al., 25 Jun 2026).

The main empirical pattern is that current MLLMs can exploit explicit cross-modal cues when the task is clearly framed, but still struggle with smoke-obscured presence detection and percentage-based coverage estimation (Habibpour et al., 25 Jun 2026). The paper identifies coverage estimation as especially problematic because it requires spatial aggregation rather than local recognition (Habibpour et al., 25 Jun 2026). It also concludes that current general-purpose MLLMs are not yet reliable for high-stakes wildfire monitoring and that domain-specific adaptation is likely necessary (Habibpour et al., 25 Jun 2026).

7. Position within the VQA literature and terminological boundaries

FlameVQA belongs to the broader VQA literature but occupies a distinct niche defined by wildfire operations, UAV sensing, and radiometric thermal supervision (Habibpour et al., 25 Jun 2026). Its benchmark design differs markedly from disaster VQA systems centered on answer generation without task-specific training. For example, the flood-disaster system ZFDDA formalizes disaster-scene VQA as free-form answer generation, uses a modular pipeline with image content extraction, chain-of-thought demonstration, and a question answering module built on Flan-Alpaca, and is evaluated on the FFD-IQA dataset with free-form, multiple-choice, and yes-no questions (Sun et al., 2023). FlameVQA, by contrast, is a wildfire benchmark with standardized multiple-choice questions and physically grounded thermal verification (Habibpour et al., 25 Jun 2026).

It also differs from retrieval-augmented VQA methods such as FLMR, which improve knowledge retrieval for knowledge-based VQA by replacing DPR-style single-vector retrieval with fine-grained late interaction and aligned visual features on benchmarks such as OK-VQA (Lin et al., 2023). FlameVQA does not center on external document retrieval; its central innovation is physically grounded supervision from radiometric UAV thermal data (Habibpour et al., 25 Jun 2026).

A further distinction concerns interaction. ClearVQA studies ambiguity in VQA and trains VLMs to ask clarification questions before answering, targeting referential ambiguity, intent underspecification, and spelling ambiguity in an interactive setting (Jian et al., 18 Jul 2025). FlameVQA does not formulate wildfire VQA as interactive clarification; its focus is operational wildfire intelligence under multimodal sensing constraints (Habibpour et al., 25 Jun 2026).

Finally, FlameVQA should not be conflated with Flame, a large vision-LLM for front-end code generation trained on synthesized image-text-code data and evaluated with Flame-React-Eval (Ge et al., 3 Mar 2025). The shared string “Flame” denotes unrelated artifacts in different subfields. In the wildfire literature, FlameVQA specifically denotes the physically grounded UAV wildfire VQA benchmark introduced in 2026 (Habibpour et al., 25 Jun 2026).