---
title: 'FlameVQA: Wildfire VQA Benchmark'
url: https://www.emergentmind.com/topics/flamevqa
type: topic
---

# FlameVQA: Wildfire VQA Benchmark

Searching arXiv for the FlameVQA paper and closely related VQA benchmark/retrieval/clarification work to ground the article with current sources.
FlameVQA is a multiple-choice visual question answering benchmark for UAV-based wildfire intelligence built on FLAME 3, leveraging paired RGB imagery and radiometric thermal TIFFs for temperature-grounded, safety-critical reasoning [2606.27128]. It was introduced for wildfire monitoring scenarios in which smoke, scale variation, and occlusions limit RGB-only interpretation, and it evaluates whether multimodal large language models can reason about wildfire scenes in ways that support operational tasks such as detection, localization, distribution or coverage estimation, cross-modal reasoning, and flight planning [2606.27128].

## 1. Definition and motivating problem

FlameVQA is a wildfire-focused VQA benchmark designed specifically for UAV imagery [2606.27128]. The benchmark is positioned as the first dedicated VQA benchmark for wildfire monitoring, and its central aim is to move beyond generic “what is in the image?” style VQA toward physically grounded, operational reasoning for disaster response [2606.27128].

The motivating premise is that wildfire scenes are not reliably interpretable from RGB alone. Flames may be hidden by smoke, residual heat may persist after visible fire weakens, and hotspots may be invisible in the RGB view but obvious in thermal measurements [2606.27128]. FlameVQA therefore targets hazard-centric tasks in which thermal evidence, visibility limitations, and safety constraints are essential, rather than optional auxiliary cues [2606.27128].

The benchmark was created in response to a gap in existing remote sensing VQA resources. Existing datasets such as RSVQA and HRVQA mostly emphasize general scene understanding and typically assume RGB-only imagery, whereas wildfire intelligence requires reasoning about fire and smoke, hotspot visibility, affected coverage, and UAV flight-related judgments [2606.27128]. This suggests that FlameVQA is intended not merely as a dataset of image-question pairs, but as an evaluation substrate for operational wildfire scene understanding.

## 2. Data foundation and physically grounded supervision

FlameVQA is derived from the public FLAME 3 computer vision subset, which provides synchronized aerial RGB imagery and radiometric thermal data collected by UAVs over prescribed burns [2606.27128]. The benchmark uses paired RGB and IR images from three burn sites: Sycan Marsh, Willamette Valley, and Shoetank [2606.27128]. In total, it uses roughly 6,100 paired RGB–thermal frames [2606.27128].

For each frame, FlameVQA provides three aligned representations: an RGB image, a color-mapped thermal JPEG for visual context, and the underlying radiometric thermal TIFF for exact temperature grounding [2606.27128]. The radiometric TIFFs contain actual per-pixel temperature values, enabling precise thermal reasoning and deterministic verification of labels rather than dependence on palette-colored thermal renderings alone [2606.27128].

This physically grounded design is a defining property of the benchmark. The paper emphasizes that answers can be verified against thermal physics using radiometric TIFFs and metadata, not just subjective annotation [2606.27128]. A canonical thermal hotspot rule is described in which per-pixel temperature in °C is thresholded by a fixed fire threshold, and connected components of the resulting binary hot mask are used as candidate hotspots [2606.27128]. The paper also states that altitude-based pixel area correction can be applied using a field-of-view–based GSD approximation to suppress spurious detections and keep hotspot estimates consistent across UAV heights [2606.27128].

A plausible implication is that FlameVQA treats thermal sensing not as a visualization aid but as a supervisory signal that constrains the semantics of the benchmark.

## 3. Benchmark composition and operational capability groups

FlameVQA contains 34 multiple-choice questions per image, organized into six operational capability groups [2606.27128]. The benchmark uses curated distractors and standardized answer options to reduce ambiguity and make evaluation reproducible [2606.27128]. Some questions are intentionally impossible to answer from RGB alone and require cross-modal inference [2606.27128].

| Capability group | Primary focus | Representative scope |
|---|---|---|
| Presence/Detection | Whether wildfire cues or operationally relevant elements are present | Fire or smoke detection, thermal activity, assets, safety-relevant context |
| Classification | Discrete scene-state labels | Fire behavior, vegetation, fuel condition, accessibility, visible assets |
| Distribution/Segmentation | Scene-wide spread and percentage estimates | Fire, smoke, fuels, vegetation, heat distribution |
| Localization/Direction | Coarse spatial grounding and directional interpretation | Hotspot location, dominant fire activity, smoke origin, man-made elements |
| Cross-Modal Reasoning | Comparison across RGB and thermal imagery | Modality gaps, occlusions, thermal-only fire evidence |
| Flight Planning | UAV-centric decision support | Viewpoint, altitude, constraints, risk near flames or smoke |

The Presence/Detection group determines whether wildfire cues or operationally relevant elements are present, including cases such as active fire despite heavy smoke or occlusion [2606.27128]. Classification assigns discrete labels describing scene state, including fire behavior, vegetation, fuel condition, accessibility, and visible assets [2606.27128]. Distribution/Segmentation focuses on how fire, smoke, fuels, vegetation, and heat are spread over the scene, including pattern descriptors and coverage estimates above thermal thresholds; the paper identifies this as one of the most difficult groups because it requires spatial aggregation rather than only object detection [2606.27128].

Localization/Direction tests coarse spatial grounding and directional interpretation, including where hotspots or dominant fire activity are located [2606.27128]. Cross-Modal Reasoning requires comparing RGB and thermal imagery and probes modality gaps, occlusions, and whether fire is visible in one modality but not the other [2606.27128]. Flight Planning evaluates UAV-centric decision support, including viewpoint or altitude considerations, environmental constraints, risk near flames or smoke, and flight safety [2606.27128].

The benchmark structure is intentionally broader than standard VQA. The paper explicitly frames it as mirroring what wildfire operators actually need, rather than serving as a generic multimodal QA suite [2606.27128].

## 4. Annotation pipeline and consistency control

FlameVQA uses a hybrid annotation pipeline that combines MLLM assistance, deterministic thermal rules, cross-question consistency checks, and human auditing [2606.27128]. This pipeline is designed to improve label reliability in a setting where wildfire interpretation can be subjective if based only on visible imagery [2606.27128].

The MLLM-assisted stage uses Gemini 2.5 Pro with representative FLAME 3 samples to generate an initial pool of about 50 candidate questions, after which the questions are curated by removing ones that are unanswerable, unsupported by the data, or not operationally useful [2606.27128]. The answer options are also refined so that distractors are informative and non-trivial, and expert-designed questions are added to fill gaps [2606.27128]. For dataset-wide labeling, Gemini 2.5 Pro is used again to generate initial answers across all samples, conditioned on the RGB image, the color-mapped thermal visualization, and a compact thermal summary derived from the radiometric TIFF, such as minimum or maximum temperature and threshold exceedance rates [2606.27128].

Deterministic thermal verification is then applied. For temperature-critical labels, the radiometric TIFF is directly read using NumPy to inspect per-pixel temperatures and validate answers [2606.27128]. The paper also uses metadata-based logic, including computation of UAV altitude from EXIF GPS or altitude metadata and a terrain elevation model, so altitude categories are not based on subjective visual inference [2606.27128].

Cross-question consistency checks impose logical constraints between related questions [2606.27128]. The paper gives several explicit examples: if **No fire** is selected, hotspot-related questions must be **No hotspots**; if **No smoke** is selected, smoke coverage must be **None**; and if **No structures** are detected, structure localization must be **No structures visible** [2606.27128]. Violations are flagged for manual expert review and correction [2606.27128].

The final stage is human auditing. Human review is used to audit difficult cases and assess reliability, including a targeted human evaluation on seven especially challenging questions in the Willamette Valley subset that stress subjectivity, aggregation, and cross-modal interpretation [2606.27128].

## 5. Reliability assessment and label quality

The benchmark reports a targeted human evaluation on the challenging Willamette Valley subset, where the automated labels matched a human expert in 70.78% of cases overall [2606.27128]. Performance varies substantially by task, indicating that label reliability is not uniform across capability groups [2606.27128].

The hardest case is **PD5** (active fire despite heavy smoke), with 36.12% agreement [2606.27128]. **DS5** (smoke coverage) reaches 45.96%, while **LD3** (dominant fire location) reaches 75.48% and **DS4** (fire coverage) 77.68% [2606.27128]. Agreement is higher for explicitly cross-modal tasks: **CMR1** (fire observable in RGB or only via thermal) reaches 82.68%, **DS6** (hotspot coverage) 86.53%, and **CMR3** (RGB/thermal consistency) 90.98% [2606.27128].

The paper interprets most mismatches as near-boundary errors, such as adjacent percentage bins or partial-versus-full occlusion disagreements, rather than major semantic failures [2606.27128]. This suggests that the quality-control pipeline is strongest when the decision boundary is tightly tied to thermal evidence or modality comparison, and weaker when the task depends on coarse aggregation or severe smoke-induced ambiguity.

## 6. Baseline MLLM performance and observed failure modes

FlameVQA evaluates two representative open-source multimodal LLMs on the Willamette human-evaluated subset: LLaVA-1.6-7B and Qwen3-VL-8B-Instruct [2606.27128]. The reported results are bucketed by capability type and expose a clear asymmetry between explicit cross-modal comparison and spatial aggregation [2606.27128].

| Model | Cross-modal reasoning | Distribution/segmentation | Combined PD+LD | Overall |
|---|---:|---:|---:|---:|
| LLaVA-1.6-7B | 85.51% | 18.07% | 14.18% | 36.23% |
| Qwen3-VL-8B-Instruct | 87.19% | 25.18% | 45.33% | 48.65% |

Both models perform relatively strongly on cross-modal reasoning, with 85.51% for LLaVA and 87.19% for Qwen-VL [2606.27128]. By contrast, distribution or segmentation is very weak, at 18.07% for LLaVA and 25.18% for Qwen-VL [2606.27128]. Presence or localization under smoke is also difficult, especially for LLaVA, whose combined Presence/Detection plus Localization/Direction accuracy is 14.18%, compared with 45.33% for Qwen-VL [2606.27128]. Overall accuracy across all seven challenging questions is 36.23% for LLaVA and 48.65% for Qwen-VL [2606.27128].

The main empirical pattern is that current MLLMs can exploit explicit cross-modal cues when the task is clearly framed, but still struggle with smoke-obscured presence detection and percentage-based coverage estimation [2606.27128]. The paper identifies coverage estimation as especially problematic because it requires spatial aggregation rather than local recognition [2606.27128]. It also concludes that current general-purpose MLLMs are not yet reliable for high-stakes wildfire monitoring and that domain-specific adaptation is likely necessary [2606.27128].

## 7. Position within the VQA literature and terminological boundaries

FlameVQA belongs to the broader VQA literature but occupies a distinct niche defined by wildfire operations, UAV sensing, and radiometric thermal supervision [2606.27128]. Its benchmark design differs markedly from disaster VQA systems centered on answer generation without task-specific training. For example, the flood-disaster system ZFDDA formalizes disaster-scene VQA as free-form answer generation, uses a modular pipeline with image content extraction, chain-of-thought demonstration, and a question answering module built on Flan-Alpaca, and is evaluated on the FFD-IQA dataset with free-form, multiple-choice, and yes-no questions [2312.01882]. FlameVQA, by contrast, is a wildfire benchmark with standardized multiple-choice questions and physically grounded thermal verification [2606.27128].

It also differs from retrieval-augmented VQA methods such as FLMR, which improve knowledge retrieval for knowledge-based VQA by replacing DPR-style single-vector retrieval with fine-grained late interaction and aligned visual features on benchmarks such as OK-VQA [2309.17133]. FlameVQA does not center on external document retrieval; its central innovation is physically grounded supervision from radiometric UAV thermal data [2606.27128].

A further distinction concerns interaction. ClearVQA studies ambiguity in VQA and trains VLMs to ask clarification questions before answering, targeting referential ambiguity, intent underspecification, and spelling ambiguity in an interactive setting [2507.13773]. FlameVQA does not formulate wildfire VQA as interactive clarification; its focus is operational wildfire intelligence under multimodal sensing constraints [2606.27128].

Finally, FlameVQA should not be conflated with **Flame**, a large vision-language model for front-end code generation trained on synthesized image-text-code data and evaluated with **Flame-React-Eval** [2503.01619]. The shared string “Flame” denotes unrelated artifacts in different subfields. In the wildfire literature, FlameVQA specifically denotes the physically grounded UAV wildfire VQA benchmark introduced in 2026 [2606.27128].

Source: https://www.emergentmind.com/topics/flamevqa