- The paper presents a physically-grounded UAV wildfire VQA benchmark that integrates paired RGB and radiometric thermal data.
- The study employs a hybrid pipeline combining MLLM outputs, deterministic rules, and expert audits for robust label verification.
- Key evaluations reveal strong cross-modal reasoning in current MLLMs while highlighting weaknesses in spatial aggregation under challenging scenarios.
FlameVQA: A Physically-Grounded Benchmark for UAV Wildfire Visual Question Answering
Introduction
FlameVQA presents a comprehensive multiple-choice visual question answering (VQA) benchmark centered on UAV-based wildfire monitoring, uniquely integrating paired RGB and radiometric thermal data. Recognizing the pervasive limitations of RGB-only or palette-based thermal imagery in UAV wildfire intelligence, the authors leverage the FLAME 3 dataset to establish temperature-grounded, operationally significant VQA tasks. The dataset is designed explicitly for multimodal LLMs (MLLMs) and targets the unique challenges introduced by complex wildfire environments, including smoke-induced occlusions, heterogeneous fuel dynamics, and safety-critical reasoning requirements.
Benchmark Design and Methodology
FlameVQA compiles paired RGB and thermal imagery collected during prescribed burns at multiple distinct sites, notably Sycan Marsh, Willamette Valley, and Shoetank. Each sample includes RGB frames, color-mapped thermal images, and radiometric thermal TIFFs with per-pixel temperature values, permitting deterministic, physics-based verification of thermal phenomena such as hotspots, smoldering, and occluded fire zones.
The VQA suite encompasses 34 carefully curated multiple-choice questions per image, partitioned into six operational groups: presence/detection, classification, distribution/segmentation, localization/direction, cross-modal reasoning, and UAV-centric flight planning. Questions are meticulously engineered to minimize subjectivity and to maximize operational relevance. Distractor options are selected to ensure non-triviality and mitigate LLM biases, while prompt generation leverages Gemini 2.5 Pro for initial question/answer pools—subsequently refined through expert review and deterministic rules.
Crucially, label generation in FlameVQA employs a hybrid pipeline: initial MLLM outputs are subjected to deterministic physical rules (e.g., thermal thresholding for hotspot segmentation using radiometric TIFF data), systematic cross-question logical checks (ensuring label consistency across mutually dependent questions), and expert human audits for ambiguous or high-stakes samples.
Physically-Grounded Label Verification
Physical grounding is operationalized by direct computation from radiometric TIFFs, ensuring labels for temperature-critical questions possess objective and reproducible foundations. For instance, thermal hotspots are algorithmically extracted via per-pixel thresholding, and their ground-projected area is estimated using field-of-view and ground sample distance metadata. Cross-modal and aggregation-based tasks, such as quantifying occluded fire presence or smoke coverage, rely on both thermal signatures and consistency constraints across associated VQA items. The pipeline flags and audits logical contradictions—e.g., frames labeled "no fire" cannot simultaneously support positive hotspot detection.
Baseline Evaluation and Empirical Findings
FlameVQA includes a large-scale evaluation of two open-source MLLMs, LLaVA-1.6-7B and Qwen3-VL-8B-Instruct, on a human-audited subset of the benchmark. The evaluation is conducted on the Willamette Valley subset, intentionally selected for its prevalence of challenging scenarios—dense smoke, occlusion, and ambiguous fire/smolder dynamics.
Key results:
- The automated (MLLM + deterministic rules) labeling pipeline matches human expert judgment in 70.78% of cases across the most challenging question types. Agreement rates are high for cross-modal reasoning (up to 90.98% on consistency questions), but dramatically lower for active fire detection under heavy smoke (36.12%), highlighting intrinsic ambiguities and limitations in RGB/thermal fusion, even for expert annotators.
- Among MLLMs, Qwen-VL achieves significantly higher overall accuracy (48.65%) than LLaVA (36.23%) on the hardest subset. Cross-modal reasoning is handled robustly by both (Qwen-VL: 87.19%, LLaVA: 85.51%), but coverage/distribution estimation remains a consistent failure point (Qwen-VL: 25.18%, LLaVA: 18.07%), pinpointing a gap in models' spatial aggregation and numeric reasoning capabilities.
- Presence and localization questions remain nontrivial due to occlusion, thermal confusion, and limitations in leveraging UAV flight metadata.
These findings empirically support the need for wildfire-specific adaptation and physically-supervised training, as off-the-shelf MLLMs are insufficient for consistent, safety-critical reasoning in operational wildfire contexts.
Implications and Future Directions
FlameVQA sets a new standard for physical and operational grounding in disaster-oriented multimodal VQA. By integrating radiometric thermal supervision, the benchmark enables more rigorous and reproducible evaluation compared to prior RGB-only or palette-based datasets, aligning MLLM development with real-world requirements for disaster intelligence, hazard localization, and flight safety.
The primary practical implication is the clear demand for advanced VLM architectures capable of leveraging physically accurate supervisory signals—moving beyond visual pattern recognition to robust, quantitative, and cross-modal understanding. The persistent weakness of SOTA MLLMs on spatial aggregation and percentage reasoning underscores a need for architectures that align visual and numeric understanding under physical constraints.
Theoretically, FlameVQA highlights the challenge of multimodal representation fusion in high-uncertainty and low-visibility scenarios. The benchmark further motivates research into physically-constrained learning, aggregation over spatial distributions, logical consistency enforcement, and the development of confidence-aware VQA agents suitable for safety-critical aerial domains.
Expanding site diversity, increasing expert-labeled subsets for high-stakes tasks, and integrating fine-grained flight and environmental metadata are all predicted to further catalyze robust, generalizable MLLM development for disaster management. The open-source release of both dataset and quality-control code establishes a reproducible standard for future benchmarking, model development, and evaluation.
Conclusion
FlameVQA is a physically-grounded, operationally realistic VQA benchmark that directly addresses the needs of UAV-based wildfire intelligence. By coupling radiometric thermal supervision with a hybrid MLLM-deterministic-human annotation pipeline, the benchmark exposes critical weaknesses in current multimodal models and establishes a platform for the advancement of reliable, physically-informed VQA systems in high-stakes, disaster-oriented applications.