Fire360: A Benchmark for Robust Perception and Episodic Memory in Degraded 360-Degree Firefighting Videos

Abstract: Modern AI systems struggle most in environments where reliability is critical - scenes with smoke, poor visibility, and structural deformation. Each year, tens of thousands of firefighters are injured on duty, often due to breakdowns in situational perception. We introduce Fire360, a benchmark for evaluating perception and reasoning in safety-critical firefighting scenarios. The dataset includes 228 360-degree videos from professional training sessions under diverse conditions (e.g., low light, thermal distortion), annotated with action segments, object locations, and degradation metadata. Fire360 supports five tasks: Visual Question Answering, Temporal Action Captioning, Object Localization, Safety-Critical Reasoning, and Transformed Object Retrieval (TOR). TOR tests whether models can match pristine exemplars to fire-damaged counterparts in unpaired scenes, evaluating transformation-invariant recognition. While human experts achieve 83.5% on TOR, models like GPT-4o lag significantly, exposing failures in reasoning under degradation. By releasing Fire360 and its evaluation suite, we aim to advance models that not only see, but also remember, reason, and act under uncertainty. The dataset is available at: https://uofi.box.com/v/fire360dataset.

• The paper introduces a novel benchmark dataset capturing 360° firefighting videos under degraded conditions.
• It evaluates AI performance across tasks like VQA, action captioning, and object localization under real-world challenges.
• Results reveal a significant performance gap between current AI models and human experts, highlighting key areas for improvement.

Fire360: A Benchmark for Robust Perception and Episodic Memory in Degraded Firefighting Videos

Introduction and Objectives

"Fire360" introduces a sophisticated dataset and benchmark for evaluating AI capabilities in challenging firefighting scenarios, where environmental conditions pose significant threats to human responders. The paper highlights the critical need for AI systems that can perceive, reason, and act in real-world settings characterized by smoke, poor visibility, and structural challenges using 360-degree video data. It offers a robust framework across five specific tasks: Visual Question Answering (VQA), Temporal Action Captioning, Object Localization, Safety-Critical Reasoning, and Transformed Object Retrieval (TOR), aiming to advance AI’s ability to operate under uncertainty and degradation.

Figure 1: Example frames from Fire360, showcasing diverse operational settings and environmental conditions: (top row) outdoor firefighting scenes in day and night conditions, (bottom row) indoor low-visibility environments with dense smoke and limited lighting.

Dataset Description and Structure

Fire360 comprises 228 360-degree videos captured during professionally conducted firefighter training sessions. The dataset is designed to reflect diverse operational environments including both indoor and outdoor settings. Each video is annotated with action segments, object locations, and degradation metadata, ensuring comprehensive coverage for the benchmark tasks.

Figure 2: Viewpoint representations derived from Fire360’s 360$^{\circ}$ footage.

The Fire360 dataset serves as a critical tool for training AI models under conditions that mimic real firefighting scenarios. It captures environments with high levels of visual degradation like smoke and thermal distortion, requiring models to handle complex perceptual tasks beyond standard datasets.

Figure 3: Fire360 content distribution. (a) Scene categories showing indoor/outdoor ratio, (b) Action categories with instance counts and percentages.

Benchmark Tasks and Evaluation

The benchmark tasks are structured to test AI systems across various competencies: spatial reasoning, temporal grounding, degradation robustness, procedural compliance, and transformation-invariant recognition.

• Visual Question Answering (VQA): Requires models to perform spatial inquiries across full panoramic frames.
• Temporal Action Captioning: Challenges models to generate descriptive captions of firefighter actions under degraded conditions.
• Object Localization: Focuses on the accurate detection of firefighting gear despite occlusion and distortion.

  Figure 4: Effect of input representation on both VQA accuracy (left) and object localization performance (right). Rectilinear projections consistently outperform equirectangular views across all models and tasks by mitigating panoramic distortion and improving robustness under degradation.

• Safety-Critical Reasoning: Assesses the model's ability to identify and reason about violations of standard safety procedures.
• Transformed Object Retrieval (TOR): Evaluates the ability of models to retrieve fire-damaged objects matched against their pristine exemplars.

  Figure 5: Degradation-aware accuracy comparison on the VQA task using equirectangular 360-degree frames.

Technical Insights and Results

The evaluation shows that current AI models struggle significantly in highly degraded settings. Human experts maintain high performance levels (over 80% accuracy in all conditions), while AI models lag, particularly in the Transformed Object Retrieval task where GPT-4o shows only 39.8% accuracy versus the human benchmark of 83.5%.

Future Directions

The paper advocates for future research to focus on embedding robustness to environmental degradation into AI models by:

1. Simulating object degradation during training to develop transformation-invariant embeddings.
2. Applying instruction-tuned conditioning to improve material state disambiguation.
3. Developing vision-language agents capable of multimodal planning and context reasoning.

Limitations

Despite its comprehensive approach, the dataset is collected from a single training institute, which could influence generalization across diverse firefighting scenarios. Furthermore, while the dataset introduces innovative tasks, these require significant computational resources for model evaluation.

Conclusion

Fire360 sets a new standard for benchmarking AI systems in safety-critical environments, catalyzing progress in understanding and navigating obscured and degraded settings typical of firefighting scenarios. By providing both the dataset and evaluation metrics, the research offers a foundation for developing AI systems capable of reliable performance in real-world applications where human safety is paramount. Through rigorous testing across various domains, Fire360 aims to close the performance gap between human experts and AI models under challenging conditions, promoting not only technical advancements but also societal benefits in emergency response.

(Figure 6 and Figure 7)

Figure 6: Illustration of the Transformed Object Retrieval (TOR) task.

Figure 7: Illustration of distortion severity in Fire360 equirectangular projections.