NADBenchmarks -- a compilation of Benchmark Datasets for Machine Learning Tasks related to Natural Disasters

Abstract: Climate change has increased the intensity, frequency, and duration of extreme weather events and natural disasters across the world. While the increased data on natural disasters improves the scope of ML in this field, progress is relatively slow. One bottleneck is the lack of benchmark datasets that would allow ML researchers to quantify their progress against a standard metric. The objective of this short paper is to explore the state of benchmark datasets for ML tasks related to natural disasters, categorizing them according to the disaster management cycle. We compile a list of existing benchmark datasets introduced in the past five years. We propose a web platform - NADBenchmarks - where researchers can search for benchmark datasets for natural disasters, and we develop a preliminary version of such a platform using our compiled list. This paper is intended to aid researchers in finding benchmark datasets to train their ML models on, and provide general directions for topics where they can contribute new benchmark datasets.

• The paper compiles benchmark datasets across disaster management phases to enhance ML model training in natural disaster scenarios.
• It introduces the NADBenchmarks web platform to facilitate dataset access without creating oversimplified leaderboards.
• It identifies research gaps, particularly in prevention and recovery phases, urging future expansion of dataset diversity.

Summary of "NADBenchmarks -- A Compilation of Benchmark Datasets for Machine Learning Tasks Related to Natural Disasters"

This paper investigates the current state of benchmark datasets for machine learning tasks related to natural disasters, addressing a critical bottleneck in research due to the lack of standardized datasets. By categorizing various datasets according to the disaster management cycle (mitigation, preparedness, response, and recovery), the paper promotes efficient data utilization in training ML models aimed at natural disaster management. A web platform, NADBenchmarks, is introduced to aid researchers in accessing these datasets without generating a leaderboard platform, thus avoiding oversimplification of research progress.

Introduction to Benchmark Datasets for Natural Disasters

The impact of climate change is heightening the frequency and severity of natural disasters globally. Despite the increasing data availability, the progress in effective management solutions using machine learning remains sluggish, significantly hindered by the absence of benchmark datasets. The paper surveys datasets relevant to the four phases of disaster management: mitigation, preparedness, response, and recovery. These phases encompass activities from forecasting potential disasters to post-disaster recovery efforts, which are crucial yet inadequately supported by standardized data collections.

Methodology

Search and Data Extraction

Utilizing databases like Google Scholar, ACM, and Scopus, along with conference proceedings (CVPR, NeurIPS, ICML, ICLR), the search focused on papers introducing datasets relevant to natural disasters over the past five years. Key parameters extracted include dataset characteristics, type of ML tasks, topic specificity, and phase categorization in the disaster cycle.

Curation of Data

Key dataset attributes such as name, ML task, natural disaster topic, and management phase were compiled to construct a comprehensive list. The web platform NADBenchmarks is created to facilitate accessibility and exploration of these datasets.

Review of Benchmark Datasets

Mitigation Phase

Datasets like EarthNet2021 focus on earth surface forecasting using satellite imagery for seasonal cycle prediction, and DroughtED facilitates drought forecasting through multiclass ordinal classification of spatio-temporal data.

Preparedness Phase

Predominant datasets include Next Day Wildfire Spread, essential for predicting wildfire progression, and MEDIC for multiclass disaster detection. Challenges like VIDI offer a video-based approach to disaster identification, expanding the modality of data used in this phase.

Figure 1: Pie-chart showing the number of papers found for each of the disaster management phases and subphases.

Response Phase

Benchmarks such as CrisisMMD and HumAID categorize social media content for post-disaster assessment using binary and multiclass classification methods, focusing on resource allocation and damage evaluation.

Recovery Phase

Though scarce, datasets like the image-sentiment dataset aim to analyze psychological impacts post-disaster through sentiment classification, underscoring a potential area for future dataset development.

Trends and Research Gaps

The analysis identifies gaps, notably the underrepresentation of benchmark datasets in prevention and recovery phases. There is overdependence on social media and Earth observation datasets, highlighting the need for diversification in data sources like cell networks and financial records. Emphasis on multitask learning frameworks is emerging, yet few benchmarks currently support such approaches.

Figure 2: Our interface with key features highlighted.

Interface Implementation

The NADBenchmarks platform adopts features from existing benchmark websites to enhance accessibility. Researchers can access detailed dataset attributes and additional information through an interactive user interface, as seen in Figure 2.

Conclusion

By systematically compiling benchmark datasets, the paper aims to accelerate research in disaster management through standardized data. Future directions include expanding dataset types, introducing comprehensive evaluation metrics, and continuous platform updates to reflect advancements in research. Stakeholder contributions are encouraged to enrich the dataset repository, driving progress in ML applications for natural disasters.