CrisisMMD: Multimodal Twitter Datasets from Natural Disasters (1805.00713v1)

Abstract: During natural and man-made disasters, people use social media platforms such as Twitter to post textual and multime- dia content to report updates about injured or dead people, infrastructure damage, and missing or found people among other information types. Studies have revealed that this on- line information, if processed timely and effectively, is ex- tremely useful for humanitarian organizations to gain situational awareness and plan relief operations. In addition to the analysis of textual content, recent studies have shown that imagery content on social media can boost disaster response significantly. Despite extensive research that mainly focuses on textual content to extract useful information, limited work has focused on the use of imagery content or the combination of both content types. One of the reasons is the lack of labeled imagery data in this domain. Therefore, in this paper, we aim to tackle this limitation by releasing a large multi-modal dataset collected from Twitter during different natural disasters. We provide three types of annotations, which are useful to address a number of crisis response and management tasks for different humanitarian organizations.

• The paper introduces CrisisMMD with over 36,000 annotated tweet-image pairs from seven major natural disasters, enhancing crisis analytics with multimodal data.
• It details a robust annotation schema that categorizes information as informative versus not, specifies humanitarian needs, and assesses damage severity.
• The dataset paves the way for advanced multimodal fusion, joint embedding models, and improved disaster response tools in crisis management.

Overview of "CrisisMMD: Multimodal Twitter Datasets from Natural Disasters"

The paper "CrisisMMD: Multimodal Twitter Datasets from Natural Disasters" by Firoj Alam, Ferda Ofli, and Muhammad Imran addresses the limitations of existing crisis-related datasets by introducing the CrisisMMD dataset. This resource provides multimodal data, combining both textual and imagery content, collected from Twitter during seven natural disasters in 2017. Notably, the authors focus on overcoming the scarcity of labeled imagery data, which impedes the development of image-based analytics for disaster response.

Dataset Composition

The CrisisMMD dataset comprises over 14 million tweets and approximately 576,000 associated images obtained from Twitter during significant natural events: Hurricane Irma, Hurricane Harvey, Hurricane Maria, the Mexico earthquake, the California wildfires, the Iraq-Iran earthquake, and the Sri Lanka floods. The authors employ a multi-step filtering process to curate relevant data, resulting in a final collection of over 36,000 tweets with corresponding images.

Annotation Schema

The dataset is uniquely annotated along three key dimensions:

1. Informative vs. Not Informative: Identifying whether tweets or images are useful for humanitarian aid.
2. Humanitarian Categories: Classifying the content into categories like infrastructure damage, rescue efforts, or affected individuals.
3. Damage Severity Assessment: Assessing the severity of damage depicted in the images.

These annotations were acquired using the Figure Eight crowdsourcing platform, ensuring high-quality labels with strong inter-annotator agreement.

Implications and Applications

The release of CrisisMMD opens up multiple avenues for research across both NLP and computer vision domains. It provides a robust foundation for investigating multimodal information fusion, enabling development in several application areas:

• Joint Embedding Models: Researchers can exploit the dataset to learn joint embeddings for paired textual and visual data, facilitating cross-modal retrieval tasks.
• Image Captioning: The dataset offers a basis for improving image-to-text generation capabilities, which is crucial for automatic reporting systems.
• Disaster Response Tools: Practically, the dataset can enhance situational awareness tools by filtering and prioritizing vital information for humanitarian organizations, potentially aiding in better resource allocation and emergency response strategies.

Despite significant progress in text analysis for disaster response, the incorporation of images paves the way for a more comprehensive understanding and management of disaster scenarios. The dataset thus enhances the potential for automated systems to process real-time data more effectively, providing critical insights into on-ground realities during crises.

Future Directions

The authors suggest several future research directions including:

• Developing enhanced multimodal fusion strategies to leverage text and image synergies.
• Integrating the datasets for deploying real-time analytics platforms to aid decision-making during emergencies.
• Advancing image damage severity models to accurately classify and prioritize responses to infrastructure damages.

In conclusion, the CrisisMMD dataset represents a significant step forward in multimodal analysis within the context of disaster management, offering a rich resource for both theoretical exploration and practical application in AI-driven crisis response.