CrisisMMD: Multimodal Crisis Dataset

• CrisisMMD is a multimodal Twitter dataset featuring paired text and image annotations from seven major 2017 disasters for actionable crisis response.
• It supports key tasks such as informativeness filtering, humanitarian category classification, and damage severity assessment to guide disaster management.
• The dataset employs a rigorous filtering and annotation pipeline, addressing modality mismatches and class imbalances to set early benchmark protocols.

CrisisMMD is a publicly released multimodal Twitter dataset collected during seven natural disasters in 2017, created to support crisis-response research on paired tweet text and images with human annotations for multiple humanitarian tasks (Alam et al., 2018). It was introduced to address a specific gap in crisis informatics: earlier crisis datasets such as CrisisLex and CrisisNLP were primarily text-only, while CrisisMMD was positioned as the first publicly shared multimodal Twitter crisis dataset with ground-truth annotations for both text and images (Alam et al., 2018). In later benchmark taxonomies, it is treated as an early benchmark in the response phase of the disaster management cycle, specifically under “Post-disaster Coordination and Response,” where it supports assessing informativeness and categorizing humanitarian tasks from social media (Proma et al., 2022).

1. Origins and benchmark role

CrisisMMD emerged from the observation that disaster-response research had focused overwhelmingly on textual social media analysis even though images shared during disasters can be useful for situational awareness, identifying infrastructure damage, assessing severity of impact, locating affected populations, and understanding rescue and donation activities (Alam et al., 2018). The dataset therefore formalized a multimodal benchmark centered on tweet-level alignment between text and images, with labels designed for operational humanitarian use rather than generic media understanding.

The original paper framed the resource around three operationally relevant tasks: high-level informativeness filtering, humanitarian category classification, and image-based damage severity assessment (Alam et al., 2018). That design gave CrisisMMD a dual identity. On one side, it is a dataset paper without baseline model results or official benchmark splits. On the other, it became a reference point for later multimodal crisis classification work, explainability studies, augmentation studies, domain generalization, and zero-shot multimodal reasoning (Gupta et al., 2024).

A later survey of natural-disaster machine-learning benchmarks explicitly places CrisisMMD in the response phase, under post-disaster coordination and response, and describes it as an earlier benchmark in which tweets from seven natural disasters during 2017 were annotated for informativeness as a binary classification problem and across eight humanitarian categories as a multiclass classification problem (Proma et al., 2022). That placement is consequential because it distinguishes CrisisMMD from response-phase datasets aimed primarily at physical damage mapping from satellite or UAV imagery. CrisisMMD is instead centered on extracting actionable information from crisis communications.

2. Data acquisition, filtering, and annotation pipeline

The dataset covers seven natural disasters: Hurricane Irma, Hurricane Harvey, Hurricane Maria, the Mexico earthquake, the California wildfires, the Iraq-Iran earthquake, and the Sri Lanka floods (Alam et al., 2018). Data were collected from Twitter using event-specific keywords and hashtags. Image URLs were extracted from the extended_entities field so that tweets with multiple attached images could be retained (Alam et al., 2018).

Across all seven events, the raw crawl yielded 14,223,141 tweets and 576,294 images (Alam et al., 2018). The preprocessing pipeline then retained only tweets with at least one image URL, removed non-English tweets using Twitter-provided language metadata, enforced a minimum textual-content requirement of at least two words or hashtags, and removed duplicates using cosine similarity over tweet text, with tweets above 0.7 treated as duplicates (Alam et al., 2018). After filtering, 36,403 tweets remained; because of annotation-budget constraints, the authors sampled 16,097 tweets, associated with 18,126 images, for manual labeling (Alam et al., 2018).

Event	Sampled tweets	Images
Hurricane Irma	4,041	4,525
Hurricane Harvey	4,000	4,443
Hurricane Maria	4,000	4,562
California wildfires	1,486	1,589
Mexico earthquake	1,239	1,382
Iraq-Iran earthquake	499	600
Sri Lanka floods	832	1,025

Annotation was conducted on Figure Eight with at least 40 test questions per task, English-language expertise requirements, and three annotators per item (Alam et al., 2018). A central design decision was that text and images were annotated separately. Task 1 was performed first; Task 2 was applied only after informativeness assessment; Task 3 was restricted further to images already labeled as infrastructure and utility damage (Alam et al., 2018). This staged design preserved modality-specific signal and allowed later work to study aligned, partially aligned, and mismatched text-image labels.

3. Annotation schema and later benchmark protocols

The original CrisisMMD schema defines three tasks. Task 1 is informativeness classification with labels Informative, Not informative, and Don’t know or can’t judge. Task 2 is humanitarian category classification for informative content, with eight categories: Infrastructure and utility damage, Vehicle damage, Rescue, volunteering, or donation effort, Injured or dead people, Affected individuals, Missing or found people, Other relevant information, and Not relevant or can’t judge. Task 3 is damage severity assessment, applied only to images labeled as infrastructure and utility damage, with labels Severe damage, Mild damage, Little or no damage, and Don’t know or can’t judge (Alam et al., 2018).

Two structural details of the original schema shaped nearly all subsequent work. First, text and image annotations are separate, so a post may have informative text but a non-informative image, or the reverse. Second, the original dataset paper does not report train/dev/test splits, which means later studies had to define task-specific evaluation protocols themselves (Alam et al., 2018).

Subsequent CrisisMMD papers commonly operate on reduced or filtered label spaces rather than the full original annotation inventory. CrisisKAN presents a standard three-task multimodal setup in which Task 1 is informative versus non-informative, Task 2 is a five-class humanitarian or impact classification problem with infrastructure damage, vehicle damage, rescue efforts, affected individuals—including injury, dead, missing, found, and related cases—and others, and Task 3 is a three-class severity problem with severe, mild, and little/no damage (Gupta et al., 2024). It further distinguishes two training settings: Setting A uses only image-text pairs with identical labels, whereas Setting B trains on all labeled image-text pairs even when image and text labels differ; for Setting A, the reported train/validation/test counts are 9601/1573/1534 for Task 1, 2874/477/451 for Task 2, and 2461/529/530 for Task 3 (Gupta et al., 2024).

Other studies adopt still narrower humanitarian subsets. CrisisSpot reports 9,601/1,573/1,534 splits for the informative task and 6,126/998/955 for the humanitarian task, and explicitly notes that some prior work does not use the official split from the CrisisNLP repository, restricting comparison accordingly (Dar et al., 2024). The augmentation study on CrisisMMD uses the “agreed-upon annotation subset” for five-way humanitarian classification with 6,126 training, 998 validation/dev, and 955 test instances; in that subset, Affected Individuals is extremely sparse at 71/9/9 across train/dev/test, while Not Humanitarian dominates with 3252/521/504 (Urse et al., 4 Oct 2025). The explainable VLTCrisis framework also focuses on a five-label humanitarian formulation—infrastructure damage, affected individuals, rescue effort, other relevant info, and not humanitarian—and follows the filtering and split setup of Ofli et al. (2020), reporting 5,263/998/955 text instances and 6,126/998/955 image instances (Nguyen et al., 19 Mar 2026).

These later protocols are not contradictions of the original dataset; they are downstream task formulations imposed on a dataset whose original release emphasized annotation breadth over standardized evaluation partitions. A plausible implication is that “CrisisMMD” refers both to a fixed underlying corpus and to a family of benchmark protocols built from that corpus.

4. Modeling trajectories on CrisisMMD

The most visible line of work on CrisisMMD concerns multimodal fusion under noisy, short, and partially mismatched tweet-image pairs. CrisisKAN addresses three problems simultaneously: sparse tweet context, semantic mismatch between image and text features, and the need for interpretability in high-stakes settings (Gupta et al., 2024). Its architecture combines a DenseNet visual encoder, an ELECTRA text encoder enriched with Wikipedia-derived text, guided cross-attention for multimodal fusion, and Grad-CAM-based image explanations. On the CrisisMMD three-task setup, it reports Setting A accuracies of 91.7 for informativeness, 93.6 for humanitarian category classification, and 73.1 for severity assessment, and an MTMS score of 87.1; under Setting B, it remains ahead of the compared baselines on the reported tasks (Gupta et al., 2024).

A later fusion line shifts away from task-specific backbones toward stronger pretrained vision-language representations. “Differential Attention for Multimodal Crisis Event Analysis” replaces DenseNet and Electra with frozen CLIP vision and text encoders, enriches the text side with LLaVA-generated image-grounded descriptions, and combines Guided Cross Attention with an additional Differential Attention layer (Munia et al., 7 Jul 2025). Its results show a clear pattern rather than a single uniformly dominant recipe: for Task 1, the best configuration is CLIP Vision + CLIP text + LLaVA + Guided CA with 92.91 accuracy and 91.91 macro F1; for Task 2, Guided CA alone gives the highest accuracy at 94.02, while CLIP + Wiki + Guided CA + Diff Attn gives the highest macro F1 at 71.04; for Task 3, Guided CA and Guided CA plus Differential Attention are nearly tied around 69 accuracy and 55 macro F1 (Munia et al., 7 Jul 2025). The paper explicitly concludes that Guided Cross Attention remains the dominant contributor and that Differential Attention yields only modest, task-dependent gains.

CrisisSpot extends CrisisMMD modeling beyond text-image fusion by adding graph-derived structure and social-context features (Dar et al., 2024). Its architecture uses BERT text features, ResNet50 image features, an Inverted Dual Embedded Attention mechanism to model harmonious and contrary cross-modal interactions, a CLIP-based similarity graph propagated with GraphSAGE, and a 21-dimensional social holistic vector built from sentiment, emotion, crisis lexicon counts, user engagement, and user informativeness signals (Dar et al., 2024). On the informative task it reports 97.58 accuracy, 96.70 precision, 99.80 recall, and 98.23 F1; on the humanitarian task it reports 90.01 accuracy and 90.13 F1, exceeding the strongest fully reported baseline in that comparison set (Dar et al., 2024). Its ablations show that the graph-learning module, the social-context module, and the IDEA attention mechanism all contribute measurable gains.

Taken together, these studies show a progression in what later papers treat as the central technical difficulty of CrisisMMD: first multimodal alignment, then knowledge infusion and pretrained vision-language semantics, then richer sample-to-sample structure and social context. This suggests that CrisisMMD has functioned less as a benchmark for one fixed architecture class than as a stress test for increasingly elaborate ways of reconciling noisy social media text with equally noisy disaster imagery.

5. Explainability, robustness, and generalization

Explainability has become a major secondary axis of CrisisMMD research. VLTCrisis is explicitly interpretable-by-design: it uses ViLT to learn a joint text-image representation, supervises token-level text rationales, transfers those rationales to image patches through IPOT-based cross-modal alignment, and then classifies using only the extracted rationales rather than the full opaque input (Nguyen et al., 19 Mar 2026). On the five-class humanitarian task it reports Macro-F1 of 0.822, above all unimodal baselines and above several multimodal baselines, though slightly below FCLIP at 0.834 (Nguyen et al., 19 Mar 2026). It also reports Token-F1 of 0.826 for text rationale extraction, comprehensiveness of 0.514 and sufficiency of 0.072, and human preference for its image rationales over RNET/RISE in 81% of pairwise comparisons; annotators could infer the class from its image rationales in 74% of cases versus 62% for the post-hoc baseline (Nguyen et al., 19 Mar 2026). A plausible implication is that CrisisMMD’s paired but separately annotated text and image channels are sufficiently aligned, on average, to support explanation transfer across modalities.

Robustness under domain shift is addressed directly by CAMO, which reformulates CrisisMMD as a multimodal domain generalization benchmark with disaster type as the domain variable: earthquakes, hurricanes, floods, and wildfires (Ma et al., 8 Dec 2025). CAMO factorizes modality-specific features into modality-general and modality-specific components, aligns the shared space with supervised contrastive learning, and then further decomposes the fused representation into a domain-specific latent variable and a domain-invariant causal variable, formalized through the independence relation $X_c \perp D_l$ (Ma et al., 8 Dec 2025). Under leave-one-domain-out evaluation, it reports the best average informative-task accuracy at 0.903, above VGG+BERT, SCBD, CLMC, SimMMDG, and CMRF (Ma et al., 8 Dec 2025). The humanitarian results are more ambiguous: the table reports CLMC at 0.823 average accuracy and CAMO at 0.801, even though the prose claims best overall performance, so the informative-task evidence is clearer than the humanitarian-task evidence (Ma et al., 8 Dec 2025).

Class imbalance and data scarcity have motivated a parallel line of augmentation work. The five-way humanitarian subset studied in “Multimodal Learning with Augmentation Techniques for Natural Disaster Assessment” is extremely skewed, especially for Affected Individuals at 71/9/9 against Not Humanitarian at 3252/521/504 (Urse et al., 4 Oct 2025). In that setting, back-translation and transformer-based paraphrasing reliably improve text-only models, caption-based text augmentation hurts because captions are added only during training, diffusion-based image augmentation helps selectively and in an architecture-sensitive way, and multimodal fusion remains stronger than unimodal learning overall (Urse et al., 4 Oct 2025). The paper also shows that its multi-view formulation underperforms because additional augmented views appear at training time but are absent at test time. This makes CrisisMMD a benchmark not only for multimodal fusion quality but also for how models react to severe imbalance and distribution mismatch.

6. Benchmark ecology, reuse, and limitations

CrisisMMD occupies a specific place in the broader natural-disaster benchmark ecosystem. NADBenchmarks lists it under Response → Post-disaster Coordination and Response, and the same survey states that HumAID “builds on concepts from CrisisMMD,” offering a larger dataset for humanitarian aid classification with tweets for 19 natural disasters across ten categories (Proma et al., 2022). The survey also positions CrisisBench and MEDIC downstream in the same lineage, moving toward combined public datasets, multilabel settings, and multitask learning (Proma et al., 2022). In that sense, CrisisMMD is not simply one benchmark among many; it is an anchor point in a sequence of increasingly larger and more structured crisis-response datasets.

At the same time, later reuse often departs substantially from the original task design. A cross-lingual and cross-domain crisis classification study incorporates CrisisMMD only as one component of a much larger unified text-only Multi-Crisis Dataset, uses 11,400 English tweets from CrisisMMD, ignores images entirely, and remaps some labels into a binary related/not-related formulation (Sánchez et al., 2022). RoadFed uses CrisisMMD as a public multimodal benchmark for its detector, but does not specify the exact CrisisMMD task, label mapping, subset, or split, even though it reports 92.00 accuracy and 91.52 F1 for MRHD; the paper itself therefore treats CrisisMMD more as external evidence for multimodal classification than as a fully specified road-hazard benchmark (Yuan et al., 14 Feb 2025). HM-RAG applies a hierarchical multi-agent multimodal RAG pipeline to zero-shot CrisisMMD classification and reports 72.06 on Task 1, 51.50 on Task 2, 52.09 on Task 2 Merged, and 58.55 average accuracy, but leaves CrisisMMD-specific retrieval-database construction details unspecified (Liu et al., 13 Apr 2025). These cases show how benchmark portability can also produce protocol drift.

Several limitations derive directly from the original release. CrisisMMD is English-only, restricted to tweets with images, and limited to seven events from 2017 (Alam et al., 2018). The original paper makes clear that class imbalance is substantial, that some classes such as Missing or found people are rare, that text and image labels may diverge because the modalities were annotated separately, and that no official train/dev/test splits were provided (Alam et al., 2018). Later papers repeatedly encounter the consequences of those design choices: different humanitarian taxonomies, different filtering rules, Setting A versus Setting B protocols, imbalance-driven augmentation studies, and partial inconsistencies in cross-paper comparison.

The dataset’s legacy is therefore twofold. First, it made multimodal crisis informatics empirically tractable by releasing aligned tweet text and images with humanitarian annotations (Alam et al., 2018). Second, it exposed a persistent methodological problem: the same underlying corpus supports multiple legitimate but non-identical benchmark constructions. CrisisMMD remains important precisely because later research continues to use it as a common reference point while negotiating the unresolved issues it made visible—modality mismatch, label sparsity, split standardization, explainability, and generalization across unseen disasters.