OpenEvents V1: Event-Centric Multimodal Dataset

• OpenEvents V1 is a large-scale multimodal benchmark dataset that unifies editorial images with narrative news articles to enable event-centric vision–language understanding.
• It integrates automated dense image descriptions and LLM-generated narrative synthesis, validated by human review for accuracy in temporal and factual details.
• The benchmark defines dual tasks—context-rich image captioning and narrative-driven image retrieval—to enhance multimodal reasoning over complex real-world events.

OpenEvents V1 is a large-scale multimodal benchmark dataset designed to advance event-centric vision–language understanding through contextual and temporal grounding. Distinct from conventional image captioning corpora that focus on surface-level descriptions, OpenEvents V1 centers on real-world news events by pairing hundreds of thousands of editorial images with corresponding narrative-style news articles and associated metadata. The benchmark formalizes two principal tasks: (1) generating event-aware, context-rich image captions grounded in both vision and external narrative evidence; and (2) retrieving event-relevant images from a massive index in response to natural language event queries. These tasks foster progress in multimodal modeling by demanding deep reasoning over event context, temporal anchoring, and named-entity recognition, setting a foundation for complex event-grounded vision–language research (Nguyen et al., 23 Jun 2025).

1. Dataset Construction and Content

OpenEvents V1 comprises 415,324 in-the-wild news images and 202,803 distinct news articles, sourced from global outlets such as CNN (24,200 articles; 89,596 images, spanning 2011–2022) and The Guardian (178,603 articles; 325,728 images, spanning 2019–2025) (Nguyen et al., 23 Jun 2025). The domain coverage includes breaking news (≈50% of images), sports (20–25%), politics, lifestyle, health, arts, and culture, as well as long-tail event types such as festivals, natural disasters, and demonstrations.

Annotation proceeds via a human–agentic framework:

1. Dense Visual Description: Automated dense image descriptions using large vision–LLMs (Molmo).
2. Contextual Question Generation: LLM-generated prompts focus on “who/what/when/where/why.”
3. Evidence-Based Answer Extraction: LLMs identify article evidence at the sentence level for each prompt or return “no information.”
4. Narrative Caption Synthesis: Fuses validated evidence and dense description into a paragraph-style, event-rich caption.
5. Human Review & Refinement: Editors verify factual reliability, disambiguate named entities and temporal information, and correct or reject erroneous samples.

For each article, structured metadata includes a precise publication timestamp (event timing), explicit location data (city, venue), and named-entity annotations of principal participants (persons, teams, organizations) extracted via off-the-shelf NER applied to the article text (Hien et al., 16 Jun 2026).

Loose image-to-text alignment is characteristic: frequently only one or two sentences in an article explicitly refer to an image, intensifying the cross-modal grounding challenge.

2. Benchmark Tasks and Formal Definitions

The OpenEvents V1 benchmark defines two central, tightly coupled tasks:

Task A: Event-Aware Image Captioning

Given an input image $I$ and, optionally, its associated article text $T$, the objective is to produce a narrative caption $C$ recounting the depicted event, including named entities, temporal anchors, causality, and outcomes.

• Input: $I \in Images$; $T \in Articles$
• Output: $C$, paragraph-style caption grounded in both $I$ and $T$
• Function: $f_{caption} : (I, T) \rightarrow C$

Task B: Narrative-Style Image Retrieval

Given a narrative-style caption query $C$ describing a real-world event, the goal is to retrieve and rank images $T$0 that are most semantically aligned with the event description.

• Input: $T$1 (event-centric text)
• Output: Ranked image list $T$2
• Function: $T$3

These dual tasks are unified by the necessity for multimodal cross-referencing and event grounding, as visual content alone is insufficient for full event understanding.

3. Evaluation Protocols and Metrics

Evaluation rigorously assesses both the semantic fidelity of event-enriched captions and retrieval precision under standardized protocols.

Captioning Metrics (Task A)

• BLEU-4: Modified n-gram precision up to four grams, including brevity penalty.
• METEOR: Harmonic mean of unigram precision/recall subject to a fragmentation penalty.
• CIDEr: TF-IDF weighted n-gram consensus over reference captions:

$T$4

• CLIPScore: Vision–text encoder similarity:

$T$5

Retrieval Metrics (Task B)

• Mean Average Precision (mAP):

$T$6

• NDCG@K: Normalized Discounted Cumulative Gain.
• Nearest-Neighbor Accuracy (Recall@1): Fraction of queries for which the top candidate matches the ground truth.
• AUC: Area under the precision–recall curve.

Task definitions and metric selection emphasize the inherent complexity of aligning event-rich textual content with real-world images anchored in time, location, and participants (Nguyen et al., 23 Jun 2025, Hien et al., 16 Jun 2026).

4. Baseline and Advanced Multimodal Approaches

Initial results across both tasks demonstrate the essential role of narrative evidence for event-centric benchmarking.

Captioning Baselines

Method	CLIPScore	CIDEr	BLEU-4	METEOR
SmolVLM	0.4609	0.0044	0.0155	0.0789
SmolVLM + Article	0.5552	0.0170	0.0229	0.0738
Qwen	0.5283	0.0282	0.0256	0.1320
Qwen + Article	0.5855	0.0565	0.0419	0.1383
Gemma	0.5945	0.0111	0.0243	0.1322
Gemma + Article	0.6634	0.0184	0.0341	0.1453

Performance is substantially higher for image+article pipelines, with relative gains of 20–50% in CIDEr and BLEU. However, even the strongest baseline (Gemma+Article) does not approach human-level event understanding, indicating the need for advanced retrieval-augmented frameworks (Nguyen et al., 23 Jun 2025).

Retrieval Baselines

No.	Method	mAP	NDCG	NN	AUC
1	CLIP	0.2467	0.3407	0.1586	0.0302
2	OpenCLIP	0.1845	0.2703	0.1845	0.0185
3	SBERT + Flan-T5	0.2134	0.2837	0.1376	0.0220
4	SBERT + BART	0.2840	0.3628	0.1863	0.0372
5	SBERT + Pegasus	0.2868	0.3665	0.1930	0.0362
6	SBERT+Flan-T5 + CLIP re-rank	0.2795	0.3408	0.1986	0.0303
7	SBERT+BART + CLIP re-rank	0.3232	0.3978	0.2226	0.0436
8	SBERT+Pegasus + CLIP re-rank	0.3216	0.3986	0.2173	0.0450

Direct CLIP matching underperforms for queries necessitating non-visual reasoning, while SBERT-based article retrieval pipelines with summarization and hybrid CLIP re-ranking outperform pure vision–language approaches by 15% mAP (0.32 vs. 0.25). This suggests that semantic filtering and document-level context extraction are critical for bridging the visual–narrative gap (Nguyen et al., 23 Jun 2025).

Advanced Pipeline: CIAN

The Contextual Image-Article Narrator (CIAN) implements a four-stage, retrieval-augmented generation pipeline: SigLIP-based context retrieval, BART summarization, LoRA-fine-tuned Qwen narrative generation, and n-gram-aware refinement to optimize CIDEr and CLIPScore. CIAN achieves mAP = 0.979, Recall@1 = 0.969, Recall@10 = 0.996 in retrieval, and CLIPScore = 0.820, CIDEr = 0.094 in captioning on the public-test split, surpassing all prior baselines (Hien et al., 16 Jun 2026).

5. Usage Guidelines and Data Access

OpenEvents V1 is publicly available at https://ltnghia.github.io/eventa/openevents-v1 (Nguyen et al., 23 Jun 2025). The release contains:

• Data files:
  • train.json, public_test.json, private_test.json (image_id, caption)
  • database.json (full article text, associated image_id list)
  • Image archive (>415k images, named by image_id)

Recommended Workflow

1. Text preprocessing (tokenization, summarization).
2. Visual feature extraction (e.g., CLIP, ViT embeddings).
3. Training multimodal models with cross-attention to image and text streams.
4. Evaluation on public_test with official metrics; submission of private_test predictions for leaderboard ranking.

Pairing of images and captions is via image_id; retrieval pipelines index database.json for full-scale ranking.

6. Research Directions and Open Problems

OpenEvents V1 foregrounds several unsolved research problems and methodological frontiers:

• Temporal reasoning: Modeling sequences and temporal causality across news events.
• Cross-document event linking: Clustering and associating image–article pairs by shared event entities or themes.
• Retrieval-augmented generation: Fusing retrieved, contextually relevant articles with vision–LLMs following the retrieval-augmented generation paradigm.
• Fact verification: Detecting hallucinated or spurious facts in generated event descriptions.
• Multimodal question answering: Expanding intermediate QA steps into full joint vision–text event QA models.
• Zero-shot and multilingual grounding: Extending event understanding to new languages and informal or historical corpora.

A plausible implication is that large-scale, contextually-annotated event data paired with strong retrieval–generation hybrids will catalyze progress in grounded multimodal reasoning and cross-modal event understanding.

7. Significance and Impact

By unifying massive-scale, real-world news imagery, deeply annotated narrative captions, and structured metadata, OpenEvents V1 establishes a new standard for benchmarks in event-grounded multimodal research. Its dual tasks—event-enriched captioning and narrative-guided image retrieval—provide both the scale and contextual complexity necessary for robust evaluation and enable the development of models capable of bridging raw visual evidence and semantically rich event narratives (Nguyen et al., 23 Jun 2025, Hien et al., 16 Jun 2026). The dataset's emphasis on human-validated evidence, temporal and spatial metadata, and open evaluation protocols is poised to drive advances in event-aware vision–language modeling and downstream applications requiring factual, temporally-anchored, and context-rich multimodal reasoning.