- The paper introduces CIAN, a multi-stage framework for event-enriched image captioning that leverages dense retrieval and retrieval-augmented generation for improved narrative coherence.
- It employs targeted dataset enhancements and SigLIP-based vision-language alignment to significantly boost retrieval accuracy (mAP 0.979) and caption quality (CIDEr increased from 0.014 to 0.094).
- The framework combines LoRA-fine-tuning with N-gram refinement to generate fluent, context-aware captions, though it still faces challenges with entity grounding and dataset alignment.
Multi-Stage Event-Enriched Image Captioning with Retrieval-Augmented Generation
Introduction
The paper proposes CIAN (Contextual Image-Article Narrator), a multi-stage framework designed to address the limitations of existing image captioning systems in capturing event-centric, non-visual information. CIAN targets the task of event-enriched image captioning, wherein generated descriptions are expected to convey not only the explicit visual content but also implicit narrative context—such as event significance, temporal-spatial properties, and the roles of participants—grounded in external textual knowledge. The approach utilizes dense retrieval to select relevant news articles, summarizes event information, generates context-aware captions with a LoRA-fine-tuned vision-LLM, and refines output using N-gram-based mechanisms. Evaluations on the OpenEvents-V1 benchmark demonstrate that CIAN achieves substantial improvements in both retrieval accuracy and caption quality, validating the effectiveness of retrieval-augmented generation and linguistic post-processing in multimodal narrative understanding.
Methodology
Dataset Curation and Enhancement
CIAN begins by addressing the misalignment issue pervasive in OpenEvents-V1, where images and paired articles often lack tight semantic correspondence. To mitigate alignment noise, the framework employs targeted web crawling (focused on CNN image galleries flagged by “hide caption” markers) to build a high-quality supplement of approximately 23,000 explicit image-caption pairs. These are visually matched to existing dataset entries with SigLIP to maximize relevance, thereby enhancing the precision of subsequent retrieval and captioning stages.
Context Retrieval with SigLIP
The core retrieval module utilizes SigLIP for dense vision-language alignment. Each query image is embedded, and the top-K visually-similar images are retrieved from the database. Associated articles are scored using aggregated embeddings (title and body, with a weighted combination), and final relevance is computed via cosine similarity between the image and article representations. This context retrieval pipeline enables the assembly of event-centric knowledge directly tied to the query image, rather than relying solely on pixel-level cues.
Narrative Generation with LoRA-fine-tuned Qwen
Article context is first condensed using BART-based summarization. Subsequently, a LoRA-fine-tuned Qwen2.5-VL model produces the initial event-enriched caption. The prompt includes both the query image and the salient summary, guiding the model toward human-like, coherent narrative construction that integrates visual and textual signals. LoRA adaptation ensures the generative model is efficiently specialized for this downstream task without full-parameter finetuning.
N-Gram-based Refinement
To further align generated captions with human stylistic and lexical preferences (as measured by metrics like CIDEr), the final stage computes high-frequency n-gram distributions from the training set. During inference, a refiner prompt encourages rewriting of the caption to softly incorporate these n-grams. Controlled rewriting is performed by a dedicated QwenRefine model, improving fluency, informativeness, and alignment with ground truth without sacrificing semantic fidelity.
Experimental Results
CIAN achieves superior document-level retrieval on OpenEvents-V1, with a mean Average Precision (mAP) of 0.979, Recall@1 of 0.969, and Recall@10 of 0.996. Comparative analysis demonstrates that SigLIP outperforms established alternatives such as CLIP, BLIP, and DINOv2, validating its robustness in event-driven cross-modal retrieval.
Caption Quality
For captioning, the system reports a CLIP Score of 0.82 and a CIDEr score of 0.094 (up from 0.014 for the baseline). Each staged component of CIAN contributes cumulatively to performance: dataset enhancement improves CIDEr; the narrative generation stage yields substantial gain; and n-gram-based refinement further boosts n-gram overlap, capturing human-descriptive conventions.
Ablation Analysis
The ablation study quantifies the impact of each pipeline stage. The baseline (image + article, simple prompt) is notably weak on reference metrics. Dataset enhancement increases CIDEr due to better alignment. Narrative generation produces higher-quality descriptions, while n-gram refinement achieves the highest CIDEr and CLIP scores, underscoring the necessity of both contextual integration and linguistic post-processing in event-enriched captioning.
Analysis of Failure Modes
Despite strong overall results, CIAN is subject to several limitations. Retrieval failures occur due to residual dataset noise and visual domain augmentations. Entity grounding is brittle: the system often describes scenes accurately but fails to link visual subjects to named entities within the textual narrative, undermining deep event understanding. The dataset curation pipeline, hinging on the "hide caption" marker, cannot address all instances of weak image-article alignment, leaving room for improvement in dataset construction and entity linking.
Implications and Future Directions
CIAN demonstrates the efficacy of combining dense retrieval, prompt engineering, parameter-efficient vision-LLM adaptation, and controlled linguistic refinement in the context of event-enriched image captioning. The pipeline serves as a blueprint for extending traditional caption generation to incorporate knowledge-driven, context-aware, and narrative-rich descriptions. Future research should address shortcomings in entity grounding through multimodal entity linking, explore advanced fusion mechanisms in MLLMs, and introduce reinforcement learning objectives targeting cross-modal narrative coherence. Furthermore, more general and scalable dataset curation strategies will be critical for broad deployment in knowledge-intensive multimodal tasks.
Conclusion
CIAN provides a technically robust, retrieval-augmented framework for event-enriched image captioning. Its multi-stage architecture effectively leverages external documents to produce captions that are both descriptive and contextually sophisticated. The empirical improvements on OpenEvents-V1 validate the approach, while also highlighting challenges for future work in multimodal entity grounding and dataset alignment. The study advances the field toward deeper integration of visual, textual, and narrative information in automated image understanding.
