- The paper presents a novel hierarchical retrieval-augmented framework that integrates visual and textual signals for context-rich, knowledge-grounded news captioning.
- It employs a three-stage pipeline—structured visual context extraction, focused context retrieval, and LLM-based synthesis—to ensure factual accuracy and detailed caption generation.
- Empirical analysis on event datasets demonstrates significant gains in retrieval metrics and caption quality, validating the method's impact in automated journalism.
Hierarchical Multi-Modal Retrieval-Augmentation for Knowledge-Grounded News Image Captioning
Traditional image captioning systems, predominantly relying on visual cues, frequently fail to produce contextually informative and knowledge-grounded descriptions, especially when critical details are not overtly visible in the image itself. This shortcoming is most pronounced in news and event-centric scenarios, where accurate captioning requires synthesis of visual understanding with external, factual knowledge—attributes, event specifics, and contextual narratives. The paper presents a retrieval-augmented image captioning framework, designed to systematically overcome these deficits by leveraging hierarchical multi-modal article retrieval, enabling generation of context-rich, factually detailed captions tailored for news domains (2606.18553).
Hierarchical Multi-Modal Article Retrieval
The retrieval module departs from simplistic text-based approaches and treats news articles as structured documents, decomposing them into semantic components—headlines, leads, body text, and image captions—with learned differential weighting. Textual features are encoded via pre-trained LLMs (BERT, Sentence Transformers), while visual placement features are integrated using CLIP-based encoders. The retrieval pipeline employs three distinct similarity computation channels:
- Content-Visual Alignment: Measures semantic similarity between the query image's textual representation and article components.
- Visual-Visual Coherence: Computes cross-image similarity between the query image and each article-embedded image, maximizing relevance.
- Discourse Positioning Score: Quantifies the correlation between image placement and event centrality within article discourse structure.
The composite retrieval score is obtained via a learned combination of these similarity channels. Contextual relevance refinement is further executed through temporal clustering and citation network analysis, enhancing accuracy by exploiting inter-article temporal and citation relationships. This multi-faceted, hierarchical retrieval methodology ensures that selected articles are optimally aligned with the underlying journalistic semantics and visual content.
Context-Aware Caption Generation Pipeline
Following article retrieval, the captioning module synthesizes visual evidence and external knowledge via a novel three-stage pipeline:
- Structured Visual Context Extraction: A Vision-LLM generates a multifaceted description of the image, encompassing objective details, inferred context, mood/atmosphere, and a prospective headline. This phase produces an interpretable, visually-grounded prior (Dvisual) that anchors subsequent synthesis.
- Relevant Context Extraction: The structured visual context is encoded using M3-embedding (text embeddings outperforming multimodal representations), used to query the top-1 retrieved article. The article is split at sentence level, cosine similarity is computed against Dvisual, and the top-3 most relevant sentences (along with neighboring context) are aggregated. This design ensures focused, noise-free evidence selection.
- Knowledge-Grounded Caption Synthesis: An LLM, prompted as an expert caption writer, is tasked to synthesize a comprehensive, factually precise caption using both Dvisual and retrieved article context. The prompt mandates explicit grounding in visual evidence, injecting factual details (identities, locations, dates, event significance) directly from retrieved text, minimizing hallucination and maximizing factual accuracy.
This decomposition guarantees both visual fidelity and knowledge integration, producing captions that transcend generic object-level descriptions and deliver event-aware, contextually enriched narratives.
Empirical Analysis
Ablation studies conducted on the OpenEvents V1 dataset demonstrate clear superiority of the hierarchical, multi-modal retrieval approach. Structural content similarity markedly outperforms CLIP-only baselines; incorporation of visual-visual coherence yields mAP increases from 0.15 to 0.95; the full model with discourse positioning achieves optimal metrics (mAP: 0.97, R@1: 0.956, R@10: 0.991). The three-stage captioning pipeline significantly outperforms image-to-text-only methods, as shown by improved CIDEr and CLIP Scores, confirming its effectiveness at generating visually grounded, factually precise captions.
On the EVENTA Grand Challenge private test set, the system achieved 5th place with an overall score of 0.2824. While a gap to the leading submissions exists, the strong CLIP Score (0.783) validates semantic relevance; CIDEr score (0.081) indicates challenges with n-gram overlap inherent to reference captions in news, especially when integrating external knowledge. Qualitative results confirm the model's ability to generate accurate, event-aware captions that identify individuals, context, and significance—evidence of successful retrieval augmentation.
Implications and Future Directions
The hierarchical retrieval-augmented framework addresses major limitations in knowledge-grounded image captioning, establishing robust mechanisms for integrating document structure, visual placement cues, and contextual article relationships. The implications span improved automation for news agencies, accessibility applications, and enhanced factuality in automated journalism. The demonstrated synergy between multi-modal retrieval and LLM-driven synthesis sets a new paradigm for knowledge-intensive generative tasks.
Further research opportunities include generalization to other document types, integration of additional relevance signals (e.g., entity linking, salience modeling), and optimization for computational efficiency via parallelized generation and advanced embedding strategies. The persistent challenge of aligning generated captions lexically with human references calls for evolution in evaluation metrics, favoring semantic and factual alignment over n-gram based scores.
Conclusion
The paper introduces a systematic, hierarchical multi-modal retrieval-augmented framework for knowledge-grounded news image captioning, effectively combining structured article analysis, visual contextual extraction, and knowledge synthesis. Empirical results on large-scale event datasets validate the efficacy of the approach, demonstrating superior retrieval accuracy and contextually enriched captions. The methodology significantly advances the state-of-the-art for factually grounded image-to-text generation, with direct applicability to automated journalism, accessibility, and information retrieval.
