Captioning in Multimodal Research

• Caption is a textual description linked to visual content, now encompassing enriched, region- and object-controlled outputs and scientific figure narrations.
• Enriched captioning leverages expert modules like object detection, OCR, and large language models to fuse diverse data into comprehensive and faithful descriptions.
• Recent approaches focus on retrieval, prompt design, and multimodal profile conditioning to improve accuracy and contextual relevance in document-centric captioning.

to=arxiv_search.search 利盛json strict=false content='{"^^^^^^^^3query3 Leveraging LLMs for Enriched Fused Image Captions\"^^^^^^^^3 OR abs:\3^^^^^^^^"FuseCap\"","max_results":5,"sort_by":"relevance"}' to=arxiv_search.search 大发官网json strict=false content='{"^^^^^^^^3query3 scientific figures chart captioning controllable image captioning arXiv","max_results":^^^^3ti:\3query3^^^^,"sort_by":"relevance"}' Caption denotes a textual description linked to visual, audiovisual, or document content. In current multimodal research, the term no longer refers only to a single sentence describing an image; it spans enriched image description, region- and object-controlled captioning, scientific figure caption generation, dense chart captioning, grounded captions with explicit object and action references, object-centric video captioning, 3D object captioning, and adaptive non-speech captions for deaf and hard of hearing viewers (&&&^^^^3query3^^^^&&&, &&&^^^^3ti:\3^^^^&&&, &&&^^^^3 OR abs:\3^^^^&&&, Lim et al., 5 Aug 2025, Oliveira et al., 19 Feb 2025, Tang et al., 7 Apr 2025, Luo et al., 2023, Huang et al., 27 Aug 2025). Across these settings, captioning functions as both an output modality and a supervisory signal: captions can be generated from images, fused from multiple experts, summarized from document context, or used as weak labels for robust visual learning (Yao et al., 2022, Feuer et al., 2022).

^^^^3ti:\3^^^^. Task families and problem formulations

Recent work treats captioning as a heterogeneous problem family defined by modality, control surface, and fidelity requirement. Image captioning remains the canonical case, but several papers specialize the task according to what must be described and how the description is constrained. "FuseCap" enriches generic image captions with outputs from a frozen object detector, an attribute recognizer, and OCR, then uses a LLM to produce comprehensive image descriptions; it reports a training set of ^^^^3ti:\3 OR abs:\3^^^^M image-enriched caption pairs and uses the resulting data to train a BLIP-based captioning model (&&&^^^^3query3^^^^&&&). "Caption AnyThing" formulates controllable captioning with both visual controls and language controls, while "CAT-V" extends this logic to spatiotemporal object-centric video description (&&&^^^^3ti:\3^^^^&&&, Tang et al., 7 Apr 2025).

Scientific and technical captioning forms a distinct branch. "SciCap" addresses caption generation for scientific figures extracted from arXiv computer science papers and establishes graph-plot captioning baselines (&&&^^^^3 OR abs:\3^^^^&&&). "Summaries as Captions" argues that figure caption generation can be more effectively treated as abstractive summarization of figure-referencing paragraphs, noting that nearly 75% of the tokens in an author-written caption already occur in the sentences or paragraphs that mention the figure (&&&^^^^3ti:\3^^^^4&&&). "Figuring out Figures" augments figures with title, abstract, and in-text references through MetaSciCap, while "LaMP-Cap" frames figure caption generation as personalization using multimodal figure profiles drawn from the same document (&&&^^^^3ti:\3^^^^5&&&, &&&^^^^3ti:\3^^^^6&&&).

Other variants further specialize captioning by verification or downstream use. "ChartCap" targets dense chart captioning with reduced hallucination and introduces captions that exclude extraneous information not inferable from chart pixels (Lim et al., 5 Aug 2025). "GroundCap" requires captions to be explicitly grounded to detected objects, actions, and background elements through persistent IDs and XML-like tags (Oliveira et al., 19 Feb 2025). "CapGeo" uses captions as an intermediate representation for geometric reasoning, and "ReCap" uses retrieved articles to generate event-enriched captions for news-like images (&&&^^^^3ti:\3^^^^9&&&, &&&^^^^3 OR abs:\3query3^^^^&&&).

Task family	Representative work	Core conditioning signal
Enriched image captioning	FuseCap	Original caption + object detector + attribute recognizer + OCR
Controllable captioning	Caption AnyThing, CAT-V	Points, boxes, trajectories, irregular regions, language prompts
Scientific figure captioning	SciCap, Summaries as Captions, MetaSciCap, LaMP-Cap	Figure image, mentions, abstracts, references, profile figures
Faithful grounded captioning	ChartCap, GroundCap, CapGeo	Type-specific structure, grounded IDs, keypoints
Context-aware captioning	ReCap, NewsCaption, Caption Injection	Retrieved articles, named entities, injected captions
3D and on-image captioning	Cap3D, CapOnImage	Multi-view renders, localized text-box locations

^^^^3 OR abs:\3^^^^. Data construction, supervision, and caption enrichment

A central theme in the literature is that caption quality is often limited by the supervision source. FuseCap explicitly traces generic captions to image-text datasets whose captions provide a general description but frequently omit salient details; its response is automated reannotation through vision experts and LLM fusion rather than manual relabeling at scale (&&&^^^^3query3^^^^&&&). CapEnrich addresses the same genericity problem with an automatic data-building strategy: from multi-caption datasets it selects the shortest caption as a generic sentence PRESERVED_PLACEHOLDER_^^^^3query3^^^^, extracts missing relation and attribute triples from the remaining captions, concatenates them into a detail string PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^, and trains prompts on a frozen VLP model to predict those details (Yao et al., 2022). On MSCOCO this produces approximately PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3^^^^ enrichment pairs, and on Flickr3^^^^3query3^^^^K approximately $1.0\times 10^5$ (Yao et al., 2022).

Scientific figure work shows a different supervision bottleneck. SciCap begins with ^^^^3ti:\3^^^^,9^^^^3 OR abs:\3ti:\3^^^^,^^^^3 OR abs:\3^^^^87 PDFs from a Dec ^^^^3 OR abs:\3 OR abs:\3^^^^ ^^^^3 OR abs:\3query3 OR abs:\3query3^^^^ arXiv snapshot, filters to ^^^^3 OR abs:\3^^^^95,^^^^3query3 OR abs:\3^^^^8 papers in ^^^^3 OR abs:\3query3ti:\3query3^^^^–^^^^3 OR abs:\3query3 OR abs:\3query3^^^^ cs.* and stat.ML, and extracts ^^^^3 OR abs:\3^^^^,^^^^3ti:\3query3query3^^^^,7 figure-caption pairs with PDFFigures ^^^^3 OR abs:\3^^^^.^^^^3query3^^^^; after figure-type classification and subfigure removal it obtains ^^^^3ti:\333^^^^,543 non-subfigure graph plots for baseline graph-plot captioning (&&&^^^^3 OR abs:\3^^^^&&&). "Summaries as Captions" then reconstructs context-to-caption pairs by redownloading arXiv PDFs, parsing them with Grobid, identifying mention sentences with regex, and appending OCR output from EasyOCR; after filtering figures without mentions and re-splitting at the paper level it reports 86,8^^^^3 OR abs:\3^^^^5 train, ^^^^3ti:\3query3^^^^,833 validation, and ^^^^3ti:\3query3^^^^,763 test figures (&&&^^^^3ti:\3^^^^4&&&). MetaSciCap similarly attaches title, abstract, and in-text reference windows to each figure, producing tuples of image plus textual metadata for caption prediction (&&&^^^^3ti:\3^^^^5&&&).

Several datasets are explicitly built to improve fidelity or controllability. ChartCap assembles a 565K-pair dataset of real-world charts with dense, type-specific captions through a four-stage pipeline beginning from 3.^^^^3ti:\3^^^^ million chart images and ending with 5^^^^3query3^^^^9K training examples and a 56K human-verified test set produced through cycle consistency–based human verification (Lim et al., 5 Aug 2025). GroundCap contains 5^^^^3 OR abs:\3^^^^,^^^^3query3ti:\3^^^^6 movie frames from 77 films, 5^^^^3 OR abs:\3^^^^,^^^^3query3ti:\3^^^^6 automatically generated grounded captions, and 344 human-refined captions, with ^^^^3ti:\33 OR abs:\3^^^^ object classes and 5^^^^3ti:\3^^^^ action classes (Oliveira et al., 19 Feb 2025). CapGeo-Bench contains 4,64^^^^3ti:\3^^^^ clean geometry figures with bilingual captions, organized into Plane Geometry, Analytic Geometry, and Solid Geometry (&&&^^^^3ti:\3^^^^9&&&). LaMP-Cap contributes ^^^^3ti:\3ti:\3query3^^^^,8^^^^3 OR abs:\3^^^^8 target-figure examples, each paired with up to three profile figures from the same paper, enabling caption generation conditioned on author-specific multimodal context (&&&^^^^3ti:\3^^^^6&&&).

Caption supervision also appears as a learning paradigm rather than an output format. "Caption supervision enables robust learners" defines caption supervision as using image-linked free-form text as weak supervision, either through VL contrastive losses or by subset matching that converts captions into integer class labels. CaptionNet adds over 5^^^^3query3^^^^,^^^^3query3query3query3^^^^ new human-labeled ImageNet-compliant samples with web-scraped captions to support controlled robustness studies (Feuer et al., 2022). This suggests that captions serve not only as generated descriptions but as a bridge between unstructured web text and discriminative visual learning.

3. Architectural patterns and conditioning mechanisms

The architectural spectrum ranges from classical encoder–decoder models to chained foundation-model systems. SciCap uses a ResNet-^^^^3ti:\3query3ti:\3^^^^ image encoder, optional LSTM text encoder for in-figure text, and a single-layer LSTM decoder with Luong attention over a $7\times 7$ image feature grid, trained with the standard negative log-likelihood objective

$$\mathcal{L}(\theta) = -\sum_{t=1}^T \log P_\theta(y_t \mid y_{<t}, x)$$

and doubly-stochastic regularization on the attention weights (&&&^^^^3 OR abs:\3^^^^&&&). Face-Cap injects facial expression features into an attention-based image captioner by extracting per-face probabilities over seven universal expressions, collapsing them into a one-hot vector, and using that vector either at every decoding step or only for LSTM initialization (Nezami et al., 2018).

Transformer and foundation-model designs dominate more recent work. "Caption AnyThing" is explicitly training-free: SAM produces a binary mask from points, boxes, or trajectories; BLIP^^^^3 OR abs:\3^^^^ generates a raw caption from the masked region; and ChatGPT refines that caption under language controls such as sentiment, length, language, and factuality (&&&^^^^3ti:\3^^^^&&&). CAT-V preserves this modular logic but replaces the static-image pipeline with a Segmenter based on SAMURAI, a Temporal Analyzer powered by TRACE-Uni, and an InternVL-^^^^3 OR abs:\3^^^^.5 Captioner guided by a chain-of-thought structure over attributes, actions, statuses, interactions, environments, and events (Tang et al., 7 Apr 2025). Cap3D applies a related decomposition to 3D assets: it renders $M=8$ views, samples $N=5$ BLIP^^^^3 OR abs:\3^^^^ captions per view, reranks them with CLIP, and uses GPT-4 to consolidate the selected view captions into one concise description (Luo et al., 2023).

Multimodal fusion varies substantially by domain. MetaSciCap concatenates CLIP-ViT/B-3^^^^3 OR abs:\3^^^^ image tokens with SciBERT metadata encodings and decodes with GPT-^^^^3 OR abs:\3^^^^ cross-attention (&&&^^^^3ti:\3^^^^5&&&). CapOnImage uses a single Transformer over image patches, neighbor-enhanced location embeddings, auxiliary product text, and autoregressive caption tokens, with joint caption generation and caption matching pre-training (Gao et al., 2022). ReCap retrieves a related article using DINOv^^^^3 OR abs:\3^^^^ global embeddings and patch-level mutual nearest neighbor similarity, synthesizes a generic visual caption, web-scraped caption, and article summary, and generates a final event-enriched caption with Qwen3-^^^^3ti:\3^^^^4B plus Semantic Gaussian Normalization (&&&^^^^3 OR abs:\3query3^^^^&&&). GroundCap, by contrast, uses a three-stage Pixtral-^^^^3ti:\3 OR abs:\3^^^^B pipeline in which scene captions, object-specific captions, and detection metadata are fused into a grounded caption containing object, action, and location tags (Oliveira et al., 19 Feb 2025).

A recurrent pattern is the replacement of learned end-to-end fusion with chained expert modules. Caption AnyThing, FuseCap, Cap3D, ReCap, and Caption Injection all rely on frozen or off-the-shelf experts connected by prompting or reranking rather than a newly trained multimodal backbone (&&&^^^^3ti:\3^^^^&&&, &&&^^^^3query3^^^^&&&, Luo et al., 2023, &&&^^^^3 OR abs:\3query3^^^^&&&, Chen et al., 6 Nov 2025). This suggests a broad methodological shift toward compositional caption systems whose behavior is controlled by data curation, prompt design, or retrieval context.

4. Scientific, technical, and document-centered captioning

Scientific captioning differs from generic image captioning because the surrounding document text often contains the necessary semantics. SciCap’s baseline results show both opportunity and difficulty: on graph plots, Vision-Only BLEU-4 reaches .^^^^3query3 OR abs:\3ti:\3^^^^9 on the First Sentence split, .^^^^3query3 OR abs:\3query3^^^^7 on the Single-Sentence split, and .^^^^3query3ti:\3query3 OR abs:\3^^^^ on the $\le 100$-word split, with common failures including overly generic statements, wrong or missing axis and legend references, loss of numeric details through normalization, and incoherent clause ordering (&&&^^^^3 OR abs:\3^^^^&&&). The paper explicitly identifies extremely low BLEU-4 values of .^^^^3query3 OR abs:\3^^^^–.^^^^3query33^^^^ as evidence of the task’s difficulty (&&&^^^^3 OR abs:\3^^^^&&&).

"Summaries as Captions" responds by reframing the problem as scientific summarization. Fine-tuning PEGASUS on Paragraph+OCR input yields ROUGE-^^^^3ti:\3^^^^ F^^^^3ti:\3^^^^ = ^^^^3query3^^^^.38^^^^3ti:\3 ROUGE-^^^^3 OR abs:\3^^^^ = ^^^^3query3^^^^.^^^^3 OR abs:\3ti:\3 OR abs:\3^^^^, ROUGE-L = ^^^^3query3^^^^.34^^^^3query3 MoverScore = ^^^^3query3^^^^.57^^^^3ti:\3 and BERTScore = ^^^^3query3^^^^.685, outperforming vision-only baselines such as TrOCR and BEiT+GPT^^^^3 OR abs:\3^^^^ on the reported evaluation (&&&^^^^3ti:\3^^^^4&&&). Human evaluation ranks the longer-caption "Better" variant statistically tied with original captions and superior to the shorter Pegasus variant on helpfulness-oriented comparisons (&&&^^^^3ti:\3^^^^4&&&). The same study also reports that in a sample of cs.CL line-chart captions, 53.9% were rated unhelpful, making training data quality a first-order issue rather than a secondary nuisance (&&&^^^^3ti:\3^^^^4&&&).

MetaSciCap pushes this argument further by attaching title, abstract, and in-text reference windows to the figure. Its most striking result is that SciBERT+GPT-^^^^3 OR abs:\3^^^^, using only textual metadata, achieves BLEU 6.7^^^^3ti:\3^^^^ and ROUGE-L ^^^^3query3^^^^.3^^^^3query3 outperforming the CLIP+SciBERT+DistilGPT-^^^^3 OR abs:\3^^^^ multimodal variant at BLEU 4.9^^^^3 OR abs:\3^^^^ and ROUGE-L ^^^^3query3^^^^.^^^^3 OR abs:\3^^^^6 and the CLIP+GPT-^^^^3 OR abs:\3^^^^ image-only model at BLEU ^^^^3ti:\3^^^^.^^^^3query3 OR abs:\3^^^^ and ROUGE-L ^^^^3query3^^^^.^^^^3ti:\33^^^^ (&&&^^^^3ti:\3^^^^5&&&). The paper offers three explanations: near-regurgitation from in-text references, parameter mismatch between DistilGPT-^^^^3 OR abs:\3^^^^ and GPT-^^^^3 OR abs:\3^^^^, and visual information loss caused by resizing figures to $224\times 224$ (&&&^^^^3ti:\3^^^^5&&&). A plausible implication is that in document-centric captioning, the central challenge is often retrieval and compression of latent textual context rather than visual scene description alone.

LaMP-Cap introduces personalization into this document setting. Each example includes the target figure image and mentioning paragraphs together with up to three profile figures from the same paper, each with its image, paragraphs, and caption (&&&^^^^3ti:\3^^^^6&&&). Across GPT-4o, Llama-4 Scout, Gemini ^^^^3 OR abs:\3^^^^.5 Flash, and GPT-4.^^^^3ti:\3^^^^ Mini, adding one profile figure nearly doubles BLEU-4 and ROUGE-^^^^3 OR abs:\3^^^^ over the no-profile condition, and using all available profiles yields further gains; for GPT-4o, BLEU-4 rises from ^^^^3query3^^^^.^^^^3query3 to ^^^^3query3^^^^.^^^^3ti:\3query33^^^^ to ^^^^3query3^^^^.^^^^3ti:\3ti:\3 across No Profile, One Profile, and All Profile (&&&^^^^3ti:\3^^^^6&&&). The ablation shows that removing the profile caption is most damaging, removing the profile image is also harmful, and removing the paragraph causes only a slight drop (&&&^^^^3ti:\3^^^^6&&&). This indicates that authorial style and within-document consistency can be operationalized as multimodal profile conditioning.

5. Faithfulness, grounding, and evaluation

A major research tension concerns whether a caption is merely fluent or actually verifiable. ChartCap is organized around the claim that faithful chart descriptions must avoid information that cannot be directly inferred from the pixels and must explicitly cover both structural components and data-driven insights (Lim et al., 5 Aug 2025). Its Visual Consistency Score reconstructs a chart from the caption using generated Matplotlib code and compares the reconstructed image PRESERVED_PLACEHOLDER_^^^^3ti:\3query3^^^^ with the original PRESERVED_PLACEHOLDER_^^^^3ti:\3ti:\3^^^^ through cosine similarity of SigLIP^^^^3 OR abs:\3^^^^ embeddings:

PRESERVED_PLACEHOLDER_^^^^3ti:\3 OR abs:\3^^^^

The accompanying OCRScore measures textual fidelity between OCR strings extracted from the original and reconstructed charts (Lim et al., 5 Aug 2025). VCS and OCRScore achieve the highest agreement with human judgments, approximately 79% for informativeness, approximately 77% for accuracy, and approximately 77% for fewer hallucinations, outperforming BLEU, ROUGE, METEOR, and BERTScore on the reported head-to-head study (Lim et al., 5 Aug 2025).

GroundCap addresses verifiability by embedding grounding directly in the caption text. Each detected object receives a persistent ID such as person-^^^^3query3^^^^, and the caption contains XML-like tags: <gdo> for objects, <gda> for actions linked to object IDs, and <gdl> for background regions (Oliveira et al., 19 Feb 2025). The proposed gMETEOR combines caption quality with grounding accuracy, with the paper using a harmonic mean between METEOR and grounding F^^^^3ti:\3^^^^ (Oliveira et al., 19 Feb 2025). On the ^^^^3ti:\3query3^^^^K-image test split, Pixtral fine-tuned on the automatic captions reaches PRESERVED_PLACEHOLDER_^^^^3ti:\33^^^^, PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^4, PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^5, METEOR = ^^^^3query3^^^^.^^^^3 OR abs:\3^^^^4, CIDEr = ^^^^3query3^^^^.46, and gMETEOR PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^6 (Oliveira et al., 19 Feb 2025). Human evaluation assigns the highest Overall score, 4.34, to human-refined captions, compared with 4.^^^^3 OR abs:\3 OR abs:\3^^^^ for the Pixtral model fine-tuned on human data and 4.^^^^3query3^^^^7 for the automatic pipeline (Oliveira et al., 19 Feb 2025).

CapGeo demonstrates that caption fidelity can be evaluated by downstream reasoning rather than text overlap alone. CapGeo-Bench extracts keypoints from captions in three dimensions—elements, spatial relations, and numerical relations—and scores recall over matched keypoints (&&&^^^^3ti:\3^^^^9&&&). This metric correlates strongly with downstream CapGeo performance, and caption assistance dramatically improves geometry reasoning: on MathVerse, Qwen^^^^3 OR abs:\3^^^^.5-VL-7^^^^3 OR abs:\3^^^^B-Instruct rises from 8.6% in the vision-only setting to 59.^^^^3query3^^^^% with GPT-o3 captions, while Claude-Opus-4 rises from 44.8% to 73.^^^^3query3^^^^% (&&&^^^^3ti:\3^^^^9&&&). Even so, the benchmark shows that numerical recall remains substantially weaker than element or relation recall; for GPT-o3, PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^7 (&&&^^^^3ti:\3^^^^9&&&). This suggests that the hardest captioning failures are often located in fine-grained relational and numerical content rather than in object naming.

The literature also records explicit metric skepticism. GroundCap notes that standard scores correlate poorly with human judgment, with reported PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^8 for gMETEOR in its discussion of limitations (Oliveira et al., 19 Feb 2025). "Summaries as Captions" argues that caption helpfulness depends strongly on "Takeaway" and "Visual-Description," not merely fluency or surface overlap (&&&^^^^3ti:\3^^^^4&&&). Together these results indicate that evaluation is moving from reference matching toward grounded reconstruction, keypoint coverage, and task-conditioned usefulness.

6. Control, personalization