---
title: Captioning in Multimodal Research
url: https://www.emergentmind.com/topics/caption
type: topic
---

# Captioning in Multimodal Research

to=arxiv_search.search  利盛json  strict=false  content='{"query":"ti:\"FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions\" OR abs:\"FuseCap\"","max_results":5,"sort_by":"relevance"}'
to=arxiv_search.search  大发官网json  strict=false  content='{"query":"captioning scientific figures chart captioning controllable image captioning arXiv","max_results":10,"sort_by":"relevance"}'
Caption denotes a textual description linked to visual, audiovisual, or document content. In current multimodal research, the term no longer refers only to a single sentence describing an image; it spans enriched image description, region- and object-controlled captioning, scientific figure caption generation, dense chart captioning, grounded captions with explicit object and action references, object-centric video captioning, 3D object captioning, and adaptive non-speech captions for deaf and hard of hearing viewers [2305.17718] [2305.02677] [2110.11624] [2508.03164] [2502.13898] [2504.05541] [2306.07279] [2508.19971]. Across these settings, captioning functions as both an output modality and a supervisory signal: captions can be generated from images, fused from multiple experts, summarized from document context, or used as weak labels for robust visual learning [2211.09371] [2210.07396].

## 1. Task families and problem formulations

Recent work treats captioning as a heterogeneous problem family defined by modality, control surface, and fidelity requirement. Image captioning remains the canonical case, but several papers specialize the task according to what must be described and how the description is constrained. "FuseCap" enriches generic image captions with outputs from a frozen object detector, an attribute recognizer, and OCR, then uses a large language model to produce comprehensive image descriptions; it reports a training set of 12M image-enriched caption pairs and uses the resulting data to train a BLIP-based captioning model [2305.17718]. "Caption AnyThing" formulates controllable captioning with both visual controls and language controls, while "CAT-V" extends this logic to spatiotemporal object-centric video description [2305.02677] [2504.05541].

Scientific and technical captioning forms a distinct branch. "SciCap" addresses caption generation for scientific figures extracted from arXiv computer science papers and establishes graph-plot captioning baselines [2110.11624]. "Summaries as Captions" argues that figure caption generation can be more effectively treated as abstractive summarization of figure-referencing paragraphs, noting that nearly 75% of the tokens in an author-written caption already occur in the sentences or paragraphs that mention the figure [2302.12324]. "Figuring out Figures" augments figures with title, abstract, and in-text references through MetaSciCap, while "LaMP-Cap" frames figure caption generation as personalization using multimodal figure profiles drawn from the same document [2407.11008] [2506.06561].

Other variants further specialize captioning by verification or downstream use. "ChartCap" targets dense chart captioning with reduced hallucination and introduces captions that exclude extraneous information not inferable from chart pixels [2508.03164]. "GroundCap" requires captions to be explicitly grounded to detected objects, actions, and background elements through persistent IDs and XML-like tags [2502.13898]. "CapGeo" uses captions as an intermediate representation for geometric reasoning, and "ReCap" uses retrieved articles to generate event-enriched captions for news-like images [2510.09302] [2509.01259].

| Task family | Representative work | Core conditioning signal |
|---|---|---|
| Enriched image captioning | FuseCap | Original caption + object detector + attribute recognizer + OCR |
| Controllable captioning | Caption AnyThing, CAT-V | Points, boxes, trajectories, irregular regions, language prompts |
| Scientific figure captioning | SciCap, Summaries as Captions, MetaSciCap, LaMP-Cap | Figure image, mentions, abstracts, references, profile figures |
| Faithful grounded captioning | ChartCap, GroundCap, CapGeo | Type-specific structure, grounded IDs, keypoints |
| Context-aware captioning | ReCap, NewsCaption, Caption Injection | Retrieved articles, named entities, injected captions |
| 3D and on-image captioning | Cap3D, CapOnImage | Multi-view renders, localized text-box locations |

## 2. Data construction, supervision, and caption enrichment

A central theme in the literature is that caption quality is often limited by the supervision source. FuseCap explicitly traces generic captions to image-text datasets whose captions provide a general description but frequently omit salient details; its response is automated reannotation through vision experts and LLM fusion rather than manual relabeling at scale [2305.17718]. CapEnrich addresses the same genericity problem with an automatic data-building strategy: from multi-caption datasets it selects the shortest caption as a generic sentence $G$, extracts missing relation and attribute triples from the remaining captions, concatenates them into a detail string $D$, and trains prompts on a frozen VLP model to predict those details [2211.09371]. On MSCOCO this produces approximately $3.6\times 10^5$ enrichment pairs, and on Flickr30K approximately $1.0\times 10^5$ [2211.09371].

Scientific figure work shows a different supervision bottleneck. SciCap begins with 1,921,287 PDFs from a Dec 22 2020 arXiv snapshot, filters to 295,028 papers in 2010–2020 cs.* and stat.ML, and extracts 2,170,719 figure-caption pairs with PDFFigures 2.0; after figure-type classification and subfigure removal it obtains 133,543 non-subfigure graph plots for baseline graph-plot captioning [2110.11624]. "Summaries as Captions" then reconstructs context-to-caption pairs by redownloading arXiv PDFs, parsing them with Grobid, identifying mention sentences with regex, and appending OCR output from EasyOCR; after filtering figures without mentions and re-splitting at the paper level it reports 86,825 train, 10,833 validation, and 10,763 test figures [2302.12324]. MetaSciCap similarly attaches title, abstract, and in-text reference windows to each figure, producing tuples of image plus textual metadata for caption prediction [2407.11008].

Several datasets are explicitly built to improve fidelity or controllability. ChartCap assembles a 565K-pair dataset of real-world charts with dense, type-specific captions through a four-stage pipeline beginning from 3.1 million chart images and ending with 509K training examples and a 56K human-verified test set produced through cycle consistency–based human verification [2508.03164]. GroundCap contains 52,016 movie frames from 77 films, 52,016 automatically generated grounded captions, and 344 human-refined captions, with 132 object classes and 51 action classes [2502.13898]. CapGeo-Bench contains 4,641 clean geometry figures with bilingual captions, organized into Plane Geometry, Analytic Geometry, and Solid Geometry [2510.09302]. LaMP-Cap contributes 110,828 target-figure examples, each paired with up to three profile figures from the same paper, enabling caption generation conditioned on author-specific multimodal context [2506.06561].

Caption supervision also appears as a learning paradigm rather than an output format. "Caption supervision enables robust learners" defines caption supervision as using image-linked free-form text as weak supervision, either through VL contrastive losses or by subset matching that converts captions into integer class labels. CaptionNet adds over 50,000 new human-labeled ImageNet-compliant samples with web-scraped captions to support controlled robustness studies [2210.07396]. This suggests that captions serve not only as generated descriptions but as a bridge between unstructured web text and discriminative visual learning.

## 3. Architectural patterns and conditioning mechanisms

The architectural spectrum ranges from classical encoder–decoder models to chained foundation-model systems. SciCap uses a ResNet-101 image encoder, optional LSTM text encoder for in-figure text, and a single-layer LSTM decoder with Luong attention over a $7\times 7$ image feature grid, trained with the standard negative log-likelihood objective
$$
\mathcal{L}(\theta) = -\sum_{t=1}^T \log P_\theta(y_t \mid y_{<t}, x)
$$
and doubly-stochastic regularization on the attention weights [2110.11624]. Face-Cap injects facial expression features into an attention-based image captioner by extracting per-face probabilities over seven universal expressions, collapsing them into a one-hot vector, and using that vector either at every decoding step or only for LSTM initialization [1807.02250].

Transformer and foundation-model designs dominate more recent work. "Caption AnyThing" is explicitly training-free: SAM produces a binary mask from points, boxes, or trajectories; BLIP2 generates a raw caption from the masked region; and ChatGPT refines that caption under language controls such as sentiment, length, language, and factuality [2305.02677]. CAT-V preserves this modular logic but replaces the static-image pipeline with a Segmenter based on SAMURAI, a Temporal Analyzer powered by TRACE-Uni, and an InternVL-2.5 Captioner guided by a chain-of-thought structure over attributes, actions, statuses, interactions, environments, and events [2504.05541]. Cap3D applies a related decomposition to 3D assets: it renders $M=8$ views, samples $N=5$ BLIP2 captions per view, reranks them with CLIP, and uses GPT-4 to consolidate the selected view captions into one concise description [2306.07279].

Multimodal fusion varies substantially by domain. MetaSciCap concatenates CLIP-ViT/B-32 image tokens with SciBERT metadata encodings and decodes with GPT-2 cross-attention [2407.11008]. CapOnImage uses a single Transformer over image patches, neighbor-enhanced location embeddings, auxiliary product text, and autoregressive caption tokens, with joint caption generation and caption matching pre-training [2204.12974]. ReCap retrieves a related article using DINOv2 global embeddings and patch-level mutual nearest neighbor similarity, synthesizes a generic visual caption, web-scraped caption, and article summary, and generates a final event-enriched caption with Qwen3-14B plus Semantic Gaussian Normalization [2509.01259]. GroundCap, by contrast, uses a three-stage Pixtral-12B pipeline in which scene captions, object-specific captions, and detection metadata are fused into a grounded caption containing object, action, and location tags [2502.13898].

A recurrent pattern is the replacement of learned end-to-end fusion with chained expert modules. Caption AnyThing, FuseCap, Cap3D, ReCap, and Caption Injection all rely on frozen or off-the-shelf experts connected by prompting or reranking rather than a newly trained multimodal backbone [2305.02677] [2305.17718] [2306.07279] [2509.01259] [2511.04080]. This suggests a broad methodological shift toward compositional caption systems whose behavior is controlled by data curation, prompt design, or retrieval context.

## 4. Scientific, technical, and document-centered captioning

Scientific captioning differs from generic image captioning because the surrounding document text often contains the necessary semantics. SciCap’s baseline results show both opportunity and difficulty: on graph plots, Vision-Only BLEU-4 reaches .0219 on the First Sentence split, .0207 on the Single-Sentence split, and .0172 on the $\le 100$-word split, with common failures including overly generic statements, wrong or missing axis and legend references, loss of numeric details through normalization, and incoherent clause ordering [2110.11624]. The paper explicitly identifies extremely low BLEU-4 values of .02–.03 as evidence of the task’s difficulty [2110.11624].

"Summaries as Captions" responds by reframing the problem as scientific summarization. Fine-tuning PEGASUS on Paragraph+OCR input yields ROUGE-1 F1 = 0.381, ROUGE-2 = 0.212, ROUGE-L = 0.340, MoverScore = 0.571, and BERTScore = 0.685, outperforming vision-only baselines such as TrOCR and BEiT+GPT2 on the reported evaluation [2302.12324]. Human evaluation ranks the longer-caption "Better" variant statistically tied with original captions and superior to the shorter Pegasus variant on helpfulness-oriented comparisons [2302.12324]. The same study also reports that in a sample of cs.CL line-chart captions, 53.9% were rated unhelpful, making training data quality a first-order issue rather than a secondary nuisance [2302.12324].

MetaSciCap pushes this argument further by attaching title, abstract, and in-text reference windows to the figure. Its most striking result is that SciBERT+GPT-2, using only textual metadata, achieves BLEU 6.71 and ROUGE-L 0.30, outperforming the CLIP+SciBERT+DistilGPT-2 multimodal variant at BLEU 4.92 and ROUGE-L 0.26 and the CLIP+GPT-2 image-only model at BLEU 1.02 and ROUGE-L 0.13 [2407.11008]. The paper offers three explanations: near-regurgitation from in-text references, parameter mismatch between DistilGPT-2 and GPT-2, and visual information loss caused by resizing figures to $224\times 224$ [2407.11008]. A plausible implication is that in document-centric captioning, the central challenge is often retrieval and compression of latent textual context rather than visual scene description alone.

LaMP-Cap introduces personalization into this document setting. Each example includes the target figure image and mentioning paragraphs together with up to three profile figures from the same paper, each with its image, paragraphs, and caption [2506.06561]. Across GPT-4o, Llama-4 Scout, Gemini 2.5 Flash, and GPT-4.1 Mini, adding one profile figure nearly doubles BLEU-4 and ROUGE-2 over the no-profile condition, and using all available profiles yields further gains; for GPT-4o, BLEU-4 rises from 0.063 to 0.103 to 0.115 across No Profile, One Profile, and All Profile [2506.06561]. The ablation shows that removing the profile caption is most damaging, removing the profile image is also harmful, and removing the paragraph causes only a slight drop [2506.06561]. This indicates that authorial style and within-document consistency can be operationalized as multimodal profile conditioning.

## 5. Faithfulness, grounding, and evaluation

A major research tension concerns whether a caption is merely fluent or actually verifiable. ChartCap is organized around the claim that faithful chart descriptions must avoid information that cannot be directly inferred from the pixels and must explicitly cover both structural components and data-driven insights [2508.03164]. Its Visual Consistency Score reconstructs a chart from the caption using generated Matplotlib code and compares the reconstructed image $\hat I_i$ with the original $I_i$ through cosine similarity of SigLIP2 embeddings:
$$
\mathrm{VCS} = \frac{1}{N}\sum_{i=1}^N \mathrm{Sim}(I_i,\hat I_i).
$$
The accompanying OCRScore measures textual fidelity between OCR strings extracted from the original and reconstructed charts [2508.03164]. VCS and OCRScore achieve the highest agreement with human judgments, approximately 79% for informativeness, approximately 77% for accuracy, and approximately 77% for fewer hallucinations, outperforming BLEU, ROUGE, METEOR, and BERTScore on the reported head-to-head study [2508.03164].

GroundCap addresses verifiability by embedding grounding directly in the caption text. Each detected object receives a persistent ID such as `person-0`, and the caption contains XML-like tags: `<gdo>` for objects, `<gda>` for actions linked to object IDs, and `<gdl>` for background regions [2502.13898]. The proposed gMETEOR combines caption quality with grounding accuracy, with the paper using a harmonic mean between METEOR and grounding F1 [2502.13898]. On the 10K-image test split, Pixtral fine-tuned on the automatic captions reaches $P=0.61$, $R=0.95$, $F1=0.70$, METEOR = 0.24, CIDEr = 0.46, and gMETEOR $\approx 0.35$ [2502.13898]. Human evaluation assigns the highest Overall score, 4.34, to human-refined captions, compared with 4.22 for the Pixtral model fine-tuned on human data and 4.07 for the automatic pipeline [2502.13898].

CapGeo demonstrates that caption fidelity can be evaluated by downstream reasoning rather than text overlap alone. CapGeo-Bench extracts keypoints from captions in three dimensions—elements, spatial relations, and numerical relations—and scores recall over matched keypoints [2510.09302]. This metric correlates strongly with downstream CapGeo performance, and caption assistance dramatically improves geometry reasoning: on MathVerse, Qwen2.5-VL-72B-Instruct rises from 8.6% in the vision-only setting to 59.0% with GPT-o3 captions, while Claude-Opus-4 rises from 44.8% to 73.0% [2510.09302]. Even so, the benchmark shows that numerical recall remains substantially weaker than element or relation recall; for GPT-o3, $(S_E,S_R,S_N)=(0.634,0.561,0.260)$ [2510.09302]. This suggests that the hardest captioning failures are often located in fine-grained relational and numerical content rather than in object naming.

The literature also records explicit metric skepticism. GroundCap notes that standard scores correlate poorly with human judgment, with reported $|r|<0.2$ for gMETEOR in its discussion of limitations [2502.13898]. "Summaries as Captions" argues that caption helpfulness depends strongly on "Takeaway" and "Visual-Description," not merely fluency or surface overlap [2302.12324]. Together these results indicate that evaluation is moving from reference matching toward grounded reconstruction, keypoint coverage, and task-conditioned usefulness.

## 6. Control, personalization

Source: https://www.emergentmind.com/topics/caption