Caption Injection in Multimodal Systems

• Caption Injection is a technique that embeds caption-like descriptions derived from images into text to enhance visibility in generative search outputs.
• It employs a three-stage pipeline—structural generation, alignment refinement, and semantic injection—to optimize text while keeping the visual input unchanged.
• The method contrasts early injection with late fusion architectures and has been extended for enhanced caption enrichment, object grounding, and style or metadata control.

Caption injection denotes a family of conditioning strategies in which caption-like textual descriptions, or representations derived from them, are inserted into a downstream generation, retrieval, or captioning pipeline to transfer visual semantics into a language-dominant interface. In the narrowest recent usage, “Caption Injection” is a multimodal generative search engine optimization method that extracts captions from images and injects them into source text to increase a source’s “subjective visibility” in a generative search engine response (Chen et al., 6 Nov 2025). In a broader architectural sense, related work treats injection as a way of introducing visual, regional, object-level, or stylistic signals into caption generators, often contrasting early conditioning with later fusion (Tanti et al., 2017).

1. Formalization in multimodal generative search

In the multimodal generative search engine setting, a user query $q$ is mapped to a retrieved source set

$S=\mathrm{Retrieval}(q)=\{s_1,s_2,\dots,s_N\},$

after which a LLM synthesizes a free-form, citation-rich response

$r=\mathrm{generate}(S,q)=\{\ell_1,\dots,\ell_m\}.$

Each sentence $\ell_k$ may cite one or more sources, and each source $s_i$ may contain text $t_i$ and an image $I_i$. The optimization target is not rank position but “subjective visibility,” because the response is presented as holistic paragraphs rather than as an ordered list. Generative SEO therefore aims to transform a source $s_i$ into an optimized version $s_i'=\mathcal{O}(s_i)$ such that

$$\mathrm{impression}_{s_i'}(r')>\mathrm{impression}_{s_i}(r),$$

where $S=\mathrm{Retrieval}(q)=\{s_1,s_2,\dots,s_N\},$0 and $S=\mathrm{Retrieval}(q)=\{s_1,s_2,\dots,s_N\},$1 measures how prominently source $S=\mathrm{Retrieval}(q)=\{s_1,s_2,\dots,s_N\},$2 appears in the generated response (Chen et al., 6 Nov 2025).

Caption Injection instantiates this objective by keeping the image unchanged and rewriting only the textual side of the source. For each source $S=\mathrm{Retrieval}(q)=\{s_1,s_2,\dots,s_N\},$3, the optimized source is

$S=\mathrm{Retrieval}(q)=\{s_1,s_2,\dots,s_N\},$4

with

$S=\mathrm{Retrieval}(q)=\{s_1,s_2,\dots,s_N\},$5

No explicit parameter-learned loss is introduced for this G-SEO formulation; the operative objective is to maximize post-optimization subjective visibility through prompt-based calls to off-the-shelf VLMs and LLMs (Chen et al., 6 Nov 2025).

2. Three-stage pipeline, benchmark, and evaluation

The method is organized as a three-stage pipeline. In Structural Generation, a VLM such as BLIP-2 receives the image $S=\mathrm{Retrieval}(q)=\{s_1,s_2,\dots,s_N\},$6 with the prompt “Generate a concise and objective caption describing the main objects, actions, and scene,” producing a structural caption framed as an $S=\mathrm{Retrieval}(q)=\{s_1,s_2,\dots,s_N\},$7 abstraction. In Alignment Refinement, an LLM rewrites that structural caption using the original text $S=\mathrm{Retrieval}(q)=\{s_1,s_2,\dots,s_N\},$8, with instructions to “rewrite the caption to be more expressive and attention-grabbing, retaining core subject, action, scene, and enriching only with the most relevant information from $S=\mathrm{Retrieval}(q)=\{s_1,s_2,\dots,s_N\},$9.” In Semantic Injection, the LLM receives $r=\mathrm{generate}(S,q)=\{\ell_1,\dots,\ell_m\}.$0 and the refined caption and is instructed to “Insert the refined caption at the single best position in $r=\mathrm{generate}(S,q)=\{\ell_1,\dots,\ell_m\}.$1 so that context remains coherent. Do not otherwise alter.” The output is the augmented text $r=\mathrm{generate}(S,q)=\{\ell_1,\dots,\ell_m\}.$2, which is then rerun through the RAG+LLM pipeline (Chen et al., 6 Nov 2025).

Subjective visibility is measured by G-Eval 2.0, an LLM-based protocol that rates seven subdimensions: Relevance, Fluency, Diversity, Uniqueness, Click-follow likelihood, Positional salience, and Content volume. Each dimension is scored from 0 to 5, and the overall visibility score is the average of the seven. Relative improvement is defined as

$r=\mathrm{generate}(S,q)=\{\ell_1,\dots,\ell_m\}.$3

All scores are averaged over three independent runs to reduce randomness (Chen et al., 6 Nov 2025).

The benchmark is MRAMG, containing 4 800 query-content pairs across six subsets: Wit (600), Wiki (600), Web (650), Arxiv (200), Recipe (1 500), and Manual (1 450). Difficulty levels are designated Easy for Wit, Wiki, and Web; Medium for Arxiv; and Hard for Recipe and Manual. In the unimodal setting, only the text $r=\mathrm{generate}(S,q)=\{\ell_1,\dots,\ell_m\}.$4 of each source is fed into GLM-4-9B with a standard prompt. In the multimodal setting, the engine receives $r=\mathrm{generate}(S,q)=\{\ell_1,\dots,\ell_m\}.$5, where $r=\mathrm{generate}(S,q)=\{\ell_1,\dots,\ell_m\}.$6 is the original alt-text, a VLM-generated caption, or the refined caption; no direct image encoder is used. Missing captions are filled by Qwen-2.5-VL-7B. The text-only baselines are Traditional SEO, Fluency Expression Optimization, Quotation-based Addition, and Statistic-based Addition, all reproduced under the same GLM-4-9B backbone (Chen et al., 6 Nov 2025).

Quantitatively, Caption Injection (“capt_addi”) reports relative improvement values of 1.09 on Arxiv, $r=\mathrm{generate}(S,q)=\{\ell_1,\dots,\ell_m\}.$7 on Manual, 1.85 on Recipe, 0.18 on Web, 1.10 on Wiki, and 0.68 on Wit, for an average of 0.47. The reported observations state that Caption Injection achieves the highest gains on five of six subsets and the best overall average; all methods degrade on the hardest Manual subset, but Caption Injection still outperforms; and traditional SEO yields almost zero or negative gains, which the paper interprets as evidence that text-only keyword tricks fail under generative search. An ablation on Arxiv compares original captions, VLM-generated captions, and refined captions, with average scores of 1.18, 1.06, and 1.11 respectively, and the reported interpretation is that refined captions strike the best balance across dimensions, confirming the value of the alignment refinement stage (Chen et al., 6 Nov 2025).

3. Architectural antecedents: injection versus late fusion

A foundational antecedent is the systematic comparison by Tanti et al. between architectures that inject image information into an RNN LLM and architectures that merge image information later in the prediction stack. The paper distinguishes three inject variants—init-inject, pre-inject, and par-inject—from a merge architecture. In init-inject, the initial hidden state is set from image features; in pre-inject, the image is treated as word 0; in par-inject, an image projection is concatenated to each token embedding at every step. In merge, the RNN encodes only the linguistic prefix, while the image projection is concatenated with the hidden state at the output layer before softmax prediction (Tanti et al., 2017).

All experiments in that comparison use fixed VGG-19 fc7 image features $r=\mathrm{generate}(S,q)=\{\ell_1,\dots,\ell_m\}.$8, a learned linear projection $r=\mathrm{generate}(S,q)=\{\ell_1,\dots,\ell_m\}.$9, trained word embeddings, and a GRU. Hyperparameters are tuned on Flickr8K and then fixed for Flickr8K, Flickr30K, and MSCOCO. The optimal layer sizes differ sharply by architecture: 512 for init-inject and pre-inject, 256 for par-inject, and 128 for merge. Because merge only needs layer size 128, its total parameter count is roughly 3–4$\ell_k$0 smaller than init- or pre-inject, and merge trains about 2–3$\ell_k$1 faster than the inject variants (Tanti et al., 2017).

Empirically, the paper reports that it is “not especially detrimental to performance whether one architecture is used or another.” On MSCOCO, for example, CIDEr is 0.818 for init-inject, 0.807 for pre-inject, 0.791 for merge, and 0.774 for par-inject; BLEU-4 is 0.271, 0.267, 0.262, and 0.265 respectively. Diversity metrics favor merge: on MSCOCO it uses 7.91% of the vocabulary versus 7.26% for init-inject, 6.59% for pre-inject, and 5.00% for par-inject, and it reproduces training captions less frequently than the inject variants except on Flickr8K. A visual-information-retention analysis shows that merge retains the largest fixed difference between the multimodal vectors for a correct image and a random image, whereas init-inject and pre-inject exhibit stronger decay of image dependence over time (Tanti et al., 2017).

This comparison is significant because it frames a recurring design question that later caption-injection systems revisit in different forms: whether visual information should be bound into the generative state early, or delivered later as a modular conditioning signal. A common misconception is that tighter joint encoding is necessarily superior. The evidence summarized by Tanti et al. instead supports late fusion as an efficient, modular, and often sufficient alternative (Tanti et al., 2017).

4. Caption enrichment and regional caption injection

Two later strands extend injection from whole-image conditioning to caption enrichment and region-level semantics.

System	Injected element	Primary target
FuseCap	vision-expert outputs fused with original captions	enriched image descriptions
Segment and Caption Anything	caption query tokens and task tokens on top of SAM	regional captioning

FuseCap addresses a data-centric bottleneck in image captioning: image-text datasets often provide only general descriptions and omit salient details. The method uses “frozen” vision experts—an object detector, an attribute recognizer, and an OCR system—and fuses their outputs with the original captions using a LLM. The result is a large-scale corpus of 12M image-enriched caption pairs, which is then used to train a BLIP-based captioning model. The abstract reports that this model outperforms current state-of-the-art approaches and produces more precise and detailed descriptions, and that the enriched dataset is released to the community (Rotstein et al., 2023).

“Segment and Caption Anything” injects captioning ability into the Segment Anything Model by preserving SAM’s ViT-based image encoder, prompt encoders, two-layer query-based feature mixer, and mask decoder as frozen components, and then adding $\ell_k$2 learnable caption query tokens, a deeper feature mixer $\ell_k$3 with $\ell_k$4 transformer blocks, a frozen causal LLM such as GPT2-large or LLAMA-3B, and task tokens of length 6. Only about 19M parameters in the new mixer and tokens are trainable, while the SAM encoder, prompt encoders, and LM remain frozen. Training proceeds in two stages: weak-supervision pretraining for 100K steps on Objects365 detection and COCO-Panoptic segmentation using single-word category names, followed by 100K steps of region-caption finetuning on Visual Genome dense captions. On the VG test set with ground-truth boxes, the method reports CIDEr-D 149.8, METEOR 17.5, SPICE 31.4, and BLEU@4 12.2, outperforming training-free SAM+BLIP/GIT pipelines with less than 70 CIDEr and exceeding GRiT at 142.2 CIDEr. The ablations indicate that weak pretraining improves CIDEr from 127.9 to 134.5 on Objects365 pretraining, and that a 12-layer mixer outperforms 2-layer and 24-layer alternatives (Huang et al., 2023).

These two systems represent different meanings of caption injection. FuseCap enriches the training text itself by inserting details from visual experts, whereas Segment and Caption Anything injects captioning capacity into a segmentation foundation model by adding learned query tokens and a lightweight mixer. This suggests that caption injection can target either the dataset layer or the model-conditioning layer, depending on where the semantic bottleneck is located.

5. Object-, style-, and metadata-level injection

A separate line of work uses caption injection to control style, metadata realization, or object grounding rather than only content coverage.

“Social Media Ready Caption Generation for Brands” factorizes generation as

$\ell_k$5

where $\ell_k$6 is one of {Sincerity, Excitement, Competence, Sophistication, Ruggedness}, $\ell_k$7 is a set of hashtags, $\ell_k$8 a set of social-media handles, and $\ell_k$9 a set of named entities, optionally with URLs. Stage 1 uses BLIP-2 with frozen CLIP ViT-B/32 and FlanT5-XXL to produce a neutral caption $s_i$0. Stage 2 injects brand personality and arbitrary user-specified content using either an instruction-tuned FlanT5-XL or zero-/few-shot GPT-3.5-turbo prompts. The fine-tuned FlanT5-XL is trained for 80K steps with cross-entropy loss, batch size 4, and learning rate $s_i$1. On the selective variants, the paper reports for fine-tuned FlanT5 a CLIPScore of 0.830, G-Eval accuracy 42.34%, cosine similarity 0.591, and coverage of 88.7% for hashtags, 74.5% for entities, 91.4% for usernames, and 81.8% for URLs (Maheshwari et al., 2024).

“OPCap: Object-aware Prompting Captioning” injects object-level prompts into a Transformer captioner to mitigate hallucination. A pre-trained or fine-tuned object detector outputs bounding boxes $s_i$2 and class labels $s_i$3, which are embedded as

$s_i$4

An attribute predictor produces

$s_i$5

after which prompt tokens are constructed via concatenation and gating,

$s_i$6

These prompts are fused with image features through cross-attention and consumed by the text decoder. On COCO val, the baseline ViT captioner reports BLEU-4 22.64, CIDEr 63.85, and SPICE 16.43, while ViT+OPCap reaches BLEU-4 24.87, CIDEr 71.26, and SPICE 17.82. On 3 000 COCO “no-people” test images, the baseline mentions “person” in 4.23% of captions, versus 2.80% for ViT+OPCap (Huang, 2024).

Taken together, these systems broaden the meaning of caption injection beyond simple insertion of descriptive text. In one case the injected content is a stylistic and metadata constraint routed through an intermediate caption; in the other it is a structured object prompt intended to improve grounding. A plausible implication is that “caption injection” is less a single algorithm than a general interface pattern for converting visual or control signals into forms a LLM can reliably exploit.

6. Limitations, misconceptions, and open directions

The literature identifies several recurring limitations. In generative search, the absolute gains of Caption Injection in multimodal settings remain modest compared to unimodal settings, the current fusion is described as “shallow” because it is textual injection rather than tight joint encoding of vision and language, and the internal preference or black-box biases of the GSE are not explicitly modeled. The paper therefore proposes deeper cross-modal feature fusion, cross-model adaptation to interpret latent visibility bias, and long-text optimization for dense sources such as manuals (Chen et al., 6 Nov 2025).

For regional captioning, Segment and Caption Anything notes incorrect attributes, confusions among visually similar objects such as lemon versus orange, and the absence of explicit alignment between masks and region tokens, which can produce mismatches between mask and caption. The proposed directions are to scale weakly supervised data such as OpenImages and image-caption corpora, to use self-training or distillation with synthetic captions, and to explore chat-style or instruction-tuning for interactive multi-round region dialogue (Huang et al., 2023).

For brand-conditioned caption generation, the main constraints are noisy labels, high evaluation cost when using GPT for G-Eval, sensitivity to prompt-slot coverage, and the quality of Instagram-scraped training data. The paper reports that human agreement on brand captions is approximately 32%, which it uses to contextualize a maximum fine-tuned accuracy of approximately 43% (Maheshwari et al., 2024).

A further misconception is that injection should always be understood as early fusion inside the decoder’s recurrent or autoregressive state. The inject-versus-merge comparison indicates that late multimodal integration can preserve performance while reducing model size and training cost substantially, and several later systems in fact rely on frozen backbones plus lightweight adapters, query tokens, or prompt-level textual insertion rather than full joint retraining (Tanti et al., 2017). This suggests that the central research question is not whether to inject, but what to inject—captions, object prompts, regional tokens, or style constraints—and where in the stack the injected signal is most effective.