Semantic-Based Adaptive Caption Retrieval

• Semantic-based adaptive caption retrieval is a method that dynamically selects, generates, or refines captions based on the semantic structure of visual queries to overcome static matching limitations.
• It employs multi-stage processes such as coarse-to-fine similarity search, adaptive segmentation, and conditional re-embedding to address challenges like viewpoint shifts and mixed sub-events.
• Empirical results demonstrate that tailored retrieval strategies significantly boost performance metrics like CIDEr and recall, underlining their practical impact.

Across the systems considered here, semantic-based adaptive caption retrieval refers to retrieval augmentation in which captions, rewritten descriptions, or other textual surrogates are selected, generated, or refined according to the semantic structure of the current visual query rather than treated as static auxiliary text. The retrieved unit may be a caption for a Top-20 aerial-image candidate, a segment-conditioned caption pool for dense video, a slot-conditioned set of relation tuples, or article sentences selected after a VLM description; in each case, the objective is finer semantic alignment than coarse global image–text matching typically provides (Zhang et al., 3 Oct 2025, Jeon et al., 4 Sep 2025, Long et al., 19 Sep 2025, Nguyen et al., 17 Jun 2026).

1. Problem setting and motivating failures

The central motivation is the mismatch between visual content and textual supervision under conditions where simple joint embedding is not sufficiently discriminative. In Team Xiaomi EV-AD VLA’s Caption-Guided Retrieval System (CGRS), fine-grained semantic matching between a free-form text query and an aerial or drone image is described as notoriously difficult because aerial views exhibit large viewpoint shifts, dramatic scale variations, and complex spatial layouts such as “a sports field top-left of a parking lot”; direct mapping of text and image into a joint embedding often overlooks such nuances (Zhang et al., 3 Oct 2025).

Comparable mismatch appears in other modalities. In dense video captioning, Sali4Vid identifies a failure mode in retrieving captions from fixed-size video chunks, because fixed windows overlook scene transitions and can mix unrelated sub-events (Jeon et al., 4 Sep 2025). In video–text retrieval, ExCae frames the problem as an information mismatch caused by personalized and inadequate textual descriptions of videos, arguing that the substantial information gap between the two modalities hinders effective cross-modal representation alignment (Yang et al., 5 Feb 2025). In retrieval-augmented image captioning, RACap further isolates a relation-modeling bottleneck: semantic prompts may be too coarse-grained to capture fine-grained relationships, and explicit modeling of image objects and their semantic relations may be absent (Long et al., 19 Sep 2025).

A recurring theme is therefore not merely missing information, but missing structure. Spatial relations, object interactions, temporal boundaries, discourse position, and annotator-specific variation all interfere with retrieval if the system assumes that a single global visual embedding and a fixed caption pool are sufficient. This suggests that “adaptive” in this literature denotes changes in what is retrieved, when retrieval is performed, and how retrieved text is incorporated.

2. Core retrieval formulations

Most systems instantiate retrieval in a shared embedding space and use cosine similarity as the primary scoring primitive. SmallCap uses a frozen CLIP model with image and text encoders, computes $$s(I,c)=\langle E_{\text{image}}(I),E_{\text{text}}(c)\rangle / (\|E_{\text{image}}(I)\|\,\|E_{\text{text}}(c)\|)$$, and inserts caption embeddings into a FAISS index for sublinear nearest-neighbour search; EXTRA likewise embeds images and captions with pretrained CLIP encoders, retrieves the top-$k$ captions with FAISS IndexFlatIP, and shows that direct Image–Text retrieval outperforms Image–Image retrieval (Ramos et al., 2022, Ramos et al., 2023).

CGRS makes the retrieval pipeline explicitly two-stage. The coarse stage adopts the GeoText-1652 baseline with $L^2$-normalized text and image embeddings $f_q(q)\in\mathbb{R}^d$ and $f_i(i)\in\mathbb{R}^d$, uses cosine similarity

$$\mathrm{sim}(u,v)=\frac{u\cdot v}{\lVert u\rVert_2\;\lVert v\rVert_2},$$

and selects the Top-20 gallery images

$$\mathcal{C}_{20}=\mathrm{TopK}_{I\in\mathcal{D}}\bigl(\mathrm{sim}(f_q(q),f_i(I)),K=20\bigr).$$

The refinement stage embeds the original query and a generated caption $c_k$ with a sentence encoder, computes a coarse score $s_{\mathrm{coarse}}(q,I_k)$ and a semantic score $s_{\mathrm{sem}}(q,c_k)$, and combines them as

$k$0

with $k$1 in practice (Zhang et al., 3 Oct 2025).

Sali4Vid introduces a different form of adaptivity before retrieval begins. Using CLIP ViT-L/14 frame features $k$2 with $k$3, it defines a frame-to-frame difference score

$k$4

then sets an adaptive threshold

$k$5

with $k$6, where $k$7 and $k$8 are the mean and standard deviation of $k$9. A running-average segment prototype $L^2$0 suppresses over-segmentation, after which each semantic segment $L^2$1 is average pooled into $L^2$2 and used to retrieve top-$L^2$3 captions from an external datastore by cosine similarity (Jeon et al., 4 Sep 2025).

These formulations show that semantic adaptation is often implemented as a thin layer around standard similarity search: reranking, threshold adaptation, or conditional re-embedding frequently yields the key effect without replacing cosine-based retrieval itself.

3. Granularity of adaptation

A defining property of the field is that adaptation occurs at multiple semantic granularities.

At the candidate level, CGRS generates detailed captions for only the Top-20 coarse candidates. GPT-4o with vision receives each raw image plus a fixed prompt template that asks for spatial positions such as “center building”, “left”, and “top-right landmark”, and produces captions of roughly 120–150 words. Reranking then becomes text-to-text comparison between the original query and these generated captions, with the coarse visual score retained as a modest anchor (Zhang et al., 3 Oct 2025).

At the segment level, Sali4Vid replaces fixed temporal chunks with semantically coherent segments discovered from frame similarity. Retrieved captions are attached to segments rather than to uniform windows, and the pooled retrieval vectors are concatenated in temporal order with saliency-reweighted video features and transcript features before decoding (Jeon et al., 4 Sep 2025).

At the object and relation level, RACap introduces an object-aware module that learns $L^2$4 slot embeddings to discover heterogeneous visual entities, retrieves the most relevant textual features for each slot from two pools—raw captions $L^2$5 and Subject–Predicate–Object–Environment tuples $L^2$6—and forms relation-aware prompts through cross-attention. The retrieval database includes approximately 120K COCO captions, tuple parsing uses a lightweight BERT-based parser, missing tuple components are replaced with a “[MISSING]” token, and both caption and tuple embeddings are indexed with FAISS (Long et al., 19 Sep 2025).

At the sample-specific fusion level, ExCae’s Expertized Caption Selection does not simply average derived captions. Instead, each video has original frame embeddings and $L^2$7 LLM-derived caption embeddings, which are routed through $L^2$8 experts with top-$L^2$9 masking. The reported experimental setting uses 8 sampled frames, $f_q(q)\in\mathbb{R}^d$0 captions per video, $f_q(q)\in\mathbb{R}^d$1 experts, and top-$f_q(q)\in\mathbb{R}^d$2 active per sample, so the retrieved caption evidence survives only insofar as routers deem it relevant to the specific sample (Yang et al., 5 Feb 2025).

At the document and discourse level, the knowledge-grounded news captioning pipeline retrieves whole articles using article structure-aware textual features and visual-placement features, then performs a second-stage sentence selection. The VLM first produces a structured visual context paragraph; the top-1 article is split into sentences, each sentence is embedded with M3-Embedding, the top-3 most similar sentences are selected, each is augmented with its immediate predecessor and successor sentence, and the resulting chunks are reordered in original document order to form the textual evidence fed to the LLM (Nguyen et al., 17 Jun 2026).

At the language canonicalization level, MoCHA adapts the text side before retrieval or alignment by mapping a raw caption $f_q(q)\in\mathbb{R}^d$3 to a canonical form $f_q(q)\in\mathbb{R}^d$4 that preserves motion-recoverable semantics while suppressing annotator-specific style and hallucinated context. The method is presented as a preprocessing step compatible with any retrieval architecture, and the combined training loss blends the canonicalized and original views with $f_q(q)\in\mathbb{R}^d$5 (Warner et al., 24 Mar 2026). A related but distinct variant appears in STiTch, where an LLM-generated composed caption embedding $f_q(q)\in\mathbb{R}^d$6 is refined toward the CLIP text embedding of the modification instruction, $f_q(q)\in\mathbb{R}^d$7, and retrieval is reformulated as bidirectional transportation between discrete text and image distributions rather than point-to-point alignment (Li et al., 20 May 2026).

The literature therefore treats “caption retrieval” broadly: the retrieved object may be an existing caption, a tuple, a generated description, a sentence excerpt, or a canonicalized text surrogate, provided it improves semantic correspondence for downstream ranking or generation.

4. Integration with caption generation and cross-modal alignment

The downstream role of retrieved text varies across architectures.

SmallCap uses retrieved captions as prompts to a lightweight captioning model. It combines three frozen backbones—CLIP image, CLIP text, and GPT-2—with newly introduced cross-attention layers, and the only learned parameters are in those inserted layers and associated LayerNorm parameters. The total trainable bridge can be as small as 10 M parameters, while CLIP and the approximately 355M-parameter GPT-2 remain frozen. Because the CLIP weights are never updated, adaptation to a new domain is achieved by rebuilding or extending the caption datastore rather than by finetuning (Ramos et al., 2022).

Ramos et al.’s EXTRA integrates retrieval more tightly with a multimodal encoder. Retrieved captions are tokenized and jointly processed with visual region features by a pretrained LXMERT encoder, and a 4-layer GPT-2–style decoder cross-attends to the fused encoder states. Training proceeds with a cross-entropy stage and a Self-Critical Sequence Training stage, and cross-attention analysis indicates that by decoder layer 4 over 80% of the attention mass is on textual representations, showing that retrieved captions actively steer generation rather than serve as passive context (Ramos et al., 2023).

RACap occupies a middle position between caption retrieval and relation-aware prompting. CLIP and GPT-2 are frozen backbones; only approximately 10.8M parameters are trained, namely the slot-attention blocks, fusion cross-attention layer and MLP, and the GPT-2 cross-attention projection layers. Retrieved caption and tuple features are fused into relation-aware prompts through multi-head cross-attention before autoregressive decoding, and the training objective is the standard negative log-likelihood of the target caption (Long et al., 19 Sep 2025).

Sali4Vid keeps retrieval external to optimization. Its only explicit training objective is the standard cross-entropy loss on the joint output sequence of timestamps and captions, and there is no separate contrastive or ranking loss for the retrieval component; retrieval is “frozen” at inference, and the model learns to condition on retrieved hints via standard cross-entropy (Jeon et al., 4 Sep 2025).

In retrieval-heavy pipelines for non-captioning tasks, the retrieved text may function primarily as an alignment aid. CGRS uses generated captions to rerank cross-view image retrieval candidates rather than to produce final captions (Zhang et al., 3 Oct 2025). ExCae inserts Caption Self-Improvement and Expertized Caption Selection between an off-the-shelf video–text backbone and a standard contrastive-loss objective, broadening the video-side textual description and then filtering it through personalized experts (Yang et al., 5 Feb 2025). MoCHA uses canonicalized text to produce tighter positive clusters in a shared embedding space, again without changing the underlying retrieval architecture (Warner et al., 24 Mar 2026).

5. Reported empirical behavior

Reported gains differ by task, but a consistent pattern is that semantic adaptation improves either retrieval ranking, caption quality, or both.

System	Adaptive retrieval unit	Reported result
CGRS (Zhang et al., 3 Oct 2025)	Top-20 candidates reranked by VLM-generated captions	$f_q(q)\in\mathbb{R}^d$8: $f_q(q)\in\mathbb{R}^d$9; $f_i(i)\in\mathbb{R}^d$0: $f_i(i)\in\mathbb{R}^d$1; $f_i(i)\in\mathbb{R}^d$2: $f_i(i)\in\mathbb{R}^d$3
Sali4Vid (Jeon et al., 4 Sep 2025)	Semantic video segments	YouCook2 validation: CIDEr $f_i(i)\in\mathbb{R}^d$4; F1 $f_i(i)\in\mathbb{R}^d$5
RACap (Long et al., 19 Sep 2025)	Slot-conditioned caption and tuple retrieval	CIDEr $f_i(i)\in\mathbb{R}^d$6 on COCO; Overall $f_i(i)\in\mathbb{R}^d$7 on NoCaps
SmallCap (Ramos et al., 2022)	Top-$f_i(i)\in\mathbb{R}^d$8 retrieved captions from a datastore	Peak COCO CIDEr at $f_i(i)\in\mathbb{R}^d$9: $$\mathrm{sim}(u,v)=\frac{u\cdot v}{\lVert u\rVert_2\;\lVert v\rVert_2},$$0; blacked-out visual features still yield CIDEr $$\mathrm{sim}(u,v)=\frac{u\cdot v}{\lVert u\rVert_2\;\lVert v\rVert_2},$$1
EXTRA (Ramos et al., 2023)	Top-$$\mathrm{sim}(u,v)=\frac{u\cdot v}{\lVert u\rVert_2\;\lVert v\rVert_2},$$2 CLIP-retrieved captions	COCO ablation: $$\mathrm{sim}(u,v)=\frac{u\cdot v}{\lVert u\rVert_2\;\lVert v\rVert_2},$$3, $$\mathrm{sim}(u,v)=\frac{u\cdot v}{\lVert u\rVert_2\;\lVert v\rVert_2},$$4, $$\mathrm{sim}(u,v)=\frac{u\cdot v}{\lVert u\rVert_2\;\lVert v\rVert_2},$$5 CIDEr
News image captioning (Nguyen et al., 17 Jun 2026)	Hierarchical article retrieval plus sentence selection	Private test: Overall $$\mathrm{sim}(u,v)=\frac{u\cdot v}{\lVert u\rVert_2\;\lVert v\rVert_2},$$6; Retrieval mAP $$\mathrm{sim}(u,v)=\frac{u\cdot v}{\lVert u\rVert_2\;\lVert v\rVert_2},$$7, $$\mathrm{sim}(u,v)=\frac{u\cdot v}{\lVert u\rVert_2\;\lVert v\rVert_2},$$8, $$\mathrm{sim}(u,v)=\frac{u\cdot v}{\lVert u\rVert_2\;\lVert v\rVert_2},$$9

The drone-navigation case is especially explicit about the effect of semantic reranking. CGRS reports a consistent approximately 5% absolute gain on all key metrics relative to the GeoText-1652 baseline and earned second place out of eight teams in RoboSense 2025 Track 4 (Zhang et al., 3 Oct 2025). In dense video captioning, Sali4Vid reports that fixed-chunk retrieval with $$\mathcal{C}_{20}=\mathrm{TopK}_{I\in\mathcal{D}}\bigl(\mathrm{sim}(f_q(q),f_i(I)),K=20\bigr).$$0 gives CIDEr $$\mathcal{C}_{20}=\mathrm{TopK}_{I\in\mathcal{D}}\bigl(\mathrm{sim}(f_q(q),f_i(I)),K=20\bigr).$$1 on YouCook2, whereas the adaptive segmentation scheme yields CIDEr $$\mathcal{C}_{20}=\mathrm{TopK}_{I\in\mathcal{D}}\bigl(\mathrm{sim}(f_q(q),f_i(I)),K=20\bigr).$$2 without additional learnable parameters, and the full model reaches CIDEr $$\mathcal{C}_{20}=\mathrm{TopK}_{I\in\mathcal{D}}\bigl(\mathrm{sim}(f_q(q),f_i(I)),K=20\bigr).$$3 when combined with saliency reweighting (Jeon et al., 4 Sep 2025).

The image-captioning literature also shows that retrieval quantity and datastore composition matter. SmallCap reports that varying $$\mathcal{C}_{20}=\mathrm{TopK}_{I\in\mathcal{D}}\bigl(\mathrm{sim}(f_q(q),f_i(I)),K=20\bigr).$$4 yields a peak CIDEr on COCO at $$\mathcal{C}_{20}=\mathrm{TopK}_{I\in\mathcal{D}}\bigl(\mathrm{sim}(f_q(q),f_i(I)),K=20\bigr).$$5 with $$\mathcal{C}_{20}=\mathrm{TopK}_{I\in\mathcal{D}}\bigl(\mathrm{sim}(f_q(q),f_i(I)),K=20\bigr).$$6, dropping to $$\mathcal{C}_{20}=\mathrm{TopK}_{I\in\mathcal{D}}\bigl(\mathrm{sim}(f_q(q),f_i(I)),K=20\bigr).$$7 at $$\mathcal{C}_{20}=\mathrm{TopK}_{I\in\mathcal{D}}\bigl(\mathrm{sim}(f_q(q),f_i(I)),K=20\bigr).$$8; the system can also exploit human-labeled and web data in a training-free fashion, and the web datastore gives the best average out-of-domain performance, including a gain of +5 CIDEr on MSR-VTT (Ramos et al., 2022). EXTRA reports a parallel result: retrieving $$\mathcal{C}_{20}=\mathrm{TopK}_{I\in\mathcal{D}}\bigl(\mathrm{sim}(f_q(q),f_i(I)),K=20\bigr).$$9 captions outperforms $c_k$0 and $c_k$1, and oracle substitution with ground-truth references raises CIDEr by +1.8 or +8.3 depending on how many references are injected, demonstrating substantial headroom for better retrieval (Ramos et al., 2023).

Beyond caption generation, semantic text adaptation also changes embedding geometry. MoCHA reports that canonicalization reduces within-motion text-embedding variance by 11–19%, improves HumanML3D text-to-motion $c_k$2 from $c_k$3 to $c_k$4 for the LLM variant, and improves KIT-ML from $c_k$5 to $c_k$6 (Warner et al., 24 Mar 2026). Although this is not caption retrieval in the narrow image-captioning sense, it reinforces the same principle: stabilizing the textual side of supervision can materially improve retrieval.

6. Misconceptions, limitations, and likely directions

A recurrent misconception is that retrieval augmentation is exhausted by static nearest-neighbour lookup over a caption datastore. The surveyed systems instead implement adaptation at several levels: Top-20 reranking with generated captions in CGRS, semantic video segmentation in Sali4Vid, slot-conditioned caption and tuple retrieval in RACap, temporal and citation reweighting in news article retrieval, expert routing in ExCae, canonicalization in MoCHA, and bidirectional transportation between caption and image distributions in STiTch (Zhang et al., 3 Oct 2025, Jeon et al., 4 Sep 2025, Long et al., 19 Sep 2025, Nguyen et al., 17 Jun 2026, Yang et al., 5 Feb 2025, Li et al., 20 May 2026).

A second misconception is that retrieving more text is always beneficial. The available evidence is more selective. SmallCap finds peak CIDEr at $c_k$7 and a drop at $c_k$8 (Ramos et al., 2022). EXTRA likewise shows improvement from $c_k$9 to $s_{\mathrm{coarse}}(q,I_k)$0, but the reported emphasis is on retrieving a sufficient number of captions, not on unbounded growth in $s_{\mathrm{coarse}}(q,I_k)$1 (Ramos et al., 2023). This suggests that semantic relevance, noise control, and fusion strategy are at least as important as raw retrieval volume.

Several limitations are repeatedly documented. STiTch explicitly observes that LLM-generated captions can introduce unexpected features from the reference image because the image contains much more detail than the text modification (Li et al., 20 May 2026). Sali4Vid shows that fixed-size chunk retrieval mixes unrelated sub-events (Jeon et al., 4 Sep 2025). MoCHA shows that backtranslation, used as a negative control, fails to improve retrieval, indicating that arbitrary text transformation is not enough; denoising must preserve recoverable semantics (Warner et al., 24 Mar 2026). EXTRA’s oracle experiments reveal that even effective systems remain retrieval-limited, because replacing retrieved captions with ground-truth references yields up to +8.3 CIDEr (Ramos et al., 2023).

A plausible implication is that future semantic-based adaptive caption retrieval systems will continue to combine three design choices already present in the literature: lightweight or frozen foundation models, semantically structured retrieval units, and explicit mechanisms for suppressing irrelevant textual variance. The existing record across drone navigation, dense video captioning, retrieval-augmented image captioning, knowledge-grounded news description, video–text retrieval, and motion–text retrieval indicates that improvements often arise not from replacing retrieval, but from making retrieval conditional on the right semantic unit at the right stage of the pipeline (Zhang et al., 3 Oct 2025, Jeon et al., 4 Sep 2025, Ramos et al., 2022, Ramos et al., 2023, Nguyen et al., 17 Jun 2026).