---
title: Caption-Guided Retrieval Systems
url: https://www.emergentmind.com/topics/caption-guided-retrieval-system-cgrs
type: topic
---

# Caption-Guided Retrieval Systems

Caption-Guided Retrieval System (CGRS) denotes a family of multimodal retrieval architectures in which captions or caption-like textual descriptions act as retrieval queries, reranking signals, semantic bridges, or retrieval-conditioned grounding representations. The term is used most explicitly for a two-stage coarse-to-fine reranking system for natural language-guided cross-view image retrieval in drone navigation [2510.02728]. In a broader systems sense, related work extends the pattern to caption-first lifelog retrieval, retrieval-augmented event captioning, adaptive caption retrieval for video, retrieval-aware caption optimization for text-video search, and caption-assisted face matching [2510.04010]. The common premise is that raw visual embeddings alone often under-specify fine-grained semantics, whereas captions can expose object identity, spatial layout, event structure, named entities, or narrative context in a form that is easier to compare, fuse, or audit [2509.16560].

## 1. Conceptual scope and system boundaries

A strict CGRS, in the narrow sense, uses captions directly in retrieval or reranking. The clearest example is the RoboSense 2025 Track 4 system, where a text query first retrieves a coarse top-20 image shortlist and generated captions for those candidates are then used for text-text semantic reranking [2510.02728]. A looser family of systems uses captions after retrieval, as semantic explanations or context-conditioned outputs, without feeding captions back into search. ZSE-Cap is explicit on this distinction: retrieval is image-to-image, captioning is used after retrieval, and “prompts/captions do not directly affect retrieval” [2507.20564]. ReCap follows the same broader pattern: image retrieval is performed first, and caption generation then fuses article-derived evidence with a generic image caption [2509.01259].

This boundary matters because the literature contains both genuine caption-guided rerankers and retrieval-augmented captioning pipelines. The former use captions as an active scoring signal; the latter use captions as a downstream grounded description. A plausible systems interpretation is that CGRS spans both, but the coupling strength varies substantially.

| Setting | Caption role | Representative work |
|---|---|---|
| Cross-view drone retrieval | Candidate reranking signal | [2510.02728] |
| Visual lifelog retrieval | Primary retrieval index representation | [2510.04010] |
| Event-enriched image understanding | Retrieval-conditioned grounded output | [2507.20564], [2509.01259] |
| Dense video captioning | Segment-level retrieved semantic memory | [2509.04602] |
| Face matching | Auxiliary multimodal supervision and fusion | [2308.06866] |

A common misconception is that any retrieval-plus-captioning pipeline is automatically caption-guided retrieval. The EVENTA systems make clear that this is not so. ZSE-Cap is described as a “retrieval-augmented captioning pipeline,” not a true caption-driven retrieval loop, because retrieval uses only image embeddings and L2 distance across CLIP, SigLIP, and DINOv2 [2507.20564]. ReCap likewise uses image-only retrieval with DINOv2 global similarity and patch-level mutual nearest-neighbor reranking; the generated caption is an intermediate semantic artifact, not the retrieval query itself [2509.01259].

## 2. Canonical architectural patterns

The dominant CGRS pattern is two-stage retrieval. In the drone-navigation formulation, the first stage is coarse retrieval with a trained baseline model, and the second stage is caption-guided reranking. The coarse model retrieves the top 20 candidates,
\[
\mathcal{C}_{20} = \text{TopK}_{I \in \mathcal{D}, K=20} \, s(T, I),
\]
after which a VLM generates one caption \(C_k\) for each candidate image and the final score fuses the original text-image score with caption-query semantic similarity,
\[
s_{\text{final}(T, I_k) = \alpha \cdot s_{\text{coarse}(T, I_k) + (1-\alpha) \cdot s_{\text{sem}(T, C_k),
\]
with \(\alpha = 0.3\) in the reported system [2510.02728]. This makes caption matching responsible for 70% of the reranking weight while preserving 30% of the original retrieval signal.

A second pattern is caption-first indexing. CIVIL, for lifelog retrieval, does not rely primarily on direct image-text embeddings. Instead, it first generates captions for first-person wearable-camera frames or frame groups, embeds those captions and user queries in a shared text space using GTE-Large or BGE-M3, and ranks frames by caption-query similarity [2510.04010]. The method includes three caption granularities—single, collective, and merged—and also reports gains from averaging similarity scores across complementary caption types. This turns retrieval into text-to-text semantic search over generated interpretations of personal experience rather than direct scene matching.

A third pattern is adaptive caption memory. Sali4Vid, although framed as dense video captioning rather than retrieval ranking, segments videos into semantically coherent temporal units and retrieves top-\(k\) caption embeddings for each segment from an external datastore,
\[
\mathcal{R}_{S_m} = \operatorname*{Top-k}_{r_r \in R} \left( \text{sim}(r_r, \boldsymbol{z}_{S_m}) \right),
\qquad
\tilde{r}_m = \frac{1}{k} \sum_{r \in \mathcal{R}_{S_m} r .
\]
Here caption retrieval is auxiliary semantic guidance for generation, but the mechanism is directly transferable to video-segment CGRS designs [2509.04602].

A fourth pattern is retrieval-grounded caption synthesis. ReCap retrieves a candidate article with DINOv2 global similarity, reranks the top-100 by patch-level mutual nearest-neighbor similarity, then constructs a context bundle from a generic image caption, an article-page image caption, the article title, and an article summary before generating the final event-aware caption with Qwen3 [2509.01259]. In medical imaging, retrieval-guided generation uses cosine similarity over visual embeddings to retrieve the top-3 histopathology cases and then summarizes their expert-authored captions into a final caption [2605.00893]. These systems are not pure caption-guided retrievers, but they establish a recurring design principle: use retrieval to ground caption formation, then use the caption as an auditable semantic bottleneck.

## 3. Representations, scoring functions, and optimization

CGRS implementations differ mainly in where caption information enters the scoring function. In the Xiaomi system, query text and generated candidate captions are encoded by BERT and compared with cosine similarity,
\[
s_{\text{sem}(T, C_k) = \frac{\mathbf{e}_T \cdot \mathbf{e}_{C_k}{||\mathbf{e}_T||_2 ||\mathbf{e}_{C_k}||_2},
\]
then fused with the coarse retrieval score [2510.02728]. No extra temperature scaling, min-max normalization, or learned calibration is reported there; the reranker is explicitly a two-score fusion.

Other systems optimize the captions themselves for retrieval utility. CaRe-DPO argues that off-the-shelf MLLM-generated video captions are often too generic and observes a Pearson correlation as low as 0.30 between top-1 caption rankings under BLEU and retrieval relevance. Its Dual-Group Direct Preference Optimization constructs preferred and dispreferred captions using retrieval scores and optimizes caption generation directly against retrieval relevance rather than captioning metrics alone [2509.16560]. The retrieval model then scores a video, an auxiliary caption, and a query jointly, using the contrastive inference score
\[
s(\mathbf{v}, \mathbf{c}, \mathbf{t})= \log \frac{P(y^+| \mathbf{v}, \mathbf{c}, \mathbf{t})}{P(y^{-}| \mathbf{v}, \mathbf{c}, \mathbf{t})}.
\]
A notable systems lesson from this work is that captions may be ignored unless the model marks their role explicitly; role-embeddings produced an average improvement of +1.8 over random-caption controls, whereas without role-embeddings the effect was about +0.1 [2509.16560].

Caption quality can also be optimized indirectly by making generated captions more discriminative. “Guiding Image Captioning Models Toward More Specific Captions” adapts classifier-free guidance to autoregressive captioning and shows that increasing the guidance scale improves caption\(\rightarrow\)image retrieval in CLIP space. On MS-COCO, CLIP-space recall@1 rises from 26.5 at \(\gamma=1.0\) to 44.6 at \(\gamma=2.0\), while CIDEr falls from 126.1 to 78.6 [2307.16686]. The retrieval interpretation is explicit in the paper: guidance trades off \(p(\mathrm{caption}\mid \mathrm{image})\) against \(p(\mathrm{image}\mid \mathrm{caption})\), thereby favoring less ambiguous captions.

Several papers treat caption supervision itself as noisy. MoCHA formalizes captions as draws from \(p(t\mid s,a)\), where \(s\) is motion-recoverable semantics and \(a\) is annotator-specific nuisance, and proposes canonicalization to reduce within-item caption variance before contrastive retrieval training [2603.23684]. In audio retrieval, “Estimated Audio-Caption Correspondences Improve Language-Based Audio Retrieval” replaces one-hot off-diagonal negatives with soft correspondence estimates distilled from an ensemble of retrieval models, improving ClothoV2 mAP@10 from 35.46 to 40.14 in the scaled setting and surpassing the previous state of the art by 1.6 percentage points [2408.11641]. These results suggest that CGRS performance depends not only on caption availability but also on whether caption supervision preserves graded semantic overlap rather than forcing brittle binary distinctions.

## 4. Domain-specific instantiations

In robotics and drone navigation, CGRS is used for natural language-guided cross-view image retrieval across drone, satellite, and ground-camera imagery. The RoboSense Track 4 system uses GPT-4o to generate detailed aerial captions of top-20 candidates, with a prompt emphasizing the central building, relative positions, landmarks such as parking lots or sports fields, and overall spatial layout [2510.02728]. The emphasis is fine-grained spatial disambiguation in cluttered aerial scenes.

In lifelog retrieval, CIVIL addresses first-person wearable-camera data, where the retrieval target is not merely the visible scene but the individual’s experience. It prepends “The individual’s experience:” to generated captions before embedding, uses single-frame, eight-frame collective, and merged-caption strategies, and reports a best average \(P@10 = 0.73\) from combining single and collective captions, compared with 0.58 for the strongest direct image-embedding baselines [2510.04010]. This establishes caption-mediated interpretation as a viable replacement for direct image-text matching in first-person memory retrieval.

In news and event understanding, both ZSE-Cap and ReCap show how caption generation can recover event semantics absent from image-only descriptions. ZSE-Cap ensembles CLIP, SigLIP, and DINOv2 for image retrieval, then uses Gemma 3 with an explicitly journalistic prompt to connect visible content to article-level events, named entities, and narrative significance [2507.20564]. ReCap adds article-summary generation, web-caption extraction from source pages, and Semantic Gaussian Normalization to keep event-aware captions within a CIDEr-favorable length range while preserving or adding named entities [2509.01259]. These systems are especially relevant where retrieval must bridge visual perception and document context, even if captions are not yet used directly in the retrieval loop.

In video, two distinct patterns appear. Sali4Vid retrieves captions from an external datastore for adaptively segmented video units, using them as semantic memory for dense video captioning [2509.04602]. CaRe-DPO instead treats auxiliary captions as retrieval objects that must be optimized for text-video matching, not just language fluency [2509.16560]. Together they show both directions of coupling: captions can guide retrieval, and retrieval can supervise caption quality.

In faces, caption-guided face recognition uses natural-language facial descriptions as auxiliary information for verification and identification. The proposed framework refines BERT text features through DAMSM and CMPC, aggregates local word-region and global image-caption interactions through CFAM, and substantially improves both ArcFace and AdaFace under degraded-image settings [2308.06866]. This is not standard text-to-image retrieval, but it is a strong template for caption-assisted re-ranking over a face gallery.

In safety-critical medical settings, retrieval-guided captioning uses retrieved expert text rather than de novo free generation. On ARCH histopathology, retrieval-guided generation via UNI2 reaches mean BioBERT cosine similarity 0.6039 versus 0.4732 for MedGemma, with non-overlapping 95% confidence intervals [2605.00893]. The paper frames the language model as a “summarizer rather than a free generator,” highlighting a CGRS design principle in which captions serve as auditable summaries of retrieved expert evidence.

## 5. Empirical performance and what it demonstrates

The strongest direct evidence for CGRS comes from the drone-navigation benchmark. The Xiaomi EV-AD VLA system achieved 2nd place among 8 teams on the official RoboSense Track 4 leaderboard, with \(R@1 = 31.33\), \(R@5 = 49.09\), and \(R@10 = 57.15\). Relative to the official baseline team RoboSense2025, that is an improvement of 5.89 percentage points at \(R@1\), 8.48 at \(R@5\), and 8.05 at \(R@10\) [2510.02728]. The improvement is attributed to caption-guided reranking, not to changes in the coarse retriever alone.

Lifelog retrieval provides complementary evidence that caption-first indexing can beat strong direct embedding baselines. CIVIL reports that ViT-SO400M-14-SigLIP-384 and DFN5B-CLIP-ViT-H-14 each achieve average \(P@10 = 0.58\), whereas the best caption-guided combination—InternLM-XComposer2-VL-7B single captions plus Video-LLaVA collective captions with GTE-Large—reaches \(P@10 = 0.73\) [2510.04010]. The system also reports a GPT-4o reranking experiment that improves \(P@10\) from 0.58 to 0.66 on top of embedding retrieval, which suggests a viable retrieval-plus-LLM-reranker deployment pattern.

Event-grounded systems show a related but distinct effect: retrieval quality and caption grounding reinforce one another even without a true caption-guided loop. ZSE-Cap reports public-test retrieval scores of mAP 0.994, \(R@1 = 0.990\), and \(R@10 = 0.999\) for its weighted ensemble of CLIP, SigLIP, and DINOv2, and reaches a final private-test score of 0.42002, securing 4th place [2507.20564]. ReCap reports private-test mAP 0.982, \(R@1 = 0.977\), \(R@10 = 0.988\), CLIPScore 0.870, CIDEr 0.205, and overall 0.54666, ranking 2nd on the private leaderboard [2509.01259]. In both cases, the empirical message is that high retrieval accuracy is a prerequisite for grounded caption quality.

Retrieval-aware caption optimization also shows sizable gains. CaRe-DPO reaches text-to-video \(R@1 = 85.1\) on DiDeMo, 79.2 on ActivityNet, and 64.1 on MSRVTT, and reports average percentage gains over MM-Embed of 3.9%, 3.1%, and 5.1%, respectively [2509.16560]. The same paper shows that simple supervised fine-tuning of the captioner yields only +0.2 average \(R@1\), whereas SG-DPO gives +0.9 and DG-DPO +1.3, indicating that the benefit comes from retrieval-aligned preference learning rather than caption finetuning alone.

Caption specificity by decoding also has a large retrieval effect. With classifier-free guidance, CLIP caption\(\rightarrow\)image recall@1 on MS-COCO increases from 26.5 to 44.6 at guidance scale 2, and to 49.4 at scale 3, even though reference-based caption metrics deteriorate [2307.16686]. This provides unusually direct evidence that captions optimized for discriminativeness can be better retrieval queries than captions optimized for reference overlap.

Audio retrieval extends the pattern beyond vision. A PaSST-based text-audio retrieval system trained on large audio-caption corpora improves ClothoV2 mAP@10 by 5.6 percentage points over the previous state of the art [2308.04258]. The follow-up method with estimated audio-caption correspondences further improves ClothoV2 mAP@10 by 1.6 percentage points over the then-current best system [2408.11641]. Although these systems are not labeled CGRS in the narrow vision-centric sense, they reinforce the same proposition: caption supervision becomes more useful when correspondence structure is modeled more carefully.

## 6. Limitations, controversies, and open directions

The most persistent limitation is weak coupling between retrieval and captioning. ZSE-Cap states the issue plainly: the system is a pipeline, not a loop, with no caption-informed retrieval, no caption-based reranking, and no end-to-end optimization [2507.20564]. ReCap shares this limitation, relying on image-only retrieval despite later generating text rich enough to guide document matching [2509.01259]. This leaves unresolved the central CGRS question of when and how generated captions should be fed back into search.

A second bottleneck is shortlist dependence. In the Xiaomi system, CGRS reranks only the top 20 coarse candidates; if the correct image is absent from \(\mathcal{C}_{20}\), the caption stage cannot recover it [2510.02728]. This is the classic reranking ceiling and suggests that caption-guided systems remain hostage to the recall of the first-stage retriever.

A third issue is caption reliability. CIVIL identifies text-embedding retrieval errors, lack of specific details, over-interpretation, and error propagation across grouped captions [2510.04010]. In histopathology, retrieval-guided generation reduces hallucination relative to free generation but introduces new failure modes such as concept mixing and inherited over-specific diagnostic labeling [2605.00893]. These findings matter because captions are often treated as semantically cleaner than images, yet the literature shows that captions can be generic, noisy, style-variable, over-specific, or simply wrong.

A fourth controversy concerns optimization targets. CaRe-DPO and the classifier-free-guidance paper converge on the same conclusion from different directions: fluent or reference-matching captions are not necessarily retrieval-useful captions [2509.16560]. Retrieval-oriented caption optimization improves ranking, but can reduce CIDEr, BLEU, or grammaticality [2307.16686]. This creates a methodological split between systems optimized for user-facing caption quality and systems optimized for retrieval discriminativeness.

A fifth limitation is supervision noise. MoCHA shows that captions are often stochastic descriptions rather than deterministic labels, and that canonicalization can reduce within-item text-embedding variance by 11–19% while improving transfer [2603.23684]. The audio correspondence-distillation work makes a similar point from the opposite side: many off-diagonal “negatives” are only partially negative, so one-hot contrastive supervision can damage retrieval geometry [2408.11641]. For CGRS, this suggests that caption normalization and soft correspondence modeling are not ancillary refinements but core training concerns.

Open directions are already explicit in the surveyed work. The drone-navigation paper suggests supervised rerankers over query, caption, and image score, multiple captions per image, and stronger sentence embedding models [2510.02728]. ZSE-Cap’s error analysis points toward “incorporating textual signals earlier in the retrieval process” [2507.20564]. ReCap naturally suggests image-plus-caption hybrid retrieval, caption-conditioned reranking against article summaries and page captions, and entity-aware query refinement [2509.01259]. Scene-graph-based caption retrieval proposes a different path altogether: represent both captions and images as scene graphs, match by graph subsumption, and, when the query is underspecified, ask interactive follow-up questions about missing attributes or relations [1911.00850]. A plausible synthesis is that the next generation of CGRS will combine dense multimodal candidate generation, caption-aware reranking, structured graph constraints, and iterative retrieve-caption-rerank loops rather than treating captioning and retrieval as adjacent but separate modules.

Source: https://www.emergentmind.com/topics/caption-guided-retrieval-system-cgrs