---
title: Query-Guided Captions
url: https://www.emergentmind.com/topics/query-guided-captions
type: topic
---

# Query-Guided Captions

Query-guided captions are textual descriptions generated or selected to be maximally informative with respect to a specific query or question—rather than providing generic, question-agnostic descriptions. This paradigm is central to modern multimodal reasoning systems, notably visual question answering (VQA), data visualization, scene-text image captioning, video retrieval, and multi-image or multi-modal LLM-based reasoning. Recent architectures achieve state-of-the-art performance by conditioning caption generation and selection on the query itself, either via targeted attention, explicit prompt decomposition, adaptive filtering, or gradient-based alignment. This article synthesizes the dominant methodologies and empirical findings from key research such as "Generating Question Relevant Captions to Aid Visual Question Answering" [1906.00513], "Narrating the Video" [2503.05186], "QG-CoC: Question-Guided Chain-of-Captions" [2511.03206], and related works.

## 1. Formal Definition and Task Scope

Query-guided captioning generalizes conventional captioning by conditioning the output on both the multimodal input (image, audio, data, video, code) and a user-specified query or task. The output is a caption $C$ that maximizes the conditional probability $P(C | X, Q)$, where $X$ is the content (e.g. image(s), video), and $Q$ is the query.

- In VQA: captions are generated (or selected) to support answering a visual question about an image.
- In data visualization: captions are produced to highlight analysis aspects relevant to a user-specified instruction or question [2212.14047].
- In multi-image LLMs: the QG-CoC protocol decomposes complex queries into sub-questions and generates targeted captions for each visual input [2511.03206].
- In scene-text tasks: question-controlled captioning explicitly fuses questions with OCR and object features to provide personalized descriptions [2108.02059].

A generic formulation is:  
$$
C^* = \arg\max_{C} P(C\,|\,X, Q)
$$
where $C^*$ should focus exclusively on evidence and relationships required to resolve $Q$.

## 2. Joint Architectures and Attention Mechanisms

Architectures for query-guided captioning invariably integrate cross-modal encodings and query-centric attention. The approach in [1906.00513] exemplifies this with (i) a bottom-up image encoder (Faster R-CNN+ResNet-101), (ii) a GRU-encoded question vector $q$, and (iii) top-down attention $A^{qv}$ generating question-conditioned region features $V^q$. Captioning is performed by a GRU with word-wise attention over $V^q$, supervised only on the most query-relevant caption per (image, question) pair.

The VQA module further embeds generated/human captions using a dual-GRU attention over words, modulating the image features via a second attention mechanism $A^{cv}$:
$$
a^{cv}_k = f(f(c) \circ f(v^q_k)), \qquad \overline{v}^{qc} = \sum_k \alpha^{cv}_k v^q_k
$$
where $c$ is the caption embedding and $\alpha^{cv}_k$ are softmax-normalized weights. The joint representation for answer prediction is:
$$
h = q \circ (f(\overline v^{qc}) + f(c))
$$
which enforces joint reasoning over question, caption, and conditioned image regions.

In video retrieval, NarVid [2503.05186] applies frame-level captioning via LLMs, then performs cross-modal co-attention between frame features $V$ and their captions $N$, followed by query-aware adaptive filtering that prunes irrelevant frames based on cosine similarity to the query.

## 3. Caption Selection and Alignment Strategies

A critical aspect is selecting or generating captions actually beneficial to the task—whether VQA accuracy, retrieval, or multi-step reasoning. [1906.00513]’s gradient-based online selection is archetypal: for $C$ human captions per image, it computes the dot-product between gradients of the answer prediction logit ($G^{vqa}_k$) and the caption generation log-probability ($G^{c_i}_k$) with respect to shared image features. The optimal caption $i^*$ satisfies:
$$
i^* = \arg\max_i \sum_{k=1}^K (G^{vqa}_k)^\top (G^{c_i}_k), \qquad \sum_k (G^{vqa}_k)^\top (G^{c_i}_k) > \xi
$$
where $\xi > 0$ is a threshold, and the loss is omitted if no caption meets the constraint.

Alternative selection mechanisms include query-aware nucleus sampling in video (NarVid)—sorting frame/caption pairs by query similarity and retaining only the nucleus for further matching.

## 4. Prompt-based and Chain-of-Captions Methods

Zero-shot prompting methods for query-guided captioning, notably QG-CoC [2511.03206], structure the reasoning pipeline as:
1. Decompose the query $Q$ into sub-questions $\{q_1, ..., q_K\}$.
2. For each image $I_i$ and sub-question $q_j$, generate a focused caption $C_{i,j}$ answering $q_j$ for $I_i$.
3. Aggregate these captions as evidence to answer each sub-question.
4. Integrate the sub-answers as the final response.

The method leverages prompt templates:
```
Step 1) Decompose the question into q1, ..., qK.
Step 2) For each qj, write a caption for each image focusing on visual details that help answer qj.
Step 3) Answer each sub-question using the captions.
Step 4) Integrate to answer the original question.
```
This yields gains of up to +12% accuracy on multi-image reasoning benchmarks (MUIR, MuirBench) for open-source and closed-source MLLMs.

## 5. Empirical Performance Across Domains

Empirical studies demonstrate substantial task-specific improvements from query guidance. [1906.00513] reports VQA test-standard accuracy of 68.37% for its single model—surpassing baselines by 2–3%, and ensembles reach 69.66%. Ablations show question-agnostic captions provide far less improvement than targeted ones (Up-Down only: 63.2%; + question-agnostic: 64.6%; + question-relevant: 65.8%; + COCO captions: 67.1%).

NarVid’s comprehensive query-guided caption integration increases video retrieval R@1 from 44.5% (CLIP4Clip) to 51.0%, with 1–4pt jumps per module (cross-modal, query filtering, dual-modal, hard negatives) [2503.05186].

QG-CoC yields up to +12% accuracy gain for multi-image tasks. Question-controlled scene-text captioning (GQAM, [2108.02059]) outperforms non-control and naïve baselines in CIDEr (+129–140 pts), SPICE (+17–18 pts), and answer recall (+10–17 pp.).

## 6. Mechanistic Analysis: Why Query Guidance Works

Query-guided captions provide multiple mechanistic benefits:
- Additional Knowledge: Captions inject rich attributes, object relations, counts, and non-visual priors beyond pure region features, directly supporting answers to “Num” and “Other” categories [1906.00513].
- Focused Information: Caption selection conditioned on the query suppresses irrelevant, noisy text, concentrating model attention on necessary evidence [1906.00513], [2503.05186].
- Modulated Attention: Caption embeddings steer visual attention modules toward regions mentioned in the caption, improving alignment with human annotations and interpretability (measured by Earth Mover's Distance; caption-attention reduces EMD by 0.08) [1906.00513].
- Consistent Optimization: Gradient-based or classifier-based selection ensures loss signals do not conflict, yielding stable training and superior generalization [1906.00513].
- Reasoning Structure: Multi-stage, prompt-based chains like QG-CoC ensure fact extraction and aggregation is explicitly guided by the decomposed query, reducing the likelihood of missing edge cases or fine-grained heterogeneity [2511.03206].

## 7. Limitations and Prospects

Limitations include dependency on high-quality, query-relevant caption collections, computational cost for online gradient methods, and degradation for models with weak captioning ability [2511.03206]. Query guidance requires the system to understand the semantic relationships encoded in the query—prompt misunderstanding or under-specified queries can lead to sub-optimal outputs. The approach is also sensitive to domain distribution; cross-domain generalization favors stylistic diversity in training corpora [2012.02339].

Ongoing research is extending query-guided captioning to video streams, sequential multi-image reasoning, and code-to-text paradigms (SQL2Text), employing graph-aware selection and iterative refinement to maximize informativeness and alignment with user intent [2501.03166]. Benchmarks and datasets are available to evaluate new methods under diverse constraint regimes.

---

In sum, query-guided captioning is a convergent frontier for multimodal systems, leveraging question- or query-conditioned generation, targeted attention, and structured reasoning to produce captions optimized for downstream understanding and task-specific inference. Methodological innovation spans gradient-based supervision, prompt engineering, attention modulation, and multi-stage reasoning chains, with documented impact on VQA, data visualization, scene-text captioning, video retrieval, and code understanding [1906.00513], [2503.05186], [2511.03206], [2108.02059], [2212.14047], [2501.03166].

Source: https://www.emergentmind.com/topics/query-guided-captions