---
title: 'Question-Based Captioning: Bridging Vision & Reasoning'
url: https://www.emergentmind.com/topics/question-based-captioning-qbc
type: topic
---

# Question-Based Captioning: Bridging Vision & Reasoning

Question-Based Captioning (QBC) denotes a family of vision-language formulations in which caption generation is shaped by a question, a distribution of anticipated questions, or an iterative questioning process, so that the resulting text emphasizes evidence useful for answering rather than producing a generic scene summary. In this literature, the caption is typically treated as an intermediate semantic bottleneck between perception and reasoning: it may be directly conditioned on a question prompt, induced by question-aware grounding and filtering, synthesized from question-answer interaction, or optimized for downstream answerability. The paradigm appears in image, video, audio, and remote-sensing settings, and is especially prominent in zero-shot and knowledge-based VQA pipelines where a text-only or multimodal language model reasons over captions rather than raw pixels [2211.09699] [2505.19354] [2402.10698].

## 1. Conceptual scope and neighboring formulations

QBC is best understood as a spectrum rather than a single canonical task. At one end are direct formulations of \(P(S \mid I, Q)\), where an image or video and a question jointly determine a caption. At another are distributional variants in which captions are optimized for the likely question distribution of a user population rather than for one explicit question, and interactive variants in which a system asks questions in order to construct or refine a caption. The common principle is that captions are evaluated by their usefulness, relevance, or faithfulness with respect to questions, not only by generic overlap with reference descriptions [2011.04264] [2303.06594].

This distinguishes QBC from standard captioning, which ordinarily models a generic description of salient content, and from VQA, which ordinarily returns a short answer rather than a coherent descriptive sentence. It also distinguishes QBC from conventional controllable captioning based on object tokens or regions: several works argue that natural-language questions are a more realistic control interface, especially when the desired information is unknown in advance or dispersed across scene text, attributes, relations, or temporal events [2108.02059].

A concise way to organize the field is to separate direct question-conditioned captioning from adjacent formulations that nevertheless share QBC’s content-selection logic.

| Formulation | Representative work | Defining mechanism |
|---|---|---|
| Direct question-conditioned captioning | "PromptCap: Prompt-Guided Task-Aware Image Captioning" [2211.09699] | Caption prompt contains the downstream question |
| Question-guided captioning for VQA | "Generating Question Relevant Captions to Aid Visual Question Answering" [1906.00513] | Generate captions targeted to help answer a specific visual question |
| Interactive questioning for caption enrichment | "ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptions" [2303.06594] | Generate a caption by iterative question generation, answering, and summarization |
| Question-controlled text-aware captioning | "Question-controlled Text-aware Image Captioning" [2108.02059] | Questions control which scene texts and object-text relations appear in the caption |
| QA-based caption evaluation | "QACE: Asking Questions to Evaluate an Image Caption" [2108.12560] | Convert a caption into answer-aware questions and validate them against text or image |

A recurring implication is that QBC is as much about **content selection** as about language generation. Several papers treat captions not as ends in themselves but as structured evidence packages for reasoning, retrieval, accessibility, or interaction.

## 2. Operational mechanisms

The central technical question in QBC is how a question enters the captioning pipeline. The literature instantiates this at several levels: prompt conditioning, visual grounding, question-guided attention, multi-caption generation, semantic reranking, synthetic QA generation, and iterative dialogue. The most explicit recent formulation is the four-stage GC-KBVQA pipeline, where the question determines **what gets grounded, what gets captioned, and what gets selected** [2505.19354].

In GC-KBVQA, Stage 1 is **Keyword-Guided Visual Grounding**. The question is distilled by KeyBERT; only words and phrases with relevance score \(> 0.4\) are kept. These keywords prompt Grounding DINO, detections with confidence \(> 0.25\) are retained, and overlapping boxes are pruned so that if the intersection exceeds \(0.9\), only the larger box remains. Bounding boxes are then expanded to preserve contextual cues. Stage 2 is **Caption Generation and Filtering**: grounded regions are captioned by both LLaVA and InstructBLIP, multiple captions are generated per region, the question is distilled with the exact prompt `"Determine the main idea of this question in short:"`, captions and distilled question are embedded with all-MiniLM-L6-v2 into a 384-dimensional dense vector space, ranked by cosine similarity, and the top \(3\) captions are kept. Stage 3 generates exactly \(2\) synthetic QA pairs from the top three captions with LLaMA-3-8B-Instruct. Stage 4 performs answer generation from the original question, the top three captions, and the two QA pairs [2505.19354].

A closely related design appears in GeReA, where question-aware prompt captions are generated from **question-relevant image regions** and **question-specific manual prompts**. Region relevance is derived from the Image grounded Text Encoder in BLIP-2 via cross-attention and GradCAM-style weighting, then each selected region is paired with six prompts spanning unconstrained description, noun-verb phrase conditioned description, question inquiry, and question-relevant description. The result is a set of answer-bearing or explanation-bearing captions rather than a single generic sentence [2402.02503].

Prompt-only conditioning also remains effective. PromptCap simply concatenates the question prompt and image representation in OFA and trains on synthesized task-aware captions, while Q-ViD applies the exact frame-level instruction `"Provide a detailed description of the image related to the question: {Q}"` to InstructBLIP for each sampled frame, then concatenates the resulting descriptions in temporal order [2211.09699] [2402.10698].

Question-aware control can also be imposed at the representation level rather than only in the prompt. Qc-TextCap’s GQAM first fuses object and OCR regions with a geometry-informed visual encoder, then applies a **Question-guided Encoder** in which question token embeddings query the visual representation to select the most relevant object and scene-text features before decoding a caption over a joint fixed-plus-OCR vocabulary [2108.02059].

These mechanisms differ in engineering detail, but they converge on one principle: QBC rarely succeeds by one-shot caption generation alone. The effective systems either localize, filter, or iteratively interrogate evidence before realizing the final text.

## 3. Image-based lineages

The image-based literature shows a clear progression from joint captioning-and-VQA systems toward explicit question-conditioned captioning. Early work established that question-related captions improve answering performance even when the primary benchmark is VQA rather than caption quality. "Joint Image Captioning and Question Answering" uses question and image features to generate question-related captions and reports **65.8%** on VQA v2 validation using generated captions, **69.1%** using annotated captions, **68.4%** on the test-standard set, and **69.7%** with an ensemble of 10 models [1805.08389].

"Generating Question Relevant Captions to Aid Visual Question Answering" makes this coupling more explicit. It selects question-relevant human captions using an online gradient-based method, generates question-relevant captions, embeds them, and uses them both as semantic features and to adjust visual attention. On VQA v2 test-standard it reports **68.37%** for a single model and **69.66%** for an ensemble, and its ablation shows that question-relevant generated captions outperform generic Up-Down captions for VQA [1906.00513].

Prompt-based QBC becomes more explicit in PromptCap. Its task-aware prompt contains the downstream question, and it is trained on **443,757 examples** synthesized from VQAv2 on COCO 2014 train using GPT-3 plus **20 human-written examples**. For each question-answer pair, GPT-3 samples **5 candidate captions**; candidates are filtered by soft VQA accuracy and then by CIDEr tie-break. In the downstream GPT-3 QA pipeline, PromptCap reports **60.4%** on OK-VQA and **59.6%** on A-OKVQA, outperforming a matched generic-caption baseline [2211.09699].

GeReA and GC-KBVQA then push QBC into knowledge-based and zero-shot regimes. GeReA’s question-aware prompt captions yield **66.5%** on OK-VQA and **63.3%** on A-OKVQA, and the paper’s ablations show that generic captions have very low answer hit rate relative to question-aware prompt captions [2402.02503]. GC-KBVQA, although framed as KB-VQA, is operationally a QBC pipeline centered on question-aware caption generation and filtering; on OK-VQA, A-OKVQA, and VQAv2 it reports **54.57**, **53.87**, and **67.96**, respectively, with ablations showing that **No Caption** drops to **18.81 / 11.01 / 27.85**, **1 Caption** yields **40.38 / 47.51 / 53.15**, **2 Captions** yields **42.62 / 51.97 / 58.83**, and **3 Captions** gives the best balanced result [2505.19354].

A useful historical summary is that image-based QBC shifted from **joint auxiliary captioning for VQA** to **question-conditioned caption generation as the primary visual-language bridge**. The question stopped being only a downstream consumer of the caption and became the signal that selects regions, prompts captioners, filters outputs, and structures final reasoning.

| Paper | Question-conditioning mechanism | Reported result |
|---|---|---|
| "Joint Image Captioning and Question Answering" [1805.08389] | Question and image features generate question-related captions | **65.8%** val with generated captions; **69.1%** with annotated captions |
| "Generating Question Relevant Captions to Aid Visual Question Answering" [1906.00513] | Question-attended image features and gradient-based caption selection | **68.37%** VQA v2 test-standard single model |
| "PromptCap" [2211.09699] | Prompt contains the question; OFA fine-tuned on synthesized task-aware captions | **60.4%** OK-VQA; **59.6%** A-OKVQA |
| "GeReA" [2402.02503] | Question-relevant regions + six manual prompts produce question-aware prompt captions | **66.5%** OK-VQA; **63.3%** A-OKVQA |
| "GC-KBVQA" [2505.19354] | Keyword-guided grounding, multi-caption generation, semantic ranking, synthetic QA pairs | **54.57 / 53.87 / 67.96** on OK-VQA / A-OKVQA / VQAv2 |

## 4. Video, audio, and other multimodal extensions

Video QBC makes the question-aware bottleneck more explicit because temporal redundancy, action transitions, and distractor events make generic captioning especially lossy. Q-ViD is one of the clearest direct formulations: it uniformly samples **64 frames** per video, prompts InstructBLIP on each frame with the target question, limits each caption to **30 tokens maximum**, uses \(top_p = 0.7\), concatenates captions in temporal order, and performs multiple-choice reasoning with Flan-T5. Its prompt ablation shows that question-dependent prompts outperform generic ones for the larger model by **+3.5** on NExT-QA and **+4.2** on STAR, and the full system reaches **66.3** on NExT-QA, **45.7** on STAR, **71.4** on How2QA, **41.0** on TVQA, and **63.6** on IntentQA [2402.10698].

The 2025 VideoQA synthesis paper explicitly names **Question-Based Captioning (QBC)** as one of its two supervisory paradigms. There, QBC generates a natural-language rationale for each video-question-answer triple using the prompt: “Given a video, a question, and its answer, generate a natural language caption that highlights the visual content most relevant to justifying the answer. The caption should be a descriptive proof grounded in visual evidence, NOT a direct restatement of the answer.” Human evaluation gives QBC rationales **4.35 ± 0.48** for factual consistency, **4.38 ± 0.52** for visual grounding, and **4.91 ± 0.19** for fluency. Scaling QBC synthesized from NExT-QA from **3.5k** to **29k** rationale samples improves cross-dataset STAR performance from **66.5%** to nearly **70.0%** [2509.24445].

Some adjacent systems are not direct QBC but are structurally relevant. QCaption repeats the same prompt over sampled frames, obtains frame-level image-LMM outputs, and then aggregates them with an LLM; it reports up to **44.2%** improvement in video captioning and **48.9%** in video QA relative to Video-LLaVA, suggesting that question-aware framewise evidence extraction plus text-level fusion is a practical precursor to video QBC [2601.06566]. VersaVid-R1 is likewise not explicit QBC, but it unifies multiple-choice QA, open-ended QA, and captioning under a Reason-Then-Respond protocol with `<think>` and `<answer>` tags, and its caption reward combines AutoDQ with a keyword reward that promotes temporal coherence and suppresses speculative content [2506.09079]. iPerceive is earlier and more indirect, but its combination of dense video captioning, question-aware VideoQA, and common-sense reasoning is a clear foundation for explanatory or causal QBC in video [2011.07735].

Beyond video, MU-LLaMA shows that QA-enhanced captioning extends to music. It generates QA pairs from MusicCaps captions and MagnaTagATune tags, trains a unified multimodal LLaMA with a frozen MERT encoder and a trainable music understanding adapter, and reports improvements in both music QA and music captioning. This suggests that QBC-style supervision can be useful even where captions are scarce and questions decompose attributes such as mood, tempo, instrumentation, and overall tone [2308.11276].

## 5. Data, supervision, and evaluation

A persistent problem in QBC is the scarcity of gold question-aligned captions. Much of the field therefore relies on synthetic supervision, weak alignment, or QA-based evaluation. PromptCap synthesizes question-aware captions from existing COCO captions and VQA question-answer pairs without collecting a dedicated QBC dataset, while the 2019 gradient-alignment model induces question-caption alignment online from generic image captions rather than from human-labeled question-specific captions [2211.09699] [1906.00513].

Two datasets explicitly operationalize question-conditioned captioning. Qc-TextCap introduces **ControlTextCaps** and **ControlVizWiz** by converting text-aware captioning data into tuples \(\langle I, C^{ini}, \mathcal{Q}, Y \rangle\). ControlTextCaps contains **65,192** train tuples, **4,992** validation tuples, and **5,000** test tuples; ControlVizWiz contains **25,339** train tuples, **1,962** validation tuples, and **1,956** test tuples [2108.02059]. In remote sensing, JL1-CC&QA extends JL1-CD with **17,021** selected change captions and **20,060** QA pairs over **5,000** bi-temporal image pairs, with eight question types: **YES/NO**, **WHAT**, **WHERE**, **HOW MUCH**, **BEFORE/AFTER**, **CAUSE**, **DETAIL**, and **COMPARE**. Although not a QBC model paper, it provides aligned caption-and-QA supervision on the same grounded image set [2606.31745].

CapWAP reframes supervision at the objective level rather than the dataset level. Instead of fitting captions to references, it optimizes captions against downstream answerability under a target question distribution \(p(\mathbf{q},\mathbf{a}\mid \mathbf{x})\). Evaluation is likewise QA-based: on held-out user questions, it measures whether a QA model can answer from the generated caption alone, reporting EM and token-level F1 on converted VQA, GQA, Visual7W, and VizWiz datasets [2011.04264].

QACE pushes evaluation further by turning a candidate caption into a set of answer-aware questions and validating them on a reference caption or the image itself. Its core score is
\[
QACE = \frac{\sum_{i=1}^{M} f(QA(q_i, x), QA(q_i, ctx))}{M},
\]
and its reported results include **75.1** on Pascal50s for QACE-Ref and **70.0** for QACE-Img [2108.12560]. This evaluation view aligns naturally with QBC because it measures whether the content asserted by a caption survives question-level probing.

Human-centered evaluation also remains important because standard caption metrics can be misaligned with question-focused informativeness. ChatCaptioner therefore emphasizes human preference and object coverage: across COCO, WikiArt, and Conceptual Captions, it receives **65%** of votes on average for providing the richest image information, compared with **18.8%** for BLIP-2 and **16.2%** for ground-truth captions; on Pascal VOC it covers **586** of **1154** objects (**50.8%**) versus **383** (**33.2%**) for BLIP-2 alone [2303.06594].

| Resource or protocol | QBC role | Reported scale or score |
|---|---|---|
| ControlTextCaps / ControlVizWiz [2108.02059] | Question-controlled text-aware caption datasets | **65,192 / 25,339** train tuples |
| JL1-CC&QA [2606.31745] | Aligned caption + QA benchmark for remote-sensing change understanding | **5,000** image pairs; **17,021** captions; **20,060** QA pairs |
| CapWAP [2011.04264] | QA-driven caption objective and evaluation | EM/F1 from held-out QA over generated captions |
| QACE [2108.12560] | Question-based caption evaluation | **75.1** QACE-Ref; **70.0** QACE-Img on Pascal50s |
| ChatCaptioner [2303.06594] | Human-preference and object-coverage evaluation of question-driven caption enrichment | **65%** average human preference; **53.0% improvement** in object coverage |

## 6. Limitations, misconceptions, and open directions

A common misconception is that QBC is merely generic captioning with a question appended. The evidence argues otherwise. In GC-KBVQA, performance depends not only on generating captions but on removing irrelevant words via KeyBERT, excluding irrelevant image regions via grounding, semantically ranking captions against a distilled question, and limiting the final prompt to the **top three captions** and **two QA pairs**; the authors explicitly state that adding too many captions introduces less question-focused content and degrades reasoning [2505.19354]. GeReA reports that using only one manual prompt lowers accuracy by at least **4.96%** relative to six prompts, indicating that multi-view prompting can be decisive [2402.02503].

Hallucination remains a central failure mode. GC-KBVQA’s appendix includes examples where captions inconsistently identify the same fruit as a cherry or a grape and the final answer becomes **Cherry**, and examples where plausible but image-unsupported knowledge such as “100 lbs” or “1 to 2 liters” is injected into captions [2505.19354]. ChatCaptioner shows cascading error from the answerer: only about **66.7%** of ChatGPT’s questions are correctly answered by BLIP-2, and among incorrect final captions, **94%** are attributed to BLIP-2’s wrong answers [2303.06594]. The video QBC synthesis paper identifies a distinct failure mode it calls “justified fabrication,” where answer-conditioned rationale generation can invent plausible-looking visual details to support the correct answer [2509.24445].

Over-focusing is another recurrent issue. Q-ViD shows that question-dependent prompting helps the larger model but not the smaller XL variant, which the authors attribute to weaker instruction following [2402.10698]. The GQA-based question-driven captioning study reports that question-driven captions improve overall accuracy from **48.06** to **49.50** and reduce “not mentioned/not visible” answers from **1.7%** to **0.5%**, but they hurt on **global** questions because keyword emphasis can suppress whole-scene information [2404.08589]. QCaption likewise finds that LLM aggregation helps on YouCook2 and ActivityNet-QA but hurts on MSR-VTT, where short captions are apparently over-abstracted by the aggregator [2601.06566].

Another limitation is that not all papers labeled here are “pure” QBC systems. CapWAP is a distributional, question-evaluable captioning framework rather than per-question caption generation; QACE is an evaluation method; JL1-CC&QA is a benchmark; iPerceive, QCaption, and VersaVid-R1 are foundations or adjacent architectures rather than explicit question-conditioned caption generators [2011.04264] [2108.12560] [2606.31745] [2011.07735] [2601.06566] [2506.09079]. This suggests that QBC should be treated as a broader research program encompassing generation, supervision, reasoning, and evaluation.

The open technical direction is therefore not only to generate better captions, but to better control **relevance**, **faithfulness**, and **compression**. The literature repeatedly indicates that question-aware localization, question-aware caption selection, multi-view prompting, synthetic QA scaffolds, and QA-based evaluation are as important as the caption decoder itself. A plausible implication is that mature QBC systems will combine targeted grounding, heterogeneous caption candidates, answer-aware verification, and explicit safeguards against speculative world knowledge rather than relying on a single monolithic captioning model.

Source: https://www.emergentmind.com/topics/question-based-captioning-qbc