Question-Based Captioning: Bridging Vision & Reasoning
- Question-Based Captioning (QBC) is a method where caption generation is guided by questions, ensuring the output emphasizes evidence crucial for answering queries.
- It integrates techniques like visual grounding, iterative prompting, and synthetic QA, optimizing captions for relevance and faithfulness.
- QBC enhances multimodal reasoning in systems such as VQA and video QA by structuring captions as focused evidence for downstream decision-making.
Question-Based Captioning (QBC) denotes a family of vision-language formulations in which caption generation is shaped by a question, a distribution of anticipated questions, or an iterative questioning process, so that the resulting text emphasizes evidence useful for answering rather than producing a generic scene summary. In this literature, the caption is typically treated as an intermediate semantic bottleneck between perception and reasoning: it may be directly conditioned on a question prompt, induced by question-aware grounding and filtering, synthesized from question-answer interaction, or optimized for downstream answerability. The paradigm appears in image, video, audio, and remote-sensing settings, and is especially prominent in zero-shot and knowledge-based VQA pipelines where a text-only or multimodal LLM reasons over captions rather than raw pixels (Hu et al., 2022, Moradi et al., 25 May 2025, Romero et al., 2024).
1. Conceptual scope and neighboring formulations
QBC is best understood as a spectrum rather than a single canonical task. At one end are direct formulations of , where an image or video and a question jointly determine a caption. At another are distributional variants in which captions are optimized for the likely question distribution of a user population rather than for one explicit question, and interactive variants in which a system asks questions in order to construct or refine a caption. The common principle is that captions are evaluated by their usefulness, relevance, or faithfulness with respect to questions, not only by generic overlap with reference descriptions (Fisch et al., 2020, Zhu et al., 2023).
This distinguishes QBC from standard captioning, which ordinarily models a generic description of salient content, and from VQA, which ordinarily returns a short answer rather than a coherent descriptive sentence. It also distinguishes QBC from conventional controllable captioning based on object tokens or regions: several works argue that natural-language questions are a more realistic control interface, especially when the desired information is unknown in advance or dispersed across scene text, attributes, relations, or temporal events (Hu et al., 2021).
A concise way to organize the field is to separate direct question-conditioned captioning from adjacent formulations that nevertheless share QBC’s content-selection logic.
| Formulation | Representative work | Defining mechanism |
|---|---|---|
| Direct question-conditioned captioning | "PromptCap: Prompt-Guided Task-Aware Image Captioning" (Hu et al., 2022) | Caption prompt contains the downstream question |
| Question-guided captioning for VQA | "Generating Question Relevant Captions to Aid Visual Question Answering" (Wu et al., 2019) | Generate captions targeted to help answer a specific visual question |
| Interactive questioning for caption enrichment | "ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptions" (Zhu et al., 2023) | Generate a caption by iterative question generation, answering, and summarization |
| Question-controlled text-aware captioning | "Question-controlled Text-aware Image Captioning" (Hu et al., 2021) | Questions control which scene texts and object-text relations appear in the caption |
| QA-based caption evaluation | "QACE: Asking Questions to Evaluate an Image Caption" (Lee et al., 2021) | Convert a caption into answer-aware questions and validate them against text or image |
A recurring implication is that QBC is as much about content selection as about language generation. Several papers treat captions not as ends in themselves but as structured evidence packages for reasoning, retrieval, accessibility, or interaction.
2. Operational mechanisms
The central technical question in QBC is how a question enters the captioning pipeline. The literature instantiates this at several levels: prompt conditioning, visual grounding, question-guided attention, multi-caption generation, semantic reranking, synthetic QA generation, and iterative dialogue. The most explicit recent formulation is the four-stage GC-KBVQA pipeline, where the question determines what gets grounded, what gets captioned, and what gets selected (Moradi et al., 25 May 2025).
In GC-KBVQA, Stage 1 is Keyword-Guided Visual Grounding. The question is distilled by KeyBERT; only words and phrases with relevance score are kept. These keywords prompt Grounding DINO, detections with confidence are retained, and overlapping boxes are pruned so that if the intersection exceeds $0.9$, only the larger box remains. Bounding boxes are then expanded to preserve contextual cues. Stage 2 is Caption Generation and Filtering: grounded regions are captioned by both LLaVA and InstructBLIP, multiple captions are generated per region, the question is distilled with the exact prompt "Determine the main idea of this question in short:", captions and distilled question are embedded with all-MiniLM-L6-v2 into a 384-dimensional dense vector space, ranked by cosine similarity, and the top $3$ captions are kept. Stage 3 generates exactly $2$ synthetic QA pairs from the top three captions with LLaMA-3-8B-Instruct. Stage 4 performs answer generation from the original question, the top three captions, and the two QA pairs (Moradi et al., 25 May 2025).
A closely related design appears in GeReA, where question-aware prompt captions are generated from question-relevant image regions and question-specific manual prompts. Region relevance is derived from the Image grounded Text Encoder in BLIP-2 via cross-attention and GradCAM-style weighting, then each selected region is paired with six prompts spanning unconstrained description, noun-verb phrase conditioned description, question inquiry, and question-relevant description. The result is a set of answer-bearing or explanation-bearing captions rather than a single generic sentence (Ma et al., 2024).
Prompt-only conditioning also remains effective. PromptCap simply concatenates the question prompt and image representation in OFA and trains on synthesized task-aware captions, while Q-ViD applies the exact frame-level instruction "Provide a detailed description of the image related to the question: {Q}" to InstructBLIP for each sampled frame, then concatenates the resulting descriptions in temporal order (Hu et al., 2022, Romero et al., 2024).
Question-aware control can also be imposed at the representation level rather than only in the prompt. Qc-TextCap’s GQAM first fuses object and OCR regions with a geometry-informed visual encoder, then applies a Question-guided Encoder in which question token embeddings query the visual representation to select the most relevant object and scene-text features before decoding a caption over a joint fixed-plus-OCR vocabulary (Hu et al., 2021).
These mechanisms differ in engineering detail, but they converge on one principle: QBC rarely succeeds by one-shot caption generation alone. The effective systems either localize, filter, or iteratively interrogate evidence before realizing the final text.
3. Image-based lineages
The image-based literature shows a clear progression from joint captioning-and-VQA systems toward explicit question-conditioned captioning. Early work established that question-related captions improve answering performance even when the primary benchmark is VQA rather than caption quality. "Joint Image Captioning and Question Answering" uses question and image features to generate question-related captions and reports 65.8% on VQA v2 validation using generated captions, 69.1% using annotated captions, 68.4% on the test-standard set, and 69.7% with an ensemble of 10 models (Wu et al., 2018).
"Generating Question Relevant Captions to Aid Visual Question Answering" makes this coupling more explicit. It selects question-relevant human captions using an online gradient-based method, generates question-relevant captions, embeds them, and uses them both as semantic features and to adjust visual attention. On VQA v2 test-standard it reports 68.37% for a single model and 69.66% for an ensemble, and its ablation shows that question-relevant generated captions outperform generic Up-Down captions for VQA (Wu et al., 2019).
Prompt-based QBC becomes more explicit in PromptCap. Its task-aware prompt contains the downstream question, and it is trained on 443,757 examples synthesized from VQAv2 on COCO 2014 train using GPT-3 plus 20 human-written examples. For each question-answer pair, GPT-3 samples 5 candidate captions; candidates are filtered by soft VQA accuracy and then by CIDEr tie-break. In the downstream GPT-3 QA pipeline, PromptCap reports 60.4% on OK-VQA and 59.6% on A-OKVQA, outperforming a matched generic-caption baseline (Hu et al., 2022).
GeReA and GC-KBVQA then push QBC into knowledge-based and zero-shot regimes. GeReA’s question-aware prompt captions yield 66.5% on OK-VQA and 63.3% on A-OKVQA, and the paper’s ablations show that generic captions have very low answer hit rate relative to question-aware prompt captions (Ma et al., 2024). GC-KBVQA, although framed as KB-VQA, is operationally a QBC pipeline centered on question-aware caption generation and filtering; on OK-VQA, A-OKVQA, and VQAv2 it reports 54.57, 53.87, and 67.96, respectively, with ablations showing that No Caption drops to 18.81 / 11.01 / 27.85, 1 Caption yields 40.38 / 47.51 / 53.15, 2 Captions yields 42.62 / 51.97 / 58.83, and 3 Captions gives the best balanced result (Moradi et al., 25 May 2025).
A useful historical summary is that image-based QBC shifted from joint auxiliary captioning for VQA to question-conditioned caption generation as the primary visual-language bridge. The question stopped being only a downstream consumer of the caption and became the signal that selects regions, prompts captioners, filters outputs, and structures final reasoning.
| Paper | Question-conditioning mechanism | Reported result |
|---|---|---|
| "Joint Image Captioning and Question Answering" (Wu et al., 2018) | Question and image features generate question-related captions | 65.8% val with generated captions; 69.1% with annotated captions |
| "Generating Question Relevant Captions to Aid Visual Question Answering" (Wu et al., 2019) | Question-attended image features and gradient-based caption selection | 68.37% VQA v2 test-standard single model |
| "PromptCap" (Hu et al., 2022) | Prompt contains the question; OFA fine-tuned on synthesized task-aware captions | 60.4% OK-VQA; 59.6% A-OKVQA |
| "GeReA" (Ma et al., 2024) | Question-relevant regions + six manual prompts produce question-aware prompt captions | 66.5% OK-VQA; 63.3% A-OKVQA |
| "GC-KBVQA" (Moradi et al., 25 May 2025) | Keyword-guided grounding, multi-caption generation, semantic ranking, synthetic QA pairs | 54.57 / 53.87 / 67.96 on OK-VQA / A-OKVQA / VQAv2 |
4. Video, audio, and other multimodal extensions
Video QBC makes the question-aware bottleneck more explicit because temporal redundancy, action transitions, and distractor events make generic captioning especially lossy. Q-ViD is one of the clearest direct formulations: it uniformly samples 64 frames per video, prompts InstructBLIP on each frame with the target question, limits each caption to 30 tokens maximum, uses , concatenates captions in temporal order, and performs multiple-choice reasoning with Flan-T5. Its prompt ablation shows that question-dependent prompts outperform generic ones for the larger model by +3.5 on NExT-QA and +4.2 on STAR, and the full system reaches 66.3 on NExT-QA, 45.7 on STAR, 71.4 on How2QA, 41.0 on TVQA, and 63.6 on IntentQA (Romero et al., 2024).
The 2025 VideoQA synthesis paper explicitly names Question-Based Captioning (QBC) as one of its two supervisory paradigms. There, QBC generates a natural-language rationale for each video-question-answer triple using the prompt: “Given a video, a question, and its answer, generate a natural language caption that highlights the visual content most relevant to justifying the answer. The caption should be a descriptive proof grounded in visual evidence, NOT a direct restatement of the answer.” Human evaluation gives QBC rationales 4.35 ± 0.48 for factual consistency, 4.38 ± 0.52 for visual grounding, and 4.91 ± 0.19 for fluency. Scaling QBC synthesized from NExT-QA from 3.5k to 29k rationale samples improves cross-dataset STAR performance from 66.5% to nearly 70.0% (Liang et al., 29 Sep 2025).
Some adjacent systems are not direct QBC but are structurally relevant. QCaption repeats the same prompt over sampled frames, obtains frame-level image-LMM outputs, and then aggregates them with an LLM; it reports up to 44.2% improvement in video captioning and 48.9% in video QA relative to Video-LLaVA, suggesting that question-aware framewise evidence extraction plus text-level fusion is a practical precursor to video QBC (Wang et al., 10 Jan 2026). VersaVid-R1 is likewise not explicit QBC, but it unifies multiple-choice QA, open-ended QA, and captioning under a Reason-Then-Respond protocol with > and <answer> tags, and its caption reward combines AutoDQ with a keyword reward that promotes temporal coherence and suppresses speculative content (Chen et al., 10 Jun 2025). iPerceive is earlier and more indirect, but its combination of dense video captioning, question-aware VideoQA, and common-sense reasoning is a clear foundation for explanatory or causal QBC in video (Chadha et al., 2020).
Beyond video, MU-LLaMA shows that QA-enhanced captioning extends to music. It generates QA pairs from MusicCaps captions and MagnaTagATune tags, trains a unified multimodal LLaMA with a frozen MERT encoder and a trainable music understanding adapter, and reports improvements in both music QA and music captioning. This suggests that QBC-style supervision can be useful even where captions are scarce and questions decompose attributes such as mood, tempo, instrumentation, and overall tone (Liu et al., 2023).
5. Data, supervision, and evaluation
A persistent problem in QBC is the scarcity of gold question-aligned captions. Much of the field therefore relies on synthetic supervision, weak alignment, or QA-based evaluation. PromptCap synthesizes question-aware captions from existing COCO captions and VQA question-answer pairs without collecting a dedicated QBC dataset, while the 2019 gradient-alignment model induces question-caption alignment online from generic image captions rather than from human-labeled question-specific captions (Hu et al., 2022, Wu et al., 2019).
Two datasets explicitly operationalize question-conditioned captioning. Qc-TextCap introduces ControlTextCaps and ControlVizWiz by converting text-aware captioning data into tuples . ControlTextCaps contains 65,192 train tuples, 4,992 validation tuples, and 5,000 test tuples; ControlVizWiz contains 25,339 train tuples, 1,962 validation tuples, and 1,956 test tuples (Hu et al., 2021). In remote sensing, JL1-CC&QA extends JL1-CD with 17,021 selected change captions and 20,060 QA pairs over 5,000 bi-temporal image pairs, with eight question types: YES/NO, WHAT, WHERE, HOW MUCH, BEFORE/AFTER, CAUSE, DETAIL, and COMPARE. Although not a QBC model paper, it provides aligned caption-and-QA supervision on the same grounded image set (Liu et al., 30 Jun 2026).
CapWAP reframes supervision at the objective level rather than the dataset level. Instead of fitting captions to references, it optimizes captions against downstream answerability under a target question distribution . Evaluation is likewise QA-based: on held-out user questions, it measures whether a QA model can answer from the generated caption alone, reporting EM and token-level F1 on converted VQA, GQA, Visual7W, and VizWiz datasets (Fisch et al., 2020).
QACE pushes evaluation further by turning a candidate caption into a set of answer-aware questions and validating them on a reference caption or the image itself. Its core score is
and its reported results include 75.1 on Pascal50s for QACE-Ref and 70.0 for QACE-Img (Lee et al., 2021). This evaluation view aligns naturally with QBC because it measures whether the content asserted by a caption survives question-level probing.
Human-centered evaluation also remains important because standard caption metrics can be misaligned with question-focused informativeness. ChatCaptioner therefore emphasizes human preference and object coverage: across COCO, WikiArt, and Conceptual Captions, it receives 65% of votes on average for providing the richest image information, compared with 18.8% for BLIP-2 and 16.2% for ground-truth captions; on Pascal VOC it covers 586 of 1154 objects (50.8%) versus 383 (33.2%) for BLIP-2 alone (Zhu et al., 2023).
Resource or protocol QBC role Reported scale or score ControlTextCaps / ControlVizWiz (Hu et al., 2021) Question-controlled text-aware caption datasets 65,192 / 25,339 train tuples JL1-CC&QA (Liu et al., 30 Jun 2026) Aligned caption + QA benchmark for remote-sensing change understanding 5,000 image pairs; 17,021 captions; 20,060 QA pairs CapWAP (Fisch et al., 2020) QA-driven caption objective and evaluation EM/F1 from held-out QA over generated captions QACE (Lee et al., 2021) Question-based caption evaluation 75.1 QACE-Ref; 70.0 QACE-Img on Pascal50s ChatCaptioner (Zhu et al., 2023) Human-preference and object-coverage evaluation of question-driven caption enrichment 65% average human preference; 53.0% improvement in object coverage 6. Limitations, misconceptions, and open directions
A common misconception is that QBC is merely generic captioning with a question appended. The evidence argues otherwise. In GC-KBVQA, performance depends not only on generating captions but on removing irrelevant words via KeyBERT, excluding irrelevant image regions via grounding, semantically ranking captions against a distilled question, and limiting the final prompt to the top three captions and two QA pairs; the authors explicitly state that adding too many captions introduces less question-focused content and degrades reasoning (Moradi et al., 25 May 2025). GeReA reports that using only one manual prompt lowers accuracy by at least 4.96% relative to six prompts, indicating that multi-view prompting can be decisive (Ma et al., 2024).
Hallucination remains a central failure mode. GC-KBVQA’s appendix includes examples where captions inconsistently identify the same fruit as a cherry or a grape and the final answer becomes Cherry, and examples where plausible but image-unsupported knowledge such as “100 lbs” or “1 to 2 liters” is injected into captions (Moradi et al., 25 May 2025). ChatCaptioner shows cascading error from the answerer: only about 66.7% of ChatGPT’s questions are correctly answered by BLIP-2, and among incorrect final captions, 94% are attributed to BLIP-2’s wrong answers (Zhu et al., 2023). The video QBC synthesis paper identifies a distinct failure mode it calls “justified fabrication,” where answer-conditioned rationale generation can invent plausible-looking visual details to support the correct answer (Liang et al., 29 Sep 2025).
Over-focusing is another recurrent issue. Q-ViD shows that question-dependent prompting helps the larger model but not the smaller XL variant, which the authors attribute to weaker instruction following (Romero et al., 2024). The GQA-based question-driven captioning study reports that question-driven captions improve overall accuracy from 48.06 to 49.50 and reduce “not mentioned/not visible” answers from 1.7% to 0.5%, but they hurt on global questions because keyword emphasis can suppress whole-scene information (Özdemir et al., 2024). QCaption likewise finds that LLM aggregation helps on YouCook2 and ActivityNet-QA but hurts on MSR-VTT, where short captions are apparently over-abstracted by the aggregator (Wang et al., 10 Jan 2026).
Another limitation is that not all papers labeled here are “pure” QBC systems. CapWAP is a distributional, question-evaluable captioning framework rather than per-question caption generation; QACE is an evaluation method; JL1-CC&QA is a benchmark; iPerceive, QCaption, and VersaVid-R1 are foundations or adjacent architectures rather than explicit question-conditioned caption generators (Fisch et al., 2020, Lee et al., 2021, Liu et al., 30 Jun 2026, Chadha et al., 2020, Wang et al., 10 Jan 2026, Chen et al., 10 Jun 2025). This suggests that QBC should be treated as a broader research program encompassing generation, supervision, reasoning, and evaluation.
The open technical direction is therefore not only to generate better captions, but to better control relevance, faithfulness, and compression. The literature repeatedly indicates that question-aware localization, question-aware caption selection, multi-view prompting, synthetic QA scaffolds, and QA-based evaluation are as important as the caption decoder itself. A plausible implication is that mature QBC systems will combine targeted grounding, heterogeneous caption candidates, answer-aware verification, and explicit safeguards against speculative world knowledge rather than relying on a single monolithic captioning model.