SCRA-VQA: Enhanced Zero-shot VQA via Caption Reranking

• The paper demonstrates a novel zero-shot VQA pipeline that transforms images into curated textual evidence using question-relevant caption summarization and reranking.
• SCRA-VQA employs a two-stage approach—image-to-text transformation followed by caption refinement—to enhance reasoning for knowledge-based VQA benchmarks like OK-VQA and A-OKVQA.
• Empirical results indicate that filtering noisy captions and providing contextual QA examples significantly improve the performance of frozen LLMs in caption-augmented VQA tasks.

Searching arXiv for the cited work to ground the article in current paper metadata. {"query":"SCRA-VQA Summarized Caption-Rerank for Augmented LLMs in Visual Question Answering arXiv (Zhang et al., 25 Sep 2025) PromptCap (Hu et al., 2022) Generating Question Relevant Captions to Aid Visual Question Answering (Wu et al., 2019)", "max_results": 10} {"query":"(Zhang et al., 25 Sep 2025)", "max_results": 5} Summarized Caption-Rerank Augmented VQA (SCRA-VQA) is a zero-shot knowledge-based visual question answering pipeline that improves frozen LLM performance by transforming an image into question-relevant textual evidence, generating contextual question-answer examples from that evidence, summarizing and reranking the resulting captions, and then answering from the curated prompt context rather than from raw captions alone (Zhang et al., 25 Sep 2025). It was introduced for knowledge-based VQA settings such as OK-VQA and A-OKVQA, where answers often require external or commonsense knowledge beyond direct visual recognition, and where naive image-to-caption conversion can leave the LLM with excessive irrelevant information and weak task grounding (Zhang et al., 25 Sep 2025).

1. Problem setting and design rationale

SCRA-VQA is formulated for knowledge-based visual question answering, in which an image and a natural-language question must be mapped to an answer that often depends on information not explicitly visible in the image. The method focuses on OK-VQA and A-OKVQA, and its motivating diagnosis is that caption-based frozen-LLM pipelines fail for two coupled reasons: captions frequently contain excessive noise irrelevant to the question, and LLMs generally do not comprehend VQA tasks well when given only loosely structured image-derived text (Zhang et al., 25 Sep 2025).

The method therefore treats the image-to-text interface as the central bottleneck. Rather than using a single whole-image caption, it first identifies question-relevant regions, then captions those regions, then refines the resulting text through two operations: summarization, which condenses the main content of the captions, and reranking, which reorders captions by question relevance. It also injects contextual similar question-answer pairs derived from captions so that the frozen answering model receives not only image evidence in text form, but also exemplars of how that evidence can support VQA-style reasoning (Zhang et al., 25 Sep 2025).

This design places SCRA-VQA within a broader line of caption-augmented VQA methods, but with a more explicit emphasis on prompt structuring. The key claim is not merely that captions help VQA. It is that captions must be filtered, ordered, and contextualized before they can reliably support a frozen LLM on knowledge-heavy visual reasoning tasks (Zhang et al., 25 Sep 2025).

2. Pipeline architecture and component functions

SCRA-VQA is organized into two macro-stages: image-to-text transformation and image caption refinement. The first stage uses BLIP to identify question-relevant image regions and generate captions for sampled segments. The second stage derives similar QA examples from captions, summarizes caption content with Gemma2b, reranks captions with BGE, and constructs the final prompt for a frozen answering LLM, primarily OPT 6.7B (Zhang et al., 25 Sep 2025).

Component	Function	Model or mechanism
Image-to-text transformation	Question-guided region selection and region captioning	BLIP 446M with Image-Text Encoder and GradCAM-style relevance
Similarity question generation	Caption-to-QA conversion for contextual examples	VQ2A-style three-step process
Caption summarization	Distill caption set into concise context	Gemma2b
Caption rerank	Score caption relevance to the question	BGE-Rerank-large, BGE-Rerank-base, CohereAI compared
Answer generation	Final KB-VQA inference from curated prompt	OPT 6.7B

The question-guided visual selection step computes cross-attention between question and image features and then derives patch relevance scores. The paper gives the cross-attention and GradCAM-style relevance as:

$$W=\mathrm{softmax}\left(\frac{f_qW_Q(W_K)^\top(f_v)^\top}{\sqrt{D_q}}\right),$$

$$R=\frac{1}{G}\sum_{g=1}^G\sum_{l=1}^N\min\left(0,\frac{\partial\text{sim}(v,q)}{\partial W_{l}^g}\right)W_{l}^g.$$

After computing $R$, the system performs Top-K sampling over image segments; the implementation reports $K=20$ sampled segments (Zhang et al., 25 Sep 2025).

Caption reranking is formulated as a one-to-one question-caption scoring problem. For each caption $c$ and question $q$, the relevance score is:

$$s=\text{score}(q,c)=\mathbf{v}_{p}^{\top} cls \left(\text{BGE}\left(\text{concat}(q,c)\right)\right).$$

Captions are then sorted by score, and the method text states that the top-5 captions are retained for the final prompt (Zhang et al., 25 Sep 2025).

The similarity-question-generation module is adapted from VQ2A. Its reported procedure is: extract candidate answers from captions using syntactic parsing, generate contextually consistent questions via a neural model, and filter question-answer pairs through consistency validation. The generated QA examples are not the final output; they are prompt-side scaffolding for the answering LLM (Zhang et al., 25 Sep 2025).

3. Caption refinement, prompt construction, and prompt ordering

The “summarized caption-rerank” part of SCRA-VQA consists of two complementary refinements. First, Gemma2b summarizes the generated captions to distill essential information and reduce redundancy. Second, BGE reranking removes captions that are less relevant to the target question. The method argues that these two steps are needed because even question-guided region captioning can still produce too many captions, and because a frozen LLM may fail to infer the relations among them if they are presented as an unstructured list (Zhang et al., 25 Sep 2025).

The final prompt template is explicitly organized around instruction plus context. The paper gives the structure as:

Please reason the answers to the questions according to the contexts. Contexts: Rerank_Caption: $e$ + Summary: $s$ + Similar_Question: $q^i$ + Similar_Answer: $$R=\frac{1}{G}\sum_{g=1}^G\sum_{l=1}^N\min\left(0,\frac{\partial\text{sim}(v,q)}{\partial W_{l}^g}\right)W_{l}^g.$$0 + Question: $$R=\frac{1}{G}\sum_{g=1}^G\sum_{l=1}^N\min\left(0,\frac{\partial\text{sim}(v,q)}{\partial W_{l}^g}\right)W_{l}^g.$$1 + Answer:

This arrangement is intended to make the LLM interpret the input as a reasoning problem grounded in curated image-derived evidence rather than as generic text continuation (Zhang et al., 25 Sep 2025).

Prompt ordering is treated as an ablation variable, not as a fixed stylistic choice. The paper compares repetitive templates such as S+C+QA and C+QA+S against grouped templates such as MC+MQA+S, S+MC+MQA, and MC+S+MQA, where MC denotes multi-caption, MQA denotes multi-QA, and S denotes summary. The best-performing format is MC+S+MQA, with reported accuracies of 38.8 on OK-VQA and 34.6 on A-OKVQA (Zhang et al., 25 Sep 2025).

The caption-count and summary-length studies show that prompt compression is not monotonic. The paper reports that BGE-Rerank-large is the best reranker among those tested; that the best caption quantity occurs at 20 on OK-VQA and 5 on A-OKVQA; that Gemma is the best summarizer among Gemma, GPT, and Llama; and that summary length is best at 140 on OK-VQA and 100 on A-OKVQA, while 160 generally hurts performance by reintroducing irrelevant information (Zhang et al., 25 Sep 2025). This indicates that SCRA-VQA treats prompt composition as a capacity-allocation problem: too little text omits evidence, and too much text restores the noise it was designed to remove.

4. Empirical performance and ablation structure

Using a frozen 6.7B LLM setup, SCRA-VQA reports 38.8% accuracy on OK-VQA and 34.6% on A-OKVQA (Zhang et al., 25 Sep 2025). In the paper’s comparison table, this exceeds Img2Prompt 7B at 38.2 and 33.3, and PNP-VQA 11B at 35.9 and 33.8, while PICa 175B is reported at 17.7 on OK-VQA (Zhang et al., 25 Sep 2025).

Method	OK-VQA	A-OKVQA
PICa 175B	17.7	–
PNP-VQA 11B	35.9	33.8
Img2Prompt 7B	38.2	33.3
SCRA-VQA 6.7B	38.8	34.6

The component ablation is especially revealing because it isolates the role of each prompt ingredient. Starting from instruction only, performance is 15.21 on OK-VQA and 2.33 on A-OKVQA. Adding captions raises this to 35.32 and 29.50. Adding question-answer pairs yields 38.29 and 33.32. Replacing raw captions with summary plus QA gives 38.00 and 33.42. The full design with reranked captions, summary, and QA pairs reaches 38.82 and 34.61 (Zhang et al., 25 Sep 2025).

Configuration	OK-VQA	A-OKVQA
I	15.21	2.33
I + C	35.32	29.50
I + C + QAP	38.29	33.32
I + S + QAP	38.00	33.42
I + C + S + QAP	38.10	33.10
I + RC + S + QAP	38.82	34.61

These numbers show three consistent patterns. First, image-derived text is the dominant source of lift over instruction-only prompting. Second, similar QA examples materially improve task adaptation. Third, the best result is obtained only when caption relevance control and summarization are both present, rather than when either is used in isolation (Zhang et al., 25 Sep 2025).

The reported failure analysis also emphasizes that the method remains bottlenecked by upstream image-to-text quality. A qualitative failure case attributes an error to BLIP mistaking a small kite for a string, after which downstream summarization and reranking cannot recover the missing visual evidence (Zhang et al., 25 Sep 2025).

5. Intellectual lineage and relation to adjacent methods

SCRA-VQA extends a longer development in which captions are increasingly treated as question-conditioned evidence rather than as generic scene descriptions. “Generating Question Relevant Captions to Aid Visual Question Answering” showed that question-relevant captions, selected by an online gradient-based method and jointly trained with VQA, can improve VQA v2 performance to 68.4% on the test-standard split using a single model (Wu et al., 2019). “PromptCap: Prompt-Guided Task-Aware Image Captioning” later argued that generic single-sentence captions are a bottleneck for knowledge-based VQA, introduced prompt-guided caption generation conditioned on the question, and reported 60.4% on OK-VQA and 59.6% on A-OKVQA in a GPT-3 pipeline (Hu et al., 2022). “Enhancing Visual Question Answering through Question-Driven Image Captions as Prompts” further showed that keyword-guided question-driven captions improve zero-shot GQA over generic captions, from 48.06 to 49.50, and reduce “not mentioned / not visible” answers from 1.7% to 0.5% (Özdemir et al., 2024). Together, these papers supply the conceptual basis for the summarized-caption part of SCRA-VQA.

A second line of work established that captions can function as auxiliary evidence even when not directly optimized as the final answer channel. “Joint Image Captioning and Question Answering” used question-related captions as additional features and reported 65.8% on VQA v2 validation with generated captions and 69.1% with annotated captions (Wu et al., 2018). “Improving Visual Question Answering by Referring to Generated Paragraph Captions” introduced a Visual and Textual Question Answering model that fuses image features, generated paragraph captions, and object property sentences, reaching 46.86 test accuracy on its Visual Genome setup (Kim et al., 2019). “Leveraging Visual Question Answering for Image-Caption Ranking” is relevant in a different way: it uses VQA plausibility scores as semantic features for image-caption compatibility, showing that caption usefulness can be judged by structured QA consistency rather than by generic descriptiveness alone (Lin et al., 2016).

The rerank aspect of SCRA-VQA also sits within a broader family of verification-oriented systems. “VQA4CIR” converts a relative caption into a small set of yes/no QA constraints, evaluates each retrieved image with VQA, aggregates consistency as $$R=\frac{1}{G}\sum_{g=1}^G\sum_{l=1}^N\min\left(0,\frac{\partial\text{sim}(v,q)}{\partial W_{l}^g}\right)W_{l}^g.$$2, and reranks candidates through a penalty $$R=\frac{1}{G}\sum_{g=1}^G\sum_{l=1}^N\min\left(0,\frac{\partial\text{sim}(v,q)}{\partial W_{l}^g}\right)W_{l}^g.$$3 (Feng et al., 2023). “ScaleCap” approaches captioning itself as an inference-time evidence extraction and calibration problem, using heuristic question answering and contrastive sentence rating to produce richer and less hallucinated captions, and explicitly evaluates replacing images with captions in downstream VQA-like settings (Xing et al., 24 Jun 2025). “Enhanced Multimodal RAG-LLM for Accurate Visual Question Answering” does not use captions, but it serializes scene-graph evidence into textual chunks and retrieves the top-4 question-relevant chunks before answering, reinforcing the same architectural principle that grounded intermediate text improves VQA when raw end-to-end perception is brittle (Xue et al., 2024). “Solution for Meta KDD Cup'25” similarly operationalizes a three-step query-generation, retrieval, reranking, and answer-generation pipeline for vision question answering (Zhang et al., 29 Jul 2025).

This lineage suggests that SCRA-VQA is best understood as a modular synthesis. It inherits question-aware captioning from earlier caption-to-VQA work, borrows rerank logic from retrieval and verification pipelines, and combines these with prompt engineering for frozen LLMs rather than with expensive end-to-end multimodal training.

6. Scope, limitations, and broader significance

SCRA-VQA is explicitly a zero-shot, frozen-LLM augmentation method rather than an end-to-end multimodal architecture. It reuses pretrained BLIP, Gemma2b, BGE reranking, and OPT 6.7B, and the paper emphasizes that it avoids expensive end-to-end training while remaining runnable on 2 × Nvidia RTX4090 GPUs (Zhang et al., 25 Sep 2025). A common misconception is therefore to treat it as another large multimodal model; in the paper’s own framing, it is instead a prompt-and-pipeline strategy for upgrading a frozen LLM with curated image-derived text (Zhang et al., 25 Sep 2025).

Its main limitation is dependence on the fidelity of the image-to-text stage. If region selection or caption generation omits or mislabels the answer-bearing cue, later summarization and reranking can only reorganize an already defective evidence pool. The paper also shows sensitivity to prompt structure, caption count, and summary length, which means the method’s reported gains depend on careful prompt budgeting rather than on a universally robust caption abstraction (Zhang et al., 25 Sep 2025).

A second limitation is that the final reasoning stage remains text-only. This means that some visual nuances may be lost during conversion, and it helps explain why the system is targeted at KB-VQA rather than at general VQAv2-style direct recognition; the paper explicitly does not evaluate on VQAv2 because its emphasis is external knowledge and logical reasoning rather than pure visual recognition (Zhang et al., 25 Sep 2025).

The broader significance of SCRA-VQA lies in how it operationalizes a specific view of vision-language interfacing: the crucial object is not the caption as such, but the curated caption context. PromptCap had already argued that the bottleneck in knowledge-based VQA is often the lossy text interface between image and LLM rather than the LLM’s world knowledge (Hu et al., 2022). SCRA-VQA makes that claim concrete by adding summarization, reranking, and contextual QA exemplars on top of region captions, and by showing that these refinements can outperform strong frozen-LLM baselines on OK-VQA and A-OKVQA without end-to-end multimodal retraining (Zhang et al., 25 Sep 2025).

A plausible implication is that future systems in this family will further tighten the connection between caption utility and downstream answerability, either by stronger inference-time caption candidate selection or by more explicit answer-aware reranking. That implication is consistent with the surrounding literature, but the paper’s direct contribution is narrower: it demonstrates that for KB-VQA, the decisive improvement often comes not from generating more text, but from presenting less noisy, better ordered, and more task-shaped text to the answering LLM (Zhang et al., 25 Sep 2025).