OAD-Promoter: Enhancing Zero-Shot Visual QA

• The paper introduces OAD-Promoter, a framework that integrates multi-level visual descriptions and dynamic memory retrieval to mitigate language bias and enhance zero-shot visual question answering.
• It employs dedicated modules for global captioning and object-attribute example generation, leveraging models like BLIP2, VinVL, and T5 to achieve deep visual grounding.
• Experimental results show state-of-the-art performance on VQA benchmarks and robust out-of-distribution generalization compared to existing frozen LLM approaches.

The Object Attribute Description Promoter (OAD-Promoter) is a framework developed to address persistent challenges in zero-shot visual question answering (VQA) with LLMs. OAD-Promoter systematically mitigates the impact of language biases and enhances out-of-distribution (OOD) robustness by integrating multi-level visual content descriptions and a dynamic retrieval memory of object-attribute-based QA exemplars. The approach has demonstrated state-of-the-art results among frozen LLM pipelines across standard VQA, knowledge-based, and OOD benchmarks (Xu et al., 15 Nov 2025).

1. Motivation and Problem Setting

LLMs have become central tools in zero-shot and few-shot VQA, particularly for knowledge-heavy queries. However, two critical limitations inherited from large-scale pretraining persist:

• Language bias: LLMs exploit statistical question–answer associations, rather than grounding their predictions in the visual content (“shortcut learning”).
• OOD generalization: Even advanced LLMs show brittleness under domain shift, struggling to answer questions about unfamiliar visual or textual distributions.

OAD-Promoter is formulated to overcome these constraints by (1) forcing deeper visual grounding through fine-grained and global image descriptions, and (2) providing memory-assisted adaptation to new domains by selectively retrieving pertinent QA examples. This dual focus enables robust, bias-resistant reasoning even under significant visual or conceptual drift.

2. System Architecture and Modules

The OAD-Promoter framework consists of three sequentially connected modules: Object-concentrated Example Generation (OEG), Memory Knowledge Assistance (MKA), and OAD Prompt Construction.

Pipeline Overview

Textual mapping of data flow:

• Input: image $I_O$ and question $Q_O$.
• OEG Module: generates global caption $C_G$ and a set of object-attribute example triples $E_O$.
• MKA Module: retrieves a subset $E_S$ of QA memory examples from dynamic storage by feature similarity.
• OAD Prompt Construction: concatenates $I$, $C_G$, $E_O$, $E_S$, $Q_O$ into a structured prompt for the frozen LLM, which generates the final answer $A_O^{LLM}$.

2.1 Object-concentrated Example Generation (OEG)

• Global Captioning: BLIP2 (frozen) generates a holistic scene summary $C_G = \mathrm{BLIP2}(I_O)$.
• Object-Attribute Captioning: VinVL object detector localizes top-scoring objects; the VinVL captioner produces fine-grained attribute descriptions $C_j = \mathrm{VinVL}(I_{j})$ for each detected object crop $I_j$.
• Synthetic QA Triple Creation: Extract candidate answers $A_j$ via noun/verb/adjective phrase extractor. For each phrase $a$, prompt a T5-large question-generation model to synthesize a question $Q_j$ regarding $a$ in context $C_j$. Assemble each triple $E_{O,j} = (C_j, Q_j, a)$.

2.2 Memory Knowledge Assistance (MKA)

• Memory Structure and Update: The system maintains a pool $\mathcal{M}$ of triples $(C_i, Q_i, A_i)$, updating it incrementally after each inference by adding the latest object-attribute triples.
• Bias Mode Detection: Compares language-only QA inference $A_B = \mathrm{QA\_LM}(Q_O)$ against visually grounded UpDn prediction $A_O = \mathrm{UpDn}(I_O, Q_O)$. The system enters negative bias-correction mode if $A_O = A_B$ (indicating probable bias), else positive mode.
• Feature Similarity-Based Retrieval: For the new input, UpDn generates feature $f = \mathrm{UpDn.features}(I_O,Q_O)$. For each stored memory example, features $f_i$ are computed analogously. Cosine similarity $s_i$ quantifies relevance. Depending on the bias mode, either the most similar ($\mathrm{TopN}$, if positive) or least similar ($\mathrm{BottomN}$, if negative) examples are retrieved: $E_S$.

2.3 OAD Prompt Construction

A structured textual prompt for the LLM consists of:

1. Instruction $I$ (e.g., “Answer the question based on the following image descriptions and examples.”)
2. Global image description $C_G$
3. Interleaved set of $N = N_O + N_S$ QA examples (each as a contiguous block: context, question, answer)
4. Target question $Q_O$ The prompt is linearly concatenated to ensure compatibility with LLM context windows.

Token-level construction strategies empirically validated indicate grouping each example as a complete triple yields a 0.8% accuracy improvement compared to alternative prompt orderings.

3. End-to-End Inference Flow

Below is the pseudocode implementing OAD-Promoter, matching the documented architecture (Xu et al., 15 Nov 2025):

def OAD_Promoter_Inference(I_O, Q_O): # (1) OEG Module C_G = BLIP2(I_O) BBoxes = VinVL.detect(I_O) top_boxes = select_top(BBoxes) E_O = [] for b_j in top_boxes: C_j = VinVL.caption(crop(I_O, b_j)) A_j = ExtractPhrases(C_j) Q_j = T5_QG(prompt=(A_j, C_j)) E_O.append((C_j, Q_j, A_j)) # (2) MKA Module A_B = QA_LM(Q_O) A_O_est = UpDn.predict(I_O, Q_O) M_mode = 'Negative' if A_O_est == A_B else 'Positive' f = UpDn.features(I_O, Q_O) s_i_list = [] for E_i in M: f_i = UpDn.features(VinVL.crop_i, E_i.Q) s_i = cosine(f, f_i) s_i_list.append(s_i) if M_mode == 'Positive': idxs = TopN(s_i_list) else: idxs = BottomN(s_i_list) E_S = [M[idx] for idx in idxs] # (3) Update Memory M.update(E_O) # (4) Prompt Construction Prompt = concat(I, C_G, E_O, E_S, Q_O, separators=True) # (5) LLM Inference A_O_LLM = LLM.generate(Prompt) return A_O_LLM

4. Implementation Details

• Global Captioning: BLIP2 model, frozen.
• Object Detection and Captioning: VinVL, frozen for both bounding box detection and captioning.
• Question Generation: T5-large fine-tuned on SQuAD2.0, MultiRC, BookQA, CommonsenseQA, SocialIQa.
• VQA Model: UpDn (Bottom-Up/Top-Down), pretrained on VQA-v2 and Visual Genome, then fine-tuned on OKVQA (with test data exclusion).
• Language-Only QA Baseline: LMH off-shift QA head without image input.
• LLM Backend: GPT-3 (175B) and OPT (6.7B, 30B, 175B), all used as frozen models.
• Prompt Example Counts: Number of object examples $N_O=3$ and memory examples $N_S=3$; the memory bag can be seeded with $K=400$ prior examples.
• Context Window Adaptation: Example count tuned to LLM prompt window size.

5. Quantitative Results and Comparative Evaluation

OAD-Promoter has been evaluated on multiple VQA and OOD datasets using standard VQA “soft” accuracy. Performance is assessed against major baselines including Flamingo, VL-T5, FewVLM, PICa, Prophet, PromptCap, GRACE, and Img2LLM+RQP.

Method	VQA-v2 (zero-shot)	A-OKVQA	OKVQA
Img2LLM (+RQP)	59.35	43.61	45.57
OAD-Promoter (ours)	61.98 (+2.6)	41.71	45.61 (+0.04)

• On VQA-v2, OAD-Promoter achieves 61.98% accuracy, outperforming all compared frozen-LLM methods.
• Sets a new state of the art for zero-shot on OKVQA among frozen LLMs (45.61%).
• In few-shot OKVQA, OAD-Promoter (60.04%) remains highly competitive with Prophet (61.08%) and GRACE (60.29%).

For OOD evaluation on VQA-CP and GQA-OOD (GPT-4, few-shot):

• VQA-CP: 55.93% (GRACE: 57.61%)
• GQA-OOD: 50.21% (GRACE: 50.19%) This suggests strong OOD robustness, with negligible performance drop relative to baselines.

6. Ablation Studies and Analysis

Ablation results using OKVQA zero-shot accuracy illustrate the contribution of each module:

Configuration	OKVQA Accuracy (%)
Baseline (–OEG, –MKA)	42.50
+ OEG only	44.26 (+1.76)
+ MKA only	43.64 (+1.14)
Full (OEG + MKA)	45.61 (+3.11)

• Both OEG and MKA provide measurable gains, with the combined approach achieving the largest improvement.
• Memory size (number of seed examples $K$) positively correlates with accuracy up to $K=400$.
• Prompt design analysis: grouping each retrieved example as a (Context–Question–Answer) block improves performance over separated or interleaved formats.

Qualitative evidence shows OAD-Promoter consistently answers questions across a range of domains even when alternative pipelines fail and is robust to the input order of retrieved memory examples, indicating stable domain adaptation and bias mitigation.

7. Significance and Domain Implications

Empirical results confirm that multi-level visual attribute description coupled with dynamic, similarity-based QA example retrieval yields substantial reduction in language bias and improved OOD transfer, all with frozen model backbones and in the absence of explicit retraining or access to external knowledge bases. The OAD-Promoter framework constitutes an effective approach for bias and domain shift in multimodal LLM-based VQA, setting a practical paradigm for future systems targeting general, robust visual reasoning (Xu et al., 15 Nov 2025).