Visual Question–Visual Answering (VQ-VA)

• Visual Question–Visual Answering is a multimodal AI task that generates synthesized image responses to visual queries by integrating image understanding, reasoning, and compositional synthesis.
• Innovative datasets and benchmarks, such as IntelligentBench and VQ-VA World, employ rigorous expert curation and multi-stage evaluation to ensure high-quality performance assessment.
• Empirical results reveal significant performance gaps between proprietary and open-source models, highlighting the need for targeted data and refined evaluation metrics to advance the field.

Visual Question–Visual Answering (VQ-VA) is the multimodal AI task of generating an image as the answer to a question posed in visual form. Unlike traditional paradigms such as Visual Question Answering (VQA), where the system produces textual descriptive answers to image-grounded questions, VQ-VA systems must synthesize semantically appropriate visual responses. This requires a combination of image understanding, knowledge-based inference, and high-fidelity image generation capabilities. Proprietary systems (e.g., NanoBanana, GPT-Image) have demonstrated emergent VQ-VA ability by leveraging advanced model architectures and extensive private datasets. Recent open-source initiatives have sought to bridge the methodological and empirical gap, catalyzing systematic evaluation of VQ-VA across the axes of world knowledge, design knowledge, and reasoning (Gou et al., 25 Nov 2025).

1. Definition and Scope

VQ-VA tasks are characterized by the input–output mapping:

• Input: a visual question, typically comprising an image and an associated prompt.
• Output: a synthesized answer image that resolves or illustrates the correct response.

Formally, the system is required to generate an image $\text{Image}_2^\text{gen}$ given a question image $\text{Image}_1$ and free-form question $Q$, approximating a curated ground-truth response image $\text{Image}_2^\text{ref}$. Core dimensions assessed include the relevance, reasoning fidelity, and knowledge adequacy of the generated image. VQ-VA fundamentally differs from image captioning or text-based VQA by constraining the output domain to images, requiring robust pixel-level compositionality informed by implicit or explicit visual–semantic memory.

2. Dataset Construction and Benchmarking

The need for systematic evaluation and model training resources for VQ-VA led to the creation of dedicated benchmarks and data construction pipelines. "VQ-VA World" introduced an agentic pipeline leveraging large-scale web crawling to assemble approximately 1.8 million high-quality, interleaved image–text pairs, focused on knowledge-centric and design-centric content (Gou et al., 25 Nov 2025).

IntelligentBench serves as the principal benchmark for VQ-VA, comprising 360 human-curated triplets $\langle \text{Image}_1, Q, \text{Image}_2 \rangle$, uniquely annotated to probe three target capabilities:

• World Knowledge: 47.5% of items, requiring grounding in physical, scientific, temporal, or spatial facts.
• Design Knowledge: 24.4% of items, focused on part–whole relationships and compositional function.
• Reasoning: 28.1% of items, demanding multi-step or causal inference beyond recognition or retrieval.

All triplets undergo multi-stage, expert-driven auditing for semantic validity, contextual relevance, and answer specificity. Category assignment is based on whether resolution implicates real-world factuality, design/compositional analysis, or abstract reasoning (e.g., causal chains). The relation types covered (change/process, composition/spatial, function/usage, scientific/analytical, evidence/validation, comparison/contrast) ensure coverage of the diverse inferential demands of the VQ-VA paradigm.

Dataset Statistics:

Category	Number of Examples	% of Total
World Knowledge	171	47.5
Design Knowledge	88	24.4
Reasoning	101	28.1

Additional quality summaries: average question length 14.2 words; answer image median resolution 512×512 pixels (90% within 256×256–1024×1024); inter-annotator agreement by design is 100%.

3. Evaluation Protocols and Metrics

Evaluation of VQ-VA systems deploys a visual-LLM (VLM)—specifically GPT-4o—for automatic scoring, aligning with human expert judgments. For each item $$(\text{Image}_1, Q, \text{Image}_2^\text{ref}, \text{Image}_2^\text{gen})$$, the VLM outputs a discrete score $s_i \in \{0, 1, 2\}$ considering fidelity, relevance, and reasoning. The normalized aggregate across $N$ items is:

$$\text{NormalizedScore} = \frac{1}{N} \sum_{i=1}^{N} \frac{s_i}{2} \times 100.$$

For reference-style tasks admitting categorical evaluation, accuracy is also reported as:

$$\text{Accuracy} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}\{\hat{y}_i = y_i\}.$$

Automated pipelines pair the model’s generated image with reference items, passing them to the VLM with explicit rubric instructions, ensuring replicable and objective assessment across systems.

4. Empirical Performance and Model Comparisons

Systematic empirical evaluation on IntelligentBench reveals significant performance stratification between proprietary and open-source models (Gou et al., 25 Nov 2025). Leading proprietary models (GPT-Image, NanoBanana) attain normalized scores above 80, indicating high reasoning fidelity and image relevance. Open-source baselines such as LightFusion and UniWorld-V1 demonstrate substantial deficiencies (below 10 normalized), but the introduction of the “LightFusion–World” model, trained on VQ-VA World’s targeted data, dramatically improves open-source results to over 53 normalized.

Model	World Knowledge	Design Knowledge	Reasoning	Overall
GPT-Image (closed)	84.5	80.7	81.2	82.6
NanoBanana (closed)	81.6	83.0	80.7	81.7
Qwen-Image (open-wt)	38.1	33.7	32.8	34.3
UniWorld-V1 (open)	2.9	0.6	1.5	1.9
LightFusion (open)	5.3	11.9	8.4	7.8
LightFusion–World	50.6	58.0	53.0	53.1

This demonstrates that targeted, high-quality VQ-VA supervision substantially narrows the gap toward proprietary systems but also highlights that state-of-the-art open-source systems remain significantly behind closed models in comprehensive image reasoning and synthesis.

5. Task Diversity and Illustrative Examples

VQ-VA benchmarks intentionally cover a spectrum of inferential tasks:

• Example (World Knowledge): Given a photograph of a cracked window with the question, “What would you expect to see on the floor beneath this window if it had just been smashed?”, ground-truth is an image of glass shards, requiring prediction of physical outcomes.
• Example (Design Knowledge): Presented with a wheel rim, the question “What complete object is this component part of?” mandates functional and compositional inference, expecting a full racing car image as answer.
• Example (Reasoning): For a chemical reaction diagram, the prompt “Show me the state of matter after the reaction when a white precipitate forms,” expects the synthesis of a beaker with white precipitate, requiring multi-step chemical reasoning and visualization.

Category distributions ensure broad coverage: relation types such as process change, composition, usage, scientific analysis, evidential/validation, and contrast/comparison cumulatively populate the benchmark. The average question length and image resolution are calibrated to reflect realistic and non-trivial problem settings.

6. Quality Assurance and Benchmark Integrity

Rigorous multi-phase quality control is central to VQ-VA benchmarking. The annotation pipeline involves:

• Human expert curation at document selection and question-authoring stages.
• Instruction filtering using VLM-based sub-scores for question clarity, answer fidelity, and context dependence (QS, AS, CDS), keeping only items with maximal combined score (QS+AS+CDS=6).
• Cross-review: Each item is reviewed by multiple domain experts, and only unanimously approved triplets are retained.
• Final manual spot-checks to eliminate trivial or ambiguous scenarios.

A plausible implication is that such intensive scrutiny results in a benchmark that is more robust to shortcut exploitation and semantic ambiguity than auto-curated or loosely supervised alternatives.

7. Research Impact, Challenges, and Future Directions

The introduction of VQ-VA and its benchmarks has enabled principled comparison of proprietary and open-source multimodal models under unified protocols. The empirical gap observed highlights the importance of large-scale, knowledge-targeted datasets for bridging capability shortfalls in open systems (Gou et al., 25 Nov 2025). Nevertheless, VQ-VA evaluation relies on current VLMs as automatic judges, introducing potential biases; advances in human-aligned scoring and the development of fine-grained, interpretable rubric-based evaluation remain open challenges. The release of the full VQ-VA World pipeline and benchmark assets is likely to accelerate progress and innovation in multimodal reasoning, compositionality, and image synthesis under real-world cognitive constraints.

Future directions may encompass scaling up benchmark diversity, refining multi-stage reasoning metrics, and leveraging few-shot or in-context adaptation techniques. As the performance ceiling is raised, a research focus is anticipated on application-specific benchmarks, compositional generalization, and robustness to semantically adversarial queries.