VRQABench: Dual Benchmark for VQA & Spatial Reasoning

• VRQABench is a polysemous research artifact defining both a controlled spatial lookahead benchmark for puzzles and an interactive VQA evaluation platform with multi-dimensional metrics.
• The spatial lookahead task leverages interventionist world models and exact solvers to generate visual rollouts for puzzles such as mazes, irregular-mazes, and Sokoban states.
• The VQA platform integrates REST APIs and visualization tools to assess model performance across robustness, bias, and uncertainty metrics on multiple datasets.

Searching arXiv for the cited VRQABench-related papers and context. arxiv_search.query({"search_query":"id:(Zhou et al., 2 Jun 2026) OR id:(Väth et al., 2021)","start":0,"max_results":10}) VRQABench denotes two distinct research artifacts in the arXiv literature. In "World Models Meet LLMs: On the Complementarity of Concrete and Abstract Reasoning" (Zhou et al., 2 Jun 2026), VRQABench is a human-verified benchmark for controllable spatial lookahead within a controlled concrete-reasoning setting. In "Beyond Accuracy: A Consolidated Tool for Visual Question Answering Benchmarking" (Väth et al., 2021), VRQABench denotes a browser-based, interactive, multi-metric benchmarking platform for Visual Question Answering (VQA) systems. The shared name is therefore polysemous: one usage identifies a task suite over future-oriented puzzle reasoning, while the other identifies an evaluation system for cross-dataset VQA analysis.

1. Terminological scope and disambiguation

The two uses of VRQABench differ in object, methodology, and intended workload. One is a benchmark dataset and task formalization for models that may invoke a world model during inference; the other is a benchmarking tool with a REST API, database-backed storage, metric computation, and interactive visualization.

Usage	Definition	Source
VRQABench	human-verified benchmark for controllable spatial lookahead	(Zhou et al., 2 Jun 2026)
VRQABench	browser-based, interactive, multi-metric benchmarking platform for VQA systems	(Väth et al., 2021)

A plausible implication is that references to VRQABench require explicit paper-level disambiguation. In the 2026 usage, the central problem is whether a model should invoke, verify, and integrate visual future simulation. In the 2021 usage, the central problem is evaluation beyond single-metric leaderboards, including robustness, bias, uncertainty, and sample-level inspection.

2. VRQABench as a controlled spatial lookahead benchmark

In the 2026 formulation, an episode $\tau$ comprises a tuple

$\tau = (o, q, O, y^*)$

where $o \in \mathcal{O}$ is a static puzzle image, $q \in \mathcal{Q}$ is a natural-language question about a future outcome, $O=\{O_1,O_2,O_3,O_4\}$ are four text answer options, and $y^* \in \{1,2,3,4\}$ is the index of the correct option. The puzzle image may be a maze, irregular-maze, or Sokoban state. The questions target future outcomes such as “How many times does the path change direction?” (Zhou et al., 2 Jun 2026)

The model interface is explicitly interventionist. A model $f$ may optionally invoke a world model $W$ to generate a candidate video rollout $\hat u$ for concrete-reasoning. At test time, the agent: observes $(o,q,O)$; decides $\tau = (o, q, O, y^*)$0 whether to call $\tau = (o, q, O, y^*)$1; if $\tau = (o, q, O, y^*)$2, emits a prompt $\tau = (o, q, O, y^*)$3 and obtains $\tau = (o, q, O, y^*)$4; and performs abstract reasoning over $\tau = (o, q, O, y^*)$5 to predict $\tau = (o, q, O, y^*)$6. During training, ground-truth future video $\tau = (o, q, O, y^*)$7 and answer $\tau = (o, q, O, y^*)$8 are used only by the privileged teacher, whereas at test time only $\tau = (o, q, O, y^*)$9, $o \in \mathcal{O}$0, $o \in \mathcal{O}$1, and any generated $o \in \mathcal{O}$2 are available.

This formulation is embedded in the paper’s broader notion of controlled concrete reasoning. World models generate concrete visual rollouts of possible futures, while MLLMs reason abstractly over questions, goals, and rules; however, generated rollouts are stochastic and may be visually plausible but task-incorrect. The benchmark is designed around that control problem rather than around passive answer prediction alone.

3. Dataset construction, question taxonomy, and annotation protocol

The source environments are reported as Maze (2D grid): 1 756 train / 2 031 test images; Irregular–maze: 1 108 train / 1 279 test images; and Sokoban (box-pushing): 1 136 train / 1 326 test images. The same summary also reports “Total: 4 000 train, 636 eval.” The question categories are fivefold: C1_turn_count, C2_turn_direction, C3_sokoban_push, C4_direction_count, and C5_push_dir_count (Zhou et al., 2 Jun 2026).

Automated generation proceeds in three stages. First, each puzzle state is solved with an exact solver: BFS for maze and Sokoban, and geometric angle analysis for irregular-maze. Second, target statistics are extracted, including turn counts, push counts, and directions. Third, 3–4 distractors are programmatically sampled. For counting questions, the distractors are integers within $o \in \mathcal{O}$3 of ground truth with a lower bound of $o \in \mathcal{O}$4; for direction questions, all four cardinal directions are permuted.

Surface realization and filtering are also specified procedurally. A VLM Prompt, labeled “QuestionWriter,” receives the puzzle image and metadata and writes one concise natural-language question sentence. SmallModelProbe (GPT-5.4-nano) discards items that the probe answers correctly twice, with shuffling. Reviewer (GPT-5.5) rates question text quality, plausibility of distractors, and difficulty, accepting only items with score $o \in \mathcal{O}$5. Final human verification checks visual consistency, answer unambiguity, option plausibility, and absence of future leakage, and removes any failure cases.

The category design emphasizes shortest-path structure, turn geometry, and box-pushing dynamics. This suggests that the benchmark operationalizes spatial lookahead through exact, solver-backed latent structure rather than through subjective annotation alone.

4. Evaluation protocol and reported performance

The primary metric is multiple-choice accuracy. For $o \in \mathcal{O}$6 evaluation items,

$o \in \mathcal{O}$7

The paper also reports simulation-control metrics in ablations: Call Rate, Average Calls, and Decision $o \in \mathcal{O}$8, where Decision $o \in \mathcal{O}$9 measures precision and recall of “call” versus privileged evaluator “simulation-helps” labels. Under controlled rollout-quality conditions, rollout verification is assessed via Accept-rate, False-accept, False-reject, and verification precision and recall versus evaluator-derived usefulness labels (Zhou et al., 2 Jun 2026).

The reported VRQABench performance spans zero-shot baselines, workflow-agent baselines, learned-controller baselines, and PF-OPSD. The zero-shot / no-sim baselines are Gemini-3-Flash at 45.9, GPT-5.4 at 43.2, Tencent HY3 at 38.2, Qwen3.6-27B at 33.0, and Qwen3.5-9B at 33.2. Under Workflow-Agent (MLLM+Helios), Qwen3.5-9B reaches 32.6 and Gemini-3-Flash reaches 47.2. Among learned-controller baselines with Qwen3.5-9B, SFT achieves 61.8 and GRPO achieves 63.5, while PF-OPSD achieves 72.4. The absolute gain over SFT is reported as +10.6 points $q \in \mathcal{Q}$0.

The abstract further states that PF-OPSD outperforms baseline by 10.6% on VRQABench and increases robustness to noisy or conflicting rollouts. Within the reported setup, the benchmark therefore serves not only as an answer-accuracy test bed but also as a control benchmark for simulation usage, verification, and reliance.

5. Role within the controlled concrete-reasoning and PF-OPSD framework

Within the controlled concrete-reasoning framework, the rollout generator $q \in \mathcal{Q}$1 is the Helios world model, described as VR-Bench-fine-tuned and producing 100-frame video rollouts. The controller policy $q \in \mathcal{Q}$2 outputs a trajectory of decisions: $q \in \mathcal{Q}$3; if yes, up to $q \in \mathcal{Q}$4 prompts $q \in \mathcal{Q}$5 and rollouts $q \in \mathcal{Q}$6; for each rollout, $q \in \mathcal{Q}$7; after stopping at attempt $q \in \mathcal{Q}$8, rollout-reliance state $q \in \mathcal{Q}$9; and a final answer $O=\{O_1,O_2,O_3,O_4\}$0 (Zhou et al., 2 Jun 2026).

The reported confidence thresholds and hyperparameters are: retry cap $O=\{O_1,O_2,O_3,O_4\}$1 attempts per example; rollout verification decoded greedily with zero temperature for discrete decisions; advantage-distillation temperature $O=\{O_1,O_2,O_3,O_4\}$2; $O=\{O_1,O_2,O_3,O_4\}$3 text-candidate samples; and simulation-call penalty $O=\{O_1,O_2,O_3,O_4\}$4, which encourages selective calls.

Training is described in two stages. Stage 1 performs SFT on Gemini-3.1-Pro–Agent trajectories to learn the protocol format. Stage 2 performs PF-OPSD self-distillation with privileged ground-truth futures $O=\{O_1,O_2,O_3,O_4\}$5 and answers $O=\{O_1,O_2,O_3,O_4\}$6, scoring each intermediate action by

$O=\{O_1,O_2,O_3,O_4\}$7

The method then backpropagates advantage-weighted KL on discrete nodes and weighted likelihood on text nodes over on-policy rollouts. At inference, the deployable student policy $O=\{O_1,O_2,O_3,O_4\}$8 has no access to $O=\{O_1,O_2,O_3,O_4\}$9 and acts as a closed-loop agent over simulation call, verification, reliance, and answer.

This framework makes VRQABench more than a static benchmark. It functions as an instrument for studying when world-model rollouts should be invoked, when they should be trusted, and how they should affect downstream multiple-choice reasoning.

6. VRQABench as a browser-based VQA benchmarking platform

In the 2021 usage, VRQABench is a browser-based, interactive, multi-metric benchmarking platform for VQA systems. Its stated goal is to help test generalization capabilities of models across multiple datasets and to evaluate not just accuracy but also performance in more realistic scenarios such as robustness to input noise, while also including metrics that measure biases and uncertainty. The system is structured as a front-end, back-end services, database, and storage layer (Väth et al., 2021).

The browser front-end is a Single-Page Application, for example built in React or Vue, with views labeled Global Leaderboard, Metric Details, Filter View, and Sample View, and it communicates with the back-end over a REST API. The back-end includes an API Gateway / Auth Service, a Model & Dataset Registry Service, an Evaluation Service, and a Visualization Service. The database stores model metadata, dataset metadata, per-sample predictions and metric outcomes, and aggregated metrics. Storage is defined as a file store or object store for images, model artifacts, and dataset annotations.

The system’s data flow is described step by step. A user registers a new model and/or dataset via the REST API; the Registry Service persists metadata and returns unique IDs; the user or system calls /api/evaluate with model_id, dataset_id, and split; the Evaluation Service schedules a job; each sample undergoes pre-processing; the model produces a score vector over the answer vocabulary; metrics are computed per sample; the results are stored and aggregated; and the front-end queries aggregated metrics and per-sample filters via REST endpoints. The platform also exposes endpoint summaries including POST /api/models, POST /api/datasets, POST /api/evaluations, GET /api/metrics, and GET /api/filters.

A plausible implication is that this VRQABench is not a benchmark dataset in the narrow sense, but an extensible benchmarking infrastructure intended to consolidate model registration, evaluation orchestration, metric computation, and interactive analysis.

7. Metrics, visualization, and case-study findings in the 2021 system

All metrics in the 2021 system return values in $y^* \in \{1,2,3,4\}$0. Accuracy uses the single-answer formulation

$y^* \in \{1,2,3,4\}$1

Modality Bias is split into image bias and question bias, each based on unchanged predictions under replacement of one modality by an unrelated alternative. Robustness to Noise includes image-space robustness, feature-space robustness, and question-embedding robustness. Robustness to Adversarial Questions uses four SEAR rules and counts unchanged predictions. Uncertainty is estimated with Monte-Carlo dropout through Predictive Entropy, Mutual Information, and optionally Expected Calibration Error (Väth et al., 2021).

The visualization layer is organized into four main views. The Global View provides a leaderboard table with per-model averages of each metric across all datasets and expandable per-dataset breakdowns. The Metrics View provides line or bar charts for a single model, with selectable dataset and metric. The Filter View provides dropdowns for Model, Dataset, and Metric, a slider for a metric-value range, and a live-updating sample list with columns Sample ID, Question Type, Answer Type, Confidence, and Metric Value. The Sample View displays the original image and question text, ground-truth answers, the model’s top-3 predicted answers with confidences, and all predictions observed under noisy or adversarial variants.

The case study integrates four state-of-the-art VQA models and evaluates them on six datasets: VQA2, GQA, GQA-OOD, CLEVR, OK-VQA, and TextVQA. Average results across all datasets are reported for MCAN, MMNASNET, BAN-8, and MDETR. MCAN has Acc 41.30, Bias_img 4.83, Bias_ques 2.57, $y^* \in \{1,2,3,4\}$2 99.98, $y^* \in \{1,2,3,4\}$3 81.09, $y^* \in \{1,2,3,4\}$4 63.14, SEAR 58.45, Predictive Entropy 31.77, and 201 parameters in millions. MMNASNET has Acc 40.80 and similar robustness trends. BAN-8 has Acc 38.62 and the highest uncertainty entropy at 66.10. MDETR has Acc 38.82, $y^* \in \{1,2,3,4\}$5 36.33, $y^* \in \{1,2,3,4\}$6 90.10, $y^* \in \{1,2,3,4\}$7 100.00, and SEAR 100.00. The report states that all models perform poorly on TextVQA, with all below 9%, and that CLEVR performance is low, below 33% for all, with modality biases on CLEVR highest, indicating poor spatial reasoning on synthetic data.

The embedding analysis ties these findings to architectural choices. Contextual language embeddings, specifically RoBERTa in MDETR, are reported to yield perfect adversarial robustness and noise robustness in question space, whereas GloVe or FastText in other models yield approximately 60–70%. Bottom-Up Top-Down image features in MCAN, BAN, and MMNASNET produce near-perfect pixel noise robustness above 99.9%, compared to raw CNN features in MDETR. This suggests that, in the 2021 usage, VRQABench is primarily a diagnostic environment for model behavior under distribution shift, perturbation, and calibration stress, rather than a single benchmark with a single leaderboard.