OpenView-Bench: Out-of-View VQA Benchmark

• OpenView-Bench is a benchmark for out-of-view VQA that measures a model's ability to infer unseen scene elements from a single perspective using panoramic context.
• It employs a four-stage panoramic synthesis pipeline to generate contextual and directional multi-choice questions while jointly evaluating answer-choice and rationale accuracy.
• Empirical results reveal a notable gap between human performance and current MLLMs, underscoring the benchmark's potential to drive improvements in spatial extrapolation.

OpenView-Bench is a benchmark for out-of-view visual question answering (OOV VQA), introduced in the paper "OpenView: Empowering MLLMs with Out-of-view VQA" (Chen et al., 21 Dec 2025). It explicitly targets the ability of a multimodal LLM (MLLM) to reason about objects, activities, and scene layouts beyond the visible perspective frame. Within this formulation, OOV understanding is scoped as the capacity to extrapolate from the evidence in a single perspective view to infer plausible content outside that view, either in a contextual sense or in a directional sense. OpenView-Bench is paired with a panoramic synthesis pipeline and a supervised fine-tuning corpus, but its distinctive role is evaluative: it jointly measures answer-choice accuracy and rationale accuracy in order to support interpretable and diagnosable assessment of OOV reasoning.

1. Conceptual scope and task definition

OpenView-Bench is presented as the first benchmark that explicitly targets OOV VQA, rather than standard in-frame VQA or knowledge-centric VQA (Chen et al., 21 Dec 2025). In-frame VQA constrains questions to visible evidence inside the image boundaries. By contrast, OOV VQA asks whether a model can infer what lies beyond those boundaries from spatial continuity, holistic scene context, and camera-centric reasoning. The benchmark therefore isolates visual extrapolation from both ordinary object recognition and external fact retrieval.

The task is divided into two question families. Contextual questions ask what is plausible outside the given view, with the correct option aligned with evidence from adjacent neighbor views but required to be justifiable solely from cues in the chosen view. Directional questions specify camera rotations or tilts, such as turning or tilting, and ask what would come into view after that camera motion. In both cases, the benchmark is designed so that the answer is grounded in panoramic context during construction while evaluation remains restricted to a single perspective image and its associated question.

This formulation differs explicitly from out-of-knowledge VQA, including OK-VQA, because the emphasis is not on external non-visual facts. The central difficulty instead lies in extrapolating scene structure beyond the visible frame. A plausible implication is that OpenView-Bench occupies an intermediate regime between classical VQA and embodied or navigation-oriented scene reasoning, because it probes inference about unseen but spatially connected visual content.

2. Panoramic grounding and the four-stage synthesis pipeline

The benchmark is built from panoramic imagery, which provides a full $360^\circ$ view and enables free view framing while preserving global scene context (Chen et al., 21 Dec 2025). The paper describes panoramas collected from five sources: Matterport3D, Mapillary Metropolis, 360Loc, 360+x, and 360-1M. Videos are uniformly sampled, including five frames per clip for 360+x and 360-1M and a frame step of ten for 360Loc.

The associated OpenView pipeline has four stages, all assisted by Qwen2.5-VL-72B. The first stage, Data Preprocessor, filters defective panoramas, including non-equirectangular formats, watermarks or overlays, stitching or motion artifacts, low visibility, and synthetic content. For videos, it removes sources with multiple invalid frames. The second stage, Visual Analyzer, addresses equirectangular distortion by dividing each panorama into 12 perspective-projected patches arranged as a $3\times 4$ grid with slight overlap at the top and bottom. Each patch is annotated with intrinsic metadata, including diagonal field of view and aspect ratio, and extrinsic metadata, including normalized UV center for yaw and pitch derivation and zero roll. The assistant model then produces a local caption, a list of objects with coarse positions using nine tokens, and spatial relations among visible objects. These patch descriptions are condensed into a global panorama summary with an 11-way scene taxonomy and an indoor/outdoor flag.

The third stage, Proposal Generator, creates candidate OOV multi-choice VQAs. It frames a perspective view using zero roll, a center $(u_{\text{norm}}, v_{\text{norm}})$, $\text{diag\_FoV} \in [40^\circ, 100^\circ]$, and an aspect ratio chosen from $$\{4\!:\!3, 3\!:\!4, 3\!:\!2, 2\!:\!3, 16\!:\!9, 9\!:\!16, 1\!:\!1\}$$. Each VQA contains five options: A-D are plausible and mutually exclusive distractors or candidates, while E is a logical interference choice such as "None of the above," "Both A and C," or "All of the above," and is correct in some cases. The assistant also generates concise option-wise rationales, a confidence score in $[1, 3]$, and explicit chain-of-thought traces for interpretability. The fourth stage, Proposal Refiner, corrects formatting errors, retains only proposals with confidence score 3, and augments the data through option shuffling and view jittering of $\pm 3.6^\circ$ yaw and pitch.

Stage	Function
Data Preprocessor	Filters defective panoramas and inconsistent video sources
Visual Analyzer	Produces patch-level and global spatial grounding
Proposal Generator	Synthesizes contextual and directional multi-choice OOV VQAs
Proposal Refiner	Filters, fixes, and augments proposals

The paper reports that structured spatial grounding is crucial: removing the visual analysis stage in ablations consistently reduces downstream performance, and the proposal refiner improves robustness and learning effectiveness. This suggests that benchmark quality depends not only on panoramic coverage but also on explicit intermediate representations of local and global scene structure.

3. Dataset support and benchmark composition

OpenView-Bench is supported by OpenView-Dataset, a supervised fine-tuning corpus generated by the same pipeline (Chen et al., 21 Dec 2025). The dataset contains over 158,000 synthetic OOV VQAs from 16,000 filtered panoramas spanning the 11 scene categories and both indoor and outdoor settings. Each question has five options and detailed textual rationales for every option. The rationale length distribution peaks near 190 words, while questions average approximately 15 words and options approximately 4 words. The linguistic pattern is intentionally uniform so that learning emphasizes visual reasoning rather than superficial text style.

The benchmark itself comprises 1,327 manually verified OOV VQAs, evenly balanced across contextual and directional types and approximately balanced across answer positions. Construction begins by selecting seven panoramas per scene category from distinct locations, generating initial proposals, and then applying multi-round manual verification focused on view framing, question quality, and rationale correctness. GPT-4o is used as an auxiliary verifier for grammatical issues and tone diversity. Each item is validated by at least two annotators plus the GPT verifier.

The benchmark composition is reported as follows:

Component	Count
Total benchmark items	1,327
Contextual questions	665
Directional questions	662

Answer balance is also explicitly reported: A: 268, B: 269, C: 267, D: 260, and E: 263. The inclusion of option E as a logical interference choice is notable because it prevents the benchmark from reducing to a conventional four-way recognition problem. A plausible implication is that answerability depends on excluding unsupported alternatives as much as identifying supported ones.

A user study with eight participants reports human choice accuracy of 83.0%, with 80.85% on contextual items and 85.00% on directional items. The paper interprets this as evidence of reasonable difficulty and reliability.

4. Evaluation protocol and formal metrics

OpenView-Bench uses a GPT-as-the-judge evaluation setup in which answer selection and rationale quality are scored separately and jointly (Chen et al., 21 Dec 2025). Choice accuracy is computed by exact-match extraction of the predicted option through regex against the ground-truth label. Rationale accuracy is judged by DeepSeek V3.1, which returns a binary correctness label for the option-wise rationales and final justification. Joint accuracy requires both a correct answer and a correct rationale.

The reported formalization is:

$$A_{\text{choice}} = \frac{1}{N}\sum_{i=1}^{N}\mathbb{1}[\hat{y}_i = y_i]$$

$A_{\text{rat}} = \frac{1}{N}\sum_{i=1}^{N} r_i$

$$A_{\text{joint}} = \frac{1}{N}\sum_{i=1}^{N}\mathbb{1}[\hat{y}_i = y_i]\cdot r_i$$

Here, $3\times 4$0 is the predicted option, $3\times 4$1 is the ground-truth option, and $3\times 4$2 is the judge’s correctness flag for the rationale and justification.

The rationale judgment is not a semantic similarity score. Instead, it is a rubric-based binary decision emphasizing evidential support, logical consistency, completeness, and non-leakage. Long rationales are not preferred, and vague statements such as "not visible" without elaboration are judged incorrect. This makes the benchmark stricter than evaluation setups that reward generic explanatory text.

Model input and output follow a unified inference prompt. The model receives the image, the question, and the five options; it is instructed to analyze each option’s correctness with reasoning and to output the final choice in <answer>...</answer> format. The benchmark is therefore designed to assess both decision quality and the alignment between a decision and its supporting explanation.

5. Empirical performance and diagnostic findings

Reported results show a substantial gap between human performance and current MLLMs on OpenView-Bench (Chen et al., 21 Dec 2025). Among proprietary models, GPT-5 and Gemini-2.5-Flash achieve the highest overall joint accuracy, at 61.79% and 61.64%, respectively, with notably high rationale reliability. GPT-4o and GPT-4o-mini are reported at 46.04% and 42.50% overall joint accuracy. Among open-source base models, MiniCPM-V 4.5 reaches 55.91% overall joint accuracy, Qwen2.5-VL-32B reaches 51.92%, and Qwen3-VL-8B-Thinking reaches 51.85%.

The reported pattern is that many models can select plausible answers but their rationales lag, especially on directional items. Directional questions are described as harder than contextual questions, with lower joint accuracies across models. This supports the paper’s motivating claim that out-of-view reasoning is not exhausted by answer selection alone; explanation quality remains a major failure mode.

Fine-tuning on OpenView-Dataset materially improves performance for open-source models. LLaVA-NeXT-Mistral-7B increases from 15.82% to 49.74% overall joint accuracy, InternVL3.5-8B from 34.14% to 48.08%, Qwen2.5-VL-7B from 33.38% to 57.65%, and Qwen3-VL-8B-Instruct from 48.08% to 58.78%. The paper also reports an average uplift from 48.6% to 64.1% after empowerment with OpenView. For Qwen3-VL-8B-Instruct, rationale accuracy slightly decreases after fine-tuning while joint accuracy still improves, which the paper interprets as better alignment between answers and justifications.

Ablation studies further report that removing the visual analyzer degrades results and that applying the proposal refiner yields additional gains. Scaling the training set from 1k to 16k panoramas steadily improves joint accuracy, including InternVL3.5-8B from 39.71% to 48.08% and Qwen2.5-VL-7B from 42.50% to 57.65%. These findings tie benchmark performance closely to the quality of spatially grounded synthetic supervision.

6. Relation to prior benchmarks and practical usage

OpenView-Bench is positioned against earlier VQA and multimodal evaluation suites such as VQA v2, GQA, OK-VQA/A-OKVQA, MMBench, MM-Vet, SEED-Bench, and MMMU (Chen et al., 21 Dec 2025). The stated distinction is that OpenView-Bench requires explicit reasoning beyond the visible frame rather than knowledge retrieval or purely in-frame perception. It also jointly evaluates answer selection and option-specific rationales for interpretability and diagnosis, which the paper describes as largely absent from existing panoramic or occlusion benchmarks.

Panoramic grounding serves two roles simultaneously. During question design, it provides access to full-scene context and neighbor views. During evaluation, it prevents leakage by restricting the model to a single framed view. Directional questions are further described as probing camera-rotation reasoning that is closer to navigation, robotics, or view-extrapolation tasks. A plausible implication is that the benchmark can function as a bridge between static-image MLLM evaluation and embodied scene understanding.

The repository at https://github.com/q1xiangchen/OpenView provides panorama processing, view projection, generation prompts, refined JSON schemas, and evaluation scripts. The benchmark is distributed as JSON with fields for the question, five options (option_a through option_e), the ground-truth answer, option-wise rationales, and view metadata including normalized UV center, diagonal FoV, aspect ratio, and neighbor relations. The evaluation script enforces the <answer>...</answer> format, exact-match choice scoring, and DeepSeek-based rationale judgments.

Training scripts include LoRA-based fine-tuning configurations. For LLaVA-NeXT-Mistral-7B and InternVL3.5-8B, the reported settings are batch size 8, LoRA rank 8, and alpha 16. For Qwen2.5-VL-7B and Qwen3-VL-8B, the reported settings are batch size 128, rank 64, and alpha 64. Training uses one epoch, a frozen vision tower, a learning rate of $3\times 4$3, and two NVIDIA A100 80GB GPUs. Usage notes also include recommended inference settings per model and prompts for contextual and directional question synthesis.

7. Limitations, controversies, and future directions

The paper identifies several limitations of OpenView-Bench and its supporting pipeline (Chen et al., 21 Dec 2025). Although panoramas mitigate hallucination risks in question design, synthetic generation can still produce trivial or biased items if not carefully filtered. The pipeline counters this through confidence scoring, manual verification, option shuffling, and view jittering, but residual artifacts may remain. Panoramic sources also introduce domain biases, including urban streets and indoor real-estate scenes, which may not cover all deployment conditions.

A second limitation concerns rationale scoring. The benchmark relies on a GPT-as-the-judge paradigm using DeepSeek, which is intended to mitigate self-evaluation bias but is itself imperfect. The paper notes that human-verified subsets and balanced answer distributions help, yet the judge may still penalize uncommon but valid explanations. This is the principal methodological controversy surrounding the benchmark: the rationale metric is designed to be stringent and diagnostic, but its correctness signal depends on an external judge rather than direct human adjudication for every instance.

Additional limitations arise from the current camera model and geometric formalization. The dataset uses normalized UV coordinates with zero roll and conceptual yaw/pitch conversion, but the paper does not provide closed-form projection equations. Future work is suggested in richer camera motion understanding beyond zero roll, broader environments, expanded geographic and device coverage, stronger human-calibrated rationale scoring, and extensions to video generation and world modeling. The paper also notes that OOV supervision improves text guidance for panorama outpainting. Taken together, these directions indicate that OpenView-Bench is intended as a foundational protocol for OOV reasoning research rather than a completed endpoint.