OpenView: OOV Visual Question Answering
- OpenView is a framework for out-of-view VQA that infers unseen scene areas from panoramic images using spatial extrapolation.
- It employs a four-stage synthesis pipeline—data preprocessor, visual analyzer, proposal generator, and proposal refiner—to generate context-rich, multi-choice queries.
- The dataset and benchmark assess both answer selection and rationale quality, demonstrating significant improvements in spatial and directional reasoning.
OpenView is a panorama-based framework for out-of-view (OOV) visual question answering in multimodal LLMs (MLLMs), together with a synthetic training corpus and a manually verified evaluation benchmark. It targets reasoning about objects, activities, and scenes outside the currently visible image frame of a perspective view, using panoramic imagery to synthesize context-rich and spatially grounded multi-choice VQA instances with free-view framing (Chen et al., 21 Dec 2025). In this formulation, the central problem is not recognition of visible content, but inference about how a scene continues beyond the crop, how content changes under camera rotation or tilt, and what contextual cues in a partial view imply about unseen regions.
1. Conceptual scope and problem definition
OpenView introduces OOV VQA as a capability target distinct from standard in-view VQA. Conventional VQA benchmarks primarily test reasoning over content that is explicitly visible within the image frame. By contrast, OOV VQA requires a model to infer what is likely just beyond the current camera view, how scene structure extends spatially outside the crop, and what may appear after specified viewpoint changes such as turning left, turning right, or tilting upward or downward (Chen et al., 21 Dec 2025).
A central clarification in the formulation is that OOV reasoning is not the same as out-of-knowledge VQA. Knowledge-based VQA depends on external facts not contained in the image. OOV VQA instead depends on spatial extrapolation and contextual continuity from a limited visible view. The problem is therefore anchored in partial observability of a perspective frame rather than in retrieval of encyclopedic world knowledge.
The task design in OpenView is organized around two question families. Contextual questions ask what is likely or plausible outside the visible frame on the basis of scene context. Directional questions ask what will appear after a specified camera motion. This division is significant because it separates scene-level contextual inference from more explicit camera-motion reasoning, and the experimental results indicate that directional reasoning is generally harder for current MLLMs.
2. Four-stage synthesis pipeline
OpenView is implemented as a four-stage automatic pipeline that uses Qwen2.5-VL-72B as the assistant model throughout (Chen et al., 21 Dec 2025).
| Stage | Purpose | Key operations |
|---|---|---|
| Data Preprocessor | Filter panoramas | Keep valid 360° equirectangular panoramas; remove blur, artifacts, obstructions, watermarks, dark or empty content, and synthetic-rendered content |
| Visual Analyzer | Build structured scene grounding | Split panorama into 12 perspective views; extract captions, objects, locations, relations; form global summary and scene label |
| Proposal Generator | Produce candidate OOV VQAs | Free-view framing, question formulation, five-option construction, rationale generation, confidence assignment |
| Proposal Refiner | Improve quality and diversity | Formatting correction, confidence filtering, option shuffling, and view jittering |
The Data Preprocessor checks whether an input is a true 360° equirectangular panorama and removes panoramas with invalid projection formats, blur, stitching artifacts, obstructions, text or watermarks, overly dark or empty content, and synthetic-rendered content. This stage constrains subsequent synthesis to panoramas judged valid and informative.
The Visual Analyzer addresses the geometric distortion of ERP panoramas by splitting each panorama into 12 perspective-projected views arranged as a 3×4 grid. Each patch is analyzed to produce a short caption, a list of objects and coarse locations, and spatial relations among objects. These patch-level analyses are then combined into a global panorama summary and scene label. This yields an explicitly spatially organized representation of the full environment rather than a single undifferentiated panorama description.
The Proposal Generator has three substeps. In view framing, a perspective-projected view is selected using normalized center coordinates , a diagonal FoV in , and an aspect ratio drawn from a fixed set; roll is fixed to zero so that the task emphasizes OOV reasoning rather than in-camera-roll reasoning. In question formulation, the system generates a brief OOV question and five options, comprising four plausible answer candidates and one interference option such as “None of the above,” “Both A and C,” or “All of the above.” In answer elaboration, it generates concise rationales for each option and assigns a confidence score from 1 to 3.
The Proposal Refiner performs formatting correction, filters low-confidence outputs, keeps only confidence-3 proposals, applies option shuffling to reduce positional bias, and introduces view jittering. The jittering uses slight perturbations of about in yaw and pitch, intended to increase robustness while preserving OOV consistency.
3. OpenView-Dataset
The synthetic corpus produced by the pipeline is released as OpenView-Dataset (Chen et al., 21 Dec 2025). It is built from panoramas and panorama videos collected from five public datasets: Matterport3D, Mapillary Metropolis, 360Loc, 360+x, and 360-1M. The reported source scale is 5,289 panorama images and 3,034 panorama videos. The supplementary source breakdown is: Matterport3D with 2,295 panoramas, Mapillary Metropolis with 2,994 panoramas, 360Loc with 14 videos, 360+x with 232 videos, and 360-1M with 2,792 videos retained after filtering to high-resolution clips.
The sampling protocol is heterogeneous across sources. All Matterport3D panoramas are kept; 10% of Mapillary panoramas are sampled; 360Loc is sampled with frame step 10; and 360+x and 360-1M are sampled with 5 frames per video from middle segments. After filtering, the paper reports over 158k synthetic multi-choice OOV VQAs produced from 16k filtered panoramas; the appendix gives the full-scale count as 158,352 training samples.
Each sample contains a selected perspective view, a question, five options, a correct answer, rationales for all options, and view-associated metadata. The dataset spans 11 scene categories, with both indoor and outdoor diversity. The paper characterizes the dataset as high quality because panoramas provide full global context, Stage 2 supplies explicit spatial grounding, Stage 4 filters malformed or low-confidence proposals, manual and automatic refinement are both used, and the benchmark is separated from the training data and further subjected to multi-round human verification.
A notable property of the corpus is that rationales are generated for all options, including incorrect ones. This makes the supervision richer than answer-only multiple choice. A plausible implication is that OpenView is designed not merely to improve answer selection, but to shape explanation structure and option-wise comparative reasoning.
4. OpenView-Bench and evaluation protocol
The evaluation benchmark, OpenView-Bench, is a manually verified test set that measures both answer selection and rationale quality (Chen et al., 21 Dec 2025). It contains 1,327 high-quality OOV VQAs, balanced across task type, scene category, and answer distribution. It is constructed by selecting 7 panoramas per each of 11 scene categories, creating 616 initial proposals, manually revising them, and augmenting and rebalance-selecting to the final benchmark size.
The benchmark contains 665 contextual questions and 662 directional questions. Its answer distribution is nearly uniform: A: 268, B: 269, C: 267, D: 260, E: 263. This balancing is methodologically important because it reduces the extent to which a model can exploit answer-position priors.
OpenView-Bench evaluates three quantities:
Choice is checked by exact match, while rationale accuracy is scored in a GPT-as-judge setup using DeepSeek V3.1 as the default judge. The benchmark therefore does not reward merely naming the correct answer; it also requires a coherent explanation of why each option is correct or incorrect. In the reported user study, eight participants achieved 83.0% average choice accuracy, which serves as the human reference level.
5. Empirical findings
The experimental study evaluates proprietary models—Gemini-2.5-flash, GPT-4o, GPT-4o mini, and GPT-5—and open-source models including GLM-4.1V-9B-Base, GLM-4.1V-9B-Thinking, InternVL3.5-8B, LLaVA-NeXT-Mistral-7B, MiniCPM-V 4.5, Ovis2.5-9B, Qwen3-VL-8B-Instruct, Qwen3-VL-8B-Thinking, Qwen2.5-VL-3B-Instruct, Qwen2.5-VL-7B-Instruct, Qwen2.5-VL-32B-Instruct, and Qwen2.5-VL-72B-Instruct (Chen et al., 21 Dec 2025).
The baseline evaluation reveals a substantial gap between humans and MLLMs. Humans achieve 83.0% choice accuracy, whereas the strongest models remain below that level, and joint and rationale scores are notably weaker than answer-selection scores. The paper identifies a recurrent pattern: many MLLMs can often guess the answer, but rationale accuracy is substantially worse, particularly on directional tasks, where spatial inference under camera motion is harder. Among the proprietary systems, GPT-5 and Gemini-2.5-flash are strongest; among open-source systems, MiniCPM-V 4.5 and the Qwen family are strongest.
Supervised fine-tuning on OpenView-Dataset produces consistent improvements. The paper reports the following joint-accuracy changes: LLaVA-NeXT-Mistral-7B: 15.82% → 49.74%, InternVL3.5-8B: 34.14% → 48.08%, Qwen2.5-VL-7B-Instruct: 33.38% → 57.65%, and Qwen3-VL-8B-Instruct: 48.08% → 58.78%. It further states that performance is uplifted from 48.6% to 64.1% on average across models after fine-tuning.
The improvement is strongest in both contextual reasoning, where global scene understanding is beneficial, and directional reasoning, where explicit spatial supervision appears especially useful. Although directional tasks remain harder overall, the fine-tuned models improve substantially on them as well. Qualitative examples show that base models often over-rely on obvious visible cues, miss unseen continuation of the scene, and produce shallow rationales, whereas fine-tuned models better infer likely unseen objects and spatial layout, provide more scene-aware explanations, and show improved uncertainty awareness in directional reasoning.
The ablation study attributes the gains to both structure and scale. Removing the Visual Analyzer reduces performance, indicating that structured spatial annotations are essential. Applying the Proposal Refiner yields further gains, showing the importance of quality control and augmentation. Scaling from 1k panoramas to 16k panoramas improves results, with 16k panoramas giving the best overall performance at scale. The paper also presents a practical demonstration in outpainting support: the fine-tuned model produces more informative adjacent-view descriptions for panorama generation, improving text-conditioned outpainting.
6. Significance, limitations, and terminological scope
OpenView is presented as the first study on out-of-view understanding for MLLMs and, more specifically, as the first systematic study of OOV VQA built around a panorama-based generation pipeline, a supervised fine-tuning dataset, and a benchmark that jointly measures choice and rationale accuracy (Chen et al., 21 Dec 2025). Its significance lies in shifting evaluation beyond reasoning over visible pixels toward reasoning about the unseen environment, which the paper identifies as relevant to navigation, robotics, scene extrapolation, panorama understanding, and broader spatial intelligence.
The work also has clearly stated limits. The current formulation focuses on panorama-based synthetic OOV VQA, and the authors note room for expansion toward video generation, world modeling, and broader extrapolation tasks. This suggests that OpenView should be understood as a task-and-data framework for a specific aspect of spatial reasoning rather than as a complete theory of open-world visual inference.
The term also benefits from disambiguation. OpenView should not be conflated with “OpenViewer: Openness-Aware Multi-View Learning”, which addresses open-set multi-view classification and interpretability in a deep-unfolding framework (Du et al., 2024), nor with “OpenMX Viewer”, a web-based graphical user interface for crystalline and molecular visualization (Lee et al., 2019). A related but distinct line of work is “Open Panoramic Segmentation”, which studies zero-shot open-vocabulary semantic segmentation on full 360° panoramas rather than OOV question answering beyond a perspective crop (Zheng et al., 2024). OpenView’s distinctive contribution is therefore not panoramic perception in general, but the use of panoramas to create and evaluate reasoning about what lies outside the visible frame.