OpenView-Dataset: OOV VQA Corpus

• OpenView-Dataset is a synthetic multi-choice VQA corpus designed to evaluate out-of-view (OOV) reasoning using panoramic imagery and spatial context.
• It employs a four-stage pipeline—from panorama filtering to proposal refinement—that generates challenging contextual and directional questions with detailed, option-wise rationales.
• Benchmark results show significant improvements in joint accuracy for fine-tuned models, highlighting practical gains in both answer selection and rationale explanation.

OpenView-Dataset is a large-scale, synthetic, multi-choice visual question answering (VQA) corpus designed to train and evaluate “out-of-view (OOV) understanding” in multimodal LLMs (MLLMs). In the OpenView formulation, OOV understanding denotes the ability to infer plausible objects, activities, and scene elements that lie outside the visible frame of a single perspective image by using spatial context and commonsense. The dataset is introduced together with the OpenView generation pipeline and the OpenView-Bench evaluation benchmark in “OpenView: Empowering MLLMs with Out-of-view VQA” (Chen et al., 21 Dec 2025). Its central methodological premise is that panoramic imagery makes OOV supervision feasible and reliable: a $360\times 180$-degree panorama provides full scene coverage, while free-view perspective crops define the visible image shown to the model and neighboring, out-of-frame crops serve as hidden grounding for what should be present, absent, or plausible just beyond the boundaries of the crop.

1. Problem formulation and task design

OpenView-Dataset focuses on multi-choice OOV VQA with spatial grounding and free-view framing through two complementary question types: Contextual OOV and Directional OOV (Chen et al., 21 Dec 2025). Contextual OOV questions ask what is likely or unlikely outside the current view given the global scene context. Directional OOV questions ask what would come into view if the camera rotates or tilts by specified angles.

This formulation differs from conventional VQA settings because the answer is not restricted to evidence directly visible in the provided crop. Instead, the model must reason from visible cues toward hidden but spatially grounded content. The paper contrasts this with conventional VQA benchmarks, which focus on in-view evidence, and with knowledge-centric datasets, which probe out-of-knowledge reasoning rather than spatial extrapolation. A plausible implication is that OpenView-Dataset is intended to isolate a distinct failure mode of MLLMs: competent in-frame recognition may coexist with weak extrapolation across the image boundary.

The dataset standardizes this reasoning setting by adopting a fixed multi-choice format with five options, labeled A–E. Option E is always an interference option such as “None of the above,” “Both A and C,” or “All of the above,” and is sometimes correct. Each option is accompanied by a concise rationale, and the rationale design explicitly requires justifications for both the correct answer and each distractor using only cues from the base view and commonsense, without panorama leakage. This choice-and-rationale structure is central to the OpenView paradigm because the benchmark jointly evaluates answer selection and explanatory adequacy rather than only final choice prediction.

2. Dataset construction pipeline

OpenView is described as a fully agentic, four-stage pipeline that synthesizes high-quality OOV VQA from diverse real-world panoramas by leveraging Qwen2.5-VL-72B as the generation assistant and applying explicit quality control (Chen et al., 21 Dec 2025). The pipeline converts raw panorama resources into grounded multi-choice OOV questions through staged filtering, semantic analysis, proposal generation, and refinement.

Stage	Name	Function
1	Data Preprocessor	Panorama filtering
2	Visual Analyzer	Multi-view semantic and spatial annotation
3	Proposal Generator	OOV multi-choice synthesis
4	Proposal Refiner	Quality control and augmentation

The Data Preprocessor draws from five public panorama datasets: Matterport3D, Mapillary Metropolis, 360Loc, 360+x, and 360-1M. The reported collection consists of 5,289 pano images and 3,034 pano videos; videos are uniformly sampled to five frames where applicable, and the final filtered pool yields 16k panoramas used for synthesis. The panoramas are represented in equirectangular projection (ERP) with a $2{:}1$ aspect ratio, and high-resolution street and web subsets include frames above $4096\times 2048$. Prompt-based quality checks remove non-ERP formats such as dual-fisheye, cube-map, and “little planet,” along with heavy artifacts, overlays or watermarks, low-visibility scenes, and synthetic or virtual renders. For videos, additional consistency checks remove clips with multiple invalid frames.

The Visual Analyzer decomposes each ERP panorama into 12 perspective views arranged in a $3\times 4$ grid, with zero roll, mild top-bottom overlap, and explicit yaw-pitch extrinsics. For each view, the assistant model produces a concise caption, an object list localized with a fixed 9-grid directional vocabulary $$\{\text{top-left, top, top-right, left, center, right, bottom-left, bottom, bottom-right}\}$$, and spatial facts such as relative positioning or functional relations. These view-level analyses are then aggregated into a panorama summary, a scene category from an 11-class taxonomy, and an indoor/outdoor label; neighbor view indices are attached to support later grounding.

The Proposal Generator chooses a base view using normalized UV coordinates on the ERP, with roll fixed to $0^\circ$, diagonal field of view in $[40^\circ, 100^\circ]$, and aspect ratio in $$\{4{:}3, 3{:}4, 3{:}2, 2{:}3, 16{:}9, 9{:}16, 1{:}1\}$$. The stated goal is to balance context sufficiency against non-triviality. Two few-shot system prompts define contextual and directional generation rules. For directional tasks, camera rotation instructions such as “turn left $45^\circ$ and tilt up slightly” are attached, and a neighbor view is used only as hidden ground truth. The generator outputs five answer options, concise option-wise rationales, and a confidence score in $\{1,2,3\}$.

The Proposal Refiner retains only proposals with full confidence $2{:}1$0, repairs formatting to valid JSON, and removes trivial or poorly formed questions. It also performs two augmentation steps: option shuffling to remove positional bias while preserving rationales, and view jittering, where yaw and pitch are perturbed by $2{:}1$1 without changing semantics. The accompanying ablation study reports that removing the Visual Analyzer harms performance, while omitting the Proposal Refiner degrades robustness; option shuffling and view jittering improve generalization. This suggests that structured spatial annotation and post-generation quality control are not incidental preprocessing conveniences but substantive contributors to downstream OOV performance.

3. Spatial grounding, camera model, and OOV determination

OpenView-Dataset grounds OOV questions through panoramic geometry rather than through unconstrained linguistic speculation (Chen et al., 21 Dec 2025). The base perspective crop is the only image shown to a model at inference time, while neighboring crops hidden from the model guide correct and incorrect options during synthesis. Although question synthesis is LLM-driven, framing, cropping, and directional reasoning follow standard ERP and pinhole projection.

For equirectangular mapping from spherical coordinates $2{:}1$2, where $2{:}1$3, $2{:}1$4, and ERP image width and height are $2{:}1$5 and $2{:}1$6, the mapping is

$2{:}1$7

From normalized ERP coordinates $2{:}1$8, the center yaw and pitch are

$2{:}1$9

Perspective intrinsics are derived from the diagonal field of view $4096\times 2048$0 and aspect ratio $4096\times 2048$1. With $4096\times 2048$2, OpenView gives

$4096\times 2048$3

so that

$4096\times 2048$4

and the focal length satisfies

$4096\times 2048$5

The pinhole projection is written as

$4096\times 2048$6

with homogeneous normalization for pixel coordinates, where $4096\times 2048$7 is a unit 3D direction in camera coordinates, $4096\times 2048$8 is the world-to-camera rotation, and $4096\times 2048$9 is the intrinsic matrix.

OpenView formulates visibility through frustum-relative angular offsets. Given camera forward unit vector $3\times 4$0 determined by $3\times 4$1 and a unit direction $3\times 4$2 with spherical bearings $3\times 4$3, the horizontal and vertical offsets are

$3\times 4$4

A direction is visible iff

$3\times 4$5

otherwise it is OOV. For directional tasks, new view centers $3\times 4$6 are obtained by applying the rotation instruction, and correctness is tied to content that becomes visible in the rotated frustum. The system fixes roll to $3\times 4$7 to avoid mixing camera geometry complexities with OOV reasoning. This design choice simplifies the visibility test and narrows the benchmark to extrapolation under controlled free-view framing rather than full six-degree-of-freedom camera reasoning.

4. Composition, annotation schema, and corpus characteristics

OpenView-Dataset contains 158,352 OOV multi-choice VQA instances synthesized from 16k filtered panoramas (Chen et al., 21 Dec 2025). Each question has five options and option-wise rationales. The question types are balanced between contextual and directional items. Linguistically, contextual prompts frequently follow stems such as “Which of the following …,” whereas directional prompts frequently follow stems such as “If you turn …”.

The scenes span 11 categories: Civic, Rural, Nature, Culture, Heritage, Transport, Education, Hospitality, Workplace, Residential, and Commercial. The corpus includes both indoor and outdoor scenes and draws on diverse environments across the source datasets. The language of questions, options, and rationales is English. Reported lengths indicate questions around 15 words, options around 4 words, and detailed rationales peaked around approximately 190 words; the long rationale targets are intended to encourage explicit reasoning.

The training set is referred to as OpenView-Dataset, while the evaluation set is OpenView-Bench, constructed from separate panoramas and excluding training panoramas. No official train/validation/test split is mandated for the training set, whereas the benchmark is fixed. This division reflects an emphasis on benchmark standardization rather than prescriptive training partitioning.

Files are organized by panorama, with perspective crops saved under an images/ directory and annotations in JSON. An annotation entry encodes panorama metadata, the chosen base view, the question and five options, the correct option, rationales, and links to the crop. The example schema includes fields such as panorama_id, panorama_path, projection, scene_label, outdoor, a nested base_view object containing normalized coordinates, yaw, pitch, diagonal FoV, aspect ratio, crop path, and neighbor indices, as well as question, options, correct_option, rationales, and confidence_score. Because the crop path identifies the perspective image shown at inference, the annotation format preserves the distinction between hidden panorama-derived grounding and observable model input.

5. Benchmark protocol and empirical results

OpenView-Bench is a manually verified benchmark containing 1,327 OOV VQA items, specifically 665 contextual and 662 directional items, with answers distributed nearly evenly across A–E: A: 268, B: 269, C: 267, D: 260, E: 263 (Chen et al., 21 Dec 2025). Human participants achieve 83.0% choice accuracy, and the benchmark is designed to quantify both correctness and interpretability.

Three metrics are defined per model. Choice accuracy is

$3\times 4$8

Rationale accuracy is

$3\times 4$9

where rationale correctness is judged in binary form by DeepSeek V3.1 against human-revised rationales, focusing on evidential support, logical consistency, and completeness. Joint accuracy is

$$\{\text{top-left, top, top-right, left, center, right, bottom-left, bottom, bottom-right}\}$$0

Choice evaluation uses regex-based exact match, and the unified inference prompt requests option-wise reasoning first, followed by a final answer enclosed in <answer>...</answer>.

The reported baseline comparison shows a substantial gap between humans and current MLLMs on OOV understanding. Proprietary models such as Gemini-2.5-Flash and GPT-5 lead overall joint accuracy, largely because of strong rationale reliability, but they still trail human choice accuracy. Among open-source systems, supervised fine-tuning on OpenView-Dataset yields the following reported gains:

Model	Joint accuracy	Rationale accuracy
LLaVA-NeXT-Mistral-7B	15.82% $$\{\text{top-left, top, top-right, left, center, right, bottom-left, bottom, bottom-right}\}$$1 49.74%	44.59% $$\{\text{top-left, top, top-right, left, center, right, bottom-left, bottom, bottom-right}\}$$2 81.89%
InternVL3.5-8B	34.14% $$\{\text{top-left, top, top-right, left, center, right, bottom-left, bottom, bottom-right}\}$$3 48.08%	65.94% $$\{\text{top-left, top, top-right, left, center, right, bottom-left, bottom, bottom-right}\}$$4 82.22%
Qwen2.5-VL-7B-Instruct	33.38% $$\{\text{top-left, top, top-right, left, center, right, bottom-left, bottom, bottom-right}\}$$5 57.65%	61.36% $$\{\text{top-left, top, top-right, left, center, right, bottom-left, bottom, bottom-right}\}$$6 85.96%
Qwen3-VL-8B-Instruct	48.08% $$\{\text{top-left, top, top-right, left, center, right, bottom-left, bottom, bottom-right}\}$$7 58.78%	90.88% $$\{\text{top-left, top, top-right, left, center, right, bottom-left, bottom, bottom-right}\}$$8 84.05%

Across multiple MLLMs, the paper reports that OpenView fine-tuning boosts average joint accuracy from 48.6% to 64.1%. For Qwen3-VL-8B-Instruct, rationale accuracy shows a slight drop while joint accuracy improves, indicating that better answer selection and better rationale scoring need not move in lockstep. More generally, the benchmark treats rationale production as an independently diagnosable competence rather than a by-product of choice accuracy.

6. Training usage, positioning, and limitations

The recommended training setup uses the perspective crop as the image input, thereby mirroring evaluation, while panorama-derived annotations are used only for generation and quality control rather than inference (Chen et al., 21 Dec 2025). The text input consists of the question stem, five options, and rationales as targets under teacher forcing. The suggested losses are a multi-choice cross-entropy term,

$$\{\text{top-left, top, top-right, left, center, right, bottom-left, bottom, bottom-right}\}$$9

a teacher-forced rationale generation loss,

$0^\circ$0

and the joint objective

$0^\circ$1

with $0^\circ$2 reported to work well in practice.

The reported fine-tuning configuration employs LoRA with frozen vision towers. Typical settings are batch size 8 for LLaVA-NeXT-Mistral-7B and InternVL3.5-8B with LoRA rank 8 and $0^\circ$3; batch size 128 for Qwen2.5-VL-7B and Qwen3-VL-8B with LoRA rank 64 and $0^\circ$4; training for 1 epoch at learning rate $0^\circ$5 on $0^\circ$6. Practical preprocessing recommendations include verifying ERP $0^\circ$7 panoramas, rejecting non-ERP and low-quality frames, normalizing UV coordinates to $0^\circ$8, converting them to yaw and pitch with the specified formulas, deriving $0^\circ$9 and $[40^\circ, 100^\circ]$0 from $[40^\circ, 100^\circ]$1 and aspect ratio, generating zero-roll perspective crops, and using small jitter of $[40^\circ, 100^\circ]$2 for augmentation.

In comparative positioning, OpenView-Dataset is described as unique in combining ERP panoramas for full spatial coverage and consistent grounding, free-view framing with controlled camera parameters and zero roll, multi-choice design with interference options and detailed option-wise rationales, and a matched benchmark with joint choice-and-rationale metrics. This suggests a deliberate attempt to define OOV VQA as a separable research problem rather than a minor extension of standard VQA.

The paper also states several limitations. Because question synthesis is LLM-driven, subtle biases or style regularities may appear in question stems and rationales despite quality control. ERP distortion complicates low-level geometry, and the design mitigates this by fixing roll to $[40^\circ, 100^\circ]$3 and using coarse regional tokens, which may limit advanced camera reasoning. OOV correctness is grounded by neighbor views rather than by full 3D reconstruction, so failures may arise when the base view provides insufficient cues or neighboring frames are ambiguous. Future directions named in the paper include extending OOV supervision to video, strengthening world modeling, and using 3D geometry to tighten spatial visibility tests and rationale evaluation.