OneCanvas: 3D Scene Understanding via Panoramic Reprojection

Abstract: Existing approaches to 3D scene understanding in Vision-LLMs (VLMs) either rely on complex, model-specific geometry encoders or large training budgets in pursuit of spatial reasoning. Instead, OneCanvas aggregates patch features from all views onto a single equirectangular panoramic canvas. Namely, each patch is unprojected to a 3D world coordinate using its depth and camera pose, then placed on the canvas at the continuous longitude and latitude of that point as seen from the canvas origin, with no rasterization or aggregation across overlapping views. A 3D position embedding of the patch's metric coordinates is added to its feature, restoring the depth lost when collapsing the world position to an angular canvas coordinate. Patches from all frames thus share one spatial coordinate system with no fusion or major architectural modifications of the backbone. The pretrained VLM consumes this representation as if it were an ordinary image. Because the canvas can be centered on any pose of interest, the same representation directly supports situated reasoning from a specific viewpoint, a common requirement in robotics and embodied AI. Thanks to this representation, we can also introduce a spatial pretraining curriculum: by procedurally placing patch features of objects, drawn from real images, at chosen 3D world positions on an otherwise empty canvas, we generate on-the-fly supervision spanning a broad range of spatial reasoning tasks, with answer distributions controlled to reduce spatial reasoning shortcuts. OneCanvas achieves state-of-the-art accuracy on SQA3D and VSI-Bench, and generalizes to out-of-distribution data on SPBench, using an order of magnitude less training compute than the strongest competing methods.

• The paper introduces a panoramic reprojection framework that unifies multi-view 3D features into a single input canvas for vision-language models.
• The paper integrates a spatial pretraining curriculum using synthetic tasks to enhance metric, navigational, and spatial QA capabilities.
• The paper demonstrates state-of-the-art results on SQA3D, VSI-Bench, and SPBench with significantly reduced training compute.

OneCanvas: Panoramic Reprojection for Efficient 3D Scene Understanding in VLMs

Motivation and Background

Advancements in Vision-LLMs (VLMs) have precipitated the need for robust 3D scene understanding, critical for embodied AI and spatial intelligence applications. Conventional approaches augment 2D VLMs with elaborate geometric encoders and extensive spatial QA datasets, yet often fail to fully utilize geometric input, reverting instead to textual and scene priors. Recent audits indicate weak spatial reasoning despite architectural complexity and data scale (Ma et al., 6 Mar 2026). OneCanvas introduces a paradigm shift: it leverages panoramic reprojection and a unified input canvas to enable comprehensive spatial reasoning without architecture modifications, prioritizing representation design for maximal compatibility with pretrained VLMs.

Figure 1: OneCanvas aggregates per-frame patch features via backprojection to 3D, subsequent placement in a common reference frame, and reprojection onto a panoramic canvas as a single image for VLM consumption.

Method Overview

OneCanvas operates by extracting patch features from multi-view RGB-D data, projecting them to 3D world coordinates using depth and camera pose, and situating each patch at its continuous angular position (longitude, latitude) on a panoramic equirectangular canvas centered on an arbitrary viewpoint. Critical elements include:

• Continuous Placement: All lifted patches retain individual token identity and coordinate, eschewing rasterization and aggregation; co-located tokens are resolved via VLM attention.
• 3D Position Embedding: Metric coordinates are encoded per patch (sinusoids, radials, and unit-direction), supplementing the angular placement to recover depth information lost in projection.
• Native RoPE Utilization: The representation is ingested by Qwen3-VL, leveraging its 3D Rotary Positional Embedding capacities without modification.

  Figure 2: Overview of OneCanvas pipeline—multi-view features encoded, lifted, and placed onto a panoramic canvas; consumed seamlessly by VLM via attention layers.

This design aligns per-frame patch features into a shared spatial coordinate system, enabling the VLM to process the 3D scene as a familiar image input, supporting both scene-centric and agent-centric spatial reasoning.

Spatial Pretraining Curriculum

A novel spatial pretraining curriculum is enabled by the panoramic representation. Synthetic objects are procedurally placed at arbitrary 3D coordinates; corresponding patch features are referenced within prompts to enforce geometric causality in supervision. The curriculum covers:

• Metric measurements: Pairwise distances, area computation.
• Egocentric direction: Left/right/front/back queries from arbitrary viewpoints.
• Navigation: Multi-turn route planning.
• Observability: Appearance ordering, line-of-sight visibility.
• Counting: Object counts including parity/divisibility.
• Bounding-box readouts: 3D localization.

Answer distributions are tightly controlled to suppress statistical shortcuts, ensuring the model learns genuine spatial reasoning rather than scene or language priors.

Training and Adaptation

Training proceeds in two stages:

1. Stage 1: Spatial pretraining, optimizing LoRA adapters and the 3D position embedding solely via curriculum tasks, yielding geometric reading skills without scene-class biases.
2. Stage 2: Target QA adaptation by merging stage-1 weights into the base model, then fine-tuning a fresh, lower-rank LoRA adapter on spatial QA corpora (SQA3D, VSI-Bench, and ViCA).

   Figure 3: Two-stage training sequence—first stage pretrains LoRA/3D embedding on synthetic spatial tasks, second stage adapts via downstream QA fine-tuning.

Token embeddings remain frozen throughout both stages, ensuring transferability and stability.

Experimental Evaluation

OneCanvas achieves state-of-the-art accuracy on SQA3D (65.3 EM@1, +2.3pt over previous best), VSI-Bench (70.1 avg), and SPBench zero-shot (72.1 overall, +4.8pt margin), utilizing an order of magnitude less training compute compared to dominant methods.

Figure 4: Benchmark comparisons—SQA3D per-question-type accuracy, SPBench zero-shot performance, and training compute vs. VSI-Bench accuracy.

Ablation studies confirm the necessity of each design element:

• Panoramic representation significantly improves cross-scene spatial tasks.
• 3D position embedding advances metric reading capabilities.
• Spatial pretraining is indispensable for route planning and nuanced spatial queries.

Canvas origin selection directly influences situated question performance, favoring agent-centric placement for viewpoint-dependent tasks.

Practical and Theoretical Implications

OneCanvas demonstrates that carefully engineered input representations deliver genuine spatial reasoning in VLMs without reliance on architectural overhauls or extensive dataset curation. The panoramic reprojection aligns multi-view input with the VLM's native processing style, enabling efficient training and strong generalization with modest compute requirements. The spatial pretraining curriculum induces metric and geometric capabilities even in models originally agnostic to 3D input.

Practically, this approach equips VLMs for robotics, augmented reality, and navigation scenarios where flexible viewpoint-centric reasoning is required. Theoretically, it substantiates the importance of representation design in unlocking latent spatial intelligence, suggesting broader applicability for panoramic canvases and synthetic curricula across multimodal models.

Limitations and Future Directions

OneCanvas depends on depth and pose availability, a constraint mitigated but not eliminated by recent feed-forward metric reconstruction. Large-scale or outdoor scenes may tax the global panoramic canvas's spatial precision, and creation of novel spatial tasks requires explicit curriculum engineering. Current experiments are limited to Qwen3-VL backbones; extension to other architectures with appropriate positional encoding mechanisms is pending.

Potential future developments include:

• Integration with pure RGB-driven pipelines via robust monocular depth and pose estimation.
• Application to outdoor and vast multi-room environments.
• Expansion of curriculum-driven pretraining for additional spatial reasoning tasks.
• Generalization to a broader set of VLM architectures.

Conclusion

OneCanvas reframes 3D scene understanding as panoramic 2D reasoning, unifying multi-view input into a single spatially consistent canvas consumed directly by existing VLMs. Through metric position embedding and spatially causal pretraining, it achieves superior spatial intelligence with minimal compute and infrastructure overhead. The methodology emphasizes input representation as the critical enabler for geometric reasoning in vision-LLMs, opening avenues for efficient embodied AI deployment and robust spatial QA benchmarking.