Occ-VLM: Occupancy Grounded Vision Language Model for Indoor Scene Understanding

Abstract: Recently, vision-LLMs (VLMs) have made significant progress in 3D scene understanding, driving advances in applications such as embodied intelligence and robotic vision. However, existing approaches typically either rely directly on explicit 3D inputs (e.g., point clouds or RGB-D sequences), or introduce an additional 3D geometry encoder to derive 3D-aware visual tokens from 2D images. Such designs structurally decouple 3D geometric perception from the rich 2D semantics learned via vision-language pre-training, hindering the development of a unified 3D vision-language representation. In this work, we propose Occ-VLM, a novel framework for 3D scene understanding that operates purely on posed RGB images and employs a single 2D vision encoder. Specifically, Occ-VLM reconstructs 3D scene occupancy as an auxiliary geometric prior, which is utilized to spatially associate foreground 2D tokens with 3D space. These tokens are then decoded by a LLM for unified scene understanding. Extensive experiments demonstrate that Occ-VLM achieves both accurate geometric perception and robust vision-language reasoning: it attains state-of-the-art performance on multi-view occupancy prediction, while performing on par with 3D-input VLMs on 3D Visual Question Answering (VQA) and 3D dense captioning benchmarks.

• The paper introduces an RGB-only framework for unified 3D indoor scene understanding using an Occ adapter for occupancy-aware tokenization.
• It achieves significant performance gains on occupancy prediction and vision-language tasks, outperforming both RGB-only and 3D-input models.
• The model demonstrates computational efficiency by reducing FLOPs while delivering high semantic accuracy and robust spatial reasoning.

Occ-VLM: Occupancy Grounded Vision-LLM for Unified 3D Indoor Scene Understanding

Motivation and Architectural Innovations

Occ-VLM addresses structural limitations in current 3D VLM paradigms by introducing an RGB-only framework for indoor scene understanding that dispenses with explicit 3D data (point clouds, RGB-D) and avoids the architectural complexity of dual-encoder systems. Existing approaches structurally decouple 2D semantic encoding and 3D geometric perception; Occ-VLM unifies these through a single frozen 2D vision encoder, leveraging an auxiliary Occ adapter for geometric reconstruction and spatial token association. This enables state-consistent multi-view reasoning with only posed RGB images.

Figure 1: Overview of paradigm shifts in 3D VLM architectures, highlighting Occ-VLM’s unified RGB-only approach using a single 2D encoder.

Figure 2: The Occ-VLM pipeline: posed RGB images → 2D vision encoder → Occ Adapter/Occupancy Decoder → foreground token sampling → integration of 3D positional cues, culminating in vision-language reasoning via an LLM.

Methodology

Occ Adapter and Occupancy-Aware Tokenization

The Occ adapter selectively activates intermediate layers of a frozen pre-trained vision transformer to extract multi-scale semantic features, aggregates these through a feature pyramid, and projects them into spatial grids using back-projection and differentiable sampling. A 3D CNN then predicts semantic occupancy per voxel. Foreground tokens are sampled from occupied grid centers; these are injected with 3D positional encoding, preserving spatial anchoring throughout the reasoning pipeline.

Figure 3: Occ adapter architecture: multi-scale activation, token aggregation, 3D decoding with anchor projection and grid sampling.

This bidirectional 2D–3D modeling is crucial. The semantic knowledge distilled from the 2D encoder can directly enhance geometric reconstruction, while the resulting spatial priors steer foreground token selection, fostering robust spatial reasoning. The foreground tokens are treated as temporally ordered sequences compatible with video-language pre-training, allowing the LLM to exploit implicit cross-view correlations.

Empirical Evaluation

3D Semantic Occupancy Benchmark

Occ-VLM achieves an mIoU of 18.41% on the EmbodiedScan occupancy prediction benchmark, substantially outperforming both RGB-based and point cloud-based methods—demonstrating that occupancy-aware RGB-only modeling can surpass explicit 3D inputs in geometric perception. The model delivers high semantic accuracy and complete geometric reconstruction across diverse categories.

Figure 4: Qualitative occupancy prediction results for ScanNet and 3RScan scenes, demonstrating semantic accuracy.

3D Visual Language Reasoning

On ScanQA and SQA3D, Occ-VLM attains EM@1 scores of 29.6% and 58.5% respectively, exceeding all baselines among RGB-only VLMs—even those employing dual-encoder or neural rendering techniques. Occ-VLM is competitive with models utilizing explicit 3D inputs, with only marginal performance gaps (<2%) to the current 3D-input SOTA. On Scan2Cap, for dense captioning, Occ-VLM’s [email protected] is 76.5, again outperforming other RGB-only models and rivaling deep 3D-input VLMs.

Figure 5: Qualitative outputs on ScanQA, showing occupancy prediction and VQA results (scene relationships, object recognition, attribute identification, object counting).

Ablation Studies

Token Representation and Positional Encoding

Ablation demonstrates that “video” temporal ordering of tokens outperforms spatial aggregation indexed by occupancy grids. Removing 3D positional encoding results in severe degradation of captioning and reasoning metrics, confirming that explicit spatial anchoring is essential for fine-grained 3D reasoning.

Input View Number

Performance positively correlates with number of input views: more views improve occupancy prediction mIoU and downstream vision-language scores via expanded coverage and reduced occlusion. As occupancy accuracy increases, the proportion of foreground tokens decreases, indicating improved recall and background filtering.

Figure 6: Occupancy performance and foreground token ratio as a function of input view number.

Computational Efficiency

Occ-VLM avoids the computational overhead of geometry-specific encoders (e.g., VGGT, Depth-Anything v2), requiring only fractional FLOPs (2.6T vs. 38.2T) for occupancy-aware tokenization, due to its modular adapter and frozen shared vision backbone. This architectural simplicity enhances extensibility and inference speed.

Practical and Theoretical Implications

Occ-VLM demonstrates that 3D semantic reasoning and spatial understanding can be achieved via RGB-only input using a single 2D backbone, provided appropriate occupancy prediction and spatial token sampling. This enables scalable deployment of 3D scene understanding systems without depth sensors and point cloud pre-processing. The results highlight that geometric priors can be distilled from 2D semantics, unifying spatial perception and vision-language reasoning in a principled manner. This paradigm may inform future LVM design for embodied AI, robotics, and augmented reality, where deployment efficiency and robustness in RGB-only contexts are critical.

Figure 7: Qualitative results for 3D dense captioning (Scan2Cap), showing spatially grounded object descriptions.

Conclusion

Occ-VLM represents a concise, high-efficiency approach to unified 3D indoor scene understanding. By grounding geometric perception in occupancy-aware tokenization and leveraging pre-trained 2D semantic knowledge, it achieves highly competitive performance across geometric and vision-language reasoning tasks, with substantial improvements over prior RGB-only and explicit 3D-input models. The demonstrated scalability and architectural simplicity establish a new standard for vision-LLMs targeting 3D spaces, with implications for real-world embodied intelligence applications (2606.19776).