Bridging 3D Gaussians and Semantic Occupancy for Comprehensive Open-Vocabulary Scene Understanding from Unposed Images

Abstract: Comprehensive 3D scene understanding from sparse, unposed images requires a model to recover renderable geometry, open-vocabulary semantics, and free/occupied 3D space without relying on external camera calibration. Recent feed-forward Gaussian methods improve pose-free reconstruction and semantic rendering, but their Gaussian primitives are mainly optimized through image-space objectives and remain weakly constrained in unobserved regions. We propose \textit{COVScene}, a pose-free semantic Gaussian framework that couples renderable Gaussian primitives with a dense semantic occupancy field through differentiable volumetric lifting. Instead of converting Gaussians to voxels only at evaluation time, COVScene lifts the predicted semantic Gaussians inside the training computation graph, so volumetric regularization provides gradients to Gaussian opacity, geometry, and semantic features. The framework combines a semantic-aware Geometry Transformer, multi-task Gaussian decoding, geometric foundation distillation, and occupancy entropy regularization to support novel view synthesis, open-vocabulary semantic querying, and semantic occupancy prediction within a single representation. Experiments on ScanNet and ScanNet++ show that COVScene maintains competitive rendering quality, improves open-vocabulary segmentation, and achieves stronger semantic occupancy prediction than the self-supervised baseline without direct voxel-level supervision.

• The paper introduces a unified framework that couples renderable 3D Gaussians with differentiable volumetric lifting to infer semantic occupancy from unposed multi-view images.
• The methodology integrates a semantic-aware geometry transformer and multi-task decoding, generating camera parameters, depth maps, and open-vocabulary embeddings.
• The approach demonstrates superior performance in semantic segmentation, novel view synthesis, and occupancy prediction via rigorous volumetric regularization and ablation studies.

COVScene: Unified 3D Gaussian and Volumetric Schema for Pose-Free, Open-Vocabulary Scene Understanding

Introduction and Motivation

COVScene addresses the under-constrained challenge of comprehensive 3D scene understanding from sparse, unposed multi-view imagery without reliance on calibrated poses. Instead of optimizing surface-centric Gaussian primitives solely through image-space supervision, COVScene enables a principled coupling between renderable 3D Gaussians and dense volumetric semantic occupancy fields by differentiable volumetric lifting. This approach supplies direct volumetric regularization during training, propagating gradients to color, geometry, opacity, and semantic features of the Gaussian scene representation. The unified model supports novel view synthesis, open-vocabulary semantic querying, depth estimation, and semantic occupancy prediction from a single forward pass.

Figure 1: COVScene jointly enables novel view synthesis, open-vocabulary segmentation, depth estimation, and semantic occupancy prediction directly from uncalibrated multi-view images.

COVScene Architecture and Semantic Geometry Encoding

The COVScene framework introduces several key architectural mechanisms to recover globally consistent semantic geometry from unposed multi-view images:

• Semantic-Aware Geometry Transformer: Adapted from VGGT, this Transformer backbone fuses intra/inter-view attention for geometry estimation and attaches language-aligned feature maps through fusion with pretrained 2D semantic extractors.
• Multi-Task Decoding: Specialized heads produce (1) camera parameters via global tokens; (2) per-view depth maps for initial 3D coordinate inference; and (3) semantic Gaussian attributes—including open-vocabulary embeddings—allowing a consistent 3D scene field.

Differentiable volumetric lifting transforms the discrete set of predicted semantic Gaussians into a continuous, dense occupancy field. This field, constructed through analytic accumulation of Gaussian density contributions, supports assignment of open-vocabulary semantic labels to occupied voxels via feature similarity with language/text embeddings.

Figure 2: Pipeline overview, showing semantic geometry encoding from unposed images and differentiable lifting of Gaussians into a dense, language-aligned occupancy field.

Training Objectives and Physics-Informed Volumetric Regularization

Training employs a composite loss comprising photometric, semantic, geometric, and volumetric terms:

• Geometric Distillation: Pseudo labels from a geometric foundation model constrain camera and depth estimation, critical for unposed inputs where 3D correspondences are inherently ambiguous.
• Photometric and Semantic Supervision: Standard 2D rendering and cross-modal feature alignment, where rendered Gaussians are supervised to reproduce both color and semantic map features.
• Entropy-Based Occupancy Regularization: A bimodal entropy penalty on lifted occupancy probabilities sharpens the free/occupied segmentation of the predicted volumetric field. This mechanism directly suppresses ambiguously occupied regions (floaters, artifacts) typical in weakly supervised 3DGS scenarios.

Figure 3: Visualization of the effect of occupancy entropy regularization, driving the volumetric field to a physically plausible bimodal structure and eliminating floating density.

Empirical Analysis and Ablation

Open-Vocabulary Segmentation

COVScene achieves competitive and, with more views, superior 3D open-vocabulary segmentation mIoU versus 2D-only and previous 3DGS baselines. The unified 3D semantic field yields increased cross-view consistency with more context images, outperforming models that process semantics separately per image or per surface fragment.

Figure 4: Qualitative open-vocabulary segmentation across ScanNet categories, demonstrating consistent 3D-Aware semantic assignments in COVScene's unified representation.

Novel View Synthesis

While specialized NVS models such as AnySplat maintain marginally higher PSNR in certain multiview regimes, COVScene matches or exceeds all baselines for LPIPS and mIoU and—crucially—supports additional tasks without performance tradeoff, indicating no compromise in rendering quality due to the volumetric coupling or multi-task optimization.

Figure 5: Novel view synthesis outputs demonstrate the preservation of photometric fidelity while supporting augmented semantic and occupancy prediction.

Semantic Occupancy Prediction

In the absence of direct voxel-level semantic supervision, COVScene's differentiable volumetric training yields a substantial improvement in semantic occupancy mIoU and per-class IoU over self-supervised and post-voxelization methods. Zero-shot qualitative results indicate successful inference of 3D spatial structures and semantics even for in-the-wild, annotation-free images.

Figure 6: Zero-shot semantic occupancy predictions: COVScene reconstructs physically and semantically plausible scene layouts without ground-truth voxel annotations.

Ablation—Crucial Mechanisms

• Removing pose distillation dramatically degrades all metrics, confirming the foundational role of geometry priors in unposed settings.
• Bypassing differentiable lifting substantially reduces 3D mIoU and NVS quality, substantiating the necessity of training-time volumetric coupling rather than post-hoc occupancy estimation.
• Omitting entropy regularization directly increases floating artifacts, demonstrates less certain segmentation, and reduces mIoU and PSNR.

Implications and Future Directions

COVScene demonstrates that tight coupling of semantic Gaussian primitives with a differentiable occupancy field enables physically grounded, unified scene representations from sparse, unposed images. The approach provides a path toward scalable embodied 3D scene understanding without extensive 3D semantic supervision or multi-stage geometric pipelines. Practically, this can facilitate robotics, AR, and embodied AI agents that require real-time, open-vocabulary semantic spatial reasoning with minimal input constraints.

Theoretically, the results substantiate that direct volumetric regularization propagates physically meaningful priors back to discrete semantic representations, thus reconciling efficient renderable models with geometry-aware, language-aligned occupancy in a single architecture.

Limitations lie in the computational load of joint optimization and focus on static indoor scenes. Prospectively, extending efficient volumetric coupling to dynamic or outdoor settings, scaling to globally consistent reconstruction over extended spaces, and integrating these methods into streaming or online contexts for real-world robotic use are pertinent directions.

Conclusion

COVScene introduces an end-to-end, pose-free framework that unifies high-fidelity 3D Gaussian rendering, open-vocabulary semantic querying, and explicit semantic occupancy prediction. Its differentiable volumetric lifting mechanism distinguishes it from prior works reliant on surface-centric, evaluation-only occupancy inference, and is reflected in strong empirical gains on semantic occupancy mIoU and cross-view 3D segmentation. The ablation demonstrates that each volumetric and geometric regularization component is critical for robust, physically consistent 3D understanding in weakly supervised regimes. This model represents a robust step toward generalizable, scene-centric AI systems with efficient, jointly learnable geometric and semantic 3D priors.