GS-Occ3D: Vision-Only Occupancy Reconstruction

• The paper introduces GS-Occ3D, a vision-only framework that reconstructs 3D occupancy using an explicit Octree-based Gaussian surfel formulation, eliminating LiDAR dependency.
• It decomposes scenes into static background, ground plane, and dynamic objects, employing tailored modeling strategies to enhance reconstruction fidelity.
• Experiments on the Waymo dataset show state-of-the-art geometry reconstruction and competitive downstream performance with superior zero-shot generalization.

Searching arXiv for GS-Occ3D and related occupancy/Gaussian-splatting papers. GS-Occ3D is a scalable vision-only framework that directly reconstructs occupancy for autonomous driving by optimizing an explicit occupancy representation using an Octree-based Gaussian Surfel formulation, and by decompos[ing] scenes into static background, ground, and dynamic objects, enabling tailored modeling strategies (Ye et al., 25 Jul 2025). It addresses a setting in which occupancy is crucial for autonomous driving, providing essential geometric priors for perception and planning, while avoiding the dependence of existing pipelines on LiDAR-based occupancy annotations, which limits scalability and prevents leveraging vast amounts of potential crowdsourced data for auto-labeling (Ye et al., 25 Jul 2025). In the reported experiments on the Waymo dataset, the framework is presented as achieving state-of-the-art geometry reconstruction results, and the resulting vision-only binary occupancy labels are used to train downstream occupancy models with competitive transfer and superior zero-shot generalization on Occ3D-nuScenes (Ye et al., 25 Jul 2025).

1. Problem setting and motivation

Vision-only occupancy reconstruction is framed around the recovery of a binary 3D grid of “free” vs. “occupied” voxels from multi-camera imagery. In this formulation, occupancy provides a 3D geometric prior that is critical for downstream perception (object detection, semantic understanding) and planning (collision avoidance) (Ye et al., 25 Jul 2025).

The motivation in GS-Occ3D is explicitly tied to the limitations of LiDAR-centric pipelines. Existing methods are described as relying almost exclusively on LiDAR-based occupancy labels, which require expensive, specialized LiDAR rigs and limit data diversity and scalability. A vision-only alternative is therefore positioned as a route to vast amounts of inexpensive, crowdsourced RGB video and to self-supervised/auto-labeling possibilities, reducing annotation cost (Ye et al., 25 Jul 2025).

The paper emphasizes four difficulties specific to this setting. First, sparse viewpoints arise because Waymo uses only five cameras over ≈1,000 frames per scene, leaving large regions under-observed. Second, dynamic scenes & occlusions introduce time-varying visibility changes. Third, long-horizon motion causes drift and inconsistency along long trajectories. Fourth, prior vision-based approaches are characterized as relying on mesh or implicit surfaces and as suffering from holes, over-smoothness, or require expensive post-processing, which makes them ill-suited for large-scale auto-labeling (Ye et al., 25 Jul 2025).

Within that problem setting, GS-Occ3D is defined by two core responses: an explicit Octree-based Gaussian Surfel representation, and a three-way scene decomposition (static background, ground plane, dynamic objects) (Ye et al., 25 Jul 2025). This pairing is the structural basis for both reconstruction fidelity and large-scale label curation.

2. Octree-based Gaussian Surfel representation

GS-Occ3D models the scene as a hierarchy of 3D “surfels”—each surfel is a 3D Gaussian with an anisotropic covariance, a color mean, and an opacity, and these surfels are stored in the leaves of a dynamically adapting octree (Ye et al., 25 Jul 2025). The representation is explicit rather than implicit, and the octree is used to maintain efficiency across large outdoor scenes.

The hierarchy is parameterized by the minimum and maximum distances from the SfM camera centers to the sparse SfM points, denoted $d_{\min}, d_{\max}$. The number of octree levels is given as

$$K \;=\;\bigl\lfloor \log_2(\tfrac{d_{\max}}{d_{\min}})\bigr\rceil + 1\,.$$

At level $L\in\{0,\dots,K-1\}$, the base voxel size is $\epsilon/2^L$, and sparse points $\{P_i\}$ are quantized by

$$V_L \;=\;\Bigl\lfloor \tfrac{P_i}{\epsilon/2^L}\Bigr\rceil\,\cdot\,(\epsilon/2^L)\,,$$

so that voxels across levels remain spatially aligned. Each non-empty leaf is allowed up to $m$ Gaussian surfels (Ye et al., 25 Jul 2025).

A single surfel $j$, at position $\mu_j\in\mathbb R^3$, with covariance $\Sigma_j$ and color $$K \;=\;\bigl\lfloor \log_2(\tfrac{d_{\max}}{d_{\min}})\bigr\rceil + 1\,.$$0, contributes a density field

$$K \;=\;\bigl\lfloor \log_2(\tfrac{d_{\max}}{d_{\min}})\bigr\rceil + 1\,.$$1

with opacity $$K \;=\;\bigl\lfloor \log_2(\tfrac{d_{\max}}{d_{\min}})\bigr\rceil + 1\,.$$2. The resulting volumetric density is

$$K \;=\;\bigl\lfloor \log_2(\tfrac{d_{\max}}{d_{\min}})\bigr\rceil + 1\,.$$3

This is the geometric substrate that is later converted into occupancy labels (Ye et al., 25 Jul 2025).

Rendering is performed by projecting each 3D Gaussian as an anisotropic 2D Gaussian. Along a camera ray $$K \;=\;\bigl\lfloor \log_2(\tfrac{d_{\max}}{d_{\min}})\bigr\rceil + 1\,.$$4, the accumulated opacity and color are written as

$$K \;=\;\bigl\lfloor \log_2(\tfrac{d_{\max}}{d_{\min}})\bigr\rceil + 1\,.$$5

The implementation is described as efficiently discretizing the contributions of each surfel, while the rendering loss $$K \;=\;\bigl\lfloor \log_2(\tfrac{d_{\max}}{d_{\min}})\bigr\rceil + 1\,.$$6 uses an $$K \;=\;\bigl\lfloor \log_2(\tfrac{d_{\max}}{d_{\min}})\bigr\rceil + 1\,.$$7 + D-SSIM term against the input images (Ye et al., 25 Jul 2025).

3. Scene decomposition and optimization strategy

A central feature of GS-Occ3D is its decomposition of the scene into static background, ground plane, and dynamic vehicles, with a different modeling strategy for each component (Ye et al., 25 Jul 2025).

For the static background, all non-ground, non-dynamic pixels are assigned to a shared octree of Gaussian surfels. This design is described as encouraging hierarchical LOD and multi-scale fidelity (Ye et al., 25 Jul 2025).

For the ground plane, the method exploits the fact that roads often dominate large horizontal extents and are weakly textured. GS-Occ3D therefore initializes a dedicated set of ground surfels by projecting camera poses onto XY-plane and lifting with a height offset. A planar regularization term

$$K \;=\;\bigl\lfloor \log_2(\tfrac{d_{\max}}{d_{\min}})\bigr\rceil + 1\,.$$8

is used to enforce consistent, nearly planar structure across the road surface (Ye et al., 25 Jul 2025). This explicit treatment of ground is one of the mechanisms claimed to significantly improve large-area consistency (Ye et al., 25 Jul 2025).

For dynamic vehicles, instances are first detected and 3D-tracked via an RGB-only tracker (e.g. CC3DT). Each vehicle has a bounding box and an initial pose $$K \;=\;\bigl\lfloor \log_2(\tfrac{d_{\max}}{d_{\min}})\bigr\rceil + 1\,.$$9, which is then refined by learnable deltas: $L\in\{0,\dots,K-1\}$0 where $L\in\{0,\dots,K-1\}$1 arises from a learnable yaw offset $L\in\{0,\dots,K-1\}$2. Inside each box, a fixed number of Gaussians are initialized and optimized to capture the object-specific occupancy (Ye et al., 25 Jul 2025). This decoupling is introduced specifically because dynamic vehicles are separately modeled to better capture motion-related occupancy patterns (Ye et al., 25 Jul 2025).

The overall training objective is

$L\in\{0,\dots,K-1\}$3

Here, $L\in\{0,\dots,K-1\}$4 is a surfel flattening regularizer, $L\in\{0,\dots,K-1\}$5 is depth distortion, $L\in\{0,\dots,K-1\}$6 is normal–depth consistency, $L\in\{0,\dots,K-1\}$7 is entropy on the object opacity map, $L\in\{0,\dots,K-1\}$8 is the road smoothness term, and $L\in\{0,\dots,K-1\}$9 is a binary cross-entropy on rendered sky-mask (Ye et al., 25 Jul 2025). The structure of the loss reflects the paper’s attempt to combine photometric reconstruction with geometric and scene-structure constraints.

4. Occupancy label curation

After reconstruction, GS-Occ3D converts the optimized Gaussian Surfel scene into a binary occupancy target. The paper states that the model first produces a dense point cloud by sampling surfels—still sparse compared to LiDAR. The conversion to binary voxels then proceeds through frame-wise division, multi-frame aggregation, and ray-casting & voxelization (Ye et al., 25 Jul 2025).

In the frame-wise division stage, the method defines a per-frame “LiDAR-like” sweep by sampling all reconstructed points within a radius $\epsilon/2^L$0 of the camera pose, so that each frame has a roughly uniform point count (Ye et al., 25 Jul 2025).

In the multi-frame aggregation stage, static points need no aggregation, whereas dynamic object points are aggregated across all frames in the object’s track. The method transform[s] each frame’s vehicle-points into the box frame, concatenate[s], and then re-project[s] to each frame to densify the object (Ye et al., 25 Jul 2025). This is the mechanism by which temporal observations are converted into denser per-frame occupancy evidence for moving objects.

For ray-casting & voxelization, the method imposes a regular 3D grid of resolution e.g. $\epsilon/2^L$1 over $\epsilon/2^L$2 horizontally and $\epsilon/2^L$3 vertically. Any voxel containing ≥1 point is “occupied.” To distinguish free from unobserved space, the method casts a ray from the camera center to each occupied voxel center. The first occupied voxel on the ray is marked “observed.” Any further occupied voxels along the same ray are “unobserved.” Voxels never hit are “unobserved free.” The paper summarizes the result as a complete, per-frame binary occupancy label set without LiDAR (Ye et al., 25 Jul 2025).

This curation pipeline is significant because the contribution of GS-Occ3D is not limited to geometry reconstruction. It also produces the auto-labeled supervision used for downstream occupancy learning, which is the basis of the framework’s scalability claim.

5. Reported empirical results

On the Waymo Static-32 benchmark, GS-Occ3D is reported to outperform state-of-the-art implicit (NeuS, F²-NeRF, StreetSurf) and explicit (2DGS, PGSR, GVKF) methods in Chamfer Distance (CD) vs. LiDAR. The reported numbers are CD = 0.56 m for GS-Occ3D and CD = 0.82 m for the next best method, GVKF. The paper also reports PSNR = 26.9 dB, comparable training time (0.8 h), and moderate GPU/memory (Ye et al., 25 Jul 2025). The qualitative discussion states that GS-Occ3D recovers fine poles, building facades, and even extends beyond LiDAR’s range (Ye et al., 25 Jul 2025).

For downstream occupancy prediction, the paper evaluates training CVT-Occ on the vision-only labels generated by GS-Occ3D and compares them with LiDAR-supervised Occ3D labels. On Waymo-Val, the reported result is IoU = 44.7 when trained on GS-Occ3D labels and evaluated on Occ3D-Val(Waymo), compared with 57.4 for LiDAR-train. In zero-shot on nuScenes, the reported result is Ours→nuScenes IoU = 33.4 versus LiDAR→nuScenes 31.4. On the held-out Ours-Val split, training on Ours yields IoU = 46.8 vs. Occ3D labels 41.1; on Ours-Train, the figures are 50.6 vs. Occ3D-Train 48.3 (Ye et al., 25 Jul 2025).

The paper interprets these results as showing that the labels produce competitive downstream performance and even generalize better zero-shot to nuScenes (different camera rigs) than LiDAR-based labels (Ye et al., 25 Jul 2025). It also reports qualitative strengths: more complete geometry in weakly textured or distant regions, better zero-shot generalization under different camera distributions and weather (e.g. light rain), and richer semantic labeling (up to 66 classes vs. Occ3D’s 16) by leveraging Mask2Former on the underlying RGB (Ye et al., 25 Jul 2025).

In the discussion of efficiency and scale, the paper states that Octree + Gaussian surfels is far more memory- and compute-efficient than volumetric meshes or dense NeRF grids, that no expensive post-processing (meshing, hole filling, smoothing) is needed—labels are produced end-to-end, and that training each scene takes <1 GPU-hour, enabling full-Waymo reconstruction (Ye et al., 25 Jul 2025). A plausible implication is that the method is intended less as a single-scene reconstruction system than as an infrastructure for large-scale occupancy annotation.

6. Position within adjacent Gaussian-splatting occupancy research

GS-Occ3D belongs to a broader 2025 trend that combines Gaussian primitives with occupancy reasoning, but it is distinct from several adjacent systems.

AutoOcc formulates open-ended semantic occupancy annotation via vision-language guided Gaussian splatting, using semantic-aware Gaussians as intermediate geometric descriptors and a cumulative Gaussian-to-voxel splatting algorithm (Zhou et al., 7 Feb 2025). Its emphasis is open-vocabulary semantic occupancy rather than the specific large-scale vision-only occupancy reconstruction and auto-labeling pipeline used in GS-Occ3D.

GraphGSOcc addresses 3D semantic occupancy prediction by combining a semantic and geometric graph Transformer with dynamic-static decoupling for 3D Gaussian Splatting-based Occupancy Prediction, and reports an mIoU of 25.20% on SurroundOcc with GPU memory to 6.8 GB (Song et al., 13 Jun 2025). This is a prediction architecture built on Gaussian features, rather than a reconstruction-and-curation pipeline of the kind implemented in GS-Occ3D.

The literature also contains Gaussian-splatting work centered on occlusion and composition rather than autonomous-driving occupancy. In ComGS, the SOPs-based module stores lighting and occlusion information in Surface Octahedral Probes, enabling interpolation-based queries that avoid expensive ray tracing and yield real-time relighting and shadowing at ∼28 FPS (Gao et al., 9 Oct 2025). Although this work operates on visibility and scene composition, its problem definition is different from occupancy label reconstruction.

This suggests a common misconception: Gaussian splatting, occupancy reconstruction, semantic occupancy prediction, and occlusion-aware composition are closely related in representation, but they are not interchangeable tasks. Within that landscape, GS-Occ3D is specifically the framework titled “GS-Occ3D: Scaling Vision-only Occupancy Reconstruction for Autonomous Driving with Gaussian Splatting” (Ye et al., 25 Jul 2025).