GPOcc: Sparse Gaussian Occupancy Prediction

• GPOcc is a monocular 3D occupancy prediction framework that converts visible-surface depth cues into sparse, volumetric Gaussian representations for indoor scene understanding.
• It uses a ray-based sampling method that extends beyond the visible surface, transforming depth estimates into probabilistic Gaussian primitives to better capture free space.
• The model introduces a training-free incremental fusion strategy to merge per-frame Gaussians, enhancing both efficiency and accuracy compared to dense occupancy methods.

GPOcc, short for Generalizing Visual Geometry Priors to Sparse Gaussian Occupancy Prediction, is a monocular 3D occupancy prediction framework that leverages generalizable visual geometry priors (GPs) for indoor embodied scene understanding. Its central premise is that modern geometry priors such as monocular depth models and visual geometry models provide strong 3D cues, but remain fundamentally surface-centric, whereas occupancy prediction must reason about volumetric interiors and free space. GPOcc addresses this gap by extending visible surface points inward along camera rays, representing the resulting volumetric samples as Gaussian primitives, and inferring occupancy probabilistically from a sparse Gaussian set; for streaming input, it adds a training-free incremental update strategy that fuses per-frame Gaussians into a global memory bank (Zhou et al., 25 Feb 2026).

1. Problem formulation and motivation

GPOcc is defined on the task of monocular 3D occupancy prediction: given a single RGB image $\mathbf{I}$, the model predicts a voxelized semantic occupancy map

$$\mathbf{O} \in \mathbb{R}^{X \times Y \times Z \times N_c},$$

where $X,Y,Z$ define the 3D grid and $N_c$ is the number of semantic classes. The representation is intended to capture both objects and traversable/free space, which the paper positions as useful for navigation, planning, manipulation, and scene interaction in embodied settings (Zhou et al., 25 Feb 2026).

The method is motivated by two limitations in prior monocular occupancy pipelines. First, methods such as ISO lift image features into dense 3D volumes using estimated depth distributions and then rely on a heavy 3D U-Net, which the paper characterizes as computationally expensive and still tied to surface-oriented depth predictions. Second, methods such as EmbodiedOcc initialize many 3D anchors or Gaussian primitives across the whole volume, although most indoor space is empty. GPOcc therefore seeks a representation that is simultaneously geometry-guided, sparse, and better aligned with volumetric occupancy reasoning (Zhou et al., 25 Feb 2026).

A further motivation is the emergence of stronger upstream priors. The paper explicitly instantiates GPOcc with DepthAnythingV2 / DPT-style monocular depth priors and VGGT, while also discussing related visual geometry models such as DUSt3R, MASt3R, and Fast3R. The key claim is not merely that stronger priors improve occupancy, but that occupancy quality depends on how those priors are converted from visible-surface geometry into a volumetric representation.

2. Geometry-prior lifting and ray-based volumetric sampling

The first stage of GPOcc applies a pretrained geometry prior to the input image

$\mathbf{I}\in \mathbb{R}^{H\times W \times 3}.$

The prior produces feature maps

$\mathbf{F} = \text{GPs}(\mathbf{I}),$

which are then downsampled, and a lightweight MLP regresses depth $\mathbf{d}$. At a pixel $(u,v)$, the normalized camera ray direction is

$$x = \frac{u - c_x}{f_x}, \quad y = \frac{v - c_y}{f_y}, \quad \mathbf{r}_{(u,v)} = \frac{[x,\,y,\,1]^\top}{\sqrt{x^2 + y^2 + 1}},$$

and the visible surface point is

$$\mathbf{x}^{\text{surf}}_{(u,v)} = \mathbf{d}_{(u,v)} \cdot \mathbf{r}_{(u,v)}.$$

This recovers one surface point per image location from the predicted depth and camera intrinsics (Zhou et al., 25 Feb 2026).

GPOcc’s defining operation is to move beyond that first visible surface. For each ray, it samples $$\mathbf{O} \in \mathbb{R}^{X \times Y \times Z \times N_c},$$0 additional points beyond the surface point: $$\mathbf{O} \in \mathbb{R}^{X \times Y \times Z \times N_c},$$1 with offsets

$$\mathbf{O} \in \mathbb{R}^{X \times Y \times Z \times N_c},$$2

The paper states that $$\mathbf{O} \in \mathbb{R}^{X \times Y \times Z \times N_c},$$3 is dynamically predicted to adapt to varying object sizes. The intended effect is to turn surface-only predictions into a geometry-guided candidate set of interior volumetric samples.

Each sampled point is paired with feature construction based on a learnable embedding matrix

$$\mathbf{O} \in \mathbb{R}^{X \times Y \times Z \times N_c},$$4

which is added by broadcast to the downsampled image features to produce per-sample features. This lets the model distinguish samples near the visible surface from samples farther along the same ray, even when they originate from the same image feature. A plausible implication is that this mechanism gives the model a structured notion of depth-order within each ray rather than treating all along-ray samples as exchangeable.

3. Sparse Gaussian primitives and occupancy inference

GPOcc represents the scene as a set of semantic Gaussian primitives

$$\mathbf{O} \in \mathbb{R}^{X \times Y \times Z \times N_c},$$5

where each primitive is described as

$$\mathbf{O} \in \mathbb{R}^{X \times Y \times Z \times N_c},$$6

Here $$\mathbf{O} \in \mathbb{R}^{X \times Y \times Z \times N_c},$$7 is the Gaussian mean, $$\mathbf{O} \in \mathbb{R}^{X \times Y \times Z \times N_c},$$8 the scale, $$\mathbf{O} \in \mathbb{R}^{X \times Y \times Z \times N_c},$$9 the rotation, $X,Y,Z$0 the opacity, and $X,Y,Z$1 the semantic feature. The sampled 3D point induced by ray extension serves as the Gaussian center $X,Y,Z$2, while an MLP predicts the remaining attributes from the per-sample feature tensor (Zhou et al., 25 Feb 2026).

Occupancy inference follows the probabilistic Gaussian superposition view inherited from GaussianFormer2. For a query point $X,Y,Z$3, the per-Gaussian contribution is

$X,Y,Z$4

where $X,Y,Z$5 is derived from the predicted rotation and scale. A general Gaussian-to-voxel aggregation is written as

$X,Y,Z$6

The paper does not provide a more explicit class-decoding rule over $X,Y,Z$7, but it does specify that occupancy is inferred probabilistically from nearby Gaussians (Zhou et al., 25 Feb 2026).

Sparsity is enforced by opacity pruning. Gaussians with

$X,Y,Z$8

are discarded, with default threshold

$X,Y,Z$9

This pruning is central to the representation’s efficiency claim: unlike dense anchor or dense volume approaches, GPOcc retains only informative primitives concentrated near occupied regions. The paper explicitly contrasts this with EmbodiedOcc’s dense anchor initialization, where many Gaussians fall in empty space.

4. Training objective and streaming fusion

The training objective combines occupancy supervision and depth supervision. The total loss includes Focal loss, Lovász-Softmax loss, geometric scene-class affinity loss, semantic scene-class affinity loss, and a Huber depth loss. The scene-class affinity losses are said to follow EmbodiedOcc, and the paper does not restate their internal formulas. It also states that adding the depth loss enables end-to-end optimization of the entire pipeline, although it does not clearly separate which parts of the geometry-prior stack are frozen and which are finetuned (Zhou et al., 25 Feb 2026).

For streaming input, GPOcc maintains a global Gaussian memory bank

$N_c$0

At frame $N_c$1, the monocular predictor outputs a set of Gaussians $N_c$2, transforms predicted centers and rotations into world coordinates using camera pose, and merges them with neighboring memory Gaussians. If neighbors exist within spatial radius $N_c$3, fusion is performed by weighted averaging: $N_c$4 where $N_c$5, $N_c$6 is the top-1 class confidence, and $N_c$7. The paper sets

$N_c$8

so newer Gaussians receive higher weight than the existing memory entry. If no neighbors are found, new Gaussians are inserted directly into the memory bank.

This incremental update is explicitly described as training-free. It is also sparse: only local Gaussian neighborhoods are compared, and no dense global anchor bank is updated. The paper does not provide the exact value of $N_c$9, the exact value of $\mathbf{I}\in \mathbb{R}^{H\times W \times 3}.$0, the exact MLP architecture, or the exact covariance construction from $\mathbf{I}\in \mathbb{R}^{H\times W \times 3}.$1 and $\mathbf{I}\in \mathbb{R}^{H\times W \times 3}.$2.

5. Benchmarks, quantitative results, and efficiency

GPOcc is evaluated on Occ-ScanNet for monocular occupancy prediction and EmbodiedOcc-ScanNet for streaming occupancy prediction (Zhou et al., 25 Feb 2026). Occ-ScanNet contains 45,755 training samples and 19,764 testing samples, with a voxel grid of

$\mathbf{I}\in \mathbb{R}^{H\times W \times 3}.$3

covering

$\mathbf{I}\in \mathbb{R}^{H\times W \times 3}.$4

in front of the camera. EmbodiedOcc-ScanNet is reorganized from Occ-ScanNet and contains 537 training scenes and 137 validation scenes, with 30 posed frames per scene. Evaluation uses IoU and mIoU.

The implementation details reported by the paper include AdamW, weight decay 0.01, a maximum learning rate of

$\mathbf{I}\in \mathbb{R}^{H\times W \times 3}.$5

a 1000-iteration warmup, cosine decay, 10 epochs, batch size 8, 4 NVIDIA A800 GPUs, longer-side image resizing to 518 pixels, gradient clipping with max norm 1.0, default ray sample count

$\mathbf{I}\in \mathbb{R}^{H\times W \times 3}.$6

and default opacity threshold

$\mathbf{I}\in \mathbb{R}^{H\times W \times 3}.$7

Setting	Model	IoU / mIoU
Occ-ScanNet	EmbodiedOcc++	54.90 / 46.20
Occ-ScanNet	Ours-DPT	56.96 / 51.88
Occ-ScanNet	Ours-VGGT	63.14 / 56.19
EmbodiedOcc-ScanNet	EmbodiedOcc++	52.20 / 43.60
EmbodiedOcc-ScanNet	Ours-DPT	56.39 / 51.22
EmbodiedOcc-ScanNet	Ours-VGGT	61.41 / 55.39

On Occ-ScanNet, the paper reports that Ours-VGGT improves over EmbodiedOcc++ by 8.24 IoU and 9.99 mIoU. Under the same depth prior comparison, Ours-DPT improves over EmbodiedOcc by 3.41 IoU and 6.73 mIoU. On EmbodiedOcc-ScanNet, Ours-VGGT improves over EmbodiedOcc++ by 9.21 IoU and 11.79 mIoU.

The efficiency comparison is equally prominent. On Occ-ScanNet, the paper reports 3.63 FPS for ISO, 10.66 FPS for EmbodiedOcc, 28.22 FPS for Ours-DPT, and 5.26 FPS for Ours-VGGT. The same-prior comparison emphasized by the paper is that Ours-DPT is 2.65$\mathbf{I}\in \mathbb{R}^{H\times W \times 3}.$8 faster than EmbodiedOcc while also more accurate. Parameter counts are reported as 303.05M for ISO, 231.45M for EmbodiedOcc, 97.95M for Ours-DPT, and 942.31M for Ours-VGGT; the VGGT variant is therefore slower because VGGT itself is a much larger backbone.

The ablation study supports the importance of volumetric sampling and conservative pruning. For the number of ray samples, the paper reports 47.88 mIoU / 53.10 IoU at $\mathbf{I}\in \mathbb{R}^{H\times W \times 3}.$9, 56.19 mIoU / 63.14 IoU at $\mathbf{F} = \text{GPs}(\mathbf{I}),$0, and 56.72 mIoU / 63.84 IoU at $\mathbf{F} = \text{GPs}(\mathbf{I}),$1, concluding that $\mathbf{F} = \text{GPs}(\mathbf{I}),$2 is a good accuracy-efficiency tradeoff. For opacity pruning, the default threshold $\mathbf{F} = \text{GPs}(\mathbf{I}),$3 yields 56.19 mIoU / 63.14 IoU with 5876 Gaussians, whereas aggressive pruning at $\mathbf{F} = \text{GPs}(\mathbf{I}),$4 reduces performance to 52.65 mIoU / 58.31 IoU with 930 Gaussians.

6. Position within the literature, nomenclature, and limitations

GPOcc belongs to the recent family of Gaussian-based occupancy prediction methods, but its placement strategy differs from prior Gaussian pipelines. The paper contrasts it particularly with EmbodiedOcc, which initializes many predefined anchors and learns which ones matter; GPOcc instead uses geometry-guided placement, because its Gaussians are sampled from visible geometry and extended inward along camera rays. A closely related later direction is GraphGSOcc, which likewise operates on Gaussian scene primitives but adds explicit geometric and semantic graph interaction over Gaussians for autonomous-driving occupancy prediction (Song et al., 13 Jun 2025). This suggests a broader design space in which Gaussian occupancy models differ less by the primitive itself than by how Gaussian placement, interaction, and aggregation are organized.

The acronym also requires disambiguation. GPOcc is unrelated to GPPO for O-RAN resource management (Ngo et al., 1 Sep 2025), Gaussian Process Policy Optimization in model-free reinforcement learning (Rao et al., 2020), and Gaussian Primitive Optimization for retinal image registration (Tian et al., 23 Aug 2025). Those methods share only superficial acronymic similarity. Within occupancy prediction, GPOcc is specifically the framework named “Generalizing Visual Geometry Priors to Sparse Gaussian Occupancy Prediction” (Zhou et al., 25 Feb 2026).

The paper is also explicit, or nearly explicit, about several limitations. GPOcc depends on the quality of the upstream geometry prior; inward ray sampling is a heuristic approximation of object interior; and the method may be less reliable for thin structures, transparent objects, hollow objects, or severe depth ambiguity. In streaming mode it requires accurate camera poses, and repeated fusion of uncertain monocular predictions may accumulate errors. The paper further omits several implementation details needed for exact reproduction, including the exact feature dimension $\mathbf{F} = \text{GPs}(\mathbf{I}),$5, the exact MLP architecture, the exact rotation parameterization, the exact covariance construction, the exact radius $\mathbf{F} = \text{GPs}(\mathbf{I}),$6 for streaming fusion, and the exact value of $\mathbf{F} = \text{GPs}(\mathbf{I}),$7. A plausible implication is that the reported gains are tied not only to the Gaussian representation, but to a specific combination of geometry priors, ray-based sampling, and sparse fusion design that the paper only partially specifies.