ST-Occ: Scene-Level Occupancy Learning

• ST-Occ is a scene-level occupancy representation learning framework that integrates spatiotemporal memory to fuse multi-camera inputs for 3D prediction.
• It uses a persistent, scene-centered memory grid to aggregate historical data, achieving 42.13 mIoU and reducing temporal inconsistency by 29%.
• The framework employs uncertainty-aware deformable attention and efficient memory updates to handle occlusions and dynamic motions with lower GPU memory usage.

Searching arXiv for ST-Occ and closely related occupancy forecasting/tracking papers. ST-Occ is a scene-level occupancy representation learning framework for 3D occupancy prediction in autonomous driving. Introduced in "Occupancy Learning with Spatiotemporal Memory," it is designed to learn the spatiotemporal feature with temporal consistency from multi-camera input streams by combining a spatiotemporal memory with a memory attention mechanism that conditions the current occupancy representation on uncertainty and dynamic awareness (Leng et al., 6 Aug 2025). The formulation targets three coupled difficulties in temporal occupancy prediction—efficiency, uncertainty, and dynamics—and reports that it outperforms the state-of-the-art methods by a margin of 3 mIoU and reduces the temporal inconsistency by 29% (Leng et al., 6 Aug 2025).

1. Problem setting and motivation

ST-Occ addresses 3D occupancy prediction, namely the estimation of which voxels in a 3D scene are occupied and by which semantic class, from multi-camera input streams. The framework is motivated by three properties of temporal occupancy modeling: dense 3D occupancy grids impose high memory and computation costs, voxels in occluded or sensor-noisy regions are uncertain, and moving objects introduce temporal misalignment that degrades naive feature fusion (Leng et al., 6 Aug 2025). In this setting, temporal aggregation is not merely an auxiliary enhancement; it is part of the representation problem itself, because the occupancy state at time $t$ depends on historical evidence, ego-motion, and dynamic scene change.

The paper contrasts ST-Occ with queue-based recurrent or stacking strategies that store a history of separate per-frame feature cubes in ego-centric coordinates. Those approaches are described as incurring major memory and computation overhead and as being unable to model uncertainty or dynamics adequately, leading to temporal inconsistency and lower robustness (Leng et al., 6 Aug 2025). ST-Occ instead adopts a single, unified memory bank in scene-centered coordinates, so that temporal aggregation becomes scene-level rather than frame-level.

This design choice places ST-Occ within a broader transition in occupancy research from single-frame or short-horizon fusion toward explicit spatiotemporal reasoning. Earlier vision-only systems such as Multi-Scale Occ emphasized multi-scale voxel construction and long-term temporal aggregation for 3D occupancy prediction, achieving 49.36 mIoU in the CVPR 2023 3D Occupancy Prediction Challenge (Ding et al., 2023). ST-Occ extends the temporal axis further by making the memory itself persistent and scene-aligned rather than treating history as a bounded queue (Leng et al., 6 Aug 2025).

2. Architecture and scene-centered spatiotemporal memory

The ST-Occ pipeline consists of input encoding, a spatiotemporal memory, memory attention fusion, decoding, and temporal-consistency evaluation (Leng et al., 6 Aug 2025). Multi-view images at time $t$ are encoded into ego-vehicle-centered occupancy features $\mathbf{V}_t$. A large, persistent scene-aligned memory grid $\mathbf{M}$, with dimensions $H_G \times W_G \times Z_G \times C_G$, stores fused historical information. For each incoming frame, spatially corresponding regions $\mathbf{H}_t$ are sampled from memory by transforming from scene coordinates to ego-vehicle coordinates, after which the current representation is fused with historical context and decoded by task-specific heads (Leng et al., 6 Aug 2025).

For each voxel position $\mathbf{p}$, the memory stores four attributes: a historical feature vector $\mathbf{V}$, historical class activation $\mathbf{c}$, uncertainty $\delta$, and occupancy flow $t$0 (Leng et al., 6 Aug 2025). The update rules are given as

$t$1

$t$2

$t$3

The memory is described as efficient for two reasons. First, only scene regions traversed by the vehicle are stored and updated. Second, storage is largely independent of the number of temporal frames, with complexity $t$4 rather than queue-based $t$5, where $t$6 is small (Leng et al., 6 Aug 2025). Feature sampling uses ego-pose transformations together with trilinear or bilinear interpolation to retrieve the current region of interest from the scene memory (Leng et al., 6 Aug 2025).

This scene-centered design is the structural core of ST-Occ. A plausible implication is that temporal evidence is accumulated in a coordinate system closer to the physical world state than to the current sensor state, which is consistent with the framework’s emphasis on temporal consistency and long-frame scalability (Leng et al., 6 Aug 2025).

3. Memory attention, uncertainty modeling, and dynamic awareness

The memory attention module fuses current-frame occupancy features with sampled historical features according to

$t$7

where $t$8 is a multi-head deformable attention module extended with uncertainty and motion modeling (Leng et al., 6 Aug 2025). In the reported implementation, ST-Occ uses 3 layers of temporal self-attention, a 4-layer MLP for temporal attribute encoding, and parallel heads for occupancy, uncertainty, and flow prediction (Leng et al., 6 Aug 2025).

Uncertainty is predicted from historical class probabilities, averaged log-variance, and feature similarity:

$t$9

where $\mathbf{V}_t$0 is the cosine similarity between current and historical features at the same physical location (Leng et al., 6 Aug 2025). Higher uncertainty in the historical memory yields greater reliance on the current observation. This is an explicit departure from uniform temporal fusion, because reliability is estimated per voxel rather than assumed globally.

Dynamic awareness is handled through occupancy flow. Historical memory may be spatially misaligned for moving objects, so ST-Occ predicts a 2D flow vector and shifts the memory sampling points accordingly (Leng et al., 6 Aug 2025). The reported fusion form is

$\mathbf{V}_t$1

with $\mathbf{V}_t$2 denoting deformable attention, $\mathbf{V}_t$3 the voxel position, and $\mathbf{V}_t$4 the predicted flow offset (Leng et al., 6 Aug 2025). The stated effect is that static or uncertain regions adaptively shift weight between history and the present, while dynamic regions use motion-corrected sampling.

This uncertainty-aware and flow-aware temporal conditioning distinguishes ST-Occ from temporal fusion schemes that only concatenate or recurrently update features. Related occupancy systems emphasize different temporal objectives. Occ-LLM, for example, converts multiview images into 3D occupancy, separates dynamic and static voxels, patchifies latent tokens, and feeds them to Llama2 for 4D occupancy forecasting, self-ego planning, and occupancy-based scene question answering (Xu et al., 10 Feb 2025). TrackOcc, by contrast, uses 4D panoptic queries and localization-aware loss to unify panoptic occupancy segmentation and object tracking over time (Chen et al., 11 Mar 2025). ST-Occ remains focused on scene-level occupancy representation learning with temporally consistent voxel semantics (Leng et al., 6 Aug 2025).

4. Optimization objectives and evaluation criteria

ST-Occ is trained with a composite objective

$\mathbf{V}_t$5

where $\mathbf{V}_t$6 is the occupancy loss, $\mathbf{V}_t$7 is an uncertainty loss, and $\mathbf{V}_t$8 is a flow loss (Leng et al., 6 Aug 2025). The occupancy loss is described as a sum of focal loss, Lovasz-Softmax loss, affinity loss, and other geometric or semantic scalars and depth loss, while the flow loss is an $\mathbf{V}_t$9 loss for occupancy flow prediction (Leng et al., 6 Aug 2025).

The uncertainty term is a Gaussian negative log-likelihood on log-variance prediction:

$\mathbf{M}$0

This places uncertainty estimation directly in the training loop rather than treating it as a post hoc diagnostic (Leng et al., 6 Aug 2025).

Evaluation uses both precision and stability. The principal semantic metric is mIoU. Temporal consistency is measured by mean Spatiotemporal Classification Variability (mSTCV), defined as the mean proportion of classification changes at the same physical location across time for non-free voxels, using memory as the association mechanism; lower mSTCV indicates higher temporal consistency (Leng et al., 6 Aug 2025). The dual use of mIoU and mSTCV is central to the paper’s argument that occupancy quality cannot be characterized solely by per-frame semantic overlap.

This evaluation perspective resonates with adjacent 4D occupancy work. TrackOcc introduces OccSTQ, OccAQ, and OccSQ for Camera-based 4D Panoptic Occupancy Tracking, explicitly combining segmentation and association quality over space-time (Chen et al., 11 Mar 2025). ST-Occ does not perform panoptic tracking, but its adoption of mSTCV reflects the same methodological pressure toward metrics that penalize temporal flicker and unstable associations (Leng et al., 6 Aug 2025).

5. Empirical performance, ablations, and efficiency

On the Occ3D benchmark, ST-Occ improves state-of-the-art by 3 mIoU and reduces mSTCV by 29% (Leng et al., 6 Aug 2025). With a ResNet-50 backbone, the reported numbers are 42.13 mIoU and 8.68% mSTCV, compared with 39.11 mIoU and 12.18% mSTCV for FB-OCC (Leng et al., 6 Aug 2025). The paper further states that temporal modeling in FB-OCC yields +1.7 mIoU, whereas ST-Occ achieves +4.7 mIoU, making it 2.8$\mathbf{M}$1 more effective under the reported comparison (Leng et al., 6 Aug 2025).

Ablation experiments attribute gains to both dynamic modeling and uncertainty-aware attention, with the combination producing the best results in mIoU and temporal consistency (Leng et al., 6 Aug 2025). Increasing the number of fused historical frames leads to monotonically better mIoU and lower mSTCV, and the method is described as scaling gracefully to long-frame temporal windows (Leng et al., 6 Aug 2025). Qualitative results are reported to show improved prediction in occlusion, increased semantic coherence over time, and fewer flickering artifacts (Leng et al., 6 Aug 2025).

Efficiency is treated as a first-class result rather than a secondary engineering detail. The paper reports that training GPU memory is halved versus stacking and remains constant in the number of frames (Leng et al., 6 Aug 2025). This should be read together with the scene-centered memory complexity claim, because the architectural compression of temporal history is what makes long-horizon aggregation tractable.

A neighboring line of work reaches similar goals by different means. STCOcc proposes an explicit state-based modeling method for 3D occupancy and scene flow, built around sparse occlusion-aware attention, cascade refinement, and sparse temporal fusion; on Occ3D-nus it reports 41.7% RayIoU and 44.6% mIoU with 7.7GB memory, and on OpenOcc it reports 40.8% RayIoU, 0.44 mAVE, and 8.7GB memory (Liao et al., 28 Apr 2025). SHTOcc instead focuses on sparse head-tail voxel construction and decoupled learning, reporting 42.2% lower GPU memory usage, 58.6% faster inference, and about 7% higher accuracy on multiple baselines (Yu et al., 28 May 2025). These results underscore that temporal consistency, sparsity, and long-tail balancing have become partially separable design axes in occupancy learning.

6. Position within the occupancy-learning landscape

Recent occupancy research has diversified across multi-scale fusion, language-conditioned reasoning, panoptic tracking, sparse explicit renovation, long-tail balancing, and compact geometric primitives. ST-Occ occupies the specific niche of scene-level memory with temporally consistent occupancy learning.

Method	Primary focus	Representative reported result
Multi-Scale Occ (Ding et al., 2023)	Multi-scale 3D occupancy prediction with long-term temporal aggregation	49.36 mIoU, 4th place in the CVPR 2023 3D Occupancy Prediction Challenge
Occ-LLM (Xu et al., 10 Feb 2025)	Occupancy-centric LLM for 4D occupancy forecasting	32.52 IoU, 20.99 mIoU, 0.28 L2 on 3-second average
TrackOcc (Chen et al., 11 Mar 2025)	Camera-based 4D panoptic occupancy tracking	OccSTQ 20.0, OccAQ 13.5, OccSQ 29.4
STCOcc (Liao et al., 28 Apr 2025)	Sparse spatial-temporal cascade renovation for occupancy and scene flow	41.7% RayIoU and 7.7GB on Occ3D-nus
ST-Occ (Leng et al., 6 Aug 2025)	Scene-level occupancy representation learning with spatiotemporal memory	42.13 mIoU and 8.68% mSTCV with ResNet-50

Other variants extend occupancy prediction in orthogonal directions. LOcc replaces the conventional semantic head with a geometry head and a language head, using semantic transitive labeling to produce dense voxel-to-text correspondences and reporting 23.84 mIoU for LOcc-BEVDet4D on Occ3D-nuScenes among open-vocabulary approaches (Yu et al., 2024). SuperQuadricOcc adopts a superquadric-based representation for self-supervised occupancy estimation and reports a 75% reduction in memory footprint, 124% faster inference, and a 5.9% improvement in mIoU compared to previous Gaussian-based methods on Occ3D (Hayes et al., 21 Nov 2025). These developments indicate that occupancy learning is no longer a single problem class but a family of related formulations with different representations, supervision regimes, and temporal objectives.

This suggests that ST-Occ’s main contribution is not merely temporal fusion in the generic sense. Rather, it formalizes a scene-level memory substrate for occupancy learning in which historical features, class activations, uncertainty, and flow are co-stored and jointly queried (Leng et al., 6 Aug 2025). Within the current literature, that combination makes ST-Occ most directly relevant to research on long-horizon camera-only occupancy prediction, temporally stable semantic scene completion, and memory-efficient 4D world modeling for autonomous driving (Leng et al., 6 Aug 2025).