Occupancy Ray-Shape Sampling (ORS)

• Occupancy Ray-Shape Sampling (ORS) is a geometric querying technique that projects 3D occupancy grids along camera rays into compact, view-aligned feature tensors.
• It uses calibrated camera parameters and differentiable raycasting with trilinear interpolation to efficiently encode spatial structure and object semantics.
• ORS enhances autonomous driving applications by improving scene reconstruction fidelity and reducing computational overhead in forecasting and planning pipelines.

Occupancy Ray-Shape Sampling (ORS) is a class of geometric ray-based querying techniques enabling efficient, semantics-rich, and viewpoint-aligned distillation of 3D occupancy grids for downstream perception and generative tasks. Initially developed for handling dense 3D volumetric representations in the context of driving scene reconstruction and self-supervised forecasting, ORS achieves high spatial fidelity and efficiency by projecting occupancy information into per-pixel, per-ray feature tensors. This facilitates their direct use as conditioning signals in vision architectures such as diffusion models and end-to-end planning networks (Khurana et al., 2022, Li et al., 3 May 2025).

1. Problem Setting and Motivation

Traditional representations in autonomous driving—binary BEV masks, sparse bounding boxes, or full 3D voxels—exhibit limitations in fidelity, alignment, and computational efficiency. Occupancy grids $O \in \mathbb R^{H \times W \times D}$ provide dense geometry and semantics but are poorly matched to camera-centric tasks and prohibitively expensive for direct 2D-UNet consumption due to tensor size and lack of viewpoint correspondence (Li et al., 3 May 2025). ORS addresses this gap by “rendering” occupancy values along camera rays, summarizing the 3D volumetric structure encountered along each pixel’s line of sight as a more compact, view-aligned feature. In self-supervised forecasting, differentiable ray-shape sampling further enables learning scene occupancy from sequential LiDAR data, disentangling dynamic actors from ego-motion (Khurana et al., 2022).

2. Mathematical Formulation and Implementation

ORS maps each camera pixel $(i, j)$ to a unit ray $r$ parameterized using calibrated intrinsics $K$, extrinsics $T$, and camera center $p_{\text{ego}}$:

$$\begin{gathered} s_{\text{img}}\in\mathbb{R}^{3} \text{ (pixel in homogeneous coords)} \ r = \text{norm}(T^{-1} K^{-1} s_{\text{img}} - p_{\text{ego}}) \end{gathered}$$

Let $N$ denote the set of ray-sample steps (e.g., $N = \{1,\ldots,n\}$). The corresponding 3D sample locations along the ray are:

$$\hat{s}_{\text{ego}} = \{ p_{\text{ego}} + r \cdot n \mid n \in N \}$$

For each query point, trilinear interpolation retrieves occupancy values from $O$. The set of sampled occupancies per pixel forms a feature vector $v_{i,j} \in \mathbb{R}^N$. Collecting these over the entire image forms the compact ORS tensor $v \in \mathbb{R}^{U \times V \times N}$ (Li et al., 3 May 2025). Pseudocode for this pipeline, as formalized in DualDiff, is:

Inputs: O # occupancy grid, shape (H,W,D) K, T # camera intrinsics and extrinsics p_ego # camera position in ego coords U, V # target image resolution N_samples # number of points per ray depths = linspace(z_near, z_far, N_samples) Output: v # ORS feature, shape (U,V,N_samples) Procedure: for i in 0..(U-1): for j in 0..(V-1): s_img = [i, j, 1]^T dir_cam = inv(K) @ s_img dir_ego = inv(T) @ dir_cam r = dir_ego / ||dir_ego|| for n_idx in 1..N_samples: n = depths[n_idx] query_pt = p_ego + r * n ray_samples[n_idx] = trilinear_sample(O, query_pt) v[i,j,:] = ray_samples

A key distinction is that occupancy values along each ray encode free space and object semantics, preserving scene structure at the pixel-ray level (Li et al., 3 May 2025).

3. ORS in Forecasting and Planning Architectures

In the context of self-supervised occupancy forecasting, the ORS mechanism is formalized via differentiable raycasting that computes the expected hit distance $\hat{d}(r)$ of each ray, given a latent occupancy field $\hat{z}[u]$ produced by a prediction network $f(\cdot)$:

$$\hat{d}(r) = \sum_{k=1}^{K} \Delta t \left\{ \prod_{i=1}^{k-1} (1 - \hat{z}(u_i)) \right\} \cdot \hat{z}(u_k)$$

This summation models the (discrete) probability of first contact with occupied space, heavily inspired by volume rendering formulations. All interpolation, multiplication, and summation are differentiable, permitting gradient flow through the ray integration operation (Khurana et al., 2022).

For dual-branch generative models (e.g., DualDiff), masked variants of $O$ are separately ray-sampled to yield $v_b$ (background) and $v_f$ (foreground), each processed through a Semantic Fusion Attention module and injected into respective ControlNet branches for conditioning generative diffusion (Li et al., 3 May 2025).

4. Comparison with Traditional Occupancy Grids

ORS differs fundamentally from voxel-based or BEV-centric representations in three principal aspects:

Aspect	3D Occupancy Grid	ORS Feature Tensor
Size	$H \times W \times D$	$U \times V \times N$
Camera Alignment	Not viewpoint-aligned	Pixel/feature aligned with 2D images
Information Encoding	Dense per-voxel semantics	Per-ray free/occupied sequence + semantics

ORS thus facilitates compact, camera-consistent scene encoding, while retaining semantic richness. It permits efficient integration with 2D-structured neural backbones, reducing computational cost and memory demands relative to full 3D tensor inputs (Li et al., 3 May 2025).

5. Differentiable Training Objectives and Supervision

The differentiable construction of ORS enables self-supervised training by comparing rendered occupancy observations to real-world data, such as LiDAR sweeps. The representative losses include:

• Distance loss between predicted and observed hit distances along each ray:

$L_r = \sum_{r} \ell(\hat{d}(r), d(r))$

where $\ell$ may be $L_1$ or relative $L_1$.

• Per-voxel binary cross-entropy loss for free/occupied classification along rays:

$$L_{\mathrm{BCE}} = -\sum_{r} \sum_{k=1}^{K} \left[ \mathrm{fgt}_k \log (1 - \hat{z}(u_k)) + (1 - \mathrm{fgt}_k)\log \hat{z}(u_k) \right]$$

where $\mathrm{fgt}_k$ denotes freespace ground truth at that step.

These objectives make emergent occupancy fields amenable to end-to-end training and downstream use in planning, providing both scene understanding and direct constraints on ego-vehicle trajectories (Khurana et al., 2022).

6. Empirical Performance and Qualitative Impact

Quantitative ablations validate the efficacy of ORS. Substituting BEV map inputs with ORS in the MagicDrive baseline reduced FID from 16.20 to 13.26 and increased road mIoU from 61.05% to 62.19%. Full DualDiff (ORS + SFA + dual-branch + FGM) achieves FID 10.99 and state-of-the-art mIoUs on nuScenes (62.75% for road, 30.22% for vehicle) (Li et al., 3 May 2025). In the self-supervised forecasting context, ORS-enabled differentiable raycasting improved per-sweep F1 by up to 15 points and reduced downstream planner collision rates by up to 17% compared to freespace-only baselines (Khurana et al., 2022).

Qualitative studies show that ORS enables finer geometric reconstruction (e.g., road edges at night, small distant objects) not captured by alternative representations. The resultant occupancy-aware feature maps are thus critical for accurate, realistic, and safe scene generation and planning.

7. Applications and Integration into Modern Architectures

ORS has been integrated into conditional diffusion models (DualDiff) for multi-view scene generation, where it delivers semantically informed, viewpoint-consistent conditioning to dual ControlNet branches for separate foreground/background control. In planning and forecasting pipelines, differentiable ray-shape sampling enforces spatial consistency and disentangles ego-motion from environmental dynamics at scale.

The compactness, differentiability, and semantic richness of ORS position it as a central linking mechanism between high-dimensional 3D perception and tractable 2D/sequence learning models. Its capacity to encode both geometry and semantics suggests broad utility in autonomous driving, robot navigation, and feature-aligned generative modeling (Khurana et al., 2022, Li et al., 3 May 2025).