NuPlan-Occ: Occupancy-Based Planning

• NuPlan-Occ is an occupancy-centric planning approach within the nuPlan ecosystem that leverages spatial-temporal grids for collision risk assessment and trajectory refinement.
• It integrates bird’s-eye-view rasterized inputs with multi-scale feature fusion using a ResNet-50 and U-Net architecture to predict ego heatmaps and agents’ occupancy grids.
• The method improves closed-loop performance by transforming occupancy predictions into cost maps, enabling effective collision avoidance and compliant, optimized motion planning.

“NuPlan-Occ” is not introduced in the cited literature as an official benchmark title. As an Editor’s term, it most precisely denotes occupancy-centric planning formulations built on the nuPlan ecosystem: planners that consume or predict spatial-temporal grids—such as ego heatmaps, agents’ occupancy, or occupancy-derived collision fields—and then optimize ego motion under nuPlan’s closed-loop evaluation protocol. In that sense, nuPlan supplies the dataset, simulator, route-conditioned planning problem, and metrics, while occupancy-style methods define one family of scene representations and planning pipelines that operate within that framework (Caesar et al., 2021, Karnchanachari et al., 2024, Hu et al., 2023).

1. Definition and scope

The nuPlan benchmark was proposed as “the world's first closed-loop ML-based planning benchmark for autonomous driving,” with a large-scale driving dataset, a lightweight closed-loop simulator, and motion-planning-specific metrics (Caesar et al., 2021). The later benchmark paper presents nuPlan as a real-world planning dataset and benchmark with 1282 hours of diverse driving scenarios from Las Vegas, Boston, Pittsburgh, and Singapore, high-quality auto-labeled object tracks and traffic light data, a taxonomy of 73 scenario classes, and a simulation framework for closed-loop planner evaluation (Karnchanachari et al., 2024).

Within that ecosystem, an occupancy-centric formulation is exemplified by the NuPlan Challenge method “Imitation with Spatial-Temporal Heatmap,” which uses a bird’s-eye-view raster, predicts an ego spatial-temporal heatmap and surrounding-agents occupancy grids, constructs a collision probability density map, and refines a trajectory through constrained optimization (Hu et al., 2023). The same paper explicitly characterizes this design as a “NuPlan-oriented planner that uses occupancy-style grids (for both ego and agents) and a cost-based post-optimization” (Hu et al., 2023).

This suggests that “NuPlan-Occ” is best understood not as a standalone released benchmark, but as a representational and algorithmic perspective on nuPlan: planning by reasoning over space-time occupancy or occupancy-like probability fields rather than only vector trajectories or direct trajectory regression.

2. nuPlan as the substrate for occupancy-centric planning

nuPlan’s central technical contribution is the shift from open-loop trajectory comparison toward closed-loop simulation. In the 2021 proposal, the benchmark is described with 1500 hours of human driving data from four cities across the US and Asia, reactive agents, and both general and scenario-specific planning metrics (Caesar et al., 2021). The 2024 benchmark paper reports 1282 hours of real-world driving, 128 hours of raw sensor data, and a fleet of 31 vehicles with five lidars, eight RGB cameras, GNSS, and IMU; it also emphasizes auto-labeled tracks, traffic light inference, scenario mining, and a modular simulation stack (Karnchanachari et al., 2024).

For occupancy-based planners, several nuPlan assets are structurally important. The benchmark provides HD semantic maps, vectorized and rasterized map representations, object tracks, traffic light states encoded into lane connectors, ego state, and scenario tags covering maneuvers such as lane changes, merges, protected and unprotected turns, pedestrian interactions, cyclist interactions, stop-controlled intersections, double parked vehicles, and construction zones (Caesar et al., 2021, Karnchanachari et al., 2024). These ingredients can be rasterized into static and dynamic occupancy fields, semantic occupancy tensors, or occupancy-conditioned route representations.

The evaluation protocol is likewise decisive. nuPlan supports open-loop evaluation and two closed-loop modes: non-reactive log replay and reactive simulation. In the reactive setting described in the benchmark paper, vehicles follow the Intelligent Driver Model (IDM) with lane-following along map centerlines, while pedestrians and cyclists remain non-reactive log-replay agents (Karnchanachari et al., 2024). The planner outputs a future ego trajectory in $SE(2)$, and a provided controller tracks that trajectory using a kinematic bicycle model (Karnchanachari et al., 2024). This is the setting in which occupancy representations matter operationally: they are not evaluated only by reconstruction quality, but by their downstream effect on collisions, drivable-area compliance, progress, and comfort.

3. Occupancy-style representations in NuPlan planning

The clearest instantiated “NuPlan-Occ” pipeline in the cited material is the 2023 NuPlan Challenge solution “Imitation with Spatial-Temporal Heatmap” (Hu et al., 2023). Its scene representation is a rasterized ego-centric BEV covering $112\,\text{m} \times 112\,\text{m}$ at $0.5\,\text{m/pixel}$, yielding a $224 \times 224$ input with six channels: ego, road map/HD map, baseline path or lanes in range, other agents’ history, route, and ego speed (Hu et al., 2023).

The network predicts three coupled outputs. First, it regresses an initial ego trajectory $\hat{\tau} \in \mathbb{R}^{T \times 3}$, where the three dimensions are $(x, y, \text{heading})$. Second, it predicts a spatial-temporal ego heatmap $\hat{H} \in \mathbb{R}^{h \times w \times T}$, upsampled to $0.25\,\text{m/pixel}$, where each slice $\hat{H}_t$ is a confidence map over future ego positions. Third, it predicts surrounding-agents occupancy $\hat{O}_{\text{agents}}$ over space and time, described as probabilistic occupancy grids for surrounding agents (Hu et al., 2023).

Architecturally, the method uses a ResNet-50 encoder that produces multi-scale feature maps $112\,\text{m} \times 112\,\text{m}$0 through $112\,\text{m} \times 112\,\text{m}$1, followed by a U-Net neck for multi-scale feature fusion (Hu et al., 2023). The trajectory head uses two fully connected layers; the heatmap head follows the HOME idea; the occupancy head predicts an occupancy-flow-style representation similar to HOPE (Hu et al., 2023).

The representational significance of this design lies in the division between point prediction and field prediction. A single trajectory regression head yields one sequence, whereas the heatmap can carry multiple high-probability regions at a given future time. The occupancy head, combined with the static map, turns the driving scene into a time-indexed field of non-drivable or risky regions. This makes planning naturally cost-map-centric rather than purely trajectory-regressive.

4. Learning objectives and trajectory optimization

The training objective in the NuPlan Challenge solution is explicitly multi-task (Hu et al., 2023). The imitation term is a time-weighted $112\,\text{m} \times 112\,\text{m}$2 loss,

$112\,\text{m} \times 112\,\text{m}$3

which emphasizes later timesteps. The heatmap term $112\,\text{m} \times 112\,\text{m}$4 uses penalty-reduced pixelwise logistic regression with focal loss against Gaussian-centered target heatmaps, and the occupancy term $112\,\text{m} \times 112\,\text{m}$5 is binary cross-entropy between predicted and ground-truth occupancy grids. The total loss is

$112\,\text{m} \times 112\,\text{m}$6

with $112\,\text{m} \times 112\,\text{m}$7, $112\,\text{m} \times 112\,\text{m}$8, and $112\,\text{m} \times 112\,\text{m}$9 (Hu et al., 2023).

At inference time, the method converts occupancy prediction into a planning cost. It combines predicted agents’ occupancy, static objects, and the complement of drivable area to form a non-drivable map; defines an ego-shape convolution kernel at each predicted ego pose; and applies convolution—implemented efficiently with group convolution on GPU—to obtain a collision probability density map $0.5\,\text{m/pixel}$0 over BEV positions at each time step (Hu et al., 2023).

Trajectory refinement is then posed as a nonlinear optimization solved with CasADi + IPOPT. The objective includes four components: closeness to the imitation trajectory $0.5\,\text{m/pixel}$1, kinematic and comfort costs, a penalty for entering high-collision regions of $0.5\,\text{m/pixel}$2, and a reward for aligning with high-probability regions of the heatmap $0.5\,\text{m/pixel}$3 (Hu et al., 2023). Hard constraints cover vehicle dynamics, state bounds, and control bounds.

In operational terms, this is the decisive occupancy-to-planning transition. The occupancy head is not merely auxiliary scene prediction; it becomes part of a collision density map over space and time, and the ego heatmap becomes a likelihood field over feasible future positions. Planning is therefore performed over occupancy-derived fields rather than from trajectories alone.

5. Evaluation in the NuPlan Challenge

The NuPlan Challenge paper describes three official tracks: Challenge 1 open-loop, Challenge 2 closed-loop non-reactive, and Challenge 3 closed-loop reactive; the final leaderboard score is the average of these three tracks (Hu et al., 2023). Reported metrics include final score, open-loop score, closed-loop non-reactive score, closed-loop reactive score, and detailed safety and behavior metrics such as collisions, time-to-collision, drivable area compliance, comfort, ego progress, speed limit compliance, and direction compliance (Hu et al., 2023).

On the private test set, the “hoplan” system reported final score $0.5\,\text{m/pixel}$4, open-loop $0.5\,\text{m/pixel}$5, closed-loop non-reactive $0.5\,\text{m/pixel}$6, and closed-loop reactive $0.5\,\text{m/pixel}$7, placing second overall. It also reported the best comfort score $0.5\,\text{m/pixel}$8 and the best ego progress score $0.5\,\text{m/pixel}$9 among the entries shown, together with collisions $224 \times 224$0, TTC $224 \times 224$1, and drivable area $224 \times 224$2 (Hu et al., 2023).

The ablation study is especially informative for the meaning of “NuPlan-Occ.” A baseline without perturbation, heatmap, or post-solver achieved final score $224 \times 224$3. Adding perturbation raised it to $224 \times 224$4. Adding the heatmap head raised it to $224 \times 224$5, with collision metric improvement from approximately $224 \times 224$6 to $224 \times 224$7. Adding the post-solver yielded the full-system score $224 \times 224$8, with closed-loop non-reactive $224 \times 224$9, closed-loop reactive $\hat{\tau} \in \mathbb{R}^{T \times 3}$0, collisions $\hat{\tau} \in \mathbb{R}^{T \times 3}$1, and drivable $\hat{\tau} \in \mathbb{R}^{T \times 3}$2 (Hu et al., 2023). These results isolate the contribution of occupancy-style field representations and cost-based refinement to closed-loop planning quality.

The broader nuPlan benchmark results reinforce the same point from another angle. In the benchmark paper, pure ML planners perform strongly in open-loop but degrade in closed-loop, while hybrid planners remain most competitive (Karnchanachari et al., 2024). A plausible implication is that occupancy-style representations become most valuable when paired with explicit optimization, constraints, or safety filtering rather than used as a purely end-to-end surrogate for final control.

6. Relation to 3D occupancy research, LLMs, and limitations

NuPlan-Occ should not be conflated with dense 3D semantic occupancy prediction as an end in itself. EFFOcc addresses 3D occupancy prediction for autonomous driving and proposes an efficient fusion-based OccNet using simple 2D operators, reporting 21.35M parameters and 51.49 mIoU on Occ3D-nuScenes with a ResNet-18 backbone, as well as distillation for label-efficient vision-only occupancy learning (Shi et al., 2024). However, the supplied material explicitly states that it contains no planning-related discussion about NuPlan or a “Nuplan-Occ benchmark” beyond the abstracted occupancy-learning claims (Shi et al., 2024). This suggests that perception-side occupancy efficiency and planning-side occupancy use are related but distinct problems.

Occ-LLM extends the occupancy-centered perspective in another direction. It operates on 4D occupancy representations and targets 4D occupancy forecasting, self-ego planning, and occupancy-based scene question answering on nuScenes, using a Motion Separation VAE and Llama2 (Xu et al., 10 Feb 2025). The paper states that it does not mention nuPlan explicitly, but that the setting is conceptually very close to a nuPlan-Occ style setup (Xu et al., 10 Feb 2025). It also notes a key difference: Occ-LLM planning is evaluated in open-loop trajectory prediction, whereas nuPlan emphasizes closed-loop safety, comfort, and efficiency (Xu et al., 10 Feb 2025, Karnchanachari et al., 2024).

Several limitations recur across the occupancy-centric planning literature. The NuPlan Challenge solution depends on HD maps and rasterization, includes a ResNet-50 plus U-Net backbone, GPU group convolutions for the collision map, and CasADi + IPOPT optimization at inference; the authors describe real-time operation via efficient GPU implementation and pixel sampling, but the pipeline is non-trivial computationally (Hu et al., 2023). The same method is based on imitation learning, so performance is bounded by dataset coverage, and multimodality is limited by supervision from a single expert trajectory (Hu et al., 2023). At the benchmark level, nuPlan’s closed-loop simulator remains lightweight, reactive agents are approximate, and pedestrians and cyclists are non-reactive in the 2024 benchmark formulation (Karnchanachari et al., 2024). In the occupancy-LLM setting, open-loop evaluation, dependence on upstream occupancy predictors, and LLM latency further complicate deployment (Xu et al., 10 Feb 2025).

Taken together, the literature supports a precise interpretation of NuPlan-Occ: occupancy is most consequential in nuPlan when it becomes a planning representation over space and time—either as agents’ occupancy, ego likelihood heatmaps, or occupancy-derived collision costs—and when that representation is embedded in closed-loop evaluation with route following, rule compliance, safety constraints, and comfort metrics.