---
title: nuPlan Autonomous Driving Benchmark
url: https://www.emergentmind.com/topics/nuplan-5cc65a3d-2ac1-41e5-a804-67dac41b8ee8
type: topic
---

# nuPlan Autonomous Driving Benchmark

nuPlan is a closed-loop, ML-based planning benchmark for autonomous vehicles built from real-world driving logs, HD maps, and planner-in-the-loop simulation. It was introduced to address a limitation of earlier autonomous-driving benchmarks: most focused on short-term motion forecasting in open-loop and scored predictions with L2-style displacement metrics, whereas motion planning requires a high-level goal, uni-modal closed-loop execution, and metrics that capture safety, comfort, rule compliance, and goal achievement [2106.11810]. Subsequent benchmark papers describe nuPlan as the first large-scale real-world autonomous-driving dataset and benchmark specifically designed for learning-based planning, and later planner papers use it as a standardized reference for comparing rule-based, imitation-based, reinforcement-learning, diffusion-based, hybrid, and language-conditioned planners [2403.04133].

## 1. Origins, scope, and design objective

nuPlan was proposed as “the world's first closed-loop ML-based planning benchmark for autonomous driving” [2106.11810]. Its motivating argument is that open-loop evaluation has three structural deficiencies for planning: no goal or high-level navigation route, single-trajectory L2 error that penalizes valid multi-modal behaviors, and no agent interaction because a plan is never executed. nuPlan was therefore designed end-to-end around long-horizon ego planning rather than short-horizon actor forecasting [2106.11810].

The benchmark is geographically diverse. Across the summaries provided in later papers, nuPlan is consistently associated with four cities: Boston, Pittsburgh, Las Vegas, and Singapore [2106.11810]. Those cities were selected because they expose materially different traffic regimes: Boston is described with narrow streets and double-parking, Pittsburgh with custom left-turn precedence at some intersections or winding roads, Las Vegas with wide multi-lane arterials and complex pick-up/drop-off interactions, and Singapore with left-hand traffic or disciplined lane behavior [2106.11810]. This cross-city heterogeneity is central to nuPlan’s use as a generalization benchmark.

Published descriptions report slightly different aggregate dataset sizes. The original benchmark summary reports 1500 hours of human driving data [2106.11810], later planner papers frequently describe 1300 hours [2306.07962], and the journal-style benchmark exposition reports 1282 hours [2403.04133]. This suggests that different papers refer to different curation stages, public subsets, or release conventions. A similar discrepancy appears in the scenario taxonomy: one description states that nuPlan mines 73 high-precision scenario types [2403.04133], while later planner papers refer to 75 automatically labeled scenario types [2309.10443].

nuPlan was created by Motional and Mercedes-Benz as one of the first large-scale closed-loop planning benchmarks, and its public artifacts include dataset and maps at `nuplan.org`, an SDK and simulator, Docker templates, and an evaluation server for code submission [2607.07844].

## 2. Dataset composition, annotations, and scenario structure

nuPlan combines recorded driving logs, object-level annotations, traffic-light state information, and HD semantic maps. The benchmark summaries emphasize that the data are real-world and urban, with route-level diversity spanning lane following, intersections, merges, pick-up/drop-off areas, pedestrian interactions, and other common or rare maneuvers [2403.04133].

The scenario representation used by most planning papers is object-centric rather than sensor-centric. Typical scenario clips are 15 seconds long at 10 Hz and contain ego-vehicle states, other dynamic agents, HD map elements such as lane-center graphs and traffic-light signals, and a high-level route or mission [2306.07962]. Planner-facing inputs commonly include 2 seconds of history and 8 seconds of future expert trajectory for open-loop evaluation, together with map layers such as lane centerlines, drivable area polygons, crosswalks, stop lines, speed limits, and traffic-light locations [2306.07962].

Descriptions of the raw sensing stack vary by publication because different papers emphasize different releases. One benchmark summary lists a “full suite on a 200 TB subset” with `5× 64-beam spinning sensors`, `6 × 1920×1200` cameras, `100 Hz GNSS+IMU pose`, and `vehicle CAN` [2106.11810]. Another describes a stratified public subset with five spinning lidars, eight RGB cameras, and high-precision GNSS/IMU [2403.04133]. Later planning papers often operate on the processed outputs of this pipeline: offline object tracks, traffic-light state logs, semantic maps, and route plans rather than raw sensor packets [2309.10443].

Annotation is performed offline. One description states that autolabeled tracks are produced with `PointPillars + CenterPoint + MVF++ + non-causal 3D tracking` and are “near human labeling” because there are no real-time latency constraints [2106.11810]. A more detailed exposition describes an offboard pipeline with an MVF++-based multi-sweep detector, offline Kalman-filter tracking with extended memory, a global track-refinement network, and a traffic-light labeling system derived from track-map interactions rather than direct vision-only classification [2403.04133]. HD maps are provided in vector and raster form and encode lanes, lane connectors, drivable areas, intersections, crosswalks, stop lines, car parks, traffic-light locations, and speed limits [2403.04133].

Scenario mining is a defining feature. nuPlan automatically extracts scenario tags such as lane changes, merges, protected and unprotected turns, pedestrian and cyclist interactions, double-parked vehicles, construction zones, stop-controlled intersections, and high-acceleration or close-proximity interactions [2106.11810]. The benchmark paper states that each mining query was quality-assured on 100 examples and tuned to exceed 90% precision [2403.04133].

## 3. Closed-loop simulator and planner interface

The simulator is a discrete-time, lightweight planner-in-the-loop environment. At each simulation step, the planner receives the recent history of ego and surrounding agents together with static map queries, and it returns a candidate trajectory as a time-parameterized sequence of $(x, y, \psi)$ poses over a future horizon [2106.11810]. A simple controller then executes that trajectory through a kinematic bicycle model. In challenge-style setups, planners typically output an 8-second trajectory at 10 Hz, and an LQR tracker plus bicycle dynamics realize the low-level motion [2309.10443].

nuPlan supports three canonical evaluation regimes. In open-loop, the environment is replayed and only the ego future is replaced for offline scoring. In closed-loop non-reactive mode, the planner controls the ego while all other agents replay their logged trajectories. In closed-loop reactive mode, the planner controls the ego and surrounding vehicles are re-simulated, most commonly by an Intelligent Driver Model-based traffic model [2306.07962]. Later papers denote these regimes as OLS, CLS-NR, and CLS-R, or equivalently as C1, C2, and C3 [2306.07962].

A concise view of the standard protocols is as follows:

| Protocol | Ego | Other agents |
|---|---|---|
| Open-loop / OLS / C1 | Predicted offline | Logged replay |
| Closed-loop non-reactive / CLS-NR / C2 | Simulated | Logged replay |
| Closed-loop reactive / CLS-R / C3 | Simulated | Reactive model |

The step rate is ordinarily 10 Hz, and the benchmark server fixes random seeds so that runs are deterministic given a planner container [2106.11810]. One benchmark description specifies a 2-second warm-up from logged history followed by rollout for up to 8 seconds, with failure on collision, rule infraction, or off-road distance greater than `0.5 m` [2106.11810]. Other planner papers describe full 15-second episodes at 10 Hz, again with the planner repeatedly replanning and the simulator checking safety and compliance at each tick [2504.17838].

Reactive-agent modeling is one of nuPlan’s most consequential design choices. In the original stack, surrounding vehicles in reactive mode are governed by IDM-style rules and pedestrians remain non-reactive or constant-velocity depending on the implementation [2504.17838]. Later work argues that this is both computationally convenient and behaviorally restrictive, and much of the recent nuPlan literature focuses on replacing IDM with learned reactive world models [2511.10403].

## 4. Metrics, scoring, and benchmark semantics

nuPlan’s evaluation logic is planning-specific rather than purely geometric. The original benchmark summary organizes metrics into four families: safety and rule compliance, human-driving similarity, comfort and dynamics, and goal achievement [2106.11810].

Safety and rule-compliance metrics include `CollisionRate`, `OffRoadRate`, `RedLightInfractionRate`, and `TimeToCollision (TTC)`. For example,
$$
\mathrm{CollisionRate} = \frac{\text{Number of episodes with }\ge 1 \text{ ego collision}}{\text{Total episodes}},
$$
and
$$
\mathrm{RouteCompletion} = \frac{\text{Distance along route reached}}{\text{Total route length}}.
$$
Human-similarity metrics include `MeanVelError`, `LongStopPosError`, and `LatPosError`; comfort metrics include `JerkCost`, `AccelCost`, `SteeringRateCost`, and `FeasibilityViolations`; scenario-specific metrics include `LaneChangeMargin`, `MergeVelocityDifference`, `UnprotectedTurnAgreement`, and `PedestrianPassingSpeed` [2106.11810].

Later challenge-oriented papers compress these metrics into aggregate scores on a 0–100 scale. In one widely used formulation, the closed-loop score is a weighted combination of `TTC`, `Route Progress`, `Speed-limit compliance`, and `Comfort`, with multiplicative penalties that zero out the scenario score under at-fault collision or driveable-area violation [2306.07962]. Another benchmark exposition formalizes the scenario score as a product of multiplier metrics and a weighted sum of average metrics:
$$
\mathrm{score} \;=\; \prod_{i\in\mathrm{mult}} s_i \;\times\; \sum_{j\in\mathrm{avg}} w_j\, s_j,
$$
where the average metrics use weights `TTC: 5`, `progress: 5`, `speed: 4`, and `comfort: 2` [2403.04133].

This scoring design has two important consequences. First, open-loop accuracy is not equivalent to closed-loop competence: a planner can match the expert trajectory distribution yet fail once its own actions perturb future observations. Second, the hard-zero structure means that safety violations dominate final ranking even when the soft metrics are strong. Later papers on reinforcement learning and imitation learning repeatedly optimize against this structure, either by directly targeting route completion with termination penalties or by explicitly coupling prediction and safety monitoring inside the closed-loop loop [2504.17838].

## 5. Baselines, challenge results, and methodological trajectories

nuPlan quickly became a common comparison point for distinct planning paradigms. The SDK originally shipped with a rule-based “Intelligent” MPC planner, an LSTM-based imitation planner trained on `1000 h` of the training set, and a hybrid cost-map planner that fuses learned lane cost with hard constraints [2106.11810]. In an example reactive closed-loop benchmark over `500 test episodes`, the hybrid planner achieved lower `CollisionRate` and higher `RouteCompletion` than the LSTM baseline, illustrating an early advantage for hybrid planning on this benchmark [2106.11810].

A dominant theme in later results is the competition between rule-based priors and learned policies. “Parting with Misconceptions about Learning-based Vehicle Motion Planning” reports that PDM-Hybrid won the nuPlan planning challenge 2023 and obtained `CLS-R = 93`, `CLS-NR = 93`, and `OLS = 83` on the leaderboard summary presented there [2306.07962]. The same line of work argues that rule-based centerline selection plus short-horizon proposal scoring remains a strong prior for closed-loop performance. By contrast, pure imitation planners such as UrbanDriver and GC-PGP often perform better in open-loop but lag in reactive closed-loop [2306.07962].

The benchmark also became a proving ground for progressively stronger learning systems. PlanTF reported `OLS = 87.07`, `NR-CLS = 86.48`, `R-CLS = 80.59`, and `Test14-hard = 72.68` in its comparison table, supporting the claim that a well-designed purely imitation-based planner can be competitive with handcrafted systems [2309.10443]. PLUTO later reported `93.21` on the non-reactive closed-loop score on Val14, slightly exceeding `PDM-Closed` at `93.08`, and thereby framed itself as surpassing the current top-performed rule-based planner on that setting [2404.14327]. CAFE-AD subsequently reported `78.16` reactive and `76.04` non-reactive on `Test14-Hard`, exceeding both `PDM-Closed` and PLUTO in that specific benchmark configuration [2504.06584].

Reinforcement learning and world-model approaches further diversified the benchmark’s methodological landscape. CaRL reported `CLS = 91.3 (±0.3)` in non-reactive traffic and `90.6` in reactive traffic on `1 118 Val14 episodes`, while also emphasizing fast inference and a simple route-completion reward [2504.17838]. AdaptiveDriver, which integrates BehaviorNet-predicted IDM parameters into MPC rollouts, reported `95.35` on the `C3` test setting and per-city gains over PDM-C in Pittsburgh, Boston, Singapore, and Las Vegas [2406.10714]. Instruction-conditioned and language-model planners have also been evaluated on nuPlan; for example, Align2Act fine-tuned `LLaMA-2-7B with LoRA on one million scenarios from the nuPlan dataset` and reported `85.17` open-loop, `70.31` closed-loop non-reactive, and `66.96` reactive closed-loop on `Test14-random` [2510.10503].

These numbers are not directly interchangeable because they come from different splits and protocols—Val14, Test14-random, Test14-Hard, C2/C3, and derived benchmark variants—but collectively they document nuPlan’s role as the central benchmark for planner comparison across paradigms.

## 6. Critiques, benchmark evolution, and downstream extensions

A recurrent conclusion in the nuPlan literature is that open-loop imitation and closed-loop planning are fundamentally misaligned. One study reports an approximately negative correlation between mean `OLS` and mean `CLS` across methods and argues that ego-forecasting and short-term planning should be addressed independently [2306.07962]. This directly challenges the common assumption that long-horizon open-loop imitation is a reliable proxy for deployed driving quality.

A second critique concerns coverage. Although nuPlan was designed around scenario mining, later work argues that the public benchmark is still dominated by “basic driving scenarios” and under-samples edge cases such as overtaking stationary obstacles, jaywalkers triggered by ego behavior, and dense interactive lane changes [2404.07569]. interPlan was introduced as a wrapper around nuPlan to create such long-tail scenarios and showed that planners with strong standard nuPlan scores can fail sharply once interaction structure is altered [2404.07569]. A related benchmark, “Shift & Drift,” uses nuPlan-trained planners under semantic shift and actuation perturbations and reports that imitation-learning methods can degrade substantially under novel city topology and correlated control drift, while the evaluated reinforcement-learning planner degrades more gracefully [2607.07844].

A third critique concerns reactive traffic realism. nuPlan’s original reactive mode uses IDM-style agents, and multiple later papers argue that this can overestimate planner performance or induce planner overfitting to IDM idiosyncrasies [2511.10403]. nuPlan-R addresses this by replacing IDM with diffusion-based reactive agents and by adding `Success Rate (SR)` and `All-Core Pass Rate (PR)` to complement the original `CLS` [2511.10403]. A parallel line replaces IDM with SMART-reactive agents and reports that nearly all scores deteriorate when the simulator becomes more realistic, while some planners actually improve in multi-lane, interaction-heavy scenarios and closed-loop-trained methods become the most stable [2510.14677].

nuPlan has also become infrastructure for adjacent tasks. Nuplan-Occ extends it into “the largest semantic occupancy dataset to date,” with `≈ 19 000` annotated scenes and `≈ 3.6 M` annotated frames for occupancy-centric generation and downstream planning evaluation [2510.22973]. In that sense, nuPlan now functions not only as a planning benchmark but also as a substrate for scene generation, occupancy forecasting, simulator realism research, and cross-domain robustness analysis.

Taken together, these developments indicate that nuPlan’s historical importance lies less in any single leaderboard and more in the benchmark design itself: real-world logs, planner-executed closed-loop rollout, explicit route context, and a metric stack that forces simultaneous attention to safety, comfort, compliance, and progress. Subsequent work has largely treated those design choices as the baseline definition of a modern autonomous-driving planning benchmark, while revising the reactive world model, scenario coverage, and robustness protocols around them [2106.11810].

Source: https://www.emergentmind.com/topics/nuplan-5cc65a3d-2ac1-41e5-a804-67dac41b8ee8