nuPlan Autonomous Driving Benchmark
- nuPlan is a closed-loop, ML-based planning benchmark that leverages real-world driving logs and HD maps to evaluate autonomous vehicle planning.
- It introduces advanced metrics focusing on safety, comfort, rule compliance, and goal achievement during closed-loop execution.
- The benchmark evaluates diverse planning paradigms across multiple cities, integrating both rule-based and learning-based approaches.
nuPlan is a closed-loop, ML-based planning benchmark for autonomous vehicles built from real-world driving logs, HD maps, and planner-in-the-loop simulation. It was introduced to address a limitation of earlier autonomous-driving benchmarks: most focused on short-term motion forecasting in open-loop and scored predictions with L2-style displacement metrics, whereas motion planning requires a high-level goal, uni-modal closed-loop execution, and metrics that capture safety, comfort, rule compliance, and goal achievement (Caesar et al., 2021). Subsequent benchmark papers describe nuPlan as the first large-scale real-world autonomous-driving dataset and benchmark specifically designed for learning-based planning, and later planner papers use it as a standardized reference for comparing rule-based, imitation-based, reinforcement-learning, diffusion-based, hybrid, and language-conditioned planners (Karnchanachari et al., 2024).
1. Origins, scope, and design objective
nuPlan was proposed as “the world's first closed-loop ML-based planning benchmark for autonomous driving” (Caesar et al., 2021). Its motivating argument is that open-loop evaluation has three structural deficiencies for planning: no goal or high-level navigation route, single-trajectory L2 error that penalizes valid multi-modal behaviors, and no agent interaction because a plan is never executed. nuPlan was therefore designed end-to-end around long-horizon ego planning rather than short-horizon actor forecasting (Caesar et al., 2021).
The benchmark is geographically diverse. Across the summaries provided in later papers, nuPlan is consistently associated with four cities: Boston, Pittsburgh, Las Vegas, and Singapore (Caesar et al., 2021). Those cities were selected because they expose materially different traffic regimes: Boston is described with narrow streets and double-parking, Pittsburgh with custom left-turn precedence at some intersections or winding roads, Las Vegas with wide multi-lane arterials and complex pick-up/drop-off interactions, and Singapore with left-hand traffic or disciplined lane behavior (Caesar et al., 2021). This cross-city heterogeneity is central to nuPlan’s use as a generalization benchmark.
Published descriptions report slightly different aggregate dataset sizes. The original benchmark summary reports 1500 hours of human driving data (Caesar et al., 2021), later planner papers frequently describe 1300 hours (Dauner et al., 2023), and the journal-style benchmark exposition reports 1282 hours (Karnchanachari et al., 2024). This suggests that different papers refer to different curation stages, public subsets, or release conventions. A similar discrepancy appears in the scenario taxonomy: one description states that nuPlan mines 73 high-precision scenario types (Karnchanachari et al., 2024), while later planner papers refer to 75 automatically labeled scenario types (Cheng et al., 2023).
nuPlan was created by Motional and Mercedes-Benz as one of the first large-scale closed-loop planning benchmarks, and its public artifacts include dataset and maps at nuplan.org, an SDK and simulator, Docker templates, and an evaluation server for code submission (Canevaro et al., 8 Jul 2026).
2. Dataset composition, annotations, and scenario structure
nuPlan combines recorded driving logs, object-level annotations, traffic-light state information, and HD semantic maps. The benchmark summaries emphasize that the data are real-world and urban, with route-level diversity spanning lane following, intersections, merges, pick-up/drop-off areas, pedestrian interactions, and other common or rare maneuvers (Karnchanachari et al., 2024).
The scenario representation used by most planning papers is object-centric rather than sensor-centric. Typical scenario clips are 15 seconds long at 10 Hz and contain ego-vehicle states, other dynamic agents, HD map elements such as lane-center graphs and traffic-light signals, and a high-level route or mission (Dauner et al., 2023). Planner-facing inputs commonly include 2 seconds of history and 8 seconds of future expert trajectory for open-loop evaluation, together with map layers such as lane centerlines, drivable area polygons, crosswalks, stop lines, speed limits, and traffic-light locations (Dauner et al., 2023).
Descriptions of the raw sensing stack vary by publication because different papers emphasize different releases. One benchmark summary lists a “full suite on a 200 TB subset” with 5× 64-beam spinning sensors, 6 × 1920×1200 cameras, 100 Hz GNSS+IMU pose, and vehicle [CAN](https://www.emergentmind.com/topics/causal-abstraction-network-can) (Caesar et al., 2021). Another describes a stratified public subset with five spinning lidars, eight RGB cameras, and high-precision GNSS/IMU (Karnchanachari et al., 2024). Later planning papers often operate on the processed outputs of this pipeline: offline object tracks, traffic-light state logs, semantic maps, and route plans rather than raw sensor packets (Cheng et al., 2023).
Annotation is performed offline. One description states that autolabeled tracks are produced with PointPillars + CenterPoint + MVF++ + non-causal 3D tracking and are “near human labeling” because there are no real-time latency constraints (Caesar et al., 2021). A more detailed exposition describes an offboard pipeline with an MVF++-based multi-sweep detector, offline Kalman-filter tracking with extended memory, a global track-refinement network, and a traffic-light labeling system derived from track-map interactions rather than direct vision-only classification (Karnchanachari et al., 2024). HD maps are provided in vector and raster form and encode lanes, lane connectors, drivable areas, intersections, crosswalks, stop lines, car parks, traffic-light locations, and speed limits (Karnchanachari et al., 2024).
Scenario mining is a defining feature. nuPlan automatically extracts scenario tags such as lane changes, merges, protected and unprotected turns, pedestrian and cyclist interactions, double-parked vehicles, construction zones, stop-controlled intersections, and high-acceleration or close-proximity interactions (Caesar et al., 2021). The benchmark paper states that each mining query was quality-assured on 100 examples and tuned to exceed 90% precision (Karnchanachari et al., 2024).
3. Closed-loop simulator and planner interface
The simulator is a discrete-time, lightweight planner-in-the-loop environment. At each simulation step, the planner receives the recent history of ego and surrounding agents together with static map queries, and it returns a candidate trajectory as a time-parameterized sequence of poses over a future horizon (Caesar et al., 2021). A simple controller then executes that trajectory through a kinematic bicycle model. In challenge-style setups, planners typically output an 8-second trajectory at 10 Hz, and an LQR tracker plus bicycle dynamics realize the low-level motion (Cheng et al., 2023).
nuPlan supports three canonical evaluation regimes. In open-loop, the environment is replayed and only the ego future is replaced for offline scoring. In closed-loop non-reactive mode, the planner controls the ego while all other agents replay their logged trajectories. In closed-loop reactive mode, the planner controls the ego and surrounding vehicles are re-simulated, most commonly by an Intelligent Driver Model-based traffic model (Dauner et al., 2023). Later papers denote these regimes as OLS, CLS-NR, and CLS-R, or equivalently as C1, C2, and C3 (Dauner et al., 2023).
A concise view of the standard protocols is as follows:
| Protocol | Ego | Other agents |
|---|---|---|
| Open-loop / OLS / C1 | Predicted offline | Logged replay |
| Closed-loop non-reactive / CLS-NR / C2 | Simulated | Logged replay |
| Closed-loop reactive / CLS-R / C3 | Simulated | Reactive model |
The step rate is ordinarily 10 Hz, and the benchmark server fixes random seeds so that runs are deterministic given a planner container (Caesar et al., 2021). One benchmark description specifies a 2-second warm-up from logged history followed by rollout for up to 8 seconds, with failure on collision, rule infraction, or off-road distance greater than 0.5 m (Caesar et al., 2021). Other planner papers describe full 15-second episodes at 10 Hz, again with the planner repeatedly replanning and the simulator checking safety and compliance at each tick (Jaeger et al., 24 Apr 2025).
Reactive-agent modeling is one of nuPlan’s most consequential design choices. In the original stack, surrounding vehicles in reactive mode are governed by IDM-style rules and pedestrians remain non-reactive or constant-velocity depending on the implementation (Jaeger et al., 24 Apr 2025). Later work argues that this is both computationally convenient and behaviorally restrictive, and much of the recent nuPlan literature focuses on replacing IDM with learned reactive world models (Peng et al., 13 Nov 2025).
4. Metrics, scoring, and benchmark semantics
nuPlan’s evaluation logic is planning-specific rather than purely geometric. The original benchmark summary organizes metrics into four families: safety and rule compliance, human-driving similarity, comfort and dynamics, and goal achievement (Caesar et al., 2021).
Safety and rule-compliance metrics include CollisionRate, OffRoadRate, RedLightInfractionRate, and TimeToCollision (TTC). For example,
and
Human-similarity metrics include MeanVelError, LongStopPosError, and LatPosError; comfort metrics include JerkCost, AccelCost, SteeringRateCost, and FeasibilityViolations; scenario-specific metrics include LaneChangeMargin, MergeVelocityDifference, UnprotectedTurnAgreement, and PedestrianPassingSpeed (Caesar et al., 2021).
Later challenge-oriented papers compress these metrics into aggregate scores on a 0–100 scale. In one widely used formulation, the closed-loop score is a weighted combination of TTC, Route Progress, Speed-limit compliance, and Comfort, with multiplicative penalties that zero out the scenario score under at-fault collision or driveable-area violation (Dauner et al., 2023). Another benchmark exposition formalizes the scenario score as a product of multiplier metrics and a weighted sum of average metrics:
where the average metrics use weights TTC: 5, progress: 5, speed: 4, and comfort: 2 (Karnchanachari et al., 2024).
This scoring design has two important consequences. First, open-loop accuracy is not equivalent to closed-loop competence: a planner can match the expert trajectory distribution yet fail once its own actions perturb future observations. Second, the hard-zero structure means that safety violations dominate final ranking even when the soft metrics are strong. Later papers on reinforcement learning and imitation learning repeatedly optimize against this structure, either by directly targeting route completion with termination penalties or by explicitly coupling prediction and safety monitoring inside the closed-loop loop (Jaeger et al., 24 Apr 2025).
5. Baselines, challenge results, and methodological trajectories
nuPlan quickly became a common comparison point for distinct planning paradigms. The SDK originally shipped with a rule-based “Intelligent” MPC planner, an LSTM-based imitation planner trained on 1000 h of the training set, and a hybrid cost-map planner that fuses learned lane cost with hard constraints (Caesar et al., 2021). In an example reactive closed-loop benchmark over 500 test episodes, the hybrid planner achieved lower CollisionRate and higher RouteCompletion than the LSTM baseline, illustrating an early advantage for hybrid planning on this benchmark (Caesar et al., 2021).
A dominant theme in later results is the competition between rule-based priors and learned policies. “Parting with Misconceptions about Learning-based Vehicle Motion Planning” reports that PDM-Hybrid won the nuPlan planning challenge 2023 and obtained CLS-R = 93, CLS-NR = 93, and OLS = 83 on the leaderboard summary presented there (Dauner et al., 2023). The same line of work argues that rule-based centerline selection plus short-horizon proposal scoring remains a strong prior for closed-loop performance. By contrast, pure imitation planners such as UrbanDriver and GC-PGP often perform better in open-loop but lag in reactive closed-loop (Dauner et al., 2023).
The benchmark also became a proving ground for progressively stronger learning systems. PlanTF reported OLS = 87.07, NR-CLS = 86.48, R-CLS = 80.59, and Test14-hard = 72.68 in its comparison table, supporting the claim that a well-designed purely imitation-based planner can be competitive with handcrafted systems (Cheng et al., 2023). PLUTO later reported 93.21 on the non-reactive closed-loop score on Val14, slightly exceeding PDM-Closed at 93.08, and thereby framed itself as surpassing the current top-performed rule-based planner on that setting (Cheng et al., 2024). CAFE-AD subsequently reported 78.16 reactive and 76.04 non-reactive on Test14-Hard, exceeding both PDM-Closed and PLUTO in that specific benchmark configuration (Zhang et al., 9 Apr 2025).
Reinforcement learning and world-model approaches further diversified the benchmark’s methodological landscape. CaRL reported CLS = 91.3 (±0.3) in non-reactive traffic and 90.6 in reactive traffic on 1 118 Val14 episodes, while also emphasizing fast inference and a simple route-completion reward (Jaeger et al., 24 Apr 2025). AdaptiveDriver, which integrates BehaviorNet-predicted IDM parameters into MPC rollouts, reported 95.35 on the C3 test setting and per-city gains over PDM-C in Pittsburgh, Boston, Singapore, and Las Vegas (Vasudevan et al., 2024). Instruction-conditioned and language-model planners have also been evaluated on nuPlan; for example, Align2Act fine-tuned LLaMA-2-7B with [LoRA](https://www.emergentmind.com/topics/low-rank-adaptation-lora-patching) on one million scenarios from the nuPlan dataset and reported 85.17 open-loop, 70.31 closed-loop non-reactive, and 66.96 reactive closed-loop on Test14-random (Jaisankar et al., 12 Oct 2025).
These numbers are not directly interchangeable because they come from different splits and protocols—Val14, Test14-random, Test14-Hard, C2/C3, and derived benchmark variants—but collectively they document nuPlan’s role as the central benchmark for planner comparison across paradigms.
6. Critiques, benchmark evolution, and downstream extensions
A recurrent conclusion in the nuPlan literature is that open-loop imitation and closed-loop planning are fundamentally misaligned. One study reports an approximately negative correlation between mean OLS and mean CLS across methods and argues that ego-forecasting and short-term planning should be addressed independently (Dauner et al., 2023). This directly challenges the common assumption that long-horizon open-loop imitation is a reliable proxy for deployed driving quality.
A second critique concerns coverage. Although nuPlan was designed around scenario mining, later work argues that the public benchmark is still dominated by “basic driving scenarios” and under-samples edge cases such as overtaking stationary obstacles, jaywalkers triggered by ego behavior, and dense interactive lane changes (Hallgarten et al., 2024). interPlan was introduced as a wrapper around nuPlan to create such long-tail scenarios and showed that planners with strong standard nuPlan scores can fail sharply once interaction structure is altered (Hallgarten et al., 2024). A related benchmark, “Shift & Drift,” uses nuPlan-trained planners under semantic shift and actuation perturbations and reports that imitation-learning methods can degrade substantially under novel city topology and correlated control drift, while the evaluated reinforcement-learning planner degrades more gracefully (Canevaro et al., 8 Jul 2026).
A third critique concerns reactive traffic realism. nuPlan’s original reactive mode uses IDM-style agents, and multiple later papers argue that this can overestimate planner performance or induce planner overfitting to IDM idiosyncrasies (Peng et al., 13 Nov 2025). nuPlan-R addresses this by replacing IDM with diffusion-based reactive agents and by adding Success Rate (SR) and All-Core Pass Rate (PR) to complement the original CLS (Peng et al., 13 Nov 2025). A parallel line replaces IDM with SMART-reactive agents and reports that nearly all scores deteriorate when the simulator becomes more realistic, while some planners actually improve in multi-lane, interaction-heavy scenarios and closed-loop-trained methods become the most stable (Hagedorn et al., 16 Oct 2025).
nuPlan has also become infrastructure for adjacent tasks. Nuplan-Occ extends it into “the largest semantic occupancy dataset to date,” with ≈ 19 000 annotated scenes and ≈ 3.6 M annotated frames for occupancy-centric generation and downstream planning evaluation (Li et al., 27 Oct 2025). In that sense, nuPlan now functions not only as a planning benchmark but also as a substrate for scene generation, occupancy forecasting, simulator realism research, and cross-domain robustness analysis.
Taken together, these developments indicate that nuPlan’s historical importance lies less in any single leaderboard and more in the benchmark design itself: real-world logs, planner-executed closed-loop rollout, explicit route context, and a metric stack that forces simultaneous attention to safety, comfort, compliance, and progress. Subsequent work has largely treated those design choices as the baseline definition of a modern autonomous-driving planning benchmark, while revising the reactive world model, scenario coverage, and robustness protocols around them (Caesar et al., 2021).