NeuroNCAP Benchmark

• NeuroNCAP benchmark is a photorealistic, closed-loop evaluation suite for assessing autonomous driving safety with neural rendering and variable safety-critical scenarios.
• It employs neural radiance fields to generate realistic sensor data streams and simulates scenarios inspired by Euro NCAP protocols for rigorous testing.
• It quantifies performance through metrics like collision rates and impact speeds, helping improve both end-to-end driving models and neuromorphic hardware evaluation.

The NeuroNCAP benchmark is a photorealistic, scenario-driven closed-loop evaluation suite designed to rigorously probe the safety and decision-making competence of autonomous driving (AD) systems in critical situations. NeuroNCAP builds on neural radiance field (NeRF) simulation to create highly realistic sensor data streams and encompasses a family of safety-critical scenarios inspired by established standards such as Euro NCAP, with a focus on empirically quantifiable safety outcomes. The framework is widely adopted for evaluation of both end-to-end differentiable driving models and modular planning pipelines, supporting research into uncertainty quantification, neuromorphic efficiency, and cross-domain robustness (Ljungbergh et al., 2024, Xiong et al., 10 Mar 2025, Chi et al., 29 May 2025, Valkenhoef et al., 2023, Millar et al., 28 Mar 2025).

1. Simulator Architecture and Photorealistic Sensing

NeuroNCAP employs a NeRF-based simulator that reconstructs the 3D world and ego-centric sensor streams from real driving logs—in particular, from multi-camera and LiDAR sequences provided by nuScenes. The static world and rigid actors are encoded as a neural function

$$F_\Theta : (\mathbf{x}\in\mathbb{R}^3,\,\mathbf{d}\in\mathbb{S}^2) \rightarrow (\sigma,\,\mathbf{c}\in\mathbb{R}^3)$$

where $\sigma$ is volume density and $\mathbf{c}$ is view-dependent radiance. Volumetric rendering along camera rays

$$\hat{C}(r) = \int_{t_n}^{t_f} T(t)\,\sigma(r(t))\,\mathbf{c}(r(t),\mathbf{d})\,dt$$

with transmission $$T(t) = \exp\left(-\int_{t_n}^t \sigma(r(s))\,ds\right)$$, produces photorealistic images. Scene-specific NeRFs are trained to minimize image reconstruction loss,

$$L = \sum_r \|\hat{C}(r) - C_{\mathrm{gt}}(r)\|_2^2,$$

using pose refinements derived from full 6-DoF trajectories. Dynamic actors can be edited or re-posed while preserving environmental coherence, following the Neural Scene Graph paradigm.

Sensor realism is enforced by rendering with calibrated camera intrinsics/extrinsics, mirroring real nuScenes hardware. No artificial noise is added—the “real-to-sim” gap for standard perception metrics (e.g., 3D ADE) is empirically $<0.1$ m over $t=3$ s (Ljungbergh et al., 2024).

2. Scenario Generation, Safety-Critical Cases, and Closed-Loop Protocols

Inspired by Euro NCAP protocols, NeuroNCAP defines three canonical safety-critical scenarios:

• Stationary: Ego approaches a fixed obstacle in-lane.
• Frontal: Ego encounters oncoming traffic drifting into its path.
• Side: Perpendicular motion, e.g., a vehicle or agent crossing at intersection.

Each scenario is parametrized by actor pose, speed, ego state, and high-level command. Perturbations are sampled to create a distribution over initial conditions, with

$$\Delta\mathbf{p}\sim\mathrm{Uniform}([-\delta_p,\,\delta_p]^3),\ \Delta\theta\sim\mathrm{Uniform}([-\delta_\theta,\,\delta_\theta]),\ \Delta v\sim\mathrm{Uniform}([-\delta_v,\,\delta_v])$$

typically $\delta_p\approx1$ m, $\delta_\theta\approx10^\circ$, $\delta_v\approx1$ m/s. Closed-loop integration proceeds as:

1. At each time step ($\Delta t=0.1$ s), the agent receives the rendered sensor input and ego state.
2. A control policy outputs a plan (future waypoints or trajectory).
3. A low-level controller executes the next control action with the vehicle dynamics model,

$S_{t+1} = S_t + \Delta t\,f(S_t,\,\delta_t,\,a_t),$

where $f$ is the kinematic bicycle model.

1. Simulator advances and renders a new observation.
2. Episode terminates on collision or after a fixed horizon (e.g., 10 s).

This closed-loop protocol exposes distribution shift, feedback, and cascading errors not accessible via open-loop (single-pass) evaluation (Ljungbergh et al., 2024, Chi et al., 29 May 2025).

3. Behavioral Metrics and NeuroNCAP Scoring

NeuroNCAP quantifies both collision occurrence and severity via principled, interpretable metrics:

• Collision Rate (CR): fraction of scenarios with any impact,

$$\mathrm{CR} = \frac{1}{N}\sum_{i=1}^N C_i, \quad C_i \in \{0,1\}.$$

• Impact Speed ($v_i$): relative velocity at contact.
• Reference Speed ($v_r$): speed if ego vehicle does not react.
• NeuroNCAP Score (NNS): a “5-star” rating designed to penalize more severe collisions,

$s_i = \begin{cases} 5.0,& C_i=0\ (\text{no collision}), \[6pt] 4.0\cdot\max\left(0,\,1 - \frac{v_i}{v_r}\right),& C_i=1, \end{cases}$

with the averaged NNS

$\mathrm{NNS} = \frac{1}{N}\sum_{i=1}^N s_i.$

Auxiliary metrics include open-loop Average Displacement Error (ADE), but safety assessments are based strictly on closed-loop CR and NNS (Ljungbergh et al., 2024, Chi et al., 29 May 2025).

4. Empirical Results, Baselines, and Model Comparisons

Published scores reveal a stark gap between current end-to-end planners, post-processed pipelines, and rule-based controllers:

• Raw policies (e.g., UniAD, VAD) crash in 80–99 % of challenging scenarios; NNS ≈ 0.7/5.
• Naïve rule-based (brake-in-lane) achieves $>$90 % success on stationary but 0 % on side/frontal (NNS stationary $4.7/5$, others $0$).
• Collision-avoidance post-processing reduces stationary crash rate to $\sim$35 % and lifts global NNS to ≈2.8/5, but side/frontal scenarios remain failure-prone (Ljungbergh et al., 2024).

Integration of the Impromptu VLA dataset, emphasizing unstructured and corner-case scenes, yields the following improvements (Chi et al., 29 May 2025):

• Baseline Qwen2.5-VL finetuned on nuScenes: Avg. NNS = 1.77; Avg. CR = 72.5 %.
• After Impromptu VLA exposure: Avg. NNS = 2.15 (+21.5 %); Avg. CR = 65.5 % (–9.7 %).

CATPlan, a loss-prediction-based uncertainty module evaluated on NeuroNCAP, outperforms GMM-based risk proxies:

• GMM: AUROC 49.4 %, AP 43.1 %.
• CATPlan: AUROC 70.6 %, AP 66.7 %—a 54.8 % relative AP gain (Xiong et al., 10 Mar 2025).

These results highlight both progress and persistent deficiencies, particularly in generalization to rare safety-critical geometry.

5. Protocols for Low-Power NN Hardware and Neuromorphic Evaluation

NeuroNCAP metrics are increasingly adopted for benchmarking ultra-low-power NN inference hardware ($\mu$NPUs), as in (Millar et al., 28 Mar 2025), and neuromorphic computing platforms (Yik et al., 2023, Valkenhoef et al., 2023):

• End-to-end latency, power, and energy per inference ($E = P \times L_{\mathrm{infer}}$) are reported in closed-loop NeuroNCAP setups.
• MAX78000 achieves $0.37$ mJ/inference for small CNNs; HX-WE2 (Ethos-U55) can deliver $<$10 ms latencies on mid-sized models; memory I/O dominates on weight-stationary designs.
• Neuromorphic models (massively parallel spiking networks) can achieve time complexity $T_{\mathrm{neu}}(n;f) = O(n^2/f^2)$ for pattern search, and for human behavioral analogy, observed scaling on NeuroNCAP tasks falls below classical bounds and can approach the quantum-optimal $O(n)$ in select regimes (Valkenhoef et al., 2023).

Well-defined, platform-independent protocols for algorithmic and system-level metrics (e.g., throughput, energy, latency under deadline constraints) are standardized following the NeuroBench framework (Yik et al., 2023).

6. Failure Modes, Limitations, and Research Directions

Failures in NeuroNCAP scenarios frequently arise from covariate shift between training logs and safety-critical episode geometry, or from planning–perception mismatches—models may internally predict obstacles yet output unsafe motion plans. Post-hoc optimization and myopic trajectory correction cannot reliably mitigate these deficits.

NeuroNCAP’s extensibility allows continuous addition of new scenario types (e.g., pedestrian, dynamic intent, adverse weather) as neural rendering and data availability advance. The community-driven evolution is modeled on NeuroBench and MLPerf practices, emphasizing reproducibility, open scenario scripting, and comprehensive metric logging (Yik et al., 2023, Ljungbergh et al., 2024).

A plausible implication is that rigorous closed-loop sensor-realistic safety benchmarks such as NeuroNCAP are required for trustworthy deployment of data-driven AD planners, hardware accelerators, and neuromorphic control stacks.

7. Impact and Broader Context

NeuroNCAP is referenced as the standard for end-to-end model and hardware evaluation in safety-critical driving, used for diagnostic benchmarking (e.g., VLA models in (Chi et al., 29 May 2025)), uncertainty quantification trials (CATPlan in (Xiong et al., 10 Mar 2025)), and cross-architecture comparisons (neural, neuromorphic, quantum-inspired search (Valkenhoef et al., 2023)).

By codifying photorealistic, stochastic, and scenario-driven tests with end-to-end metrics, NeuroNCAP underpins quantitative research into AD safety, driving policy robustness, hardware efficiency, and algorithmic self-awareness. Release of turnkey simulation and evaluation code accelerates adoption for both academic and industrial research (Ljungbergh et al., 2024, Chi et al., 29 May 2025, Xiong et al., 10 Mar 2025, Valkenhoef et al., 2023, Millar et al., 28 Mar 2025).