NeuroNCAP: AD Safety & Neuro Imaging

• NeuroNCAP is a multidisciplinary framework combining NeRF‐based closed-loop simulation for autonomous driving safety with advanced neuroimaging methods for functional connectivity analysis.
• It employs scenario-based evaluations and transformer-enhanced risk estimation to rigorously test AD planners under realistic, adversarial conditions.
• In neuroscience, NeuroNCAP pioneers counter-condition learning and connectomics, supporting improved diagnostic accuracy and refined nonhuman primate brain mapping.

NeuroNCAP refers to distinct research efforts in neuroscience, neuroimaging, and autonomous driving safety, each leveraging advanced computational frameworks, data integration, and scenario-based evaluation protocols. The following sections detail the principal initiatives under the NeuroNCAP name, their methodologies, empirical benchmarks, and future prospects.

1. NeuroNCAP: Photorealistic Closed-Loop Simulator for Autonomous Driving

NeuroNCAP (Ljungbergh et al., 2024, Xiong et al., 10 Mar 2025) is a NeRF-based, sensor-realistic simulator designed for evaluating autonomous driving (AD) software under safety-critical, closed-loop conditions. Inspired by the Euro NCAP's vehicle safety standards, the platform enables reproducible, high-fidelity testing environments that support adversarial scenario synthesis and rigorous safety metric computation.

The core pipeline integrates:

• Volumetric Neural Radiance Field (NeRF) rendering, reconstructing RGB observations from real driving logs by simulating camera intrinsics and extrinsics via ray tracing;
• Plug-in interface for any end-to-end AD planner, which ingests rendered panoramic imagery and outputs planned trajectories;
• Linear Quadratic Regulator (LQR) controllers and a kinematic bicycle model for physically plausible vehicle actuation in closed-loop;
• Scenario creation tools allowing reconfiguration of actors and controlled injection of collision-prone events.

This architecture mirrors the causal feedback loops of real-world driving, exposing system-level failures that are otherwise masked by open-loop benchmarks. Sensor fidelity is validated by comparing planning and detection metrics (ADE, NDS) between real and simulated video, with discrepancies below 5%.

2. Safety-Critical Scenario Design and Evaluation Protocol

NeuroNCAP supports the instantiation and systematic variation of adversarial scenarios derived from the 2023 Euro NCAP Collision Avoidance Protocol. The three principal scenario classes are:

• Stationary Obstacle: The ego vehicle is confronted with a parked car in-lane.
• Frontal Drift-in: An oncoming vehicle veers into the ego lane.
• Side Cut-in: A perpendicular actor crosses the ego's path.

Each scenario is parameterized via position, heading, and velocity perturbations of actor vehicles. For any test run, neural rendering replays only the critical actors, enforcing balanced collision statistics. The experimental interface exposes APIs for rendering, planning, control, and state propagation per time step.

Evaluation metrics include:

• Collision Rate ($p_\text{coll}$): Fraction of runs with physical contact.
• Impact Speed ($v_i$): Relative velocity at collision onset.
• NeuroNCAP Score (NNS):

$$\text{NNS} = \begin{cases} 5.0, &\text{no collision}\ 4.0\,\max\bigl(0,1-\frac{v_i}{v_r}\bigr), &\text{otherwise} \end{cases}$$

with $v_r$ the reference speed in absence of intervention.

• Minimum Time-to-Collision and violation rates (e.g., percentage with $\mathrm{TTC}<1$ s).

Table: Representative Results for Baseline End-to-End Planners (Ljungbergh et al., 2024)

Model	NNS Avg.	CR Avg (%)	NNS Front	CR Side (%)
UniAD (raw)	0.73	88.6	0.10	79.6
VAD (raw)	0.66	92.5	0.04	81.6
UniAD† (post-proc)	1.84	68.7	0.66	78.8
VAD† (diff. occl.)	2.75	50.7	1.44	49.8

†Post-processing methods utilize classical optimization to refine predicted trajectories; these significantly lower collision rates for stationary obstacles but are substantially less effective in dynamic frontal/side scenarios.

Empirical results show that state-of-the-art planners, which perform near-perfectly in open-loop settings, exhibit high collision rates (80–99%) in these closed-loop, adversarial tests—exposing deficiencies in robust causal reasoning and multi-agent interaction (Ljungbergh et al., 2024).

3. Embedding-Based Collision-Risk Quantification and CATPlan Integration

Uncertainty-aware evaluation is vital for deployment-ready AD systems. NeuroNCAP supports auxiliary risk heads exemplified by CATPlan (Xiong et al., 10 Mar 2025), a transformer-based risk estimation module. CATPlan operates by:

• Extracting internal planning and agent-motion embeddings from a frozen end-to-end planner (e.g., UniAD, VAD).
• Cross-attending the planning query with agent-motion queries using a TransformerDecoder layer.
• Decoding the attended feature into a scalar collision probability via MLP and sigmoid.

The collision label is determined by the sign of the planner’s collision loss:

$$\mathcal{L}_\mathrm{collision}(\hat\tau_\mathrm{plan}) = \sum_{k=1}^T 1_{\|\hat x_\mathrm{plan}(k) - x_\mathrm{agent}(k)\| < d_\mathrm{safe}}$$

Training uses binary cross-entropy (or focal loss) on the predicted risk. CATPlan shows marked improvement over Gaussian mixture model (GMM) rule-based risk baselines:

Method	AUROC (%)	AP (%)	Pr₅₀ (%)
GMM Baseline	49.4	43.1	49.8
MLP (plan only)	64.7	56.5	56.4
CATPlan	70.6	66.7	67.5

This yields a relative average precision gain of 54.8% (from 43.1% to 66.7%). CATPlan effectively detects collision risks in complex multi-agent scenes, especially for “side” collision scenarios that otherwise confound standard planners (Xiong et al., 10 Mar 2025).

4. Neuroscientific NeuroNCAP Initiatives: Nonhuman Primate and Connectomics Frameworks

NeuroNCAP in neuroimaging refers to two parallel efforts: a learnable counter-condition analysis framework for functional connectivity-based disorder diagnosis (Kang et al., 2023) and the NonHuman Primate Neuroimaging & Neuroanatomy Project (Hayashi et al., 2020).

4.1. Counter-Condition Analysis Framework for FC-based Neurological Diagnosis

The NeuroNCAP framework (Kang et al., 2023) is an end-to-end deep learning model unifying feature selection, relational encoding, classification, and neuroscientific interpretability for functional connectivity (FC) matrices derived from fMRI.

Principal components:

• Adaptive Attention Network: Learns subject-specific binary masks for ROI connections $M\in\mathbb{R}^{R\times R}$ using a Gumbel-sigmoid hard attention scheme, highlighting connections crucial for diagnosis.
• Functional Network Relational Encoder: Stacked intra-network (MLP) and inter-network (Transformer self-attention) modules abstract local and global FC topologies. A learnable summary token aggregates global features via multihead self-attention.
• Prototype-Based Classifier: Encodes subjects into embedding space; classification relies on cosine similarity to learned class prototypes.
• Counter-Condition Learning: Shuffles summary embeddings across class labels during training, guiding the model to generate “counterfactual” FC patterns. This simulates diagnosis reversals (e.g., from patient to control) and exposes the FC alterations driving class discrimination.

Empirical performance (AUC, accuracy, sensitivity, specificity) on ABIDE (ASD vs. TD) and REST-meta-MDD (MDD vs. HC) benchmarks surpasses standard methods (STCAL, LASSO+SVM). The counter-condition decoding achieves AUC 0.945/0.971 and accuracy 92.8%/94.6% on ABIDE/MDD held-out data.

Counter-condition analysis reveals disorder-specific FC alterations (e.g., cerebellar–default mode hyper/hypoconnectivity in ASD; fronto-temporal dysconnectivity in MDD) and suggests that personalized ∆FC signatures can differentiate clinically meaningful subgroups (Kang et al., 2023).

4.2. NonHuman Primate Neuroimaging & Neuroanatomy Project

The NonHuman Primate NeuroNCAP (Hayashi et al., 2020) is a large-scale, multi-site consortium generating ground-truth validated connectomic datasets in macaque and marmoset monkeys via:

• Multi-modal MRI: Structural, functional (BOLD/MION), and diffusion MRI at submillimeter resolution, following adapted HCP protocols.
• Invasive Tract-Tracing: Classical and next-generation tracers quantify area-to-area projections at laminar and cellular resolution.
• Histological Mapping: High-throughput immunocytochemistry registers cytoarchitectonic and subcortical boundaries to MRI atlases.
• Atlas Development: Integration of histology, MRI myelin/thickness, and connectivity metrics to refine parcellations.
• Genetic and Behavioral Phenotyping: Genetic assays (e.g., AVPR1A, OPRM1) and behavioral measures (e.g., personality traits, gaze sensitivity) linked to individual connectomic features.

Advanced statistical models—linear mixed-effects, Bayesian regression, correlation metrics—calibrate MRI-derived and tracer-based connectivity. The project publicly releases both the raw and preprocessed datasets through PRIME-DE, BALSA, and related repositories, facilitating translational studies in human brain organization and neuropsychiatric disorders (Hayashi et al., 2020).

5. Limitations, Open Challenges, and Prospective Directions

Autonomous Driving NeuroNCAP

• Current releases are limited to camera data; LiDAR/radar simulation and combined multi-sensor fusion are priorities for future iterations.
• Post-processing strategies substantially improve results in stationary scenarios but remain insufficient for dynamic or multi-agent interaction failures.
• Incorporation of deformable actors, advanced vehicle dynamics, automated worst-case scenario discovery, and domain adaptation are identified as major development trajectories (Ljungbergh et al., 2024).

Functional Connectomics NeuroNCAP

• FC is treated as static; dynamic FC modeling is currently absent.
• Gumbel-sigmoid mask attention is an approximation; hard combinatorial selection remains computationally infeasible.
• The generalizability to other neurological conditions requires future validation.
• Additional integration of multi-modal data (e.g., structural MRI, DTI), hard attention mechanisms, and causal-inference constraints for counter-condition modeling are proposed enhancements (Kang et al., 2023).

NonHuman Primate Connectomics NeuroNCAP

• Determining the full alignment and discrepancies between MRI- and tracer-derived connectivity remains an active statistical and methodological concern.
• The scalability of behavioral-genetic analysis and its direct translatability to human cohorts require further elaboration (Hayashi et al., 2020).

6. Significance Across Domains

NeuroNCAP, in its various instantiations, constitutes a paradigm for integrating data-driven, interpretable, and causally rigorous testing and analysis in both cognitive neurosciences and robotics. In autonomous driving, the closed-loop NeRF-based framework provides a reproducible, extensible platform for stress-testing planners under adversarial, sensor-realistic conditions, revealing critical safety gaps and enabling benchmarking of auxiliary risk heads.

In neuroscience, the counter-condition analysis and ground-truth validated connectomics pipeline unify methodological rigor and biological insight, facilitating personalized classification, biomarker discovery, and translational research. Together, these approaches exemplify the convergence of neural computation, rigorous data integration, and scenario-based evaluation in state-of-the-art applied research.