Towards a Systematic Risk Assessment of Deep Neural Network Limitations in Autonomous Driving Perception

Abstract: Safety and security are essential for the admission and acceptance of automated and autonomous vehicles. Deep neural networks (DNNs) are widely used for perception and further components of the autonomous driving (AD) stack. However, they possess several limitations, including lack of generalization, efficiency, explainability, plausibility, and robustness. These insufficiencies can pose significant risks to autonomous driving systems. However, hazards, threats, and risks associated with DNN limitations in this domain have not been systematically studied so far. In this work, we propose a joint workflow for risk assessment combining the hazard analysis and risk assessment (HARA) following ISO 26262 and threat analysis and risk assessment (TARA) following the ISO/SAE 21434 to identify and analyze risks arising from inherent DNN limitations in AD perception.

• The paper introduces a combined HARA-TARA method to systematically map DNN limitations in autonomous driving perception to safety and cybersecurity risks.
• It classifies DNN shortcomings—such as lack of generalization, robustness, and explainability—and ties them to risk metrics like ASIL levels and high-security threats.
• The methodology provides actionable mitigation strategies, including data augmentation and runtime monitoring, to enhance both functional safety and security in AV systems.

Systematic Risk Assessment of Deep Neural Network Limitations in Autonomous Driving Perception

Introduction

Advancements in deep neural networks (DNNs) have catalyzed performance improvements in the perception subsystems of autonomous vehicles (AVs), providing robust scene understanding critical for decision-making. However, these models exhibit intrinsic and extrinsic insufficiencies that introduce unique safety and cybersecurity hazards. The paper "Towards a Systematic Risk Assessment of Deep Neural Network Limitations in Autonomous Driving Perception" (2604.20895) addresses the gap in systematic risk assessments targeting DNN-rooted failures in AV perception. It proposes a structured workflow that integrates hazard analysis and risk assessment (HARA) under ISO 26262 with threat analysis and risk assessment (TARA) under ISO/SAE 21434, explicitly designed to map the multifaceted risks posed by DNN limitations to both safety and security domains.

DNN Limitations and Taxonomy

DNN limitations in AV perception are classified along two axes: intrinsic properties—arising from model architecture or fundamental learning theory (nature)—and deficiencies derived from data or training practices (nurture). The adopted taxonomy identifies five major insufficiencies directly influencing risk propagation:

• Lack of Generalization: Propensity to underperform in long-tail, rare, or out-of-distribution conditions. This catalyzes both inadvertent failures and attack surfaces via data gaps.
• Inefficiency: High computational and memory demands undermine real-time operation and can become attack vectors for denial-of-service scenarios.
• Lack of Explainability: Opaque representations and untraceable failure modes hinder root-cause analysis, limiting safety validation, and mask stealthy adversarial attacks.
• Lack of Plausibility: Outputting high-confidence predictions for physically unrealistic objects/events reveals absence of physical consistency constraints, creating vulnerabilities exploitable by semantic attacks.
• Lack of Robustness: Susceptibility to minor or adversarial input perturbations facilitates both unintentional safety hazards and intentional security breaches.

Standards Analysis

Existing automotive safety and AI regulation standards inadequately capture these DNN-specific failure modalities. While ISO 26262 and ISO/SAE 21434 provide frameworks for HARA and TARA, direct references to DNNs are absent. Recent documents (e.g., ISO PAS 8800, ISO/IEC TR 5469, EU AI Act) emphasize the importance of robustness, generalization, and explainability, recommending mitigations such as redundancy and data augmentation. However, these frameworks lack specificity for AV perception, especially for interpreting novel or adversarial sensor signals.

Combined HARA-TARA Risk Assessment Workflow

The proposed methodology introduces a unified HARA-TARA workflow, enabling coherent identification, assessment, and treatment of DNN-induced safety and security risks in the perception pipeline. It formalizes a procedure comprising:

1. System/Item Definition: Demarcating the scope (e.g., DNN-based perception module: object detection, traffic sign recognition, semantic segmentation).
2. Function and Situation Analysis: Linking DNN insufficiencies to functional safety objectives.
3. Asset, Hazard, and Threat Identification: Mapping each DNN limitation to both traditional safety hazards and meaningful security threat scenarios, focusing on assets such as sensor data integrity, model behavior, and perception outputs.
4. Risk Assessment and Classification: Applying the Automotive Safety Integrity Level (ASIL) for safety risks and risk values (impact × feasibility) for security threats, directly correlating DNN limitations with asset vulnerabilities and mapping risk treatment strategies.
5. Safety and Security Goal Definition and Mitigation: Formulating traceable safety/security goals and partitioned mitigation strategies for each DNN shortcoming.

   Figure 1: Combined HARA-TARA workflow demonstrating interleaved safety and security risk assessment for DNN-based perception in autonomous vehicles.

Evaluation and Risk Mapping

Application of the workflow demonstrates decisive interdependencies between safety and security risk landscapes; limitations such as lack of generalization and robustness represent high-criticality hazards due to their dual role in causing unintentional failures and enabling adversarial exploits. Risk classification highlights:

• Generalization/Robustness: Underspecified training domains or sensitivity to perturbations can result in fatal perception errors (ASIL D/High-security risk). Mitigations include extensive data augmentation, out-of-distribution detection, adversarial training, and runtime input validation.
• Efficiency: Timing violations impact both safety (missed deadlines, ASIL C/B) and expose the system to resource exhaustion attacks. Runtime monitoring and prioritized scheduling are recommended.
• Explainability: Gaps in traceability complicate post-incident diagnostics and facilitate persistent, stealthy attacks. Integration of XAI tooling and decision logging are necessary for risk reduction.
• Plausibility: While direct safety consequences are usually moderate (ASIL B), security risks escalate when attackers inject physically inconsistent yet semantically plausible artifacts.

The analysis also reveals that high-severity, high-feasibility attack vectors (e.g., dataset poisoning or physical adversarial attacks) require intensified risk treatment beyond conventional HARA, embedding robust monitoring and layered defense strategies into both the development and operational lifecycles.

Theoretical and Practical Implications

The systematic integration of HARA and TARA tailored to DNN perception substantiates several theoretical advances:

• It enables rigorous mapping of cross-domain (safety and security) risk propagation pathways unique to AI-heavy pipelines, supporting development of formal safety/security cases for regulatory clearance.
• The framework accentuates essential limitations of extant standards in capturing emergent DNN risks and motivates refinement of automotive/Ai standards to mandate DNN-specific safety cases, with traceability from perception-system failure to incident consequence.
• Practically, the workflow provides a template for AV developers and safety assessors, clarifying how DNN-specific technical weaknesses must be holistically treated in both design-time (architecture, validation) and runtime (monitoring, update) risk controls.

Future Directions

The primary limitation of the current approach is reliance on qualitative, expert-driven risk assignments without integration of empirical incident datasets or systematic multi-stakeholder elicitation. Future research should close this gap via:

• Data-driven risk quantification, sourcing both pre- and post-deployment incident/attack logs.
• Extension to non-camera modalities and sensor fusion, addressing systemic risk across heterogeneous perception stacks.
• Codification of DNN-specific risk metrics and incorporation in safety case arguments accepted by international regulators.
• Automated, continuous risk assessment pipelines to support agile software releases within safety-critical operational design domains.

Conclusion

This work advances the state-of-the-art in AV safety and security engineering by formalizing a joint HARA-TARA risk assessment methodology specifically targeting DNN-related insufficiencies within the perception stack. The analysis highlights the necessity of cross-domain, DNN-aware risk modeling to capture interconnected safety and cybersecurity hazards. Integrative frameworks such as this will be essential for demonstrably safe and trusted deployment of higher-order AVs, especially as DNNs proliferate deeper across the automated driving stack.