Redefining Safety for Autonomous Vehicles (2404.16768v4)

Abstract: Existing definitions and associated conceptual frameworks for computer-based system safety should be revisited in light of real-world experiences from deploying autonomous vehicles. Current terminology used by industry safety standards emphasizes mitigation of risk from specifically identified hazards, and carries assumptions based on human-supervised vehicle operation. Operation without a human driver dramatically increases the scope of safety concerns, especially due to operation in an open world environment, a requirement to self-enforce operational limits, participation in an ad hoc sociotechnical system of systems, and a requirement to conform to both legal and ethical constraints. Existing standards and terminology only partially address these new challenges. We propose updated definitions for core system safety concepts that encompass these additional considerations as a starting point for evolving safe-ty approaches to address these additional safety challenges. These results might additionally inform framing safety terminology for other autonomous system applications.

• The paper redefines safety by critiquing traditional frameworks and proposing comprehensive, multi-dimensional definitions for autonomous vehicles.
• It rigorously analyzes standards like ISO 26262, highlighting their limitations in unpredictable, open-world environments.
• The authors advocate for continuous safety validation and lifecycle management to align technical performance with legal and ethical constraints.

Redefining Safety for Autonomous Vehicles: Critical Considerations and Proposals

The paper "Redefining Safety for Autonomous Vehicles" by Philip Koopman and William Widen presents a rigorous examination of the evolving safety landscape confronting autonomous vehicle (AV) systems. With a focus on redefining core safety terminology, the work reflects a vital need to adapt existing safety frameworks to address challenges posed by AVs, as they operate without human drivers in open-world environments and must comply with a complex interplay of legal and ethical constraints.

Analyzing Existing Safety Frameworks

The discussion begins by critiquing current safety standards, notably ISO 26262, ISO 21448, and UL 4600, assessing their adequacy for AVs. These standards traditionally emphasize preventing unreasonable risk emanating from specific hazards, relying on a human-supervised operational context. However, AVs necessitate an expanded scope, navigating environments filled with unforeseen conditions and requiring self-enforcement of operational limits. Consequently, traditional frameworks show limitations, often assuming a fully characterized operational environment and ignore prolonged exposure to unpredictable conditions.

Real-World Implications and Incidents

Empirical evidence underscores these conceptual gaps, with numerous incidents illustrating the inadequacies of current safety measures. For example, robotaxis have displayed behaviors such as inappropriate immobilization in emergency lanes and failure to yield to emergency vehicles, behaviors not typically anticipated by existing definitions of vehicle safety. The listed incidents illustrate that AV systems must engage in higher-order societal interactions and often face dilemmas beyond mere collision avoidance, such as safety trade-offs in system-of-systems contexts and reconciling legal and ethical constraints with societal norms.

Proposing Updated Definitions

The authors propose a set of augmented definitions that better reflect the unique demands of AV safety. Key among these is reorienting the notion of "safety" from simple risk mitigation to satisfying a broader array of safety constraints, encompassing legal and equity considerations. This includes:

• Safety: Defined in terms of meeting externally imposed constraints beyond simple hazard mitigation.
• Safety Case: Expanded to incorporate continuous lifecycle management and accommodate stakeholder-driven constraints.
• Risk: Encompasses both probability and the consequences of patterns of losses, taking into account stakeholder perceptions and societal impacts.

Such redefinitions aim to capture the multifaceted safety challenges inherent to autonomous systems, recognizing the operational environment's unpredictability and the ethical underpinnings of vehicle behavior in public domains.

Implications for Autonomous Vehicle Systems

The proposed terminological updates have strong implications for AV system design and certification. They necessitate comprehensive validation processes that address not only malfunction risks but also the broader societal implications of system behavior. Moreover, these changes suggest that safety engineering practices must evolve to integrate new, more dynamic forms of hazard analysis, incorporating concepts like safety performance indicators and continuous improvement mechanisms to manage risks associated with an open-world system.

Conclusions and Future Directions

The paper serves as a call to action within the safety community to reassess and augment existing standards in line with the operational realities and societal expectations of autonomous systems. While focused on AVs, the insights and proposals have broader applicability across autonomous technologies. The authors advocate for a nuanced, multi-dimensional approach to safety, reflecting the complex interplay between technology, society, and regulation. This entails a shift away from narrow, compliance-driven interpretations to a more holistic view embracing the dynamic nature of modern automated systems.