Autonomous driving technology has seen significant advancements, promising safer and more efficient transportation. However, the increasing reliance on complex, often "black-box" AI models presents a challenge: the decision-making process of autonomous vehicles (AVs) is not easily understandable by humans. This lack of transparency hinders social acceptance, trust, and regulatory compliance. This paper provides a comprehensive overview of Explainable Artificial Intelligence (XAI) approaches applied to autonomous driving, aiming to bridge this gap.
The fundamental need for XAI in AVs stems from multiple perspectives. Psychologically, accidents involving AVs erode public trust. Sociotechnically, AV design must be human-centered and account for user needs. Philosophically, explanations help understand the causal history of decisions, especially in critical situations. Legally, regulations like the EU's GDPR emphasize the "right to an explanation," and AVs must comply with standards like ISO 26262 (functional safety) and ISO/SAE 21434 (cybersecurity). Benefits of explainable AVs include enhanced human-centered design, improved trustworthiness, better traceability for post-accident analysis, and increased transparency and accountability.
Explanations must be tailored to different stakeholders, including road users, AV developers, regulators/insurers, and executive management. Delivery methods vary, often relying on human-machine interfaces (HMIs) such as display monitors, sound, lights, or vibrotactile feedback. Key questions in designing explanations include who needs them, why they are needed, what kind can be generated, and when they should be delivered. Explanations can be derived from psychological theories or focus on user needs, and their form can be visual, textual, or multimodal.
The paper surveys various XAI techniques applied to AVs, categorized by their underlying methodology:
- Visual Explanations: These methods often leverage techniques like saliency maps, CAM, and Grad-CAM to highlight image regions influencing decisions. Examples include using VisualBackProp for debugging CNN predictions in end-to-end driving [bojarski2016visualbackprop], semantic segmentation for environment perception [hofmarcher2019visual], causal attention models for steering control [kim2017interpretable], and interpretable neural motion planners visualizing 3D detections, trajectories, and cost maps [zeng2019end]. Counterfactual explanations, showing minimal changes to the input scene to alter a decision (e.g., from "stop" to "go"), are also explored using generative models [li2020make, jacob2022steex, zemni2023octet]. Datasets like BDD-X [kim2018textual] and BDD-OIA [xu2020explainable] are used for training and evaluation.
- Reinforcement Learning (RL) and Imitation Learning (IL)-based Explanations: These approaches aim to explain how perceived states map to actions. Methods include Semantic Predictive Control [pan2019semantic] for visual policy explanation, latent RL models interpreting actions via bird-eye masks [chen2021interpretable], interpretable vision-based motion planning using semantic maps and optical flow [wang2021learning], and probabilistic models integrating internal states like speed for tasks like lane merging [wang2021uncovering]. IL-based methods use visual attention models [cultrera2020explaining] or bird-eye views combined with steering angles [teng2022hierarchical] to make end-to-end driving decisions interpretable. Frameworks like PlanT use transformers for planning and object recognition for explanations [renz2022plant].
- Decision Tree-based Explanations: Decision trees are intrinsically interpretable and have been used to generate scenario-based textual explanations (Why, Why Not, What If) by mapping observations to actions according to rules [omeiza2021towards]. They are also used for goal recognition [brewitt2021grit] and interpreting tasks like car-following [cui2022interpretation].
- Logic-based Explanations: These focus on verifying the safety of AVs using formal methods. Signal temporal logic is used to describe and identify interpretable failure cases [corso2020interpretable, decastro2020interpretable]. Answer set programming is applied for online sensemaking in perception/control and common-sense reasoning in decision-making [suchan2019out, kothawade2021auto].
- User Study-based Explanations: Empirical studies with human participants help understand users' mental models, determine when and what explanations are needed in unexpected scenarios [wiegand2019drive, w2020d], and evaluate the effectiveness of different HMI modalities (visual, textual, audio, light, vibration) on trust and situation awareness [schneider2021increase, schneider2021explain, schneider2023don]. Research suggests explanations are most valuable in critical situations to avoid information overload [kim2023and].
- LLMs and Vision-LLMs (VLMs)-based Explanations: Emerging as a recent paradigm, LLMs and VLMs enable natural language explanations. Systems like Wayve's LINGO-1 and LINGO-2 provide live linguistic explanations for end-to-end driving actions and causal factors [wayve_lingo1_2023, wayve_lingo2_2024]. Video Question Answering (VideoQA) allows interactive dialogue between users and AVs about traffic scenes and decisions [xu2023drivegpt4, marcu2023lingoqa, park2024vlaad]. LLMs are also used for interpretable motion planning [mao2023gpt], chain-of-thought reasoning [sha2023languagempc, wen2023dilu], and predicting other actors' intent [dewangan2023talk2bev].
The paper proposes a conceptual framework for explainable end-to-end autonomous driving that integrates three core components:
- End-to-end Control (eeC): This maps the environment to actions, typically implemented using RL or IL, aiming for efficient, unified decision-making.
- Safety-Regulatory Compliance (srC): This component represents the function of a regulatory agency and involves verifying the safety of the eeC and actions against standards (ISO 26262) and cybersecurity requirements (ISO/SAE 21434). Compliance can be confirmed via rigorous simulation and real-world verification, akin to a "driving school" for the AV software stack.
- Explanation Component: This provides insights into real-time decisions, justifying chosen actions. Explanations must consider temporal granularity, including:
- Timing: Explanations delivered before an action is preferred for allowing human intervention [haspiel2018explanations].
- Lead Time: Sufficient time (e.g., >10 seconds for non-critical, several seconds for critical) is needed for safe human takeover requests [wan2018effects].
- Frequency: Explanations should be delivered only when required (e.g., in critical conditions) to avoid user overload [kim2023and]. Effective HMI design is crucial for conveying explanations tailored to diverse user needs and abilities.
The future direction, referred to as AV 2.0, envisions unifying vision, language, and action within an Embodied AI paradigm. This aims for adaptive learning, scaling, and generalization in complex environments. However, achieving safe and explainable AV 2.0 requires addressing fundamental AI safety challenges: avoiding negative side effects (ensuring AV actions don't indirectly cause harm to others), avoiding reward hacking (preventing the model from finding loopholes to maximize reward in unintended ways), scalable oversight (allowing humans to monitor complex behaviors), safe exploration (making safe choices even when exploring new behaviors), and robustness to distributional shift (maintaining performance from simulation to reality and in unseen scenarios). A crucial addition is the concept of "fail-safe" ability, allowing the AV to pause operations in challenging conditions.
Challenges also persist in explainability itself, particularly with emerging LLM/VLM approaches. Current models may lack human-level performance in generating explanations and struggle with timing. Hallucinations in LLMs can produce fictitious explanations with serious safety implications. Robustness against adversarial questions is also an issue, potentially leading to incorrect justifications [atakishiyev2024safety]. Regulating these models with common sense and human-defined concepts is vital for building trustworthy human-AI alignment, trust, and public acceptance.
In conclusion, the paper highlights that achieving truly explainable and trustworthy autonomous driving requires integrating sophisticated end-to-end control, robust safety verification, and context-aware explanation generation tailored to diverse users and timely delivery, while addressing inherent challenges in AI safety and the limitations of current explanation models.