- The paper challenges conventional one-dimensional automation by presenting a HCAI framework that integrates high automation with significant human control.
- The paper details methods for ensuring Reliability, Safety, and Trustworthiness through audit trails, benchmark testing, and independent oversight.
- The paper offers actionable design guidelines to mitigate risks from excessive automation or human control by emphasizing robust UI design and safety protocols.
This paper critiques the traditional one-dimensional view of automation, exemplified by the Sheridan-Verplank levels and the SAE levels for self-driving cars, which suggests an inherent trade-off between human control and computer automation (2002.04087). It argues that this view is limiting and can lead to dangerous designs, citing examples like the Boeing 737 MAX crashes and issues with Tesla's Autopilot system (2002.04087). Instead, the paper proposes a Human-Centered Artificial Intelligence (HCAI) framework aimed at creating Reliable, Safe & Trustworthy (RST) systems (2002.04087).
The core idea of HCAI is a two-dimensional framework that separates the level of human control from the level of computer automation. The goal, particularly for complex and poorly understood problems, is often to achieve high levels of human control AND high levels of automation simultaneously (2002.04087). This approach aims to amplify human performance, support self-efficacy, mastery, creativity, and responsibility, rather than simply replacing humans with autonomous systems (2002.04087).
Achieving RST systems involves three key aspects:
- Reliability: Stems from sound technical practices like maintaining detailed audit trails (similar to flight data recorders), using benchmark tests for verification, continuous data quality monitoring, and bias testing (2002.04087).
- Safety: Cultivated through management strategies that foster a culture of safety, including leadership commitment, open reporting of problems, internal reviews of failures, and public reporting (2002.04087).
- Trustworthiness: Built upon independent oversight structures such as capable companies, professional organizations (e.g., IEEE, ISO), government regulatory agencies (e.g., FDA, FAA), non-governmental certification bodies (e.g., UL), auditing firms, and insurance companies (2002.04087).
The HCAI framework visualizes design goals within four quadrants based on low/high human control and low/high computer automation:
- Low Automation / Low Human Control: Simple devices like clocks or mousetraps (2002.04087).
- High Automation / Low Human Control (Computer Control): Suitable for predictable tasks requiring rapid action where human intervention is impossible or undesirable, like airbag deployment or pacemakers. Requires extremely careful design and testing (2002.04087).
- Low Automation / High Human Control (Human Mastery): Appropriate for activities where skill development, engagement, and creativity are paramount, such as riding a bicycle or playing a musical instrument (2002.04087).
- High Automation / High Human Control (HCAI Goal for RST): The target for many complex systems. Examples include well-designed elevators, digital cameras, advanced thermostats, and future RST self-driving cars. This quadrant leverages automation to enhance human capabilities and control (2002.04087).
The framework also highlights two dangerous regions to avoid:
- Excessive Automation: Where automation lacks transparency, overrides human control unexpectedly, or lulls users into complacency (e.g., Boeing 737 MCAS, Tesla Autopilot leading to driver inattention) (2002.04087).
- Excessive Human Control: Where a lack of safeguards or automation allows for critical human errors (e.g., driving under the influence without interlocks, train speeding without Positive Train Control). Mitigated by interlocks, guards, and constraints (2002.04087).
To implement HCAI systems that empower users, the paper suggests following the Prometheus Principles for user interface design, which ensure comprehensibility, predictability, and controllability:
- Consistent interfaces for expressing intent.
- Continuous visual display of relevant information and actions.
- Rapid, incremental, and reversible user actions.
- Informative feedback confirming actions.
- Progress indicators showing system status.
- Completion reports confirming task accomplishment (2002.04087).
Several examples illustrate these concepts:
- Thermostats: Allow users to see the current state, set a desired state, and receive feedback, while automation maintains the temperature (2002.04087).
- Automotive Features: Incremental automation like ABS, cruise control, and lane assist enhances driver control and safety (2002.04087).
- Elevators: High automation coordinated with clear user controls (buttons, floor indicators) and safety interlocks results in an RST system (2002.04087).
- Digital Cameras: Combine automatic adjustments (focus, aperture, stabilization) with user controls (zoom, modes, filters, manual overrides) (2002.04087).
- Patient Controlled Analgesia (PCA): Can be designed across the quadrants, with the RST ideal involving patient input guided by automated safeguards, sensor data, and clinician monitoring (2002.04087).
In conclusion, the paper advocates for a paradigm shift from pursuing pure autonomy to designing HCAI systems. By integrating high levels of automation with high levels of human control through thoughtful interface design and robust RST practices, developers can create AI systems that effectively amplify, augment, and empower users (2002.04087).