On the Safety of Machine Learning: Cyber-Physical Systems, Decision Sciences, and Data Products (1610.01256v2)

Abstract: Machine learning algorithms increasingly influence our decisions and interact with us in all parts of our daily lives. Therefore, just as we consider the safety of power plants, highways, and a variety of other engineered socio-technical systems, we must also take into account the safety of systems involving machine learning. Heretofore, the definition of safety has not been formalized in a machine learning context. In this paper, we do so by defining machine learning safety in terms of risk, epistemic uncertainty, and the harm incurred by unwanted outcomes. We then use this definition to examine safety in all sorts of applications in cyber-physical systems, decision sciences, and data products. We find that the foundational principle of modern statistical machine learning, empirical risk minimization, is not always a sufficient objective. Finally, we discuss how four different categories of strategies for achieving safety in engineering, including inherently safe design, safety reserves, safe fail, and procedural safeguards can be mapped to a machine learning context. We then discuss example techniques that can be adopted in each category, such as considering interpretability and causality of predictive models, objective functions beyond expected prediction accuracy, human involvement for labeling difficult or rare examples, and user experience design of software and open data.

• The paper introduces a comprehensive framework that integrates risk, epistemic uncertainty, and potential harm to redefine safety in machine learning systems.
• It critiques traditional empirical risk minimization and categorizes safety strategies into inherently safe design, safety reserves, safe fail, and procedural safeguards.
• The study applies these insights to cyber-physical systems, decision sciences, and data products, emphasizing actionable approaches for mitigating real-world operational risks.

Safety Considerations in Machine Learning Systems

The paper "On the Safety of Machine Learning: Cyber-Physical Systems, Decision Sciences, and Data Products" by Kush R. Varshney and Homa Alemzadeh introduces a comprehensive examination of safety in systems utilizing ML. A central assertion is that the notion of safety, traditionally applied to engineered systems like highways or industrial plants, must also be relevantly applied to ML systems, especially as they become increasingly integrated into critical societal functions in sectors such as healthcare, finance, and transportation.

The authors make a compelling case for formalizing the definition of safety in ML by integrating concepts of risk, epistemic uncertainty, and harm due to unwanted outcomes. They convincingly argue that the prevailing framework of empirical risk minimization (ERM) is insufficient. ERM chiefly addresses regularized optimization over historical data, which may ignore critical concerns related to unquantified uncertainties in unforeseen contexts.

Framework for ML Safety

The paper leverages Moeller's decision-theoretic definition of safety, focusing on minimizing the risks and uncertainties associated with severe, unwanted outcomes. The authors suggest that achieving safety necessitates both reducing the likelihood of expected harms and accounting for the unknown probabilities of unexpected harms. Unlike analogies in classical engineering domains, ML systems require explicit consideration of the epistemic uncertainties linked to data representation and model generalization.

Categories and Strategies for Safety

The authors categorize strategies for achieving safety into four segments: inherently safe design, safety reserves, safe fail, and procedural safeguards.

1. Inherently Safe Design: This emphasizes models that are interpretable and utilize causation-aware features. By focusing on interpretable models, ML systems can support more robust diagnostics and exclusions of biased patterns learned from training data.
2. Safety Reserves: Concepts like robust optimization address the uncertainty variability between training and testing distributions directly. Additionally, fairness in decision-making processes can be framed within permissible safety margins to prevent undue risk differentials across protected groups.
3. Safe Fail: The adoption of reject options in ML offers a mechanism to prevent overconfident decisions in low-data-density regions, effectively handing over control to human operators.
4. Procedural Safeguards: Designing user interfaces that guide operators when defining mission-critical data pipelines and making sources of training data publicly available can mitigate risks from erroneous assumptions and obscure data biases.

Implications and Applications

The paper insightfully discusses applications across cyber-physical systems, decision sciences, and data products. In cyber-physical systems, ML is often used in high-stakes environments like autonomous vehicles and surgical robots, where safety-critical controls involve not just prediction accuracy but also real-time adaptive decision-making facing unexpected contingencies. Decision science applications such as predictive modeling in human resource management or financial services must consider fairness and avoid embedding institutional biases. Data products, while less critical from an immediate safety perspective, must nonetheless consider biases and potential long-term harms.

Conclusion and Future Directions

In establishing a framework for safety in ML, this paper contributes significant theoretical groundwork and practical strategies for researchers and practitioners. The work invites further exploration into the development of novel ML algorithms that intrinsically incorporate safety considerations beyond naive risk minimization. As ML continues to occupy a pivotal role in socio-technical contexts, the safety-first agenda prescribed here underscores a necessary shift towards algorithms that acknowledge and strategically mitigate both known and unknown uncertainties in their operational environment. This emphasis not only aligns with ethical imperatives but increasingly intersects with legal regimes mandating standards for algorithmic transparency and accountability.