AI Mismatches: Identifying Potential Algorithmic Harms Before AI Development

Abstract: AI systems are often introduced with high expectations, yet many fail to deliver, resulting in unintended harm and missed opportunities for benefit. We frequently observe significant "AI Mismatches", where the system's actual performance falls short of what is needed to ensure safety and co-create value. These mismatches are particularly difficult to address once development is underway, highlighting the need for early-stage intervention. Navigating complex, multi-dimensional risk factors that contribute to AI Mismatches is a persistent challenge. To address it, we propose an AI Mismatch approach to anticipate and mitigate risks early on, focusing on the gap between realistic model performance and required task performance. Through an analysis of 774 AI cases, we extracted a set of critical factors, which informed the development of seven matrices that map the relationships between these factors and highlight high-risk areas. Through case studies, we demonstrate how our approach can help reduce risks in AI development.

• The paper presents seven matrices that assess discrepancies between expected and required performance to preempt algorithmic harms in AI systems.
• It details frameworks like the Required Performance and Disparate Performance matrices to evaluate ethical risks and error costs in practical scenarios.
• The study analyzes 774 AI cases to illustrate risk factors and promotes interdisciplinary collaboration for aligning technical abilities with societal values.

A Detailed Examination of "AI Mismatches: Identifying Potential Algorithmic Harms Before AI Development"

Introduction

The paper "AI Mismatches: Identifying Potential Algorithmic Harms Before AI Development" introduces a framework for addressing AI Mismatches—discrepancies between expected AI model performance and the performance required to avoid harm and ensure value creation. It outlines a proactive approach to anticipate algorithmic risks before substantial development begins, emphasizing the importance of early-stage interventions. Through a study of 774 AI-related cases, the authors propose seven matrices that chart critical factors leading to high-risk areas in AI development.

Overview of AI Mismatches

AI Mismatches manifest when an AI system's actual performance deviates from the expected or required performance level necessary to deliver its intended benefits without causing harm. The framework advanced by the paper identifies and visualizes these mismatches across seven matrices, each probing different dimensions of AI risks, such as performance, data quality, error costs, and disparities (Figure 1).

Figure 1: Motivations and Interconnections of the Seven AI Mismatch Matrices.

Core Performance Mismatch Matrices

1. Required Performance Matrix
   • Purpose: Evaluates whether the expected model performance is sufficient to deliver the intended value.
   • Axes: Expected Model Performance vs. Minimum Required Performance.
   • Application: If expected performance is below the required threshold, reconsideration of the AI concept's feasibility is necessary. It flags potential feasibility issues early in the development process.
2. Disparate Performance Matrix
   • Purpose: Assesses expected performance disparities across different groups.
   • Axes: Expected Disparity in Performance vs. Importance of Avoiding Disparities.
   • Application: Highlights ethical and social justice considerations. A high disparity in critical applications calls for immediate action or redesign.
3. Cost of Errors Matrix
   • Purpose: Investigates the consequences of errors made by the AI system.
   • Axes: Expected Model Performance vs. Severity of Error Consequences.
   • Application: Crucial for understanding the impact of AI errors, particularly in high-stakes environments where errors could lead to severe consequences.

     Figure 2: The Cost of Errors matrix.

Supporting Matrices

1. Data Quality Matrix
   • Focuses on the relationship between data quality and the task's tolerance for low-quality data. Essential for understanding data limitations that may skew AI performance.
2. Model Unobservables Matrix
   • Evaluates how missing unobservable factors could impact model performance. It is critical in contexts where untracked variables contribute to outcomes.
3. Expectation of Errors Matrix
   • Considers user expectations of error frequency and tolerance for AI errors. Useful in gauging the user experience and system acceptability.
4. Error Detection and Mitigation Matrix
   • Assesses the ease of error detection and the effort required for mitigation. Offers insight into operational management and system refinement processes.

Application of Matrices in Practice

When visualized together, these matrices allow practitioners to assess the feasibility and risk of AI concepts systematically before resource-intensive development phases. The case studies provided in the paper illustrate the application of these matrices in real-world scenarios within domains such as child welfare and financial services, offering actionable insights into identifying and mitigating high-risk areas early on.

Implications for AI Development

The proposed matrices guide interdisciplinary collaboration among technical, ethical, and domain experts, fostering informed decision-making in AI projects. This framework aims to bridge the gap between AI’s technical performance metrics and real-world task requirements, ensuring AI systems are both effective and ethical.

Conclusion

This research establishes a foundational approach for pre-development risk assessment, enabling teams to align AI concepts with technological capabilities and ethical imperatives effectively. Future studies should expand on refining these matrices to cover a broader range of applications and further enhance AI innovation processes toward benefiting societies while minimizing potential harms.