Embracing Contradiction: Theoretical Inconsistency Will Not Impede the Road of Building Responsible AI Systems

Abstract: This position paper argues that the theoretical inconsistency often observed among Responsible AI (RAI) metrics, such as differing fairness definitions or tradeoffs between accuracy and privacy, should be embraced as a valuable feature rather than a flaw to be eliminated. We contend that navigating these inconsistencies, by treating metrics as divergent objectives, yields three key benefits: (1) Normative Pluralism: Maintaining a full suite of potentially contradictory metrics ensures that the diverse moral stances and stakeholder values inherent in RAI are adequately represented. (2) Epistemological Completeness: The use of multiple, sometimes conflicting, metrics allows for a more comprehensive capture of multifaceted ethical concepts, thereby preserving greater informational fidelity about these concepts than any single, simplified definition. (3) Implicit Regularization: Jointly optimizing for theoretically conflicting objectives discourages overfitting to one specific metric, steering models towards solutions with enhanced generalization and robustness under real-world complexities. In contrast, efforts to enforce theoretical consistency by simplifying or pruning metrics risk narrowing this value diversity, losing conceptual depth, and degrading model performance. We therefore advocate for a shift in RAI theory and practice: from getting trapped in inconsistency to characterizing acceptable inconsistency thresholds and elucidating the mechanisms that permit robust, approximated consistency in practice.

Embracing Contradiction in Responsible AI Systems

In the paper "Embracing Contradiction: Theoretical Inconsistency Will Not Impede the Road to Building Responsible AI Systems," the authors, Gordon Dai and Yunze Xiao, present a compelling argument for embracing theoretical inconsistency within Responsible AI (RAI) metrics. They propose that the inconsistencies commonly observed among fairness, privacy, and robustness metrics should be seen as advantageous features rather than defects that need correction. They argue that such inconsistencies, when managed properly, offer normative pluralism, epistemological completeness, and implicit regularization.

Key Contributions

1. Normative Pluralism: By maintaining a suite of conflicting metrics, the diverse moral perspectives and stakeholder values inherent in RAI are adequately represented. This is crucial in AI alignment, which must accommodate myriad human values.
2. Epistemological Completeness: Multiple metrics capture the complex and multifaceted nature of ethical concepts like fairness and political neutrality. Rather than oversimplifying these concepts, the use of multiple metrics preserves the breadth of meaning and intent behind them.
3. Implicit Regularization: The joint optimization for conflicting objectives can prevent overfitting to any single metric, increasing model robustness and generalization capabilities under real-world conditions. This approach leverages the mechanistic interactions between metrics as a form of semantic regularization.

Theoretical Framework

The paper delineates two forms of inconsistency in metrics: intra-concept inconsistency, where metrics derived from the same concept clash, and inter-concept tradeoffs, where optimizing for one metric degrades another. Examples such as fairness and political neutrality illustrate these types of inconsistencies.

• Intra-concept Inconsistency: The authors discuss fairness, showing that no algorithm can satisfy all fairness definitions simultaneously due to differences in group base rates. They relate this to the fairness impossibility theorem, which highlights the mathematical conflicts inherent in fairness notions such as demographic parity and equalized odds.
• Inter-concept Tradeoff: Tradeoffs between accuracy and privacy or fairness are explored with information-theoretic and empirical evidence. For example, Zhao and Gordon showed theoretically that maintaining demographic parity imposes additional error due to disparate group base rates, but practical methods can minimize these discrepancies without significant accuracy loss.

Practical Implications

The paper advocates against reducing inconsistency as a way to simplify metrics but recommends setting acceptable inconsistency thresholds. This preserves the pluralism essential in AI applied across diverse societal contexts. Additionally, maintaining theoretical inconsistencies leverages them as regularizers to enhance model robustness against adversarial environments and contribute to better generalization.

1. Empirical Evidence: The authors present empirical studies, such as those by Bell and others, to reveal that accommodating multiple metrics can lead to approximately consistent models with robust performance metrics.
2. Pluralistic Alignment: Embracing multiple inconsistent metrics supports the paradigm of pluralistic alignment, which captures the diversity of human values necessary for ethical AI engagement within specific communities.

Moving Forward in AI Theory

The paper suggests future directions for RAI theory and practice, advocating for a shift towards understanding and characterizing metric inconsistency tolerance rather than complete eradication of inconsistencies. Moreover, it proposes further empirical studies to understand human interactions with pluralistic evaluation tools and forming documentation frameworks to clearly articulate normative assumptions within evaluation metrics.

Conclusion

Dai and Xiao's paper challenges the conventional pursuit of internal consistency among Responsible AI metrics and opens a debate on the strategic value of such contradictions. By embracing theoretical inconsistencies, AI systems can better reflect diverse values, retain complex informational richness, and mitigate overfitting through implicit regularization, leading to more ethically aligned and technically robust AI systems.