Black-Box Access is Insufficient for Rigorous AI Audits (2401.14446v3)

Abstract: External audits of AI systems are increasingly recognized as a key mechanism for AI governance. The effectiveness of an audit, however, depends on the degree of access granted to auditors. Recent audits of state-of-the-art AI systems have primarily relied on black-box access, in which auditors can only query the system and observe its outputs. However, white-box access to the system's inner workings (e.g., weights, activations, gradients) allows an auditor to perform stronger attacks, more thoroughly interpret models, and conduct fine-tuning. Meanwhile, outside-the-box access to training and deployment information (e.g., methodology, code, documentation, data, deployment details, findings from internal evaluations) allows auditors to scrutinize the development process and design more targeted evaluations. In this paper, we examine the limitations of black-box audits and the advantages of white- and outside-the-box audits. We also discuss technical, physical, and legal safeguards for performing these audits with minimal security risks. Given that different forms of access can lead to very different levels of evaluation, we conclude that (1) transparency regarding the access and methods used by auditors is necessary to properly interpret audit results, and (2) white- and outside-the-box access allow for substantially more scrutiny than black-box access alone.

Rigorous AI Audits Require More Than Black-Box Access

The paper "Black-Box Access is Insufficient for Rigorous AI Audits" examines the limitations of black-box AI audits and advocates for more comprehensive white- and outside-the-box auditing practices. This discussion is timely as AI systems become more complex and influential in society, making rigorous audits an essential component of AI governance.

Black-Box Insufficiencies

The authors begin by critiquing the current trend of relying solely on black-box access, which allows auditors to query AI systems and analyze outputs without any insight into the internal workings. The paper argues that this approach is inherently limited in scope for several reasons:

• Identification of Failure Modes: Without white-box access, auditors cannot effectively diagnose complex failure modes such as backdoors or dataset biases. Existing black-box methods often depend on heuristics, which can misrepresent a model's capabilities and limitations.
• Mechanistic Insights: Evaluations limited to input-output interactions prevent auditors from understanding internal representations and mechanisms, crucial for diagnosing issues and improving models.
• Explanations and Justifications: Black-box auditing restricts explanations of decision-making processes, which are essential for accountability and user trust.

Advantages of White-Box and Outside-the-Box Access

The authors comprehensively discuss the enhancements provided by white- and outside-the-box access:

• Stronger Attack Techniques: White-box methods enable gradient-based optimization and other attacks that exploit internal model details, offering more robust identification of vulnerabilities compared to black-box methods.
• Enhanced Interpretability: Access to internal architecture facilitates interpretability methods, leading to a clearer understanding of how models make decisions. This is crucial for addressing issues like bias and ensuring compliance with legal standards.
• Dormant Capability Detection: White-box access allows fine-tuning and inspection for harmful capabilities that are not apparent through black-box testing alone, ensuring that hidden risks can be identified and mitigated.
• Contextual Evaluation: Outside-the-box access provides information on training data, methodologies, and developer evaluations, aiding auditors in designing targeted evaluations and understanding the technological and societal implications of AI systems.

Addressing Security Concerns

The authors recognize the potential risks of increased information sharing, particularly the possibility of intellectual property leakage. To mitigate these concerns, they propose several strategies:

• API-Based Solutions: By providing structured API access, auditors can perform white-box evaluations while keeping the system's parameters secure.
• Secure Research Environments: On-site audits in controlled environments can help provide comprehensive access without compromising security.
• Legal Mechanisms: Strategies from financial auditing, such as confidentiality agreements and conflict of interest regulations, can be adapted to AI audits.

Implications and Future Directions

The paper concludes that without white- and outside-the-box access, audits will remain insufficiently rigorous. As AI systems continue to evolve, both technical and policy frameworks will need to adapt. There is a clear need for continued development of auditing tools and techniques, alongside institutional and regulatory measures, to ensure audits maintain pace with technological advancements.

By advocating for enhanced access to AI systems, the authors align with an emerging consensus that regards transparency and thorough evaluation as pivotal for the responsible governance of AI. Future research could explore how to balance these needs with the proprietary interests of AI developers, and how innovative technical solutions, such as homomorphic encryption, might enable secure and comprehensive audits. As audits evolve, they will play a critical role in ensuring the safety, fairness, and accountability of AI systems.