- The paper introduces a systematic audit framework that verifies AI systems' reliability, ethics, and legal compliance across all development phases.
- It details a comprehensive lifecycle from data management to operation, along with seven audit access levels ranging from black-box to white-box.
- The framework integrates technical audits with governance, addressing trade-offs in fairness, robustness, explainability, and privacy for trustworthy AI.
This paper proposes a framework for the audit and assurance of AI algorithms, emphasizing the need for systematic verification to ensure they are lawful, ethical, and secure, similar to financial audits. As AI systems become more prevalent in critical sectors with limited human intervention, the risks of reputational or financial damage necessitate a standardized approach to assessing their integrity. Algorithm auditing aims to professionalize and industrialize AI development by providing a formal guarantee of reliability, legality, fairness, and regulatory compliance.
The paper outlines the AI application development lifecycle, which includes four phases:
- Data Management: Involves data collection, storage, processing, and documentation to ensure data pipelines are well-structured and the task is well-defined.
- Model Selection: Focuses on cross-validation, optimization, and comparison of different models.
- Development: Enhances interpretability, adds feedback mechanisms, and assesses the system's impact.
- Operation: Implements supervision and supply interfaces and records outcomes and feedback from the field.
Each of these phases can be audited, with the overall audit guided by a policy document outlining the algorithm's intended purpose.
A core concept of the framework is the acknowledgement of varying degrees of access an auditor might have to the AI system. This ranges from 'black-box' (no access) to 'white-box' (full access), with multiple 'shades of grey' in between. The paper proposes seven potential audit phases based on this access level:
- Process Access: Auditor has no direct access to the algorithm but reviews the development process using checklists and rules. Suitable for low-risk applications.
- Model Access: Auditor can make prediction calls via an API using fictitious input but has no access to underlying data patterns or training data. Focuses on API-level analysis.
- Input Access: Auditor can use real input data to make predictions but lacks access to the actual results or model parameters. Analysis focuses on bias from input distribution, property inference, or surrogate explanations.
- Output Access: Auditor has access to real input and output data used for training/verification and can make predictions. This enables model-agnostic analysis, concept drift detection, inversion attacks, and bias verification (e.g., equality of opportunity).
- Parameter Control: Auditor has administrative rights to model parameters, input/output data, and prediction ability, but not the model family or reward function. Allows for consistency, perturbation testing, and evaluation of operational theft risk.
- Learning Objective: Auditor understands the model's construction, learning goals, parameters, input/output data, and can make predictions. Full understanding of the training objective function enables detailed feedback on network size, stress testing, and trade-off analysis (bias, privacy, loss) without assumptions.
- White-box: Auditor has complete information including architecture, learning process, objectives, parameters, input/output data, and prediction access. Provides the highest level of detail for evaluation and mitigation assessment, best suited for internal audits due to required transparency.
The outcome of an algorithm audit is to increase confidence and trust, potentially leading to certification. This assurance process verifies compliance with regulatory, governance, and ethical requirements. This requires establishing standards, both general (covering privacy, explainability, safety, fairness) and sector-specific. Governance involves both non-technical aspects (decision-makers, training, human-in-the-loop) and technical aspects (ensuring robustness, bias mitigation, explainability by design). Technical audits should occur throughout development and during live monitoring, with impact evaluations preceding deployment to identify and mitigate risks.
The paper links algorithm auditing to the concept of "Trustworthy AI," highlighting four key elements that require focus:
- Accountability and Privacy: Involves data governance (quality, integrity, relevance, access procedures) and protecting against model inferences and attacks (like extracting model copies). Data Protection Impact Assessments are a standard tool.
- Discrimination and Bias: Addresses various forms of bias (from historical data, under-sampling, etc.) and the need to distinguish between personal fairness (treating similar individuals similarly) and collective fairness (achieving equality across groups based on protected characteristics). It acknowledges the theoretical impossibility of satisfying all fairness criteria simultaneously.
- Explainability and Interpretability: Crucial for building user trust and allowing developers to debug and expose unjust decisions. Interpretability is understanding the cause-effect, while explainability is describing the underlying mechanics. Approaches include intrinsic (models transparent by design) and model-agnostic methods (applied externally to any model).
- Robustness and Performance: Concerns the safety and security of the system against vulnerabilities (data poisoning, model leaking) and adversarial attacks. It also encompasses the model's accuracy (generalization performance) and reproducibility.
Finally, the paper discusses the inherent trade-offs within Trustworthy AI, such as Explainability vs. Accuracy, Explainability vs. Fairness, Fairness vs. Robustness, and Privacy vs. Fairness. It stresses that there is no one-size-fits-all solution, and trade-off analysis should be contextual, aligning with the specific application's requirements. A toolkit or roadmap approach is suggested to help practitioners prioritize risks and identify appropriate tools and methods for risk mitigation and monitoring throughout the development and deployment phases. The conclusion underscores that achieving trustworthy AI requires integrating governance structures with real-time algorithm audits.