AI Procurement Checklists: Revisiting Implementation in the Age of AI Governance (2404.14660v1)

Abstract: Public sector use of AI has been quietly on the rise for the past decade, but only recently have efforts to regulate it entered the cultural zeitgeist. While simple to articulate, promoting ethical and effective roll outs of AI systems in government is a notoriously elusive task. On the one hand there are hard-to-address pitfalls associated with AI-based tools, including concerns about bias towards marginalized communities, safety, and gameability. On the other, there is pressure not to make it too difficult to adopt AI, especially in the public sector which typically has fewer resources than the private sector$\unicode{x2014}$conserving scarce government resources is often the draw of using AI-based tools in the first place. These tensions create a real risk that procedures built to ensure marginalized groups are not hurt by government use of AI will, in practice, be performative and ineffective. To inform the latest wave of regulatory efforts in the United States, we look to jurisdictions with mature regulations around government AI use. We report on lessons learned by officials in Brazil, Singapore and Canada, who have collectively implemented risk categories, disclosure requirements and assessments into the way they procure AI tools. In particular, we investigate two implemented checklists: the Canadian Directive on Automated Decision-Making (CDADM) and the World Economic Forum's AI Procurement in a Box (WEF). We detail three key pitfalls around expertise, risk frameworks and transparency, that can decrease the efficacy of regulations aimed at government AI use and suggest avenues for improvement.

AI Procurement Checklists: Striking Balance in Governance for Public Sector AI Systems

Overview and Context

The paper discusses the rising use of artificially intelligent systems within public sectors and the accompanying need for structured governance frameworks to manage AI responsibly. It emphasizes the role of AI procurement checklists in balancing technical requirements and ethical considerations amidst regulatory, resource, and expertise constraints. Using case studies from Canada, Brazil, and Singapore, this paper addresses the importance of refining these checklists to ensure they are effectively guiding the procurement processes and safeguarding against potential harms, without stifling innovation.

Effective AI Procurement Needs Technical Expertise

Key insights from the deployment of procurement tools like the Canadian Directive on Automated Decision-Making and the World Economic Forum’s AI Procurement in a Box reflect the necessity for technical expertise in AI system evaluations and deployments. The paper argues that while procurement processes are designed to align with strategic goals, the unique technical, ethical, and operational challenges posed by AI require specialist knowledge. The paper reports significant gaps in available expertise and suggests:

• Enhanced training: Boosting the AI-savviness of governmental procurement personnel.
• Expert consultations: Leveraging external AI expertise to supplement internal capabilities.

Identification of Loopholes in Procurement Processes

Several legal and procedural loopholes currently undercut the effectiveness of procurement checklists. The authors detail examples where low-cost or in-house developed AI systems avoid scrutiny under existing frameworks due to these oversights:

• Value thresholds: AI systems below a certain cost avoid rigorous procurement evaluations.
• Non-AI initial procurements: Projects not initially defined as AI but later incorporating AI elements bypass established AI-specific checks.
• Design in-house: AI systems created within a government entity do not undergo the same rigorous procurement process as externally sourced systems.

The Essential Role of Transparency

Transparency remains a linchpin in the deployment of ethical and effective AI systems in the public sector. The paper stresses the need for both substantive and procedural transparency where:

• Substantive: Full disclosure of AI system design, including data sources, model architecture, and performance metrics to the public wherever possible to allow external scrutiny and enhancement.
• Procedural: Ensuring all AI procurement processes are consistently applied and visible to prevent discretionary practices that may circumvent formal guidelines.

Speculating on Future Developments

Looking forward, the paper suggests a potential move towards more standardized global frameworks for AI procurement, akin to international standards seen in other technical fields. There is an expressed need for collaboration across governmental, private, and academic sectors to establish common standards for AI audits and transparency. Moreover, the evolving nature of AI technology and its implications necessitates continuous updates to procurement guidelines and the inclusion of AI education at all levels of governmental operations.

By addressing these areas, the authors advocate for a balanced approach to regulating AI in public sectors - one that promotes innovation and efficiency while ensuring equity, transparency, and accountability.