Combining AI auditing tools into coherent workflows

Determine how to integrate heterogeneous and often context-specific AI auditing tools into a coherent, end-to-end AI auditing workflow so that tools created for specific application contexts and use cases can be effectively combined along the auditing process.

Background

The paper surveys existing AI auditing tools and observes that, despite a growing ecosystem, many tools are isolated, tailored to specific contexts, and not easily composable. This fragmentation hinders the development of continuous AI auditing practices that require integrated workflows.

Within the discussion of tools, the authors explicitly note that it is unclear how to combine disparate tools along an AI auditing workflow. AuditMAI is proposed partly to address this gap by outlining knowledge, process, and architecture views to guide integration.

References

While these attempts establish a useful basis by categorizing single tools, there are remaining challenges: i) tools is a fuzzy term in AI auditing, not necessarily describing executable software but also guidelines , ii) isolated approaches since often tools are created for specific application contexts and use cases, it is unclear how to combine them along an AI auditing workflow , and iii) context-dependence , e.g., the World Privacy Forum has identified that many popular fairness auditing tools implement US-specific legislation (US Four-Fifths Employment Rule for AI Fairness) , not suitable for the rest of the world.

— AuditMAI: Towards An Infrastructure for Continuous AI Auditing  (2406.14243 - Waltersdorfer et al., 2024) in Section 2.1, Background: Common Discussion Themes in AI Auditability — Tools