Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Auditing Work: Exploring the New York City algorithmic bias audit regime (2402.08101v1)

Published 12 Feb 2024 in cs.CY

Abstract: In July 2023, New York City (NYC) initiated the first algorithm auditing system for commercial machine-learning systems. Local Law 144 (LL 144) mandates NYC-based employers using automated employment decision-making tools (AEDTs) in hiring to undergo annual bias audits conducted by an independent auditor. This paper examines lessons from LL 144 for other national algorithm auditing attempts. Through qualitative interviews with 16 experts and practitioners within the regime, we find that LL 144 has not effectively established an auditing regime. The law fails to clearly define key aspects, such as AEDTs and independent auditors, leading auditors, AEDT vendors, and companies using AEDTs to define the law's practical implementation in ways that failed to protect job applicants. Contributing factors include the law's flawed transparency-driven theory of change, industry lobbying narrowing the definition of AEDTs, practical and cultural challenges faced by auditors in accessing data, and wide disagreement over what constitutes a legitimate auditor, resulting in four distinct 'auditor roles.' We conclude with four recommendations for policymakers seeking to create similar bias auditing regimes, emphasizing clearer definitions, metrics, and increased accountability. By exploring LL 144 through the lens of auditors, our paper advances the evidence base around audit as an accountability mechanism, providing guidance for policymakers seeking to create similar regimes.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (49)
  1. [n. d.]. IAAA - International Algorithmic Auditors Association. ([n. d.]). https://iaaa-algorithmicauditors.org/
  2. 2024. Code of Federal Regulations. 29 CFR Part 1607 - General Principles (2024). https://www.wired.com/story/opinion-new-york-citys-surveillance-battle-offers-national-lessons/
  3. Hiring by algorithm: predicting and preventing disparate impact. Available at SSRN (2016).
  4. Solon Barocas and Andrew D Selbst. 2016. Big data’s disparate impact. California law review (2016), 671–732.
  5. Customer-driven misconduct: How competition corrupts business practices. Management Science 59, 8 (2013), 1725–1742.
  6. AI Assurance? https://www.adalovelaceinstitute.org/report/risks-ai-systems/
  7. Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. PMLR, 77–91.
  8. Albert Fox Cahn. 2021. New York City’s Surveillance Battle Offers National Lessons. Wired (2021). https://www.wired.com/story/opinion-new-york-citys-surveillance-battle-offers-national-lessons/
  9. Pew Research Center. 2023. AI in Hiring and Evaluation of Workers: What People Think. (2023). https://www.pewresearch.org/internet/2023/04/20/ai-in-hiring-and-evaluating-workers-what-americans-think/
  10. Kathy Charmaz. 2023. Constructing grounded theory: A practical guide through qualitative analysis. sage.
  11. Understanding accountability in algorithmic supply chains. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency. 1186–1197.
  12. European Commission. 2024. The Digital Services Act package — Shaping Europe’s digital future. (2024). https://digital-strategy.ec.europa.eu/en/policies/digital-services-act-package
  13. Equal Employment Opportunity Commission. 2007. Employment Tests and Selection Procedures. (2007). https://www.eeoc.gov/laws/guidance/employment-tests-and-selection-procedures
  14. Equal Employment Opportunity Commission. 2022. Definitions of Race and Ethnicity Categories. Data Collection (2022).
  15. Equal Employment Opportunity Commission. 2023. Assessing Adverse Impact in Software, Algorithms, and Artificial Intelligence Used in Employment Selection Procedures Under Title VII of the Civil Rights Act of 1964. (2023). https://www.eeoc.gov/laws/guidance/select-issues-assessing-adverse-impact-software-algorithms-and-artificial
  16. Australian Competition and Consumer Commission. 2020. Trivago misled consumers about hotel room rates. (2020). https://www.accc.gov.au/media-release/trivago-misled-consumers-about-hotel-room-rates
  17. US Congress. 2002. The Sarbanes Oxley Act. (2002).
  18. New York City Council. 2021. A Local Law to amend the administrative code of the city of New York, in relation to automated employment decision tools. (2021). https://legistar.council.nyc.gov/LegislationDetail.aspx?ID=4344524&GUID=B051915D-A9AC-451E-81F8-6596032FA3F9&Options=Advanced&Search=
  19. Donald R Deis Jr and Gary A Giroux. 1992. Determinants of audit quality in the public sector. Accounting review (1992), 462–479.
  20. Certifying and removing disparate impact. In proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining. 259–268.
  21. Marissa Gerchick and Brooke Watson. 2023. Tracking Automated Employment Decision Tool Bias Audits. (2023). https://github.com/aclu-national/tracking-ll144-bias-audits
  22. Ellen P Goodman and Julia Trehu. 2022. Algorithmic Auditing: Chasing AI Accountability. Santa Clara High Tech. LJ 39 (2022), 289.
  23. Algorithmic impact assessment: a case study in healthcare. (2022). https://www.adalovelaceinstitute.org/wp-content/uploads/2022/02/Algorithmic-impact-assessment-a-case-study-in-healthcare.pdf
  24. Indeed. 2023. The Indeed Global AI Survey: Your Guide to the Future of Hiring. (2023). https://www.indeed.com/lead/the-indeed-ai-report?hl=en#form
  25. Irina Ivanova. 2020. New York City wants to restrict artificial intelligence in hiring. CBS News (2020). https://www.cbsnews.com/news/new-york-city-artificial-intelligence-hiring-restriction/
  26. Keeping an eye on AI. Ada Lovelace Institute (2023). https://www.adalovelaceinstitute.org/wp-content/uploads/2023/09/ALI_Keeping-an-eye-on-AI-2023.pdf
  27. Lauren Kirchner. 2023. New York City Moves to Create Accountability for Algorithms. ProPublica (2023). https://www.propublica.org/article/new-york-city-moves-to-create-accountability-for-algorithms
  28. Towards algorithm auditing: a survey on managing legal, ethical and technological risks of AI, ML and associated algorithms. (2021).
  29. David Leslie. 2019. Understanding artificial intelligence ethics and safety. arXiv preprint arXiv:1906.05684 (2019).
  30. Steve Lohr. 2023. A Hiring Law Blazes a Path for A.I. Regulation. The New York Times (2023). https://www.nytimes.com/2023/05/25/technology/ai-hiring-law-new-york.html
  31. New York City Department of Consumer and Worker Protections. 2023a. Automated Employment Decision Tools: Frequently Asked Questions. (2023). https://www.nyc.gov/assets/dca/downloads/pdf/about/DCWP-AEDT-FAQ.pdf
  32. New York City Department of Consumer and Worker Protections. 2023b. Automated Employment Decision Tools (Updated). (2023). https://rules.cityofnewyork.us/rule/automated-employment-decision-tools-updated/
  33. Colorado Division of Insurance. 2023. Regulation 10-1-1 Governance and Risk Management Framework Requirements for Life Insurers’ Use of External Consumer Data and Information Sources, Algorithms, and Predictive Models. (2023). https://drive.google.com/file/d/1dlPKJCDo76iHfJZDopQEhTDCmKbuYnNI/view?usp=embed_facebook
  34. US Department of Justice. 2015. 9-48.000 - Computer Fraud and Abuse Act. (2015). https://www.justice.gov/jm/jm-9-48000-computer-fraud
  35. Information Commissioner’s Office. 2020. Guidance on the AI auditing framework Draft guidance for consultation. Information Commissioner’s Office. (2020). https://ico.org.uk/media/2617219/guidance-on-the-ai-auditing-framework-draft-for-consultation.pdf
  36. UK Parliament. 2022. Online Safety Act 2023. (2022). https://bills.parliament.uk/bills/3137
  37. Jules Pattison-Gordon. 2023. Colorado Aims to Prevent AI-Driven Discrimination in Insurance. GovTech (2023). https://www.govtech.com/policy/colorado-aims-to-prevent-ai-driven-discrimination-in-insurance
  38. PCAOB. [n. d.]. Driving improvement in audit quality to protect investors. ([n. d.]). https://pcaobus.org/
  39. Evani Radiya-Dixit and Gina Neff. 2023. A Sociotechnical Audit: Assessing Police Use of Facial Recognition. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency. 1334–1346.
  40. Mitigating bias in algorithmic hiring: Evaluating claims and practices. In Proceedings of the 2020 conference on fairness, accountability, and transparency. 469–481.
  41. Inioluwa Deborah Raji and Joy Buolamwini. 2019. Actionable auditing: Investigating the impact of publicly naming biased performance results of commercial ai products. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. 429–435.
  42. Outsider oversight: Designing a third party audit ecosystem for ai governance. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society. 557–571.
  43. Algemene Rekenkamer. 2022. An Audit of 9 Algorithms used by the Dutch Government - Report - Netherlands Court of Audit. (2022). https://english.rekenkamer.nl/publications/reports/2022/05/18/an-audit-of-9-algorithms-used-by-the-dutch-government
  44. Yvette D. [D-NY-9] Rep. Clarke. 2022. Algorithmic Accountability Act of 2022. (2022). https://www.congress.gov/bill/117th-congress/house-bill/6580/text
  45. Andrew D Selbst. 2017. Disparate impact in big data policing. Ga. L. Rev. 52 (2017), 109.
  46. Fairness and abstraction in sociotechnical systems. In Proceedings of the conference on fairness, accountability, and transparency. 59–68.
  47. SHRM. 2022. Automation and AI in HR. (2022). https://advocacy.shrm.org/SHRM-2022-Automation-AI-Research.pdf?_ga=2.112869508.1029738808.1666019592-61357574.1655121608
  48. Nopmanee Tepalagul and Ling Lin. 2015. Auditor independence and audit quality: A literature review. Journal of Accounting, Auditing & Finance 30, 1 (2015), 101–121.
  49. The four-fifths rule is not disparate impact: a woeful tale of epistemic trespassing in algorithmic fairness. arXiv preprint arXiv:2202.09519 (2022).
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Lara Groves (2 papers)
  2. Jacob Metcalf (5 papers)
  3. Alayna Kennedy (1 paper)
  4. Briana Vecchione (7 papers)
  5. Andrew Strait (5 papers)
Citations (10)
View on Semantic Scholar

Summary

An Evaluation of NYC's Algorithmic Bias Audit Regime: Insights from Local Law 144

The paper, "Auditing Work: Exploring the New York City algorithmic bias audit regime," addresses the early implementation challenges and insights concerning New York City's Local Law 144 (LL 144). This law represents a pioneering effort to regulate algorithmic biases within automated employment decision-making tools (AEDTs) used by employers. Although LL 144's intent to introduce accountability through mandatory third-party audits is clear, practical issues in its execution reveal significant areas for improvement.

Core Findings and Challenges

LL 144 mandates that NYC employers using AEDTs conduct annual bias audits through independent auditors, aimed at identifying discriminatory impacts related to race and gender. However, the research identifies that a significant portion of stakeholders criticize the law's inefficacy due to ambiguous definitions and practical hurdles. Key among these concerns is the vague definition of what constitutes an AEDT and "independent auditor," which leaves substantial room for interpretation by companies and auditors. This vagueness perpetuates a lack of accountability, as it becomes challenging to enforce compliance effectively.

The research synthesizes interview insights from 16 practitioners within the LL 144 framework, revealing four major inadequacies. Firstly, the transparency-driven theory underpinning the law does not prevent employers from using biased AEDTs, primarily due to a low level of legislative commitment to actionable outcomes following an audit. Secondly, industry-driven narrowing of AEDT definitions has excluded numerous tools from the law’s purview, further reducing its efficacy. Thirdly, auditors face considerable challenges in data access from tool vendors and employers, which hampers the auditing process. Finally, the debate over what defines a legitimate auditor remains unresolved, with varying perceptions about the auditor's scope and independence.

Implications of Findings

The insights have significant implications for the evolving discourse on algorithmic transparency and accountability. Policymakers looking to emulate LL 144 need to provide rigorous definitions and criteria to prevent loopholes and ensure a comprehensive scope that encompasses a more extensive range of AEDTs. Moreover, compelling evidence suggests that the mere provision of an audit cannot substitute for genuine accountability mechanisms that include mandatory corrective actions for non-compliance.

On a theoretical front, the introduction of algorithm audits has engendered discussions on the systemic responsibility of algorithm developers and users alike. Who should bear the onus for ensuring fairness—developers who create these systems, or companies that deploy them? This debate urges a reevaluation of current practices to include holistic oversight that transcends jurisdictional constraints.

Speculation on Future Developments

Moving forward, the development of robust infrastructural frameworks that accommodate clearer guidelines, standards of audit practice, and roles for auditors, will likely be pivotal in shaping the efficacy of algorithm audit regimes. Auditing regimes may benefit from adopting models similar to those in traditional fields, such as financial auditing, to engender consistency and integrity in audit practices.

Policymakers are encouraged to view algorithmic auditing standards through a lens of distributed responsibility, urging collaborations where developers, vendors, and employers collectively uphold ethical standards. In the future, algorithmic auditing might expand to include a broader array of metrics beyond race and gender, as multifaceted as those found in diverse, global contexts.

Conclusion

The initial experiences with LL 144 underscore the need for evolving strategies to govern algorithmic bias auditing. While its introduction signifies a landmark in AI regulation, its practical shortfalls offer critical lessons in delineating clearer parameters and building robust accountability frameworks. By cultivating a nuanced understanding of the social dynamics surrounding AEDTs and incorporating a broader spectrum of responsible AI practices, more effective and equitable governance models can emerge.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown