Dice Question Streamline Icon: https://streamlinehq.com

Quantify LL 144 audit volume and AEDT adoption baseline

Determine the baseline rate of adoption of automated employment decision tools (AEDTs) among New York City employers, including the subset of AEDTs that fall within the scope of Local Law 144, and quantify both the expected number of required bias audits and the number of audits publicly posted, given the current absence of a central repository and the lack of available adoption data.

Information Square Streamline Icon: https://streamlinehq.com

Background

Local Law 144 mandates that New York City employers using automated employment decision tools (AEDTs) conduct annual bias audits and post the results publicly. The paper’s survey of available audits found only a small number posted and emphasizes that, due to missing system-wide data and the absence of a central repository, it is not currently feasible to assess how many audits should exist or how many have been published.

This unresolved measurement problem limits empirical evaluation of the regime’s effectiveness and undermines transparency aims, as neither researchers nor the public can reliably assess audit coverage, compliance rates, or trends over time.

References

Surveys indicates that AEDTs of all varieties are thought to be widespread, but there is no available data about the baseline rate of adoption of AEDTs generally, let alone adoption of systems that fall within the scope of LL 144 [Indeed2023,Pew2023,selbst2019fairness]. Therefore, it is not possible to know precisely how many audits we should expect to find, and lacking a central repository, it is also impossible to know precisely how many audits are published.

Auditing Work: Exploring the New York City algorithmic bias audit regime (2402.08101 - Groves et al., 12 Feb 2024) in Subsection 'Public availability of audit reports' in Section 'Background of Local Law 144'