Dice Question Streamline Icon: https://streamlinehq.com

Assess enforceability of protective measures for on‑site auditors without external legal structures

Determine whether, in the absence of external legal structures, AI developers can effectively incentivize on‑site external auditors to adhere to protective measures at levels comparable to those applicable to employees within secure research environments.

Information Square Streamline Icon: https://streamlinehq.com

Background

The paper proposes secure research environments as a way to provide auditors with full white-box access while mitigating leakage risks, drawing analogies from other industries that use on-site inspections. However, the authors caution that organizational levers used for employees (e.g., internal policies and enforcement) may not straightforwardly apply to external auditors.

They explicitly flag uncertainty about whether labs—without external legal frameworks—can achieve the same level of adherence to protective measures from auditors as they do from employees, highlighting a governance and enforcement question that affects the feasibility of on-site white-box audits.

References

However, it is unclear whether, absent external legal structures, labs could incentivize adherence to protective measures to the same extent as with employees.

Black-Box Access is Insufficient for Rigorous AI Audits (2401.14446 - Casper et al., 25 Jan 2024) in Section 6, Methods to Address Security Risks – Physical: Secure research environments