Optimal Audit-Rate Selection

Determine the optimal rate at which would-be shipped interventions should be randomized into audit experiments to balance drift detection, statistical information, traffic cost, and the validity of eval-based shipping guarantees.

Background

After deployment, interventions routed directly to shipping do not generate online outcome labels, so drift in the ship region cannot be detected without randomized audits. The paper proposes auditing a constant fraction of would-be ships and shows that any positive audit rate identifies the shipped-region outcome distribution under the working model.

The appropriate audit rate is not derived. A higher rate produces audited outcomes more quickly and improves monitoring power, whereas a lower rate reduces the traffic and experimentation cost. The paper’s simulations illustrate this trade-off but do not solve the value-of-information problem or provide a generally optimal rate.