Finite-sample validity of the prompt-clustered Gaussian multiplier audit

Establish finite-sample rejection-rate guarantees for the prompt-clustered Gaussian multiplier audit with fixed numbers of prompts, dependent sibling answers, and a finite number of multiplier draws, beyond its pointwise asymptotic size result.

Background

The paper develops a prompt-clustered kernel audit whose asymptotic validity is established when prompts are independent sampling units, while answers within a prompt may be dependent. The theoretical result concerns pointwise asymptotic size and assumes that the number of multiplier draws grows for the exact asymptotic statement.

The controlled experiments examine finite prompt counts and 1,999 multiplier draws, but the authors explicitly characterize the resulting finite-sample rejection rates as unresolved empirically. This leaves open the derivation of finite-sample validity guarantees for practical audit budgets and dependence structures.

References

With fixed $J$ and $1{,}999$ multiplier draws, finite-sample rejection rates remain an empirical question.

— Are You Sure Your LLM Is Sure? Repairing and Auditing Confidence  (2610.22139 - Dey et al., 15 Oct 2026) in Appendix, Section “Controlled residual audits: what each test can detect” (label app:controlled-audit)