Sample-size variance rate for PR-AUC permutation estimates

Determine a corresponding sample-size variance rate for PR-AUC permutation estimates, particularly for highly imbalanced output heads, analogous to the established O(1/n) variance rate for ROC-AUC under the stated row-wise inference conditions.

Background

The paper proves an O(1/n) variance bound for ROC-AUC permutation estimates when the evaluation dataset and model state are fixed, positive and negative sample counts are both proportional to the total sample size, and inference is row-wise. This result relies on ROC-AUC’s pairwise-comparison representation.

The authors note that the same argument does not extend to PR-AUC, especially when output heads are highly imbalanced. They evaluate PR-AUC variance empirically, but explicitly state that the experiments do not establish a corresponding theoretical sample-size rate. Establishing such a rate would clarify how evaluation-set size controls the precision of PR-AUC-based feature-importance estimates in production recommender systems.

References

However, the $O(1/n)$ rate in Proposition~\ref{prop:auc-permutation-variance} uses AUC's pairwise-comparison representation and does not establish a corresponding sample-size rate for PR-AUC, particularly for highly imbalanced heads.

— FORTE: A Production Framework for Feature Evaluation and Deprecation in Industrial Recommender Systems  (2610.13806 - Arun et al., 9 Oct 2026) in Section 3.5, subsection “Empirical bridge to PR-AUC”