Effectiveness of Large Conformal Prediction Sets Under Reduced Coverage

Determine whether larger conformal prediction sets—specifically Regularized Adaptive Prediction Sets (RAPS) used to advise human image labeling—remain effective at improving human decision-making when their coverage rate is reduced below the nominal guarantee (e.g., due to distribution shift or other factors).

Background

Conformal prediction sets provide distribution-free uncertainty quantification with specified marginal coverage guarantees and adaptively grow in size for difficult instances. In this paper, RAPS were evaluated against Top-k displays in AI-advised image labeling with in-distribution and out-of-distribution stimuli. The authors found that larger sets can increase cognitive load and diminish accuracy for in-distribution cases, while offering some advantages for hard out-of-distribution cases. However, the practical effectiveness of larger sets under conditions where coverage is reduced remains uncertain, motivating further evaluation of their utility when guarantees are disrupted.