Determine whether newly induced abstentions are selectively beneficial

Determine whether the abstentions induced by adding an unanswerable demonstration selectively remove weak answers rather than valid answers in extractive radiology question answering.

Background

The unanswerable demonstration caused many additional abstentions on answerable questions. Those withdrawn answers had somewhat lower prior F1 than answers that were retained, but the confidence interval was compatible with no selectivity.

The authors therefore cannot determine whether the demonstration preferentially removes low-quality answers or whether it also withdraws valid answers. This matters because the intervention reduced false answers while increasing over-abstention.

References

The CI of the difference includes zero, although it cannot exclude modest selectivity.

— Dissociating Answerability from Extraction in LLM Question Answering over Radiology Reports  (2610.17576 - Sinha et al., 13 Oct 2026) in Section 4.3, paragraph “The unanswerable demonstration is not detectably selective.”