Answer-level superiority over unchanged and output-based policies

Determine whether native readout veto policies consistently outperform unchanged model behavior for Qwen2.5-7B-Instruct and output-based veto policies for Gemma-2-9B-it in answer-level F1 on the official SQuAD 2.0 development set, under the fixed-candidate evaluation protocol.

Background

The paper compares native-trained and empty-trained answerability readouts while keeping each LLM’s generated answer candidate fixed. A veto policy either retains that candidate or replaces it with abstention according to a calibrated answerability score. Native readout policies clearly outperform empty readout policies in the reported F1 comparisons, but the broader baseline comparisons are less decisive: Qwen’s advantage over unchanged behavior does not meet the prespecified practical criterion after accounting for unfinished generations, and Gemma’s advantage over the selected output-based veto has an interval crossing zero. Consequently, the paper does not resolve whether native readouts provide a reliable answer-level advantage over these alternative policies.