Determine the precision dependence of the token-readout shift

Determine whether the 6.13-point adversarial F1 shift caused by replacing the two-token readout with the eight-token readout persists at higher numerical precision than 4-bit NF4 quantization.

Background

The primary 9,000-question audit uses a single LLaVA-1.5-7B checkpoint under 4-bit NF4 quantization. The authors find that the readout choice alone changes adversarial F1 from 0.7608 to 0.8221, but they do not establish whether the magnitude of this effect is stable under full-precision or other higher-precision inference.

Resolving this question would clarify whether the observed token-extraction confound is primarily a consequence of the evaluation protocol or whether its quantitative impact also depends materially on numerical precision.

References

A few caveats bound the result. The main 9{,}000-question audit uses one checkpoint at one quantization level (4-bit NF4), so we cannot say whether the 6.13-point shift is the same at higher precision.

— Token-Set Choice Confounds POPE: A Systematic Audit of Yes/No Extraction in VLM Hallucination Evaluation  (2610.25964 - Jayakumar et al., 19 Oct 2026) in Section 7, Limitations and Conclusion