Test whether instability tracks correctness

Investigate whether semantic response instability tracks answer correctness by evaluating a harder or more varied set of verifiable questions that produces variation in correctness across trials.

Background

All four verifiable questions in the study were answered correctly in every one of the 30 trials, so the verifiable group had no correctness variance. Consequently, the planned analysis of whether instability covaries with correctness could not be performed.

A more difficult or heterogeneous set of verifiable tasks could generate incorrect as well as correct responses, allowing the relationship between semantic instability and correctness to be tested directly.

References

The verifiable-question group showed no variance in correctness, which meant the planned check of whether instability tracks correctness could not be run; a harder or more varied set of verifiable questions would be needed to test this.

— Self-Referential Induction Increases Response Instability Relative to Unresolvable and Verifiable Questions in Large Language Models  (2608.13258 - Balani et al., 13 Aug 2026) in Section 5, Limitations