Sign of the Mean Utility Change for Huginn on CommonsenseQA

Determine the sign of the mean reference-utility change for Huginn at the 33→34 recurrent transition on the complete CommonsenseQA validation split.

Background

The paper extends its step-contraction analysis to CommonsenseQA and selects Huginn’s 33→34 transition on a training subset. When the transition is evaluated on the complete validation split, the available result does not resolve whether the mean reference utility increases or decreases.

Consequently, the direction of aggregate progress for this model–task transition remains empirically undetermined and would require further evaluation or analysis.

References

Huginn's $33\to34$ transition was selected on its training subset; evaluation on the complete validation split leaves the sign of the mean gain unresolved.

— Right Direction, Wrong Step: Geometric Analysis of Finite-Step Failure in Looped Transformers  (2609.16665 - Guo et al., 15 Sep 2026) in Appendix, Section Additional Experimental Details, subsection Task extensions and effect margins