Allocate reasoning tokens between knowledge retrieval and instruction following

Determine how reasoning tokens are allocated between knowledge retrieval and instruction following in large language models, and whether this allocation varies across model architectures.

Background

ListQA separates factual knowledge retrieval from instruction-following performance through distinct evaluation metrics. The paper observes that models differ substantially in how they divide generation between reasoning tokens and final output tokens, while greater reasoning expenditure does not necessarily produce proportionally larger accuracy gains.

The authors explicitly identify the unresolved issue of whether reasoning tokens serve knowledge retrieval, instruction following, or both, and whether different model architectures allocate those tokens differently. Resolving this issue could clarify why inference-time reasoning improves some capabilities more than others on list-formatted factual retrieval tasks.