Mechanisms underlying performance degradation with increasing instruction count

Investigate and characterize the internal mechanisms in large language models that cause instruction-following performance to degrade as the number of simultaneous instructions increases, including analysis of attention patterns and internal model representations to identify failure modes responsible for this degradation.

Background

The paper documents a robust empirical trend: even when individual instructions are simple and programmatically verifiable, models’ ability to satisfy all instructions simultaneously deteriorates as instruction count rises. Despite thorough measurement and modeling, the underlying causes of this degradation remain unexplained.

The authors call for deeper analysis of model internals—such as attention patterns and activation dynamics—to understand how and why multiple concurrent instructions lead to failures, citing related work on attention steering and activation-based interventions as potential avenues for uncovering these mechanisms.