Sequential propagation of Transformer quotient states

Establish whether Transformer representations that causally control a quotient-class prediction at one position can be maintained and sequentially updated so that a donor quotient state propagates through subsequent input positions, or whether the relevant quotient information is distributed across multiple positions.

Background

The paper shows that replacing a class-mean subspace at a single Transformer position can transfer the current quotient-class readout. However, the same intervention usually does not transfer the donor class to later positions under a shared suffix. This distinguishes a representation that affects the current output from a state representation that is carried forward and updated recurrently.

The authors explicitly state that the experiment does not rule out a representation distributed across several positions and therefore leaves the sequential implementation of quotient tracking unresolved.

References

We therefore identify a causal contribution to the current quotient readout, while leaving its sequential implementation unresolved.

— Tracking States or Tracking Cosets? An Algebraic Account of Learned State Tracking  (2609.29951 - Zhang et al., 24 Sep 2026) in Appendix, Section \ref{app:transformer-geometry-propagation}