Resolve whether middle-layer token movements are linear

Determine whether the small token-specific movements across the unresolved middle layers of Pythia-410m are generated by approximately linear transformations, as suggested by the measurable final layer, by using an instrument with sufficiently fine resolution.

Background

Most individual transitions in the middle of both investigated transformers cannot be resolved with the 4,000-token samples because token-specific movements are smaller than the sampling gap. The paper can assess some accumulated multi-layer blocks, but not the cost of most middle layers individually.

The authors hypothesize that these small movements may be linear, noting that the final layer of Pythia-410m is consistent with linear behavior at the available resolution. The conclusion explicitly identifies this as an unresolved question requiring finer measurement resolution.