Determine whether the weight-space displacement dips immediately after injection

Determine whether the weight-space displacement between an injected GPT-2 training run and its seed-matched twin decreases during the trough around training step 210, immediately after the single-row data injection at step 200.

Background

The study measures the effect of replacing one row of a 256-row batch with a 194-token passage at step 200 during GPT-2 pre-training. Per-step loss differences are available at every step and show a pronounced trough near step 210, followed by a rise toward a peak around steps 280–300 and gradual decay thereafter.

Weight-space measurements were collected only at selected checkpoints, including steps 199, 249, and 299. Because no weight-space checkpoint was retained at the trough, the paper cannot determine whether the model’s parameter displacement from its seed-matched twin also decreases at that time. The authors explicitly leave this behavior unresolved rather than attributing the loss trough to a corresponding change in displacement.

References

Therefore, the only available conclusion across the injection is 199 to 249, and we cannot say whether the displacement itself dips there. We record the trough as unresolved rather than explained.

— Learned, Then Lost: A Measured Single-Example Counterfactual in Pre-training  (2608.19168 - Speck et al., 19 Aug 2026) in Section 5, “Mapping the Disturbance Throughout Pre-training,” Interpretation paragraph