Test the consumption-point mechanism independently of depth

Determine whether transfer fragility tracks the sharpness of the consuming attention readout rather than network depth by varying attention sharpness and depth independently through grouped-query-attention group size and attention temperature.

Background

The paper attributes the contrast between cache and activation sharing to their consumption points: caches are read through normalized, competitive softmax attention, whereas activations enter through an additive pre-softmax residual pathway. Existing experiments show that activations transfer through the same depth at which caches fail, but the architectural variables associated with attention sharpness and depth are not independently controlled.

The authors therefore identify an unresolved empirical question: whether the observed fragility is genuinely caused by the sharpness of the consuming readout instead of by depth or correlated architectural factors.

References

Our account also predicts that fragility tracks the sharpness of the consuming readout and not depth, which existing evidence confounds, so varying attention sharpness and depth independently, through GQA group size and attention temperature, would test the consumption-point mechanism.

— Augment the Cache, Replace the Activation: A 2x2 Study of Internal-State Sharing  (2610.22026 - Deng et al., 15 Oct 2026) in Section 5, “Further work”