Precise delayed-recovery sample complexity for higher-order untied activations

Determine the precise weak-recovery sample complexity for untied attention with higher-order activation functions, including the increasingly slow escape from the uninformative state observed numerically.

Background

The untied-attention analysis establishes a dichotomy on the Θ(d² log d) scale when the fast mean dynamics either breaks or preserves the relevant symmetry. Numerical experiments with Hermite-type higher-order activations h₂ and h₃ show increasingly slow escape from the uninformative state.

The paper does not determine the exact scaling of this delayed recovery. This unresolved question is a concrete instance of the broader need to analyze longer timescales and information-exponent effects.

References

The linear activation h_1(x)=x develops overlap rapidly, whereas the higher-order activations h_2(x)=\frac{x2-1}{\sqrt{2} and h_3(x)=\frac{x3-3x}{\sqrt{6} exhibit increasingly slow escape from the uninformative state, whose precise sample complexity is left for future work.

— High-Dimensional Learning Dynamics of Attention-Indexed Models  (2609.03858 - Xu et al., 3 Sep 2026) in Figure caption, Figure \ref{fig:SGD_untied}, Section “Weak recovery and its sample complexity”