Convergence near the constant-gate Bayes component

Determine whether gradient flow for the population logistic risk converges near the constant-gate Bayes component of the softmax-attention parameterization for LDA, and characterize its convergence rate when the Hessian has flat directions transverse to the local minimizer manifold.

Background

The paper establishes local exponential convergence near the regular zero-amplitude Bayes manifold and global convergence on a specially constructed invariant subspace in the symmetric balanced case. It does not provide an analogous convergence theorem near the constant-gate component, where perturbations of the gate direction can be compensated to first order by changes in the affine coefficient.

The appendix shows that this component fails to satisfy the Morse–Bott property because the Hessian kernel has additional transverse flat directions. The authors explicitly state that this degeneracy does not determine convergence or its rate, leaving both questions unresolved.

References

This degeneracy alone determines neither convergence nor its rate. The slower decay in Figure~\ref{fig:lda-local} is an empirical observation consistent with the previous discussion (see Appendix~\ref{appendix:LDA_experimental_details} for a description of the numerical protocol).

— Softmax Attention on Gaussian Mixtures: Linear When It Can, Selective When It Must  (2610.11798 - Gabet et al., 8 Oct 2026) in Section 3.4, paragraph “Why the constant-gate component is different”; Appendix, Section “Constant-gate geometry”

This does not prove divergence of gradient flow, but shows why the landscape alone cannot supply a boundedness argument.

— Softmax Attention on Gaussian Mixtures: Linear When It Can, Selective When It Must  (2610.11798 - Gabet et al., 8 Oct 2026) in Section 3.3, paragraph “Why this does not establish global convergence”; Appendix, Section “Why a benign landscape is not a global convergence theorem”