Extension beyond Gaussian supervised inputs to natural self-supervised settings

Extend the high-dimensional analysis of empirical risk minimization for single-head tied attention from Gaussian input embeddings with supervised targets to natural structured inputs learned in a self-supervised manner, determining whether the derived generalization and spectral predictions continue to hold.

Background

The current analysis assumes Gaussian inputs and a supervised, attention-indexed target, which enable tractable characterization of generalization and spectral properties.

The authors explicitly state that extending their framework to natural data and self-supervised learning remains open, which would test the robustness and applicability of their predictions to realistic training regimes.