Analyze ICL beyond single-index targets

Analyze in-context learning for nonlinear target families beyond single-index targets whose teacher function satisfies the nonzero linear-component condition \(\mu_1\neq0\).

Background

The paper restricts the task family to noisy single-index targets of the form σ⋆(a⊤x/d)\sigma_\star(a^\top x/\sqrt d) and assumes μ1=E[σ⋆(G)G]≠0\mu_1=\mathbb E[\sigma_\star(G)G]\neq0, which places the problem in an O(d2)\mathcal O(d^2) learning regime.

The authors explicitly identify extending the analysis to target classes outside this single-index setting as unresolved.

References

On the theoretical side, important directions include proving the finite-dimensional results and analyzing ICL beyond single-index targets with \mu_1\neq0.

— In-context Learning of Single-index Targets: Comparing Kernel and Feature Learners  (2610.01712 - Gu et al., 1 Oct 2026) in Section “Conclusion and Future Work”