Existence and learnability of a fixed-width transformer construction for linear spike-and-slab selection

Construct, for fixed dimension \(p\), a transformer-based selection network for linear spike-and-slab regression whose width is independent of sample size \(n\), whose depth is polylogarithmic in \(n\), and whose computation represents the required least-squares statistics and the \(-\tfrac12\log n\) model-complexity penalty; determine whether training can find such weights.

Background

For linear spike-and-slab models with a g-prior, the posterior inclusion probability can be expressed through summary statistics such as n−1X⊤Xn^{-1}X^\top X, n−1X⊤yn^{-1}X^\top y, n−1y⊤yn^{-1}y^\top y, and nn. The paper argues that row attention could compute these quantities and that active-variable log-odds grow at the signal-information scale, whereas null-variable log-odds include a −12log⁡n-\tfrac12\log n penalty.

The authors conjecture that a transformer construction with width of order 2p2^p, independent of nn, and depth polylogarithmic in nn may exist. They explicitly leave unresolved whether optimization or training can discover the required weights.

References

We conjecture that such a construction exists with width independent of $n$ (of order $2p$) and depth polylogarithmic in $n$, building on in-context least squares \citep{bai2023transformers}. Whether training finds such weights is open.

— SelectPFN: Amortized Bayesian Variable Selection for Sparse Nonlinear Regression with Prior-Data Fitted Networks  (2610.21950 - Luo, 15 Oct 2026) in Appendix, Section “Interpretation and open problems,” paragraph “Conjecture (linear spike-and-slab, fixed p)”