Single-Layer Linear Transformer

• Single-Layer Linear Transformers are streamlined self-attention models that replace Softmax with bilinear, linear attention without feed-forward nonlinearities.
• They mimic one-step gradient descent and pre-conditioned updates to implement exact least-squares projections, facilitating in-context learning.
• Their design achieves lower computational cost and memory footprint in applications like regression, graph learning, and neural operators while maintaining core transformer properties.

A single-layer linear transformer is a one-layer, single-head self-attention architecture in which the attention map is linear or bilinear rather than Softmax-normalized, and in which feed-forward nonlinearities are absent or analytically suppressed. In the canonical regression model studied as a shallow linearized transformer, the context matrix $Z_0\in\mathbb{R}^{(d+1)\times(n+1)}$ is updated by

$$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$

with learned $P,Q\in\mathbb{R}^{(d+1)\times(d+1)}$, mask $M=\mathrm{diag}(I_n,0)$, and scalar prediction $\hat y(Z_0;P,Q)=-[Z_1]_{(d+1),(n+1)}$ (Ahn et al., 2023). Closely related one-layer formulations encode tokens $v_i=(x_i,y_i)\in\mathbb{R}^{d+1}$ and show that such models can implement exactly one step of gradient descent or pre-conditioned gradient descent on least-squares regression, while a particular parameter construction makes a single forward pass recover the ordinary least-squares projection itself (Mahankali et al., 2023, Tan et al., 15 Apr 2026). Across regression, graph learning, and neural operators, the single-layer linear transformer functions both as a minimal analytical model and as a practical architecture with linear-complexity attention mechanisms (Wu et al., 2024, Hu et al., 9 Nov 2025).

1. Canonical architectural forms

The core architectural simplification is the replacement of standard Transformer attention by a linear self-attention rule. In the regression formulation of "Linear attention is (maybe) all you need (to understand transformer optimization)," the model receives $d$-dimensional covariates $x_1,\dots,x_n,x_{n+1}\in\mathbb{R}^d$ and constructs

$Z_0\in\mathbb{R}^{(d+1)\times(n+1)}$

by stacking the covariates and labels so that the bottom-right entry is zero and the last column corresponds to the held-out example. The attention affinity is a plain bilinear form,

$A := Z_0^\top Q Z_0,$

rather than $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$0, and the hidden width is fixed at $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$1 (Ahn et al., 2023).

In the in-context-learning formulation of "One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention," each example becomes a token

$$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$2

with parameters $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$3 and $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$4, where $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$5. For a sequence $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$6, the layer output is

$$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$7

again using a bilinear score rather than a Softmax kernel (Mahankali et al., 2023).

In the OLS construction, the single-layer linear transformer is written in standard $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$8 notation,

$$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$9

with $P,Q\in\mathbb{R}^{(d+1)\times(d+1)}$0, $P,Q\in\mathbb{R}^{(d+1)\times(d+1)}$1, and $P,Q\in\mathbb{R}^{(d+1)\times(d+1)}$2 (Tan et al., 15 Apr 2026).

Formulation	Input representation	Core linear-attention rule
Regression testbed (Ahn et al., 2023)	$P,Q\in\mathbb{R}^{(d+1)\times(d+1)}$3 with masked held-out example	$P,Q\in\mathbb{R}^{(d+1)\times(d+1)}$4
In-context regression (Mahankali et al., 2023)	tokens $P,Q\in\mathbb{R}^{(d+1)\times(d+1)}$5, query $P,Q\in\mathbb{R}^{(d+1)\times(d+1)}$6	$P,Q\in\mathbb{R}^{(d+1)\times(d+1)}$7
OLS realization (Tan et al., 15 Apr 2026)	design matrix $P,Q\in\mathbb{R}^{(d+1)\times(d+1)}$8	$P,Q\in\mathbb{R}^{(d+1)\times(d+1)}$9

These formulations share several structural restrictions relative to standard Transformers: one head, no multi-head split, no feed-forward nonlinear layer, and no activation functions in the canonical regression models. This suggests that the term “single-layer linear transformer” refers less to one fixed implementation than to a family of one-layer self-attention systems whose expressive content is concentrated in bilinear attention and linear projections.

2. Data models, objectives, and masking conventions

The most studied training setting is synthetic regression. In the shallow linearized regression model, the task is to predict the scalar label

$M=\mathrm{diag}(I_n,0)$0

from the masked context $M=\mathrm{diag}(I_n,0)$1, under mean-squared error

$M=\mathrm{diag}(I_n,0)$2

where the expectation is over random draws of $M=\mathrm{diag}(I_n,0)$3 and $M=\mathrm{diag}(I_n,0)$4 (Ahn et al., 2023).

The in-context-learning analysis uses a more explicit generative model. The covariates satisfy

$M=\mathrm{diag}(I_n,0)$5

the hidden vector satisfies

$M=\mathrm{diag}(I_n,0)$6

and the responses are

$M=\mathrm{diag}(I_n,0)$7

The final query token is masked by setting $M=\mathrm{diag}(I_n,0)$8, and pre-training minimizes

$M=\mathrm{diag}(I_n,0)$9

over the entire data-generation process (Mahankali et al., 2023).

A significant extension concerns the response model. If the responses instead satisfy $\hat y(Z_0;P,Q)=-[Z_1]_{(d+1),(n+1)}$0, with $\hat y(Z_0;P,Q)=-[Z_1]_{(d+1),(n+1)}$1 drawn from a rotation-invariant, symmetric function family, the same one-layer linear self-attention architecture remains analyzable. The paper states that the global minimizer still implements the same un-preconditioned one-step gradient-descent rule on a least-squares linear regression objective. This is presented as evidence that, in this setting, the model cannot exploit anything beyond the second-moment structure of the responses (Mahankali et al., 2023).

Masking is therefore not incidental. In both canonical regression constructions, the held-out example is present as a query but its response is removed. The resulting architecture is simultaneously a masked predictor and an in-context learner, with the support examples furnishing the sufficient statistics that the bilinear attention operator can access.

3. Provable learned algorithms: one-step GD and pre-conditioned GD

The central theorem in the in-context-learning analysis is explicit. Under isotropic covariates $\hat y(Z_0;P,Q)=-[Z_1]_{(d+1),(n+1)}$2, noisy labels, and a one-layer, single-head linear self-attention transformer with no MLP, any global minimizer of the expected square loss implements exactly one step of gradient descent on least-squares regression (Mahankali et al., 2023). The predictor takes the form

$\hat y(Z_0;P,Q)=-[Z_1]_{(d+1),(n+1)}$3

where

$\hat y(Z_0;P,Q)=-[Z_1]_{(d+1),(n+1)}$4

Equivalently, the learned weight vector is

$\hat y(Z_0;P,Q)=-[Z_1]_{(d+1),(n+1)}$5

which is exactly one GD step from $\hat y(Z_0;P,Q)=-[Z_1]_{(d+1),(n+1)}$6 for $\hat y(Z_0;P,Q)=-[Z_1]_{(d+1),(n+1)}$7.

When the covariates are non-isotropic Gaussian, the learned update changes in a precise way. If $\hat y(Z_0;P,Q)=-[Z_1]_{(d+1),(n+1)}$8 and $\hat y(Z_0;P,Q)=-[Z_1]_{(d+1),(n+1)}$9, the global minimizer implements one step of pre-conditioned GD with preconditioner $v_i=(x_i,y_i)\in\mathbb{R}^{d+1}$0: $v_i=(x_i,y_i)\in\mathbb{R}^{d+1}$1 The paper interprets this as the transformer learning the same single-step update but whitening the inputs by $v_i=(x_i,y_i)\in\mathbb{R}^{d+1}$2 (Mahankali et al., 2023).

The proof structure is also informative. First, conditional-expectation and ridge-regression identities reduce the loss to a condition on the effective linear map $v_i=(x_i,y_i)\in\mathbb{R}^{d+1}$3. Second, rotational invariance and odd-moment cancellation reduce the Bayes-optimal target to a simpler quantity proportional to $v_i=(x_i,y_i)\in\mathbb{R}^{d+1}$4. The theorem then pins down the learned rule as a GD-like update. This suggests that, in the one-layer linear regime, in-context learning can be characterized as exact algorithm selection by the pre-training optimum rather than merely by functional approximation.

A common misconception is that such algorithmic behavior requires deep transformers or nonlinear MLP blocks. The results above do not support that view. Within the Gaussian synthetic-regression setting, the one-layer linear self-attention architecture is already sufficient for exact one-step GD and pre-conditioned GD behavior (Mahankali et al., 2023).

4. Optimization dynamics as a minimal testbed

The optimization study in "Linear attention is (maybe) all you need (to understand transformer optimization)" treats the single-layer linear transformer as a controlled surrogate for Transformer training. The training protocol is deliberately simple: SGD with momentum $v_i=(x_i,y_i)\in\mathbb{R}^{d+1}$5, Adam with $v_i=(x_i,y_i)\in\mathbb{R}^{d+1}$6, learning rates chosen by small grid search, global gradient clipping at norm $v_i=(x_i,y_i)\in\mathbb{R}^{d+1}$7, no weight decay, no learning-rate warmup, batch size $v_i=(x_i,y_i)\in\mathbb{R}^{d+1}$8, and training for a few thousand iterations averaged over multiple seeds (Ahn et al., 2023).

Within this regime, several phenomena previously associated with large NLP Transformers are reproduced.

• Adam wins: even for a single layer, Adam reaches low training loss $v_i=(x_i,y_i)\in\mathbb{R}^{d+1}$9 much faster than SGD.
• Heavy-tailed gradient noise: at initialization, the stochastic gradient noise in each parameter block has very heavy tails, observed through QQ-plots against a Gaussian reference.
• Robust condition number gap: defining $d$0, the reported values satisfy $d$1.
• Directional smoothness gap: for $d$2, the directional smoothness measure $d$3 remains significantly lower for Adam than for SGD.
• Generalized smoothness: empirically, $d$4 grows linearly with $d$5, consistent with an $d$6-smoothness law of the form

$d$7

The controlled perturbation studies refine this picture. Making the input covariates more heavy-tailed through $d$8-scaled norms exacerbates heavy-tailed gradient noise and slows both optimizers, yet the Adam/SGD gap remains. Increasing depth to $d$9 magnifies the same effects: heavier tails, a larger condition-number gap, and a wider speed gap between Adam and SGD (Ahn et al., 2023).

The paper does not prove a closed-form convergence theorem. Instead, it advances a narrower claim: the shallow linear setting may be the minimal realistic abstraction needed for a rigorous theory of Transformer optimization. The significance of the model is therefore methodological as well as architectural. It isolates heavy-tailed stochastic updates, ill-conditioning, and smoothness violations in a low-dimensional testbed while preserving several empirical hallmarks of full Transformer training.

5. Exact least-squares realization and memory decomposition

A stronger algebraic statement appears in "Ordinary Least Squares is a Special Case of Transformer." Let $x_1,\dots,x_n,x_{n+1}\in\mathbb{R}^d$0 have full column rank, let $x_1,\dots,x_n,x_{n+1}\in\mathbb{R}^d$1, and let

$x_1,\dots,x_n,x_{n+1}\in\mathbb{R}^d$2

The paper constructs a single-layer linear transformer whose forward pass is exactly $x_1,\dots,x_n,x_{n+1}\in\mathbb{R}^d$3 in one pass (Tan et al., 15 Apr 2026).

The construction begins from the empirical covariance

$x_1,\dots,x_n,x_{n+1}\in\mathbb{R}^d$4

with orthogonal $x_1,\dots,x_n,x_{n+1}\in\mathbb{R}^d$5 and diagonal $x_1,\dots,x_n,x_{n+1}\in\mathbb{R}^d$6, $x_1,\dots,x_n,x_{n+1}\in\mathbb{R}^d$7. Define

$x_1,\dots,x_n,x_{n+1}\in\mathbb{R}^d$8

so that $x_1,\dots,x_n,x_{n+1}\in\mathbb{R}^d$9. Setting

$Z_0\in\mathbb{R}^{(d+1)\times(n+1)}$0

yields

$Z_0\in\mathbb{R}^{(d+1)\times(n+1)}$1

and

$Z_0\in\mathbb{R}^{(d+1)\times(n+1)}$2

The conclusion is exact: ordinary least squares is a special case of the single-layer linear transformer (Tan et al., 15 Apr 2026).

This construction is used to define a “slow vs. fast” memory mechanism. The matrix $Z_0\in\mathbb{R}^{(d+1)\times(n+1)}$3, equivalently $Z_0\in\mathbb{R}^{(d+1)\times(n+1)}$4, is the slow memory because it is learned once from the training set and fixed at inference. The attention scores

$Z_0\in\mathbb{R}^{(d+1)\times(n+1)}$5

for a new context $Z_0\in\mathbb{R}^{(d+1)\times(n+1)}$6 are the fast memory because they are formed dynamically per context. In the special case $Z_0\in\mathbb{R}^{(d+1)\times(n+1)}$7, the OLS-transformer predicts $Z_0\in\mathbb{R}^{(d+1)\times(n+1)}$8; otherwise the output is distorted by the factor $Z_0\in\mathbb{R}^{(d+1)\times(n+1)}$9, where $A := Z_0^\top Q Z_0,$0 (Tan et al., 15 Apr 2026).

The same paper places the linear model in a Hopfield-network perspective. With a quadratic kernel $A := Z_0^\top Q Z_0,$1, the associated energy is

$A := Z_0^\top Q Z_0,$2

Replacing this with Softmax attention corresponds to the exponential energy

$A := Z_0^\top Q Z_0,$3

In that view, the passage from linear to standard Transformers increases associative-memory capacity from $A := Z_0^\top Q Z_0,$4 patterns to $A := Z_0^\top Q Z_0,$5 patterns. A plausible implication is that the single-layer linear transformer serves as a quadratic-memory prototype from which the statistical role of full Softmax attention can be read off more transparently.

6. Specialized single-layer variants in graphs and neural operators

The single-layer linear transformer has also been specialized beyond tabular regression. In graph learning, SGFormer studies a hybrid propagation layer

$A := Z_0^\top Q Z_0,$6

and proves that a $A := Z_0^\top Q Z_0,$7-layer propagation stack can be reduced to a single layer

$A := Z_0^\top Q Z_0,$8

such that, for any $A := Z_0^\top Q Z_0,$9, $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$00 (Wu et al., 2024). The graph attention mechanism normalizes $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$01 and $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$02 by Frobenius norms, forms

$$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$03

and updates features by $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$04. Crucially, the $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$05 matrix $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$06 is never formed explicitly; the exact all-pair interaction is rewritten through matrix-vector products, giving total complexity $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$07 for fixed $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$08, with no random features and no low-rank approximations. The reported empirical results include $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$09–$$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$10 faster training and inference than peer Transformers on medium-sized graphs, memory footprint below $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$11, and $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$12 accuracy on ogbn-papers100M in $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$13 on a single $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$14 GPU (Wu et al., 2024).

In neural operators for PDEs, "Transolver is a Linear Transformer" shows that the Physics-Attention mechanism of Transolver is algebraically a special case of linear attention. Starting from slice, attention, and deslice operations, the paper introduces block matrices $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$15 and rewrites the output as

$$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$16

with $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$17, $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$18, and $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$19 (Hu et al., 9 Nov 2025). This yields the canonical single-layer Linear Attention Neural Operator (LinearNO), which uses learned $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$20, row-wise Softmax normalizations, key-value aggregation $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$21, and attention output $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$22. Its per-layer cost is $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$23 when $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$24, with parameter count $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$25. The paper reports state-of-the-art performance on six standard PDE benchmarks, an average reduction of $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$26 in the number of parameters, a $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$27 reduction in computational cost, and superior performance on AirfRANS and Shape-Net Car. In one Airfoil configuration with $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$28 and $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$29, TransSolver uses about $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$30 of parameters and $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$31, while LinearNO uses about $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$32 and $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$33, approximately a $$\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,$$34 reduction in both storage and compute (Hu et al., 9 Nov 2025).

Taken together, these specialized models show that the single-layer linear transformer is not only a stripped-down analytical surrogate. It is also a constructive design principle: one layer can be sufficient for exact least-squares projection, for provable one-step in-context optimization, for approximation-free linear-complexity graph attention, and for canonical linear-attention neural operators. This suggests that depth and Softmax are not prerequisites for every computational role commonly attributed to Transformers, although the cited works equally indicate that moving beyond the linear regime changes memory capacity, symmetry properties, and the attainable class of learned interactions.