---
title: Single-Layer Linear Transformer
url: https://www.emergentmind.com/topics/single-layer-linear-transformer
type: topic
---

# Single-Layer Linear Transformer

A single-layer linear transformer is a one-layer, single-head self-attention architecture in which the attention map is linear or bilinear rather than Softmax-normalized, and in which feed-forward nonlinearities are absent or analytically suppressed. In the canonical regression model studied as a shallow linearized transformer, the context matrix \(Z_0\in\mathbb{R}^{(d+1)\times(n+1)}\) is updated by
\[
\Delta Z := \frac1n\,P\,Z_0\,M\,(Z_0^\top Q Z_0),\qquad Z_1 = Z_0+\Delta Z,
\]
with learned \(P,Q\in\mathbb{R}^{(d+1)\times(d+1)}\), mask \(M=\mathrm{diag}(I_n,0)\), and scalar prediction \(\hat y(Z_0;P,Q)=-[Z_1]_{(d+1),(n+1)}\) [2310.01082]. Closely related one-layer formulations encode tokens \(v_i=(x_i,y_i)\in\mathbb{R}^{d+1}\) and show that such models can implement exactly one step of gradient descent or pre-conditioned gradient descent on least-squares regression, while a particular parameter construction makes a single forward pass recover the ordinary least-squares projection itself [2307.03576][2604.13656]. Across regression, graph learning, and neural operators, the single-layer linear transformer functions both as a minimal analytical model and as a practical architecture with linear-complexity attention mechanisms [2409.09007][2511.06294].

## 1. Canonical architectural forms

The core architectural simplification is the replacement of standard Transformer attention by a linear self-attention rule. In the regression formulation of "Linear attention is (maybe) all you need (to understand transformer optimization)," the model receives \(d\)-dimensional covariates \(x_1,\dots,x_n,x_{n+1}\in\mathbb{R}^d\) and constructs
\[
Z_0\in\mathbb{R}^{(d+1)\times(n+1)}
\]
by stacking the covariates and labels so that the bottom-right entry is zero and the last column corresponds to the held-out example. The attention affinity is a plain bilinear form,
\[
A := Z_0^\top Q Z_0,
\]
rather than \(\mathrm{Softmax}(QK^\top)\), and the hidden width is fixed at \(d+1\) [2310.01082].

In the in-context-learning formulation of "One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention," each example becomes a token
\[
v_i=\begin{pmatrix}x_i\\ y_i\end{pmatrix}\in\mathbb{R}^{d+1},
\]
with parameters \(W_K,W_Q,W_V\in\mathbb{R}^{s\times s}\) and \(h\in\mathbb{R}^s\), where \(s=d+1\). For a sequence \((v_1,\dots,v_T)\), the layer output is
\[
a_1=0,\qquad
a_i=\sum_{j=1}^{i-1}(W_Vv_j)\,(v_j^\top W_K^\top W_Q v_i)\quad (i>1),
\]
again using a bilinear score rather than a Softmax kernel [2307.03576].

In the OLS construction, the single-layer linear transformer is written in standard \(Q,K,V\) notation,
\[
\mathrm{Attention}(X)=\frac1n\,QK^\top V,\qquad
\mathrm{Output}(X)=\mathrm{Attention}(X)\,W_{FFN}\,W_p,
\]
with \(Q=XW_Q\), \(K=XW_K\), and \(V=XW_V\) [2604.13656].

| Formulation | Input representation | Core linear-attention rule |
|---|---|---|
| Regression testbed [2310.01082] | \(Z_0\in\mathbb{R}^{(d+1)\times(n+1)}\) with masked held-out example | \(\Delta Z=\frac1n PZ_0M(Z_0^\top QZ_0)\) |
| In-context regression [2307.03576] | tokens \(v_i=(x_i,y_i)\), query \(v_{n+1}=(x_{n+1},0)\) | \(a_i=\sum_{j<i}(W_Vv_j)(v_j^\top W_K^\top W_Q v_i)\) |
| OLS realization [2604.13656] | design matrix \(X\in\mathbb{R}^{n\times k}\) | \(\mathrm{Attention}(X)=\frac1n QK^\top V\) |

These formulations share several structural restrictions relative to standard Transformers: one head, no multi-head split, no feed-forward nonlinear layer, and no activation functions in the canonical regression models. This suggests that the term “single-layer linear transformer” refers less to one fixed implementation than to a family of one-layer self-attention systems whose expressive content is concentrated in bilinear attention and linear projections.

## 2. Data models, objectives, and masking conventions

The most studied training setting is synthetic regression. In the shallow linearized regression model, the task is to predict the scalar label
\[
y = w^\top x_{n+1}
\]
from the masked context \(Z_0\), under mean-squared error
\[
f(P,Q)=\mathbb{E}_{x_1,\dots,x_{n+1},w}\Big[(\hat y(Z_0;P,Q)-w^\top x_{n+1})^2\Big],
\]
where the expectation is over random draws of \(x_i\sim D_x\) and \(w\sim D_w\) [2310.01082].

The in-context-learning analysis uses a more explicit generative model. The covariates satisfy
\[
x_i\stackrel{i.i.d.}{\sim}N(0,\Sigma),\qquad i=1,\dots,n+1,
\]
the hidden vector satisfies
\[
w_{\rm true}\sim N(0,\Sigma^{-1}),
\]
and the responses are
\[
y_i = w_{\rm true}^\top x_i + \epsilon_i,\qquad \epsilon_i\sim N(0,\sigma^2).
\]
The final query token is masked by setting \(v_{n+1}=(x_{n+1},0)\), and pre-training minimizes
\[
L(W_K,W_Q,W_V,h)=\mathbb{E}\big[(\hat y_{n+1}-y_{n+1})^2\big]
\]
over the entire data-generation process [2307.03576].

A significant extension concerns the response model. If the responses instead satisfy \(y_i=f(x_i)+\epsilon_i\), with \(f\) drawn from a rotation-invariant, symmetric function family, the same one-layer linear self-attention architecture remains analyzable. The paper states that the global minimizer still implements the same un-preconditioned one-step gradient-descent rule on a least-squares linear regression objective. This is presented as evidence that, in this setting, the model cannot exploit anything beyond the second-moment structure of the responses [2307.03576].

Masking is therefore not incidental. In both canonical regression constructions, the held-out example is present as a query but its response is removed. The resulting architecture is simultaneously a masked predictor and an in-context learner, with the support examples furnishing the sufficient statistics that the bilinear attention operator can access.

## 3. Provable learned algorithms: one-step GD and pre-conditioned GD

The central theorem in the in-context-learning analysis is explicit. Under isotropic covariates \(x_i\sim N(0,I)\), noisy labels, and a one-layer, single-head linear self-attention transformer with no MLP, any global minimizer of the expected square loss implements exactly one step of gradient descent on least-squares regression [2307.03576]. The predictor takes the form
\[
\hat y_{n+1}
=\eta\sum_{i=1}^n y_i\,(x_i^\top x_{n+1}),
\]
where
\[
\eta=\frac{\mathbb{E}[w^\top X^\top]}{\mathbb{E}[\mathrm{Tr}(XX^\top)]}.
\]
Equivalently, the learned weight vector is
\[
w_1=\eta\sum_{i=1}^n y_i\,x_i,
\]
which is exactly one GD step from \(w_0=0\) for \(\min_w \frac12\|Xw-y\|^2\).

When the covariates are non-isotropic Gaussian, the learned update changes in a precise way. If \(x_i\sim N(0,\Sigma)\) and \(w_{\rm true}\sim N(0,\Sigma^{-1})\), the global minimizer implements one step of pre-conditioned GD with preconditioner \(\Sigma^{-1}\):
\[
\hat y_{n+1}
=\eta\sum_{i=1}^n y_i\,(\Sigma^{-1}x_i)^\top x_{n+1}.
\]
The paper interprets this as the transformer learning the same single-step update but whitening the inputs by \(\Sigma^{-1}\) [2307.03576].

The proof structure is also informative. First, conditional-expectation and ridge-regression identities reduce the loss to a condition on the effective linear map \(M_{:,1:d}^\top G w\). Second, rotational invariance and odd-moment cancellation reduce the Bayes-optimal target to a simpler quantity proportional to \(X^\top\). The theorem then pins down the learned rule as a GD-like update. This suggests that, in the one-layer linear regime, in-context learning can be characterized as exact algorithm selection by the pre-training optimum rather than merely by functional approximation.

A common misconception is that such algorithmic behavior requires deep transformers or nonlinear MLP blocks. The results above do not support that view. Within the Gaussian synthetic-regression setting, the one-layer linear self-attention architecture is already sufficient for exact one-step GD and pre-conditioned GD behavior [2307.03576].

## 4. Optimization dynamics as a minimal testbed

The optimization study in "Linear attention is (maybe) all you need (to understand transformer optimization)" treats the single-layer linear transformer as a controlled surrogate for Transformer training. The training protocol is deliberately simple: SGD with momentum \(\mu=0.9\), Adam with \(\beta_1=\beta_2=0.9\), learning rates chosen by small grid search, global gradient clipping at norm \(1\), no weight decay, no learning-rate warmup, batch size \(1\), and training for a few thousand iterations averaged over multiple seeds [2310.01082].

Within this regime, several phenomena previously associated with large NLP Transformers are reproduced.

- **Adam wins**: even for a single layer, Adam reaches low training loss \(\sim 10^{-3}\) much faster than SGD.
- **Heavy-tailed gradient noise**: at initialization, the stochastic gradient noise in each parameter block has very heavy tails, observed through QQ-plots against a Gaussian reference.
- **Robust condition number gap**: defining \(R_t=\lambda_{\max}(H_t)/\mathrm{median}\,\mathrm{diag}(H_t)\), the reported values satisfy \(R_t(\mathrm{SGD})\gg R_t(\mathrm{Adam})\).
- **Directional smoothness gap**: for \(\delta=x_{t+1}-x_t\), the directional smoothness measure \(\langle \delta,H_t\delta\rangle/\|\delta\|^2\) remains significantly lower for Adam than for SGD.
- **Generalized smoothness**: empirically, \(\|H_t\|\) grows linearly with \(\|\nabla f(x_t)\|\), consistent with an \((L_0,L_1)\)-smoothness law of the form
  \[
  \|H(x)\|\le L_0+L_1\|\nabla f(x)\|.
  \]

The controlled perturbation studies refine this picture. Making the input covariates more heavy-tailed through \(\Gamma\)-scaled norms exacerbates heavy-tailed gradient noise and slows both optimizers, yet the Adam/SGD gap remains. Increasing depth to \(L=2,4,6,8\) magnifies the same effects: heavier tails, a larger condition-number gap, and a wider speed gap between Adam and SGD [2310.01082].

The paper does not prove a closed-form convergence theorem. Instead, it advances a narrower claim: the shallow linear setting may be the minimal realistic abstraction needed for a rigorous theory of Transformer optimization. The significance of the model is therefore methodological as well as architectural. It isolates heavy-tailed stochastic updates, ill-conditioning, and smoothness violations in a low-dimensional testbed while preserving several empirical hallmarks of full Transformer training.

## 5. Exact least-squares realization and memory decomposition

A stronger algebraic statement appears in "Ordinary Least Squares is a Special Case of Transformer." Let \(X\in\mathbb{R}^{n\times k}\) have full column rank, let \(y\in\mathbb{R}^n\), and let
\[
\beta^*=(X^\top X)^{-1}X^\top y,\qquad \hat y=X\beta^*=X(X^\top X)^{-1}X^\top y.
\]
The paper constructs a single-layer linear transformer whose forward pass is exactly \(\hat y\) in one pass [2604.13656].

The construction begins from the empirical covariance
\[
\Sigma=X^\top X=VAV^\top,
\]
with orthogonal \(V\) and diagonal \(A=\mathrm{diag}(\lambda_1,\dots,\lambda_k)\), \(\lambda_i>0\). Define
\[
L=VA^{-1/2},
\]
so that \(LL^\top=\Sigma^{-1}\). Setting
\[
W_Q=W_K=W_V=L,\qquad W_{FFN}=I_k,\qquad W_p=P=\frac1n\,L^\top X^\top y
\]
yields
\[
Q=XL,\qquad K=XL,\qquad V=XL,
\]
and
\[
\mathrm{Output}(X)=\mathrm{Attention}(X)\,I\,P=X\beta^*=\hat y.
\]
The conclusion is exact: ordinary least squares is a special case of the single-layer linear transformer [2604.13656].

This construction is used to define a “slow vs. fast” memory mechanism. The matrix \(L\), equivalently \(\Sigma^{-1}\), is the slow memory because it is learned once from the training set and fixed at inference. The attention scores
\[
\frac1m\,(ZL)(L^\top Z^\top)
\]
for a new context \(Z\in\mathbb{R}^{m\times k}\) are the fast memory because they are formed dynamically per context. In the special case \(Z^\top Z=X^\top X\), the OLS-transformer predicts \(Z\beta^*\); otherwise the output is distorted by the factor \(\Sigma^{-1}E_z\), where \(E_z=Z^\top Z/m\) [2604.13656].

The same paper places the linear model in a Hopfield-network perspective. With a quadratic kernel \(f(q,k)=q\cdot k\), the associated energy is
\[
E(q;\{k_i\})=-\sum_i (q^\top k_i)^2.
\]
Replacing this with Softmax attention corresponds to the exponential energy
\[
E(q)=-\log\sum_i \exp(q^\top k_i).
\]
In that view, the passage from linear to standard Transformers increases associative-memory capacity from \(O(d)\) patterns to \(O(2^{d/2})\) patterns. A plausible implication is that the single-layer linear transformer serves as a quadratic-memory prototype from which the statistical role of full Softmax attention can be read off more transparently.

## 6. Specialized single-layer variants in graphs and neural operators

The single-layer linear transformer has also been specialized beyond tabular regression. In graph learning, SGFormer studies a hybrid propagation layer
\[
Z^{(k+1)}=(1-\alpha)P_A^{(k)}Z^{(k)}W^{(k)}+\alpha P_G Z^{(k)}W^{(k)}+\beta Z^{(k)}W^{(k)},
\]
and proves that a \(K\)-layer propagation stack can be reduced to a single layer
\[
Z^*=(1-\alpha)P_A^* Z^{(0)}W^*+\alpha P_G^* Z^{(0)}W^*
\]
such that, for any \(\epsilon>0\), \(\|Z^*-Z^{(K)}\|<\epsilon\) [2409.09007]. The graph attention mechanism normalizes \(Q\) and \(K\) by Frobenius norms, forms
\[
\overline C = I_N+\frac1N\tilde Q\tilde K^\top,
\qquad
C=\mathrm{diag}(\overline C\,1_N)^{-1}\overline C,
\]
and updates features by \(Z_{AN}=CV\). Crucially, the \(N\times N\) matrix \(C\) is never formed explicitly; the exact all-pair interaction is rewritten through matrix-vector products, giving total complexity \(O(Nd^2)\simeq O(N)\) for fixed \(d\), with no random features and no low-rank approximations. The reported empirical results include \(20\)–\(30\times\) faster training and inference than peer Transformers on medium-sized graphs, memory footprint below \(2\,\mathrm{GB}\), and \(66.0\%\) accuracy on ogbn-papers100M in \(\simeq 3.5\,\mathrm{h}\) on a single \(24\,\mathrm{GB}\) GPU [2409.09007].

In neural operators for PDEs, "Transolver is a Linear Transformer" shows that the Physics-Attention mechanism of Transolver is algebraically a special case of linear attention. Starting from slice, attention, and deslice operations, the paper introduces block matrices \(A,B,R\) and rewrites the output as
\[
O=\underbrace{\mathrm{Softmax}(Q)}_{\Phi}\,
\Bigl(\underbrace{\mathrm{Softmax}(K)}_{\Psi}\Bigr)^\top
V,
\]
with \(Q=v(X)A\), \(K=v(X)B\), and \(V=v(X)R\) [2511.06294]. This yields the canonical single-layer Linear Attention Neural Operator (LinearNO), which uses learned \(W_Q,W_K,W_V,W_O\), row-wise Softmax normalizations, key-value aggregation \(S=\tilde K^\top V\), and attention output \(O=\tilde Q S\). Its per-layer cost is \(O(Nr)\) when \(r\ll N\), with parameter count \(4d^2\). The paper reports state-of-the-art performance on six standard PDE benchmarks, an average reduction of \(40.0\%\) in the number of parameters, a \(36.2\%\) reduction in computational cost, and superior performance on AirfRANS and Shape-Net Car. In one Airfoil configuration with \(d=128\) and \(M=64\), TransSolver uses about \(2.81\,\mathrm{GB}\) of parameters and \(32.4\,\mathrm{GFLOPs}\), while LinearNO uses about \(1.77\,\mathrm{GB}\) and \(21.3\,\mathrm{GFLOPs}\), approximately a \(37\%\) reduction in both storage and compute [2511.06294].

Taken together, these specialized models show that the single-layer linear transformer is not only a stripped-down analytical surrogate. It is also a constructive design principle: one layer can be sufficient for exact least-squares projection, for provable one-step in-context optimization, for approximation-free linear-complexity graph attention, and for canonical linear-attention neural operators. This suggests that depth and Softmax are not prerequisites for every computational role commonly attributed to Transformers, although the cited works equally indicate that moving beyond the linear regime changes memory capacity, symmetry properties, and the attainable class of learned interactions.

Source: https://www.emergentmind.com/topics/single-layer-linear-transformer