One-Layer Transformer

• One-layer Transformer is a shallow neural architecture featuring a single multi-head self-attention block complemented by a position-wise feed-forward network and optional normalization.
• It leverages precise mechanisms like scaled dot-product attention to achieve universal approximation, finite memorization, and in-context learning dynamics.
• The model is pivotal for analyzing resource-efficient adaptations and theoretical limitations, inspiring methods such as recurrent RingFormer for improved parameter efficiency.

A one-layer Transformer is a sequence model consisting of a single self-attention block (often multi-head), typically followed by a position-wise feed-forward network and optional normalization and residual connections. Despite its apparent simplicity compared to multi-layer deep stacks, the one-layer architecture offers a precise setting in which to analyze the expressive capacity, optimization dynamics, memory properties, and algorithmic limitations of the Transformer paradigm. Its study has produced universal approximation results, mechanistic insights into in-context learning, resource-efficient recurrent variants, and rigorous lower bounds that clarify the necessity of network depth. Below, key dimensions of the one-layer Transformer's theory and practice are surveyed with references to the primary research literature.

1. Formal Definition and Architectural Variants

A standard one-layer Transformer operates on a sequence of $n$ vector embeddings $X = (x_1, \ldots, x_n)$, each $x_i \in \mathbb{R}^d$, supplemented by position encodings $p_i$. The core computational block is multi-head attention, parameterized by $h$ parallel sets of projection matrices $W_j^Q, W_j^K, W_j^V \in \mathbb{R}^{d \times d}$ ($j = 1\ldots h$) and an output projection $W^O$:

1. Multi-Head Self-Attention:
   • $Q_j = X W_j^Q$ (queries), $K_j = X W_j^K$ (keys), $V_j = X W_j^V$ (values)
   • Scaled dot-product attention per head: $$\mathrm{softmax}\left(\frac{Q_j K_j^T}{\sqrt{d}}\right) V_j$$
   • Concatenate all head outputs, then project: $\mathrm{MHA}(X) = \mathrm{Concat}_j[\cdot] W^O$
2. Residual and Layer Norm (optional):
   • $Y = \mathrm{LayerNorm}(X + \mathrm{MHA}(X))$
3. Position-wise Feed-Forward Network:
   • $FFN(Y)_i = \mathrm{ReLU}(Y_i W_1 + b_1) W_2 + b_2$
4. Output:
   • Final output $Z = \mathrm{LayerNorm}(Y + FFN(Y))$

Several architectural modifications exist:

• Single-head vs multi-head attention (Kajitsuka et al., 2023)
• Inclusion/exclusion of position encodings (Gumaan, 11 Jul 2025)
• Layer-norm-free and feed-forward-free settings for mechanistic analysis (Mahankali et al., 2023)
• Recurrent/compressed parameterizations as in RingFormer (Heo et al., 18 Feb 2025)

2. Universal Approximation, Expressivity, and Memorization

Recent results formally demonstrate the universal approximation power of one-layer Transformers with sufficient width and head count. The single-layer structure, when equipped with softmax-attention and an adequately wide feed-forward, can approximate any continuous mapping $f: X \to Y$ on a compact subset of $\mathbb{R}^{n \times d}$ up to arbitrary precision (Gumaan, 11 Jul 2025). The core elements of this result include:

• Region encoding via heads: Each attention head can attend to a particular partition of the input space, selecting for regions in which the function $f$ is approximately constant.
• Attention sharpness: By scaling query-key products, softmax becomes arbitrarily close to a discrete selection, enabling the network to mimic lookup tables and memorize datasets (Kajitsuka et al., 2023).
• Permutation-equivariant universality: One-layer, single-head Transformers with two feed-forward networks are universal approximators for continuous permutation-equivariant functions on compact domains (Kajitsuka et al., 2023).
• Finite memorization: Arbitrary finite mappings—including lookup tables, sorting short lists, and function tables—can be implemented with one-layer models with $O(nN)$ parameters for $N$ samples of length $n$ (Kajitsuka et al., 2023, Gumaan, 11 Jul 2025).
• Experimental confirmation: Synthetic regression, function evaluation, and finite mapping tasks confirm the theory, with models able to fit functions such as $f(x_1,x_2) = \sin(x_1) + \sin(x_2)$ to high precision (Gumaan, 11 Jul 2025, Strobl et al., 28 Mar 2025).

3. Algorithmic and In-Context Learning Dynamics

In-context learning and algorithmic capabilities of one-layer Transformers have been examined for tasks ranging from nearest neighbor retrieval to online regression:

• One-nearest neighbor: When trained on prompts containing $N$ labeled pairs and one query, gradient descent leads the attention layer to concentrate weight on the nearest labeled example in the embedding space, effectively implementing the 1-NN rule (Li et al., 2024). Attention temperature and bias adjust dynamically to focus on the correct context entry.
• One-step gradient descent: With a linear self-attention mechanism (i.e., omitting softmax), one-layer Transformers trained on linear regression prompts represent exactly one step of gradient descent (GD) on least-squares, or a preconditioned version for anisotropic covariates (Mahankali et al., 2023).
• Bayes-optimal next-token prediction: In next-token prediction on specially formulated data, one-layer transformers (with ReLU or linear attention) can represent and reach Bayes-optimal predictors, with expected loss converging at a linear rate under normalized gradient descent, and generalization guarantees for unseen tokens (Nguyen et al., 21 May 2025).
• Contextual mapping: With low-rank weight matrices, even single-head, single-layer architectures can map sequences to unique representations suitable for context-dependent computation (Kajitsuka et al., 2023).
• Training dynamics: Under certain conditions (e.g., no positional encodings, long sequences, faster decoder learning), the self-attention weights evolve to increasingly focus on discriminative tokens, following a “scan and snap” dynamic (Tian et al., 2023).

4. Fundamental Limitations and Hardness Results

Despite their universality and memorization power, one-layer Transformers exhibit structural barriers for algorithmic and reasoning tasks:

• Induction heads task: For the "induction heads" copy-and-point-forward benchmark, a one-layer Transformer requires model size $\Omega(n)$ (number of heads $\times$ dimension $\times$ bits of precision) to succeed, an exponential inefficiency compared to the logarithmic resources needed by two-layer architectures; this lower bound leverages communication complexity reductions (Sanford et al., 2024).
• Function evaluation: A concise (polylog-parameter) one-layer Transformer can compute $f: [n]\to[n]$ at query $i$ only under carefully structured input encodings. If keys and values are distributed across unrelated positions (“permuted pairs”), a 1-layer model requires $\Omega(n\log n)$ bits, while two layers regain succinctness (Strobl et al., 28 Mar 2025).
• Reasoning and generalization: Provably, one-layer attention-only models can memorize but cannot perform in-context reasoning, pattern generalization, or compose chain-of-thought steps; depth is required to implement these higher-level algorithmic patterns (Chen et al., 2024).
• Structural impossibility: Template matching, multi-hop in-context QA, and “dependent” input pattern discriminations cannot be realized by a single layer—no matter how wide—due to preserved linear dependencies (Chen et al., 2024).

5. Parameter Efficiency, Training, and Structural Adaptations

One-layer Transformers have enabled new approaches to model compression, parameter sharing, and efficiency improvements:

• RingFormer: Replaces a stack of Transformer layers with recurrent reuse of a single parameter block, modulated at each unrolled iteration by lightweight, input-adaptive low-rank signals. Empirically, this architecture matches the accuracy of much deeper models in vision and translation at a fraction of the parameter cost (Heo et al., 18 Feb 2025).
• Low-rank and sparsity structure: Training with SGD on data with a small number of label-relevant patterns induces parameter updates of provably low rank, matched to the discriminative subspace of the data manifold. Furthermore, magnitude-based pruning of small output neurons after training has negligible effect on generalization (Li et al., 2024).
• Lottery ticket phenomenon: Randomly initialized one-layer Transformer networks with untrained weights can contain sparse subnetworks that match >90% of the BLEU score of a fully trained model, demonstrating a strong “lottery ticket” effect in the presence of multi-head attention and feed-forward nonlinearities (Shen et al., 2021).

Efficiency Approach	Main Mechanism	Empirical Findings
RingFormer	Recurrent single block + low-rank signals	5× param reduction, state-of-the-art BLEU/accuracy (Heo et al., 18 Feb 2025)
Low-rank adaptation	Updates span label patterns	Supports LoRA-style fine-tuning, theoretical bounds (Li et al., 2024)
Subnetwork supermasks	Sparsity via binary masking	Achieves 90–98% BLEU with 10–50% active weights (Shen et al., 2021)

6. Connections to Other Models, Theoretical Equivalences, and Open Questions

The one-layer Transformer links to classic models and raises further questions:

• RNN equivalence: A one-layer decoder-only Transformer without normalization and with a single head is mathematically equivalent to a two-layer RNN: the self-attention computation can be rewritten as two sequential recurrent state updates per token, and the position-wise feed-forward MLP as a stateless second "RNN" layer (Zhang et al., 2024).
• Certified robustness: This equivalence enables robust certification frameworks (e.g., ARC-Tran), supporting verification of classifier invariance to complex perturbations (Zhang et al., 2024).
• Role of positional encoding: The expressivity of one-layer models critically depends on positional signal injection; appropriate encoding enables even shallow models to perform complex lookups, whereas permutation-invariant models cannot realize certain position-dependent computations (Strobl et al., 28 Mar 2025, Gumaan, 11 Jul 2025).
• Compositionality and depth: While universal in the classical sense, one-layer Transformers lack the compositional algebraic step power of deeper stacks, limiting their practical effectiveness on reasoning and algorithmic tasks (Chen et al., 2024, Sanford et al., 2024).
• Open problems: Minimal architectures for chain-of-thought reasoning, expressivity of shallow networks with nonlinear feed-forward blocks, and full characterization of inference efficiency (not just capacity) remain active areas (Chen et al., 2024, Li et al., 2024).

7. Summary and Implications

The study of one-layer Transformers advances the foundational understanding of neural sequence models' approximation power, points to the outer limits of what can be achieved with shallow attention architectures, and reveals trade-offs between depth, parameter efficiency, and compositionality. While they can memorize and approximate any continuous mapping given sufficient capacity and careful input encoding, there exist hard algorithmic and scaling bottlenecks that only stacking multiple layers can overcome. Efficient adaptations such as RingFormer exploit this understanding to engineer parameter- and compute-efficient architectures for sequence modeling in NLP, vision, and beyond. Insights into algorithmic learning, in-context inference, and low-rank adaptation sharpen the theoretical and empirical design of lightweight but expressive attention models, while rigorous lower bounds and negative results clarify when depth is indispensable (Gumaan, 11 Jul 2025, Kajitsuka et al., 2023, Chen et al., 2024, Sanford et al., 2024, Heo et al., 18 Feb 2025, Li et al., 2024, Li et al., 2024, Mahankali et al., 2023, Strobl et al., 28 Mar 2025, Shen et al., 2021, Nguyen et al., 21 May 2025, Zhang et al., 2024).