Formal Algorithms for Transformers

• Formal Algorithms for Transformers are mathematically defined procedures outlining token embedding, multi-head attention, residual connections, and gradient-based training.
• They enable in-context algorithm emulation, where fixed-weight architectures use prompt design to replicate classical algorithms without feed-forward layers.
• Rigorous analysis of these methods reveals their expressivity, computational complexity, and certified robustness, guiding both theory and practical implementations.

A formal algorithm for a transformer specifies, at a mathematical level, the precise sequence of computations defining both its forward (inference) and backward (training) passes, and characterizes the functional and expressive properties of the architecture. These formalizations are foundational for analyzing, implementing, and extending transformer models in a principled and reproducible manner. Over the past several years, the transformer architecture has become the canonical deep sequence model, and its formal algorithms and theoretical underpinnings have been rigorously developed across several subfields, including deep learning optimization, in-context algorithm emulation, computational complexity, formal language theory, and verification.

1. Formal Algorithms for Transformer Forward and Backward Passes

The formal algorithm for a (standard) transformer encoder or decoder consists of a repeated stack of identical layers, each executing a precise mathematical sequence (Phuong et al., 2022):

• Input Embedding: Each token $x[t]$ is mapped to an embedding $W_e[:,x[t]]$, combined with a learned or fixed positional embedding $W_p[:,t]$, yielding input $E[:,t] = W_e[:,x[t]] + W_p[:,t]$.
• Multi-Head Attention: For each head $h$, the multi-head attention module computes projections $Q^h = W_q^h X + b_q^h$, $K^h = W_k^h X + b_k^h$, $V^h = W_v^h X + b_v^h$, and synthesizes the output as

$$\mathrm{Attention}(Q, K, V; M) = V \cdot \mathrm{softmax}\left(\frac{K^\top Q}{\sqrt{d_k}} + m\right),$$

where $M$ is an optional mask.

• Residual and LayerNorm: Residual connections and learned scale/shift per coordinate; for each position,

$W_e[:,x[t]]$0

for mean $W_e[:,x[t]]$1 and variance $W_e[:,x[t]]$2 computed over the features.

• Feed-Forward Sublayer: Position-wise MLP, $W_e[:,x[t]]$3.
• Autoregressive Decoding (decoder-only): Causal masking $W_e[:,x[t]]$4 iff $W_e[:,x[t]]$5 ensures next-token predictions depend only on past tokens.

Training Procedures

• Loss Function: Negative log-likelihood; in the autoregressive setting,

$W_e[:,x[t]]$6

• Gradient-Based Optimization: Parameter updates via gradient descent or Adam, with explicit update formulas carrying all stateful parameters through the optimization loop.

This mathematical decomposition provides unambiguous, fully specified algorithms for both forward inference and backward (gradient-based) training (Phuong et al., 2022).

2. Formal Algorithmic Emulation in Transformers

Recent work establishes that even minimal fixed-weight (frozen) transformer architectures, with suitable in-context prompts, can emulate a broad class of classical algorithms (Hu et al., 24 Aug 2025). Specifically:

• Two-Layer Attention Emulation: For any algorithm implementable by a single-head attention module, a prompt can be constructed that encodes the algorithm's $W_e[:,x[t]]$7 as part of the token embeddings. A two-layer softmax-attention stack achieves

$W_e[:,x[t]]$8

for any required precision $W_e[:,x[t]]$9.

• Single-Head Minimality: Even a single-head softmax-attention layer can, with longer prompts, carry out entire algorithm families (e.g., one-step gradient descent, least-squares, ridge regression).
• Constructive Prompt Design: Prompts are designed to create sharp dot-product gaps, making softmax function as a near-hardmax selector, precisely encoding the algorithm's logic.
• No Feed-Forward Layers Required: These constructions require only attention; all adaptation happens through the prompt sequence, not parameter updates.

This establishes a direct, constructive link between in-context learning and algorithmic emulation—transformers are, in a precise sense, prompt-programmable (Hu et al., 24 Aug 2025).

3. Expressivity and Complexity of Formal Transformer Algorithms

The computational capabilities and limitations of formal transformer algorithms can be mapped to classical complexity classes and logic (Strobl et al., 2023, Yang et al., 2024, Hao et al., 2022):

Attention/Model	Expressivity Class	Precise Bound
Hard (UHAT/GUHAT)	AC$W_p[:,t]$0	Constant-depth, polynomial-size circuits
Averaging hard (AHAT)	TC$W_p[:,t]$1 / Beyond AC$W_p[:,t]$2	Recognizes Majority, Dyck-1
Softmax (finite-depth)	TC$W_p[:,t]$3	Uniform threshold circuits
Encoder–decoder (unbounded CoT)	RE/P (if poly steps)	Turing machine (general recursion/polytime)

• Kₜ[#] and C-RASP: All counting temporal logic formulas (including Dyck-1, $W_p[:,t]$4) can be compiled into standard masked softmax transformer encoders, giving a tight lower bound on their expressivity in the infinite-input regime (Yang et al., 2024).
• Formal Language Limits: Star-free/FO[<] languages correspond to hard-attention encoders without position encodings; softmax or sinusoidal PE (modular) extend this power to FO+MOD and beyond (Strobl et al., 2023).
• Practical Complexity Ceiling: Infinite-width transformers in the lazy/rich regimes provably capture only Efficient Polynomial-Time Heuristic Schemes (EPTHS) up to $W_p[:,t]$5 or $W_p[:,t]$6, limiting learnable algorithms to low-degree tasks (e.g., sorting, copy, simple graph algorithms) (Davidovich et al., 11 Mar 2026).

4. Formal Algorithms for In-Context Machine Learning

Transformers can implement classical statistical learning algorithms via in-context processing, with explicit attention-based constructions (Bai et al., 2023):

• Least Squares, Ridge, Lasso, GLMs: Multi-layer transformers can realize in-context gradient descent, proximal steps, and convex ERM, achieving minimax-optimal learning rates given sufficient context length.
• Algorithm Selection: Mechanisms such as post-ICL validation and pre-ICL testing are formally encoded as multi-layer attention+MLP modules; a single transformer can select the best algorithm from a library based on prompt structure.
• Provable Generalization and Statistical Bounds: Generalization guarantees and error bounds are established via chaining and Rademacher complexity, fully explicit in network parameters and architecture.

Thus, formal algorithms extend beyond symbolic tasks to encompass practical in-context statistical learning, conditioned on the transformer’s formal inductive bias and constructed attention patterns (Bai et al., 2023).

5. Transformers as Formal Bayesian Networks

A recent foundational result demonstrates that every sigmoid-transformer layer with arbitrary weights implements weighted loopy belief propagation on a corresponding factor graph, where a single layer executes one round of BP (Coppola, 17 Mar 2026):

• Attention as AND / FFN as OR: Layer alternation mirrors Pearl’s gather-update (belief propagation behind graphical models)—attention gathers “logical AND,” feed-forward computes “OR” (log-sum-exp).
• Exact BP on Trees: A constructive mapping yields exact posterior inference, with transformer weights directly interpreting log-potentials and message-passing rules.
• Uniqueness: Only BP-aligned weights in sigmoid-attention architectures yield exact posteriors; no alternative parameterizations suffice.

This identifies the standard transformer layer as a formal message-passing algorithm, theoretically unifying neural attention with classical probabilistic graphical models (Coppola, 17 Mar 2026).

6. Robustness- and Verification-Oriented Formal Transformer Algorithms

Transformer algorithms yield formal procedures not just for modeling, but also for certifiable robustness. Precise algorithmic relaxations—provably sound—exist for forward and backward bound propagation in the presence of nonlinear self-attention layers. These methods derive tight, tractable certified lower bounds for adversarial robustness that are substantially superior to interval-bound propagation, by formally handling cross-variable nonlinearity and cross-position dependency introduced by attention (Shi et al., 2020).

7. Implementation and Engineering Principles

Formal transformer algorithms are architecture- and library-agnostic blueprints. They underpin:

• Batching and Masking: Explicit accounting for variable-length sequences, masked attention, and paddings in all computations.
• Mixed-Precision and Numerical Stability: Layer-norm and Adam step-sizes ($W_p[:,t]$7) are treated mathematically for stable computation in all formal pseudocode.
• Forward-Pass/Backward-Pass Separation: A single forward-pass routine, invoked under autodiff, suffices for modern frameworks, in line with the chain-rule decomposition (Phuong et al., 2022).
• Inference Acceleration: Caching of past keys and values (“attention caching”) is captured as a formal algorithmic optimization for efficient autoregressive decoding.

These engineering notes are not ad hoc: they follow strictly from the mathematically formal algorithms and guarantee correct functional equivalence (Phuong et al., 2022).

---

References:

• (Phuong et al., 2022): Formal Algorithms for Transformers
• (Hu et al., 24 Aug 2025): In-Context Algorithm Emulation in Fixed-Weight Transformers
• (Davidovich et al., 11 Mar 2026): Algorithmic Capture, Computational Complexity, and Inductive Bias of Infinite Transformers
• (Yang et al., 2024): Counting Like Transformers: Compiling Temporal Counting Logic Into Softmax Transformers
• (Hao et al., 2022): Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexity
• (Strobl et al., 2023): What Formal Languages Can Transformers Express? A Survey
• (Bai et al., 2023): Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection
• (Coppola, 17 Mar 2026): Transformers are Bayesian Networks
• (Shi et al., 2020): Robustness Verification for Transformers