Log-Precision Transformers

• Log-precision transformers are transformer models that employ O(log n)-bit internal computations, enabling simulation by constant-depth uniform threshold circuits.
• They rigorously model operations like attention, feed-forward processing, and positional encoding within a fixed precision regime that aligns with first-order logic with majority quantifiers.
• Despite their scalability and high parallelism, these architectures face inherent limits, such as inability to solve P-complete problems, illustrating a tradeoff between parallelism and computational power.

Log-precision transformers are transformer architectures studied under an asymptotic arithmetic model in which the precision of internal numerical representations grows as $O(\log n)$ with the input length $n$. In this regime, every similarity score, positional encoding, feed-forward computation, and intermediate vector is represented with $O(\log n)$ bits and computed in $O(\log n)$ space. The model is important because it admits an exact complexity-theoretic characterization: it can be simulated by logspace-uniform constant-depth threshold circuits, and it is equivalently expressible in first-order logic with majority quantification, yielding a precise account of both its capabilities and its limitations (Merrill et al., 2022, Merrill et al., 2022).

1. Formal model and precision regime

A standard definition begins with the notion of a $p$-precision function. A function

$f : \{0,1\}^* \to \{0,1\}^p$

is $p$-precision if all inputs and outputs have bit-length at most $p$, and $f$ can be computed by a Turing machine using $O(p)$ space. In the transformer model, $n$0, where $n$1 is the number of input tokens. Under this restriction, every similarity score, feed-forward net, positional encoder, and intermediate vector is represented in $n$2 bits and computed in $n$3 space (Merrill et al., 2022).

One formalization uses an input embedding

$n$4

with a typical choice given by the “fractional positional embedding”

$n$5

encoded to $n$6 bits. For an attention head with $n$7, the head output is

$n$8

where $n$9 is a $O(\log n)$0-precision similarity function and $O(\log n)$1 is approximate float addition that preserves $O(\log n)$2 bits. With $O(\log n)$3 heads and a $O(\log n)$4-precision feedforward function $O(\log n)$5, a layer update has the form

$O(\log n)$6

and a depth-$O(\log n)$7 transformer is the cascade of $O(\log n)$8 such layers preceded by $O(\log n)$9 (Merrill et al., 2022).

A second formalization places the same precision condition on a more conventional transformer stack. In that presentation, the model has fixed depth $O(\log n)$0, number of heads $O(\log n)$1, model width $O(\log n)$2, and feed-forward width $O(\log n)$3, while all additions, multiplications, divisions, exponentials, layer normalization, and ReLU operations are performed with $O(\log n)$4 bits of precision. The computation includes the usual query, key, and value projections, softmax attention weights, residual connections, feed-forward sublayers, and a final classifier acting on the last token representation (Merrill et al., 2022).

The common feature across these definitions is not low-bit engineering in the hardware sense, but an asymptotic precision discipline: the representational budget grows only logarithmically with the context length. This is the technical condition that makes the model amenable to circuit-complexity and descriptive-complexity analysis.

2. Simulation by uniform threshold circuits

The central upper bound states that log-precision transformers lie in uniform $O(\log n)$5. In non-uniform form, any $O(\log n)$6-precision depth-$O(\log n)$7 transformer on inputs in $O(\log n)$8 can be simulated by a threshold circuit family of depth

$O(\log n)$9

and polynomial size, where $p$0 is the constant depth needed to sum $p$1 floats of $p$2 precision in $p$3. The uniform version strengthens this to a logspace-uniform family of constant-depth threshold circuits of the same depth and polynomial size. Corollary 5 states succinctly that log-precision transformers are contained in logspace-uniform $p$4 (Merrill et al., 2022).

The proof proceeds by induction on the layer index. In the base case, a logspace machine constructs a depth-3 circuit for the position embedding $p$5. In the inductive step, it builds depth-3 subcircuits computing $p$6 for each head and each position pair $p$7, applies a uniform $p$8 summation subcircuit to produce the normalization term $p$9, performs the scalar multiplications and summations needed for attention outputs, and finally applies another depth-3 subcircuit for the $f : \{0,1\}^* \to \{0,1\}^p$0-precision feed-forward function. The circuit-writing Turing machine uses only $f : \{0,1\}^* \to \{0,1\}^p$1 space because it needs only counters for $f : \{0,1\}^* \to \{0,1\}^p$2, $f : \{0,1\}^* \to \{0,1\}^p$3, and $f : \{0,1\}^* \to \{0,1\}^p$4 together with the small-space procedures for $f : \{0,1\}^* \to \{0,1\}^p$5 and $f : \{0,1\}^* \to \{0,1\}^p$6 (Merrill et al., 2022).

Later work sharpened the corresponding uniform-circuit results for related transformer variants. For average-hard attention transformers (AHATs), exact rational arithmetic with unbounded precision still yields recognition in DLOGTIME-uniform $f : \{0,1\}^* \to \{0,1\}^p$7. For softmax-attention transformers (SMATs), the same paper shows that $f : \{0,1\}^* \to \{0,1\}^p$8-bit floating-point precision lies in DLOGTIME-uniform $f : \{0,1\}^* \to \{0,1\}^p$9, and that even exact-real SMATs can be approximated to absolute error $p$0 by a $p$1-computable function (Chiang, 2024).

These results locate the fixed-depth transformer, under explicit precision control, inside a highly parallel circuit class. In the original log-precision setting the emphasis is logspace-uniformity; in the later refinements the emphasis shifts to DLOGTIME-uniformity and to stronger attention and precision regimes. In both cases, the operative structural fact is constant sequential depth.

3. Logical characterization and the role of uniform attention

Log-precision transformers also admit a logical characterization. The relevant logic is first-order logic over the domain of input positions, augmented with a majority quantifier $p$2, which holds when $p$3 is true for at least $p$4 positions. The main theorem states that for every family of log-precision transformers of fixed depth, head count, width, and feed-forward size, there is an $p$5 sentence computing the same predicate; conversely, for every $p$6 sentence there is a log-precision transformer family of fixed finite depth and width computing the same predicate (Merrill et al., 2022).

The forward direction rests on the fact that arithmetic on $p$7-bit quantities can be expressed in $p$8: addition, multiplication, division, comparisons, ReLU, layer normalization, and the arithmetic needed to define softmax attention can be reduced to formulas of constant quantifier depth. By induction over layers, each coordinate of each hidden vector becomes definable by an $p$9 formula, and the final classifier becomes an $p$0 sentence (Merrill et al., 2022).

The reverse direction uses the classical correspondence between $p$1 and uniform $p$2. Existential and universal quantifiers can be implemented by attention-and-feedforward gadgets, while the majority quantifier is implemented by uniform attention: all queries and keys are set equal so that

$p$3

A final feed-forward threshold then tests whether the average of indicator values is at least $p$4 (Merrill et al., 2022).

This makes the precision threshold conceptually sharp. Finite-precision transformers in the sense of constant $p$5 are too weak because a single head can only attend to a constant number of tokens and, in particular, cannot represent uniform attention. By contrast, $p$6 bits are sufficient to represent rational weights of the form $p$7, and thus sufficient to realize majority quantification. The paper therefore treats $p$8 precision as necessary and sufficient for matching $p$9 (Merrill et al., 2022).

Taken together with the circuit upper bound, this logical equivalence places log-precision transformers at the level of uniform $f$0. The characterization is exact at the level of string predicates under the stated architectural restrictions.

4. Expressive limits, advice, and the parallelism tradeoff

Because uniform $f$1 sits strictly below $f$2 and, under standard hypotheses, far below $f$3, the circuit upper bound immediately yields impossibility results. If $f$4, then log-precision transformers cannot solve all $f$5-complete problems. In particular, they cannot perfectly decide linear equation solving $f$6, universal context-free grammar membership, and, assuming $f$7, SAT; the discussion also names Horn-SAT, AI-planning, and permanent computation as further examples (Merrill et al., 2022).

The original paper states the limitation in a stronger intuitive form: if $f$8, then transformers cannot even accurately solve linear equalities or check membership in an arbitrary context-free grammar with empty productions. The argument is not that transformers are weak in an absolute sense, but that fixed-depth, highly parallel transformer computation falls into a class whose known upper bounds exclude many problems associated with sequential composition or deeper circuit depth (Merrill et al., 2022).

The same work also gives lower-bound style completeness statements in the opposite direction. Treating a non-uniform $f$9 circuit family $O(p)$0 as advice yields

$O(p)$1

Moreover, a depth-$O(p)$2 transformer can evaluate any depth-$O(p)$3 threshold circuit, so a transformer given $O(p)$4 in its prompt can solve any language in non-uniform $O(p)$5. A related instruction-following corollary states that if an arbitrary $O(p)$6 circuit for a regular-language recognizer is encoded as the instruction, then a fixed-weight transformer can follow that instruction on any input string (Merrill et al., 2022).

These upper and lower bounds motivate what the paper calls the “parallelism tradeoff.” Complexity classes such as $O(p)$7, $O(p)$8, and $O(p)$9 capture extreme parallelizability: polynomial size and constant depth. Transformers are designed for massive data-parallel training, and the paper speculatively introduces the idea that any architecture equally parallelizable, in the sense of constant sequential depth, will obey similar limitations. This suggests that “scaling by parallelism” may incur an inherent ceiling on computable functions (Merrill et al., 2022).

The same discussion explains why higher precision changes the picture. With $n$00 or unbounded precision, a layer can pack far more information into a single word and thereby simulate richer sequential behavior inside feed-forward arithmetic. In that sense, the impossibility results are specific to the joint restriction of logarithmic precision, small-space subcomputations, and constant-depth layer composition (Merrill et al., 2022).

5. Hierarchical prerequisite reasoning as a case study

A later case study analyzes hierarchical prerequisite propagation through the same circuit-complexity lens. The central task is recursive-majority on balanced ternary trees of height $n$01, with $n$02 leaves. The function $n$03 is defined recursively by applying $n$04 at each internal node: $n$05 where

$n$06

The paper shows unconditionally that recursive-majority lies in $n$07 via $n$08-depth bounded-fanin circuits of size $n$09. It also states that separating recursive-majority from uniform $n$10, and therefore from log-precision transformers, would require progress on the open question $n$11 (Liu et al., 25 Mar 2026).

Under a monotonicity restriction, however, the paper obtains an unconditional barrier. For alternating ALL/ANY prerequisite trees, corresponding to read-once alternating $n$12 formulas of depth $n$13, the monotone threshold-circuit depth hierarchy of Yao and of Håstad and Goldmann yields explicit monotone Boolean functions $n$14 for which every monotone threshold circuit of depth $n$15 requires size

$n$16

The corollary is that layered prerequisite structure, under the natural assumption that more mastered prerequisites never hurt, resists collapse to smaller monotone threshold depth without exponential blow-up (Liu et al., 25 Mar 2026).

The empirical portion of the study complements this theoretical picture. A 4-layer transformer encoder with hidden size 128, 4 heads, and model size $n$17K parameters was trained to predict the root outcome of a ternary majority tree of depth $n$18 from the $n$19 leaf bits using only root labels. At depth $n$20, transformer accuracy was $n$21, compared with $n$22 for a permutation-invariant MLP baseline that saw only the normalized leaf sum; at $n$23, the transformer reached $n$24, compared with $n$25 for the MLP; the oracle achieved $n$26 at all depths. Under random permutation of leaf positions, transformer accuracy was unchanged within $n$27, indicating a learned permutation-invariant shortcut (Liu et al., 25 Mar 2026).

A structural scaffolding experiment altered this behavior. When subtree boundaries were exposed by level-tag tokens but no auxiliary loss was used, accuracy remained approximately at the flat baseline. When auxiliary supervision was added at each subtree closure so that the model had to predict each internal node’s majority value, root accuracy rose to $n$28 at depth $n$29 and $n$30 at depth $n$31, with auxiliary accuracy on all separators at least $n$32. Shuffling the leaves then caused a large drop in root accuracy, for example $n$33 percentage points at depth $n$34 and $n$35 percentage points at depth $n$36, confirming that the computation had become structure-dependent (Liu et al., 25 Mar 2026).

The case study therefore connects the abstract $n$37 upper bound to a concrete hierarchical reasoning setting. The theoretical boundary remains open in the unrestricted case, but the empirical results show that standard training can favor shallow, permutation-invariant shortcuts even when the target function is recursively structured.

6. Extensions via chain-of-thought and growing computational traces

Subsequent work shows that the uniform-$n$38 characterization is specific to the single-pass, fixed-depth model, not to low precision by itself. One line of work studies decoder-only transformers with chain-of-thought (CoT) computation and log precision $n$39, where $n$40 is the number of computational steps to be simulated. For a Continuous CoT transformer, each step emits both a discrete token and a fixed-dimensional soft-token vector; for a Hybrid Transformer–RNN, each step augments the transformer with a linear RNN state of constant dimension. In both settings, the model has fixed width and fixed transformer depth, yet can simulate any $n$41-step Word RAM program in $n$42 decoding steps, where $n$43 is the bit-serial Turing-machine cost of one RAM instruction. For flat instruction sets, the overhead reduces to $n$44 when each instruction costs $n$45 time, and to $n$46 when instructions such as multiplication or division cost $n$47, with $n$48 (Li et al., 18 Jun 2026).

The construction encodes the RAM’s program counter, registers, and memory as an append-only log of store summaries, using a code

$n$49

with the property that $n$50 iff $n$51, separated by a polynomially small gap otherwise. A six-phase protocol loads the program counter, dereferences addresses, serializes words bit by bit, executes a localized TM-oracle subroutine for the active instruction, deserializes the outputs, and stores new summaries. The result is an efficient Word-RAM simulation using only poly-logarithmic overhead in $n$52 (Li et al., 18 Jun 2026).

A related paper analyzes standard decoder-only softmax transformers with rounded activations and attention weights, while allowing depth and width to grow logarithmically with the context length. It first constructs hardmax transformers with ternary activations and well-separated attention scores that simulate Turing machines using CoT, then converts them to equivalent softmax transformers. In the CoT construction, depth and width scale as $n$53 for a time bound $n$54, with ternary activations and total generated length $n$55. In the summarized CoT (SCoT) construction, model size scales logarithmically in a space bound $n$56, each segment has context length $n$57, and total generation length remains $n$58 (Brösamle et al., 18 May 2026).

That work also reports a Sudoku reasoning study. SCoT learns the Sudoku-solver algorithm with as few as $n$59 layers at $n$60 accuracy, whereas ordinary CoT with the same $n$61 fails on puzzles whose solver CoT length exceeds approximately $n$62 tokens; increasing depth to $n$63 at bfloat16 precision extends learnability to the full training length $n$64, while switching the $n$65 model from bfloat16 to fp32 has negligible effect. The paper interprets this as matching the theoretical distinction that CoT requires model size scaling in the time bound $n$66, whereas SCoT requires scaling in the space bound $n$67 (Brösamle et al., 18 May 2026).

These developments indicate that the log-precision transformer upper bound is a statement about a particular architectural regime: fixed-depth, massively parallel, single-pass computation with $n$68-bit intermediate states. When the resource model is changed by adding explicit computational traces, continuous soft tokens, recurrent carry state, or logarithmic growth in architecture size, the attainable computational class changes as well.

In the current literature, “log-precision transformer” therefore names a precise theoretical object rather than a generic low-bit implementation. Its significance lies in making the transformer’s computational profile analyzable in the language of $n$69, $n$70, uniformity, and depth hierarchy, while also clarifying which additional mechanisms are needed to escape those bounds.