Fixed-Precision Transformers Explained

• Fixed-Precision Transformers are models that enforce a fixed approximation error or fixed arithmetic precision, ensuring consistent performance over arbitrary context lengths.
• They often employ uniform low-bit quantization strategies, which reduce model size and increase hardware efficiency without severely compromising accuracy.
• Formal analyses demonstrate that explicit rounding and fixed-point computations enable theoretical guarantees in universality and expressivity, despite practical deployment challenges.

Fixed-Precision Transformers is not a single technical object but a family of meanings that recur across the transformer literature. In one line of work, the phrase denotes deep transformers that attain a prescribed approximation error $\epsilon$ uniformly over arbitrary, even infinite, context lengths while keeping embedding dimension and head count fixed; in that setting, “precision” means approximation error rather than numerical arithmetic precision. In quantization and systems work, the term usually refers to uniform low-bit or integer-only implementations in which weights, and sometimes activations and KV-cache, are represented with a fixed bit-width. In formal expressivity work, it denotes transformers whose arithmetic is restricted to a finite fixed-point domain with explicit rounding, sometimes with rounding inside attention itself. These usages address different questions—universality, deployment efficiency, numerical stability, or formal computational power—and their results should be read within their respective models (Furuya et al., 2024, Xu et al., 2021, Yang et al., 13 Feb 2026).

1. Terminological scope and core distinctions

The most important distinction is between approximation precision and arithmetic precision. In "Transformers are Universal In-context Learners" (Furuya et al., 2024), a “fixed-precision” transformer is a deep architecture that, for a chosen approximation tolerance $\epsilon > 0$, uniformly approximates a continuous in-context mapping over contexts of arbitrary cardinality while keeping the embedding dimension and number of heads fixed. The paper explicitly states that “precision” there is approximation error $\epsilon$, not numerical floating-point precision.

By contrast, in quantization papers such as "Mixed Precision of Quantization of Transformer LLMs for Speech Recognition" (Xu et al., 2021), fixed precision means uniform precision: all weight parameters across the Transformer are quantized to the same $n$-bit format. The paper contrasts this with mixed precision, where bit-widths are assigned per layer or per module according to quantization sensitivity. Related systems papers extend this deployment-oriented meaning to integer-only execution, static INT4 quantization, and fixed-point accelerators (Ling et al., 2024, Marchisio et al., 2023, Breugel et al., 5 Jun 2025).

A third meaning is formal and language-theoretic. "Length Generalization Bounds for Transformers" (Yang et al., 13 Feb 2026) defines fixed-precision numbers as

$$\mathbb{F}_{p,s} = \{m \cdot 2^{-s} : m \in \mathbb{Z},\ -2^{p-1} \le m < 2^{p-1}\},$$

and studies transformers with explicit rounding both outside and inside attention. The same paper distinguishes general $(p,q)$-precision transformers from fixed-precision transformers, using “transformers” for $(1,\log n)$ and “fixed-precision transformers” for $(1,1)$.

A separate literature uses “transformers” to mean abstract transfer functions in numerical abstract interpretation rather than the Transformer neural-network architecture. That usage is explicitly distinguished in "Universal Synthesis of Differentiably Tunable Numerical Abstract Transformers" (Gomber et al., 16 Jul 2025), and it is separate from the neural models discussed here.

2. Fixed approximation precision over arbitrary context length

The approximation-theoretic formulation begins by replacing finite token sequences with probability measures on a compact token domain. Let $X \subset \mathbb{R}^d$ be compact, and encode a finite set of tokens $\{x_i\}_{i=1}^n$ as the empirical measure

$\epsilon > 0$0

An in-context mapping is then

$\epsilon > 0$1

or equivalently a distribution-to-function map $\epsilon > 0$2 defined by $\epsilon > 0$3. Continuity is imposed with respect to the weak$\epsilon > 0$4 topology on $\epsilon > 0$5, metrized on compact domains by the $\epsilon > 0$6-Wasserstein distance

$\epsilon > 0$7

The main theorem proves uniform approximation of every continuous $\epsilon > 0$8 on $\epsilon > 0$9 by a deep transformer alternating multi-head attention layers $\epsilon$0 and token-wise MLPs $\epsilon$1:

$\epsilon$2

Crucially, the construction holds for arbitrary, even infinite, context size, because contexts are measures rather than fixed-length sequences (Furuya et al., 2024).

The “fixed-precision” content of the theorem is architectural. For a given $\epsilon$3, the embedding dimension is fixed and does not depend on $\epsilon$4 or on context length; in the explicit construction, $\epsilon$5 suffices. The number of heads is fixed at $\epsilon$6, independent of $\epsilon$7 and of the number of represented tokens. Each head has key/query dimension $\epsilon$8 and head output dimension $\epsilon$9. What scales with $n$0 is depth $n$1 and the size of MLPs used to approximate multiplication, but the proof is existential and does not provide explicit bounds on $n$2 or MLP width as a function of $n$3 (Furuya et al., 2024).

The attention operator is extended from finite-token softmax attention to a measure-theoretic integral operator. For one layer,

$n$4

with an optional residual connection. Composition is defined via pushforward of the context measure:

$n$5

The proof combines Stone–Weierstrass density of cylindrical function classes, an injectivity lemma related to the Radon transform, and MLP approximation of component-wise multiplication. A central point-separation functional is

$n$6

with $n$7. The same paper formalizes a causal, masked setting by “space-time lifting,” treating sequences as measures on $n$8 and inserting the indicator $n$9 into the attention kernel, but the universality theorem itself does not cover masked attention. Relative to earlier universality results such as Yun et al. (2019) and the smoothness issues emphasized by Nath (2024), this framework establishes universality over distributions with fixed embedding and head counts at fixed approximation precision (Furuya et al., 2024).

3. Uniform low-bit precision, mixed precision, and transform-assisted quantization

In deployment-oriented work, fixed-precision transformers are usually uniformly quantized transformers. "Mixed Precision of Quantization of Transformer LLMs for Speech Recognition" (Xu et al., 2021) defines fixed precision as a single uniform bit-width across the Transformer and contrasts it with mixed precision learned by either a Hessian-trace-weighted sensitivity metric (MinSen) or mixed-precision neural architecture search (MPNAS). The quantizer is a per-layer uniform quantizer with shared scale $$\mathbb{F}_{p,s} = \{m \cdot 2^{-s} : m \in \mathbb{Z},\ -2^{p-1} \le m < 2^{p-1}\},$$0, and the loss increase under quantization noise is approximated by

$$\mathbb{F}_{p,s} = \{m \cdot 2^{-s} : m \in \mathbb{Z},\ -2^{p-1} \le m < 2^{p-1}\},$$1

The paper uses the criterion

$$\mathbb{F}_{p,s} = \{m \cdot 2^{-s} : m \in \mathbb{Z},\ -2^{p-1} \le m < 2^{p-1}\},$$2

and estimates $$\mathbb{F}_{p,s} = \{m \cdot 2^{-s} : m \in \mathbb{Z},\ -2^{p-1} \le m < 2^{p-1}\},$$3 with Hutchinson’s estimator. On Penn Treebank, uniform 2-bit yields PPL $$\mathbb{F}_{p,s} = \{m \cdot 2^{-s} : m \in \mathbb{Z},\ -2^{p-1} \le m < 2^{p-1}\},$$4 at size $$\mathbb{F}_{p,s} = \{m \cdot 2^{-s} : m \in \mathbb{Z},\ -2^{p-1} \le m < 2^{p-1}\},$$5 MB and $$\mathbb{F}_{p,s} = \{m \cdot 2^{-s} : m \in \mathbb{Z},\ -2^{p-1} \le m < 2^{p-1}\},$$6 compression, while MinSen at approximately $$\mathbb{F}_{p,s} = \{m \cdot 2^{-s} : m \in \mathbb{Z},\ -2^{p-1} \le m < 2^{p-1}\},$$7 bits yields PPL $$\mathbb{F}_{p,s} = \{m \cdot 2^{-s} : m \in \mathbb{Z},\ -2^{p-1} \le m < 2^{p-1}\},$$8 at size $$\mathbb{F}_{p,s} = \{m \cdot 2^{-s} : m \in \mathbb{Z},\ -2^{p-1} \le m < 2^{p-1}\},$$9 MB and $(p,q)$0 compression. On Switchboard, MinSen at approximately $(p,q)$1 bits yields PPL approximately $(p,q)$2, size $(p,q)$3 MB, $(p,q)$4 compression, and evaluation time approximately $(p,q)$5 s versus $(p,q)$6 s for the FP32 baseline. The paper reports up to $(p,q)$7 compression without WER degradation, and for a larger 16-layer Transformer LM, ADMM 4-bit yields a Hub5’00 swbd WER reduction of $(p,q)$8 absolute ($(p,q)$9 relative) versus the FP32 16-layer baseline (Xu et al., 2021).

"Learning Accurate Integer Transformer Machine-Translation Models" (Wu, 2020) shows that all $(1,\log n)$0 matrix multiplications in Transformer Base and Transformer Big can be converted to integer matmuls without material accuracy loss by combining range-preserving quantization for parameters with learned per-tensor scales for non-parameter tensors and staged QAT. The signed quantizer is

$(1,\log n)$1

with bankers’ rounding and clipping after rounding. Attention scores use

$(1,\log n)$2

softmax remains in FP32, and attention application uses

$(1,\log n)$3

On WMT14 En–De newstest2014 with beam $(1,\log n)$4 and length penalty $(1,\log n)$5, Transformer Base FP32 achieves $(1,\log n)$6 uncased and $(1,\log n)$7 cased BLEU, and INT8 matches them at $(1,\log n)$8 and $(1,\log n)$9. Transformer Big FP32 achieves $(1,1)$0 uncased and $(1,1)$1 cased, while INT8 gives $(1,1)$2 and $(1,1)$3. INT6 degrades more, by $(1,1)$4 BLEU for Base and $(1,1)$5–$(1,1)$6 BLEU for Big (Wu, 2020).

More recent work treats fixed precision as an optimization target rather than a rigid architectural constraint. "FPTQuant: Function-Preserving Transforms for LLM Quantization" (Breugel et al., 5 Jun 2025) introduces four function-preserving transforms: a mergeable pre-RoPE transform for queries and keys, a mergeable transform for values, a mergeable scaling transform within the MLP block, and a cheap dynamic scaling transform. These transforms preserve the full-precision function while reshaping activation distributions to be more quantization-friendly. The pre-RoPE construction exploits the commutation of block-diagonal $(1,1)$7 rotations and diagonal scalings with RoPE; the value transform exploits head-local invertibility; the MLP transform uses distributivity of diagonal scaling over the Hadamard product; and the residual scaling uses RMSNorm-derived per-token scalers. The paper targets static INT4 quantization for weights, activations, and KV-cache, requires no custom kernels, and reports static INT4 prefill speedups of $(1,1)$8–$(1,1)$9 over FP16, within $X \subset \mathbb{R}^d$0–$X \subset \mathbb{R}^d$1 of the INT4 upper bound. This does not redefine fixed precision as mixed precision; rather, it shows that fixed-bit deployment can be materially improved by transform design before quantization (Breugel et al., 5 Jun 2025).

Across these papers, a common pattern is that uniform fixed precision is hardware-friendly and analytically simple, but heterogeneous sensitivity of embeddings, attention projections, attention outputs, and FFN layers makes naive uniform low-bit quantization brittle. This suggests that “fixed precision” in practical deployment is often the baseline against which mixed precision, learned scaling, or function-preserving transforms are evaluated.

4. Integer-only and fixed-point hardware realizations

Hardware papers emphasize that fixed precision is not merely a storage format; it restructures the full inference pipeline. "Integer-only Quantized Transformers for Embedded FPGA-based Time-series Forecasting in AIoT" (Ling et al., 2024) presents a complete integer-only design for a single-encoder-layer, single-head Transformer on a Xilinx Spartan-7 XC7S15. The model uses $X \subset \mathbb{R}^d$2, $X \subset \mathbb{R}^d$3, $X \subset \mathbb{R}^d$4, $X \subset \mathbb{R}^d$5, and output projections all of dimensionality $X \subset \mathbb{R}^d$6, FFN width $X \subset \mathbb{R}^d$7, residual connections, and Batch Normalization in place of Layer Normalization. Quantization is asymmetric per-tensor for most tensors and symmetric for biases and BN offsets, with

$X \subset \mathbb{R}^d$8

The attention scaling factor $X \subset \mathbb{R}^d$9 is folded into integer scaling factors, residual additions use ApproxMul with right-shifted integer multipliers, softmax is implemented through LUTs plus a Radix-2 non-restoring divider, and BN is reparameterized into an affine map. On the AirU dataset, representative deployments include $\{x_i\}_{i=1}^n$0 with RMSE $\{x_i\}_{i=1}^n$1, $\{x_i\}_{i=1}^n$2 ms latency, and $\{x_i\}_{i=1}^n$3 mJ energy; $\{x_i\}_{i=1}^n$4 with RMSE $\{x_i\}_{i=1}^n$5, $\{x_i\}_{i=1}^n$6 ms, and $\{x_i\}_{i=1}^n$7 mJ; and $\{x_i\}_{i=1}^n$8 with RMSE $\{x_i\}_{i=1}^n$9, $\epsilon > 0$00 ms, and $\epsilon > 0$01 mJ. Compared to an 8-bit MCU baseline, the 4-bit FPGA model increases test loss by only $\epsilon > 0$02, runs $\epsilon > 0$03 faster, and is $\epsilon > 0$04 more energy-efficient. The paper also stresses that reducing bit-width does not monotonically reduce latency or energy, because datapath balance, divider cost, rescaling overhead, and model configuration can dominate the outcome (Ling et al., 2024).

"SwiftTron: An Efficient Hardware Accelerator for Quantized Transformers" (Marchisio et al., 2023) addresses a more general full-Transformer inference path in a 65 nm ASIC flow. SwiftTron uses symmetric quantization with zero-point $\epsilon > 0$05, INT8 inputs to MatMul, INT32 accumulators and nonlinear blocks, and dyadic requantization

$\epsilon > 0$06

Attention is implemented as MatMul $\epsilon > 0$07 Scale $\epsilon > 0$08 Softmax $\epsilon > 0$09 Requantization $\epsilon > 0$10 MatMul, with max-subtraction, a second-order polynomial approximation of $\epsilon > 0$11 on $\epsilon > 0$12, integer-only GELU and LayerNorm, and dyadic rescaling for residual alignment. For RoBERTa-base on SST-2 with $\epsilon > 0$13, $\epsilon > 0$14, $\epsilon > 0$15, and $\epsilon > 0$16, the table reports end-to-end inference latency of $\epsilon > 0$17 ms at 65 nm and 143 MHz, area $\epsilon > 0$18 mm$\epsilon > 0$19, and a power discrepancy: $\epsilon > 0$20 mW in the abstract versus $\epsilon > 0$21 W in the synthesis table. The paper states that the abstract’s “1.83 ns” is a typographical error and that the table’s $\epsilon > 0$22 ms value is the correct latency. Area and power are dominated by MatMul, with Softmax and LayerNorm contributing materially smaller but still nontrivial fractions (Marchisio et al., 2023).

These implementations clarify a recurrent misconception: low-bit fixed precision does not automatically imply an entirely homogeneous low-bit datapath. Integer-only systems often widen accumulators, use wider LUT outputs, or retain carefully designed normalization pathways. Conversely, matmul-only integerization, as in the machine-translation QAT approach, can already yield strong gains while leaving softmax, LayerNorm, residual adds, and related non-matmul operations in FP32 (Wu, 2020).

5. Finite precision, length generalization, and computational expressivity

Formal work on fixed precision asks what transformers can compute when arithmetic is explicitly rounded and the precision budget is part of the model. "Length Generalization Bounds for Transformers" (Yang et al., 13 Feb 2026) studies future-masked encoders over $\epsilon > 0$23 and defines a fixed-precision variant in which attention-side quantities are rounded inside attention:

$\epsilon > 0$24

The paper proves that fixed-precision transformers are equivalent, via two-way translations, to positive CRASP and to $\epsilon > 0$25. For this positive fragment, the length complexity is exponential in the size of the program, with matching upper and lower bounds. By contrast, for general CRASP—and hence for standard transformers with log-precision attention—the paper proves that non-asymptotic length generalization bounds are not computable already at two layers. Its transformer-side summary is that fixed-precision transformers have length complexity $\epsilon > 0$26 and $\epsilon > 0$27, whereas unrestricted transformers do not admit computable non-asymptotic length generalization bounds (Yang et al., 13 Feb 2026).

"Revisiting Padded Transformer Expressivity: Which Architectural Choices Matter and Which Don't" (Svete et al., 28 May 2026) places fixed precision into a circuit-complexity framework. With signed fixed-point numbers of $\epsilon > 0$28 bits and polynomial padding, it defines constant precision by $\epsilon > 0$29, width $\epsilon > 0$30, and volume $\epsilon > 0$31. Under sufficient volume $\epsilon > 0$32, polynomially padded L-uniform constant-precision transformers of constant depth are exactly L-uniform $\epsilon > 0$33, while growing-precision ones are exactly L-uniform $\epsilon > 0$34, regardless of width. With $\epsilon > 0$35 looping, constant-precision models reach FO-uniform $\epsilon > 0$36 and growing-precision models reach FO-uniform $\epsilon > 0$37. The paper’s central claim is that numeric precision and model depth are the main factors affecting expressivity, while attention type and width are comparatively robust within this padded setting (Svete et al., 28 May 2026).

A different formal model reaches a different endpoint. "Constant Bit-size Transformers Are Turing Complete" (Li et al., 22 May 2025) defines a constant bit-size transformer as one with constant parameter precision $\epsilon > 0$38 and constant parameter count $\epsilon > 0$39, so total bit-size $\epsilon > 0$40 is independent of input length. The construction uses a decoder-only autoregressive transformer with one decoder block, one hard-attention head, relative positional encoding, and a fixed vocabulary $\epsilon > 0$41 derived from a Post machine simulating a Turing machine. The context window of length $\epsilon > 0$42 acts as a queue, and the main theorem states

$\epsilon > 0$43

for any non-decreasing $\epsilon > 0$44, implying in particular $\epsilon > 0$45. The same paper proves that a constant bit-size transformer with context window $\epsilon > 0$46 can simulate a single-tape Turing machine using $\epsilon > 0$47 space after $\epsilon > 0$48 chain-of-thought steps (Li et al., 22 May 2025).

Finite-precision linear-attention models exhibit yet another profile. "Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensions" (Irie et al., 2023) studies bounded precision, real-time, fixed-size-state linear Transformers or Fast Weight Programmers. Under these assumptions, vanilla linear Transformers are equivalent to RNN-like sequence processors with a fixed-size fast weight matrix. The paper shows that normalized linear Transformers are not rationally recurrent, can recognize certain counter languages, but cannot recognize certain regular languages such as $\epsilon > 0$49 without positional encoding. Recurrent Delta and self-referential weight matrices empirically achieve $\epsilon > 0$50 Bin0/Bin1 accuracy on parity, $\epsilon > 0$51, and $\epsilon > 0$52, whereas vanilla linear attention fails to generalize on parity and reset Dyck-1. This indicates that finite precision alone does not fix expressive power; recurrence, memory rewrite, and state-update structure also matter (Irie et al., 2023).

Taken together, these results do not yield a single expressivity hierarchy for “fixed-precision transformers.” They study different architectures, different attention rules, different resources that are allowed to scale, and different notions of generalization. A plausible implication is that formal claims about fixed precision are highly model-specific: round-everywhere future-masked encoders, padded fixed-point families, hard-attention context-window simulators, and fixed-state linear Transformers occupy distinct analytical regimes.

6. Numerical stability, common misconceptions, and open technical issues

Finite-precision deployment also raises a numerical-analysis question: how forward-pass round-off errors propagate through transformer blocks. "Numerical Error Analysis of LLMs" (Budzinskiy et al., 13 Mar 2025) adopts the standard floating-point model with unit roundoff

$\epsilon > 0$53

and

$\epsilon > 0$54

Using Higham-style $\epsilon > 0$55–$\epsilon > 0$56 notation, it derives local and global forward-error bounds for affine layers, RMS normalization, LayerNorm, ReLU MLPs, self-attention, residual connections, and whole pre-norm transformer stacks. RMS normalization has a uniformly bounded componentwise condition number $\epsilon > 0$57, while standard LayerNorm becomes sensitive when the centered vector $\epsilon > 0$58 has components near zero. For RMS LN,

$\epsilon > 0$59

whereas LN carries additional factors depending on $\epsilon > 0$60. Self-attention error depends strongly on the conditioning of $\epsilon > 0$61, the magnitude of logits, and token norms. The global forward-pass bounds grow exponentially in depth $\epsilon > 0$62, and experiments reported in the paper show roughly exponential growth in mean relative componentwise error with depth, although median error remains far smaller (Budzinskiy et al., 13 Mar 2025).

This numerical work aligns with several practical design choices seen elsewhere. Pre-norm architectures reduce attention conditioning, and the paper recommends computing similarity scores and softmax in higher precision, using mixed-precision GEMM with higher-precision accumulation, and preferring RMS LN to standard LN in low-precision regimes (Budzinskiy et al., 13 Mar 2025). These recommendations are consistent with deployment papers that either retain softmax and normalization in FP32 or use widened internal formats even when matmuls are quantized (Wu, 2020, Marchisio et al., 2023).

Several recurring misconceptions follow from the terminological breadth of the field. First, fixed precision does not always mean fixed bit-width: in the universality theorem over measures, it means fixed approximation precision $\epsilon > 0$63 with fixed architectural width and head count (Furuya et al., 2024). Second, fixed precision does not always mean fully integer-only inference: many quantized systems keep softmax, LayerNorm, residual adds, or analogous nonlinear stages in FP32 or in widened integer formats (Wu, 2020, Xu et al., 2021). Third, lower bit-width does not necessarily imply lower latency or energy; the embedded FPGA study explicitly reports cases where 6-bit is slower and more energy-consuming than 8-bit because model size, cycle count, divider cost, and rescaling overhead dominate raw bit-width effects (Ling et al., 2024).

Open technical questions remain heterogeneous. In the universality setting, masked causal attention is formalized but not covered by the main theorem (Furuya et al., 2024). In quantization, activation quantization, extreme low-bit attention, and calibration of normalization layers remain difficult, and transform-assisted methods such as FPTQuant improve but do not eliminate these issues (Breugel et al., 5 Jun 2025). In formal expressivity, current bounds depend sharply on padding, rounding semantics, attention type, and context-window assumptions, and there is no single finite-precision abstraction subsuming all of the proven results (Yang et al., 13 Feb 2026, Svete et al., 28 May 2026, Li et al., 22 May 2025). The term “Fixed-Precision Transformers” therefore names a broad research area rather than a unified model class: one must always specify whether the subject is approximation at fixed $\epsilon > 0$64, uniform low-bit quantization, integer-only hardware, or explicit finite-precision computation with formal rounding semantics.