Constant Bit-Size Transformers

• Constant Bit-Size Transformers are transformer variants with fixed numerical precision or parameter bit-size independent of input length, ensuring consistent model size and efficiency.
• They employ techniques such as quantization, alternative attention kernels (e.g., T-bit XOR–popcount) and constant recurrent state to balance precision, memory, and compute.
• They demonstrate computational universality and competitive empirical performance by leveraging chain-of-thought, normalization strategies, and hardware-aware design.

Searching arXiv for the core papers on constant bit-size transformers and related low-bit Transformer architectures. Constant Bit-Size Transformers are Transformer architectures or analyses in which numerical precision, parameter bit-size, or recurrent state size is held fixed independently of input length, with efficiency or expressiveness recovered through architectural design, quantization, alternative attention kernels, or enlarged context windows. In the literature, the phrase spans several distinct but related ideas: constant-bit attention mechanisms such as Bitformer’s $T$-bit XOR–popcount attention (Duan et al., 2023); constant 1-bit or few-bit parameterizations such as BitNet, BMT, and related quantized Transformer variants (Wang et al., 2023, Zhang et al., 2023); constant-bit post-training or integer-only inference schemes (Dettmers et al., 2022, Liu et al., 2023, Chang et al., 2023, Adepu et al., 2024); and formal results showing that constant bit-size transformers remain computationally universal when the context window grows with problem space complexity (Li et al., 22 May 2025). A separate but adjacent line of work studies fixed-size recurrent state in Transformer blocks, exemplified by Tensor Memory, where memory size is constant even as sequence length increases (Swain et al., 26 May 2026). Taken together, these works characterize constant-bit-size design as a trade-off surface between precision, memory, compute, context length, and serial reasoning.

1. Conceptual scope and formal definitions

The most explicit formalization appears in "Constant Bit-size Transformers Are Turing Complete" (Li et al., 22 May 2025). There, a decoder-only transformer $\mathrm{TF}_\theta$ is specified by an embedding dimension $d$, a number $H$ of attention heads, feed-forward layers of width $O(d)$, and real-valued parameters $\theta$, each stored in $p$ bits, with total bit-size

$$\mathrm{BitSize}(\mathrm{TF}) \;=\; p \times |\theta|\,.$$

The model is constant bit-size if both $p$ and $|\theta|$ are fixed as the input length $\mathrm{TF}_\theta$0 grows (Li et al., 22 May 2025).

That paper also defines the context window length $\mathrm{TF}_\theta$1 by requiring each next token to depend only on the last $\mathrm{TF}_\theta$2 tokens, and introduces the class $\mathrm{TF}_\theta$3 of decision problems solvable by a constant bit-size transformer with context window $\mathrm{TF}_\theta$4 (Li et al., 22 May 2025). This formalization separates three resources that are often conflated in practical discussions: parameter count, numerical precision, and context window.

In empirical systems papers, “constant bit-size” is used more loosely. Bitformer uses exactly $\mathrm{TF}_\theta$5 binary bits per original float entry for $\mathrm{TF}_\theta$6, with $\mathrm{TF}_\theta$7 remaining float (Duan et al., 2023). BitNet trains 1-bit weights from scratch via BitLinear, while keeping gradients and optimizer states in FP32 and using 8-bit activations in experiments (Wang et al., 2023). Binarized Neural Machine Translation distinguishes one-bit weights from one-bit activations and reports that a one-bit weight-only Transformer can achieve the same quality as a float one, while being 16x smaller in size (Zhang et al., 2023). Scaled Quantization for the Vision Transformer fixes $\mathrm{TF}_\theta$8 bits for every tensor together with a 5-bit scale exponent, and carries every operator in integer form without intermediate floating-point computations (Chang et al., 2023).

A useful distinction is therefore between constant-bit parameters, constant-bit attention operators, constant-bit inference datapaths, and constant-size state. This suggests that the field is not organized around a single architecture class, but around a family of constraints imposed on Transformer computation.

2. Constant-bit attention mechanisms

Bitformer is an archetypal constant-bit attention design. Starting from the usual $\mathrm{TF}_\theta$9, $d$0, and $d$1, it replaces floating-point $d$2 with $d$3-step binary sequences $d$4 using a Time-Integrate-and-Fire converter (Duan et al., 2023). After pre-normalization to $d$5, the binary sequences are generated by

$d$6

$d$7

$d$8

and analogously for $d$9 (Duan et al., 2023). The attention score at time step $H$0 is computed by a Hamming-distance kernel via XOR plus popcount,

$H$1

and the final attention matrix is

$H$2

followed by $H$3 (Duan et al., 2023).

The central computational claim is a reduction from $H$4 float-MAC complexity to an effective $H$5 bitwise complexity, since each Hamming distance on $H$6 bits can be implemented with a single word-parallel popcount and $H$7 (Duan et al., 2023). The bit precision is constant because each $H$8 element is a single binary bit and exactly $H$9 bits are used per original float entry (Duan et al., 2023).

BinaryAttention, proposed for vision and diffusion transformers, is a more aggressive one-bit formulation of query–key attention (Xiao et al., 10 Mar 2026). It retains only the sign of queries and keys, replacing floating dot products with bit-wise operations. With normalized $O(d)$0, the paper rewrites sign-based similarity through Hamming geometry: $O(d)$1 so that the exponentiated similarity becomes a Hamming-distance kernel (Xiao et al., 10 Mar 2026). A Gaussian result is also given: $O(d)$2 where $O(d)$3 (Xiao et al., 10 Mar 2026). In practice the logits are

$O(d)$4

augmented by a trainable bias $O(d)$5 (Xiao et al., 10 Mar 2026).

These approaches differ in representational granularity. Bitformer uses a fixed chain of $O(d)$6 binary events to approximate real-valued attention structure (Duan et al., 2023), whereas BinaryAttention uses a direct one-bit sign representation plus bias correction (Xiao et al., 10 Mar 2026). This suggests two distinct constant-bit attention regimes: multi-bit temporal coding and single-bit sign coding.

3. One-bit and few-bit parameterizations

BitNet introduces BitLinear as a drop-in replacement of $O(d)$7 to train 1-bit weights from scratch (Wang et al., 2023). Given a real latent weight matrix $O(d)$8, it first centers and binarizes: $O(d)$9 with scaling factor

$\theta$0

Activations are quantized to $\theta$1 bits using absmax after a sub-LayerNorm: $\theta$2 The forward pass is

$\theta$3

and all non-differentiable steps use the Straight-Through Estimator (Wang et al., 2023). The method maintains a high-precision latent copy of each weight, uses a substantially larger peak learning rate, and introduces group quantization for model parallelism to avoid cross-GPU all-reduces of $\theta$4 (Wang et al., 2023).

BitNet’s empirical results frame constant 1-bit weights as a scaling-law question rather than only a compression question. With fixed training tokens, loss follows

$\theta$5

with exponents almost identical to FP16 Transformers from 0.125B to 6.7B parameters, and the gap to FP16 shrinks as $\theta$6 grows (Wang et al., 2023). Memory is reduced by 16× relative to FP16 weights and 8× relative to 8-bit integer weights, while activations remain at 8 bit in the reported experiments (Wang et al., 2023).

Binarized Neural Machine Translation studies one-bit Transformers in the translation setting and identifies a failure mode absent from many high-level summaries: inflated dot-product variance under one-bit weights and activations (Zhang et al., 2023). For inner dimension $\theta$7, after binarization the variance becomes

$\theta$8

which for typical $\theta$9 and $p$0 yields at least 256× inflation (Zhang et al., 2023). To stabilize training, the model adds extra LayerNorms, inserts a LayerNorm between ReLU and the second binarized FFN layer, and places an extra residual around the binarized output projection in attention (Zhang et al., 2023). On WMT17 En–De, the base float 6L6L Transformer has validation loss 1.39 and BLEU 26.35; weight-only binarization gives validation loss 1.38 and BLEU 25.93, while a stable 1-bit FFN gives validation loss 1.40 and BLEU 25.44 (Zhang et al., 2023).

Related application-specific evidence appears in BitParT for top tagging (Rai et al., 10 Aug 2025). There, FFNs and classifier layers are converted to binary-weight variants while attention, LayerNorm, and physics-informed modules remain full precision. On the Top Quark Tagging Reference Dataset, ParT achieves 94.00% accuracy and 0.9862 AUC, while BitParT achieves 93.99% accuracy and 0.9856 AUC, with an estimated total footprint of $p$1 MB versus $p$2 MB for full fp32 (Rai et al., 10 Aug 2025).

Sparse Binary Transformers for Multivariate Time Series Modeling combine one-bit retained weights with sparsity masks (Gorbett et al., 2023). The effective weight is

$p$3

with storage

$p$4

for a layer with $p$5 weights and keep fraction $p$6 (Gorbett et al., 2023). The paper reports up to a 53× reduction in stored bits and up to a 10.5× reduction in FLOPs (Gorbett et al., 2023).

4. Integer-only and post-training constant-bit inference

A separate tradition keeps training unchanged and imposes a fixed low-bit inference representation after the fact. LLM.int8() uses vector-wise 8-bit quantization for feed-forward and attention projection layers, combined with a mixed-precision decomposition for emergent outlier dimensions (Dettmers et al., 2022). For each row $p$7 of $p$8 and column $p$9 of $$\mathrm{BitSize}(\mathrm{TF}) \;=\; p \times |\theta|\,.$$0, the method uses separate abs-max scales

$$\mathrm{BitSize}(\mathrm{TF}) \;=\; p \times |\theta|\,.$$1

with dequantized output

$$\mathrm{BitSize}(\mathrm{TF}) \;=\; p \times |\theta|\,.$$2

Outlier feature dimensions are isolated into a 16-bit multiplication if their absolute values exceed threshold $$\mathrm{BitSize}(\mathrm{TF}) \;=\; p \times |\theta|\,.$$3 in at least 25% of layers and at least 6% of positions, usually with $$\mathrm{BitSize}(\mathrm{TF}) \;=\; p \times |\theta|\,.$$4 (Dettmers et al., 2022). This cuts memory by roughly 50%, and on a 13B model the reported C4 validation perplexity is 12.45 for FP32, 16.48 for vector-wise int8 abs-max alone, and 12.45 for LLM.int8() (Dettmers et al., 2022).

LLM-FP4 quantizes both weights and activations to 4-bit floating-point values in a post-training manner (Liu et al., 2023). It searches over FP4 formats E3M0, E2M1, and E1M2, where

$$\mathrm{BitSize}(\mathrm{TF}) \;=\; p \times |\theta|\,.$$5

The clipping range is determined by

$$\mathrm{BitSize}(\mathrm{TF}) \;=\; p \times |\theta|\,.$$6

and the method introduces a pre-shifted exponent-bias reparameterization to absorb per-channel activation scaling into weights (Liu et al., 2023). On LLaMA-13B, the reported average score across BoolQ, PIQA, HellaSwag, WinoGrande, ARC-e/c is 68.9 for full precision and 63.1 for the 4-bit weight/activation model, outperforming prior PTQ baselines by 12.7 points (Liu et al., 2023).

FrameQuant targets almost two-bit quantization using Fusion Frames (Adepu et al., 2024). Rather than quantize $$\mathrm{BitSize}(\mathrm{TF}) \;=\; p \times |\theta|\,.$$7 directly, it quantizes

$$\mathrm{BitSize}(\mathrm{TF}) \;=\; p \times |\theta|\,.$$8

after clipping at $$\mathrm{BitSize}(\mathrm{TF}) \;=\; p \times |\theta|\,.$$9 and solving a GPTQ-style reconstruction problem

$p$0

The paper reports, for example, that a 1024×1024 FP32 weight matrix of 4 MB becomes $p$1 MB under FrameQuant(2 bits, $p$2), a 13× storage reduction (Adepu et al., 2024).

Scaled Quantization for the Vision Transformer is the most explicit integer-only constant-datapath scheme (Chang et al., 2023). Every real tensor value is represented as

$p$3

with $p$4 signed integer bits and a 5-bit signed scale exponent shared across the tensor (Chang et al., 2023). Multiply, add, softmax, layer normalization, inverse square-root, and GELU are all rewritten in integer form using ScaleMultiplication, ScaleSummation, ScaleDivision, truncated Maclaurin series for $p$5, and Newton’s method for $p$6 (Chang et al., 2023). This is a purely post-training quantization method with no retraining, no mixed precision, and no float intermediates (Chang et al., 2023).

These schemes show that “constant bit-size” at inference often refers less to a universal representational theorem than to a hardware contract: each tensor or operator carries a fixed-width representation across layers.

5. Computational expressiveness and Turing completeness

The theoretical centerpiece of the subject is the result that constant bit-size transformers are Turing complete (Li et al., 22 May 2025). The main simulation theorem states that for every single-tape Turing machine $p$7 running in time $p$8 and space $p$9, there exists a transformer of constant bit-size and context window

$|\theta|$0

that produces the same final accept/reject bit, possibly after $|\theta|$1 intermediate chain-of-thought steps (Li et al., 22 May 2025). The exact characterization is

$|\theta|$2

In particular, for polynomial $|\theta|$3, the paper recovers $|\theta|$4 (Li et al., 22 May 2025).

The proof proceeds by simulating a Turing machine with a Post machine and then a Post machine with a constant-size transformer (Li et al., 22 May 2025). The vocabulary is taken as $|\theta|$5, each memory position stores a token $|\theta|$6, relative positional encodings mark the oldest and newest positions in the context window, one hardmax attention head selects the first or last token in the window, and a small constant-size ReLU network implements the finite transition map $|\theta|$7 (Li et al., 22 May 2025). The crucial point is that no parameter count or precision bits grow with $|\theta|$8; only the context window does (Li et al., 22 May 2025).

This result improves earlier universality arguments that required scaling embedding dimension or numerical precision with input length. The paper explicitly contrasts itself with previous works that required such growth and argues instead that only context window scaling is necessary, at the rate of the task’s space complexity rather than its time complexity (Li et al., 22 May 2025).

A later result, "Efficient Turing Machine Simulation with Transformers" (Li et al., 28 Sep 2025), refines this efficiency picture. For a $|\theta|$9-bounded $\mathrm{TF}_\theta$00-tape TM and fixed head-layer product $\mathrm{TF}_\theta$01, it proves the existence of a constant-bit Transformer with context window $\mathrm{TF}_\theta$02 and CoT overhead per TM step

$\mathrm{TF}_\theta$03

so that by choosing $\mathrm{TF}_\theta$04, the exponent $\mathrm{TF}_\theta$05 can be made arbitrarily small (Li et al., 28 Sep 2025). The construction uses synchronous multi-queue TMs as an intermediate model and sparse attention with fixed geometric offsets (Li et al., 28 Sep 2025). This suggests that the original $\mathrm{TF}_\theta$06 chain-of-thought overhead is not intrinsic to constant-bit universality, though the later result still relies on a growing context window.

6. Chain of thought, serial computation, and constant precision

The relationship between constant-bit precision and chain of thought is clarified by "Chain of Thought Empowers Transformers to Solve Inherently Serial Problems" (Li et al., 2024). That paper considers constant-depth decoder-only transformers with constant-bit floating-point precision, where all operations are rounded into a fixed floating-point set $\mathrm{TF}_\theta$07 with constant exponent and mantissa bits (Li et al., 2024).

Without CoT, the main upper bound is that constant-depth, constant-precision transformers decide only $\mathrm{TF}_\theta$08 languages: $\mathrm{TF}_\theta$09 with the proof relying on the fact that finite-precision iterated summation can be compiled into constant-depth Boolean circuits (Li et al., 2024). With $\mathrm{TF}_\theta$10 steps of CoT, however, the paper proves

$\mathrm{TF}_\theta$11

and in particular

$\mathrm{TF}_\theta$12

for constant-depth transformers with constant-bit precision and $\mathrm{TF}_\theta$13 embedding size (Li et al., 2024).

The empirical section reinforces the expressiveness argument. On modular addition, permutation-group composition, iterated squaring, and circuit value problems, CoT greatly improves low-depth transformers on tasks that are hard for parallel computation (Li et al., 2024). The reported excerpt gives, for instance, S$\mathrm{TF}_\theta$14 accuracy of 20% without CoT versus 92% with CoT for depth $\mathrm{TF}_\theta$15, and 10% versus 98% on CVP for depth $\mathrm{TF}_\theta$16 (Li et al., 2024).

This addresses a common misconception. Constant-bit precision by itself does not imply weak computation in the presence of autoregressive generation. Rather, finite precision sharply limits the computational class of a single forward pass, but CoT restores serial depth across steps (Li et al., 2024, Li et al., 22 May 2025). A plausible implication is that practical reasoning improvements from CoT can be interpreted as an externalization of intermediate state into the token stream, compensating for the bounded serial depth of shallow constant-precision computation.

7. Hardware consequences, empirical trade-offs, and adjacent fixed-state memory

Across the low-bit literature, the main engineering motivation is the replacement of float MACs by cheaper bitwise or integer operations. Bitformer reports on Xilinx ZCU104: BRAM increased by 3.4%, latency decreased by 14.5%, DSP usage decreased by 70.4%, FF decreased by 40.9%, and LUT decreased by 36.4% (Duan et al., 2023). Using 45 nm CMOS energy numbers, it also cites MAC at 4.6 pJ and binary AC at 0.9 pJ, with Bitformer estimated at approximately one quarter the energy of a full-precision Transformer for comparable accuracy (Duan et al., 2023). On ImageNet-1K with 224×224 input, Bitformer ($\mathrm{TF}_\theta$17) attains 80.2% top-1 at 3.1 G(S)OPs versus Swin-T at 80.9% top-1 and 4.5 GFLOPs (Duan et al., 2023).

BitNet’s energy model at 7 nm gives, for a 6.7B model with batch length 512, FP16 matrix-multiply layers at $\mathrm{TF}_\theta$18 J and $\mathrm{TF}_\theta$19 J versus BitNet at $\mathrm{TF}_\theta$20 J and $\mathrm{TF}_\theta$21 J, corresponding to approximately 200× less multiply energy and approximately 30× less add energy for the main matrix-multiply layers (Wang et al., 2023).

BinaryAttention reports a custom CUDA kernel that is 2.0×–2.2× faster than FlashAttention2 on A100 GPUs at head-size 128 across sequence lengths from 256 to 4096, and end-to-end ViT throughput at 1024×1024 input resolution that is 1.5× that of FlashAttention2 (Xiao et al., 10 Mar 2026). It also quantifies overheads: sign quantization of $\mathrm{TF}_\theta$22 takes approximately 2.8 ms, quantization of $\mathrm{TF}_\theta$23 to u8 approximately 1.9 ms, around 5% of kernel time (Xiao et al., 10 Mar 2026).

The quality side of the trade-off is heterogeneous. Bitformer reports average text-classification accuracy of 90.2% for Bitformer ($\mathrm{TF}_\theta$24) versus 89.0% for a float32 Transformer on THUCNews and IMDB; BLEU 25.0 versus 25.8 on WMT18 Zh–En; and CIFAR10 accuracy improving from 69.3% at $\mathrm{TF}_\theta$25 to 95.9% at $\mathrm{TF}_\theta$26 and 96.0% at $\mathrm{TF}_\theta$27 (Duan et al., 2023). BMT reports that one-bit activations incur varying degrees of quality drop, though architectural changes mitigate this (Zhang et al., 2023). LLM.int8() reports no degradation on zero-shot evaluation up to 175B parameters (Dettmers et al., 2022), while LLM-FP4 reports larger but controlled loss at 4-bit weight/activation precision (Liu et al., 2023).

An adjacent development is fixed-size recurrent state. Tensor Memory augments Transformer blocks with a fixed-size recurrent 3D memory tensor $\mathrm{TF}_\theta$28, updated by Gaussian soft writes, a factorized 3D operator, and ConvLSTM-style gating (Swain et al., 26 May 2026). The memory size is constant because the same tensor is reused at every step; storage remains $\mathrm{TF}_\theta$29 floats regardless of sequence length (Swain et al., 26 May 2026). This is not a quantization method, but it is a constant-size-state analogue to constant-bit-size parameterization. A plausible implication is that the broader research program is increasingly concerned with decoupling some notion of model state from input length, whether through finite precision, bounded parameter bit-size, or fixed recurrent memory.

The principal open limitations stated in the literature recur across papers. Very low $\mathrm{TF}_\theta$30 in Bitformer loses expressive power for complex tasks, and increasing $\mathrm{TF}_\theta$31 too far erodes the memory–compute advantage (Duan et al., 2023). BitNet still uses 8-bit activations and leaves full 1-bit W1A1 to future work (Wang et al., 2023). BinaryAttention quantizes only $\mathrm{TF}_\theta$32 to 1 bit and leaves $\mathrm{TF}_\theta$33 at 8 bit (Xiao et al., 10 Mar 2026). BMT shows that naïve binarization destabilizes variance and requires extra normalization and skips (Zhang et al., 2023). Theoretical universality results require sufficiently long context windows and, in their baseline form, can incur substantial chain-of-thought overhead (Li et al., 22 May 2025).

In aggregate, constant bit-size transformers should not be understood as a single settled model family. They are better described as a research area organized around fixed-width computation in Transformer architectures, with three major conclusions already established. First, low-bit and even one-bit Transformer components can remain competitive on substantial empirical workloads when accompanied by scaling, normalization, bias correction, or selective mixed precision (Duan et al., 2023, Wang et al., 2023, Zhang et al., 2023, Xiao et al., 10 Mar 2026). Second, fixed precision does not preclude strong computational expressiveness once context length and autoregressive computation are admitted (Li et al., 22 May 2025, Li et al., 2024). Third, the practical frontier lies in hardware–algorithm co-design: bitwise kernels, sparse or geometric-offset attention, integer-only softmax and normalization, and selective preservation of full precision in the most sensitive substructures (Duan et al., 2023, Chang et al., 2023, Li et al., 28 Sep 2025).