---
title: Constant Bit-Size Transformers
url: https://www.emergentmind.com/topics/constant-bit-size-transformers
type: topic
---

# Constant Bit-Size Transformers

Searching arXiv for the core papers on constant bit-size transformers and related low-bit Transformer architectures.
Constant Bit-Size Transformers are Transformer architectures or analyses in which numerical precision, parameter bit-size, or recurrent state size is held fixed independently of input length, with efficiency or expressiveness recovered through architectural design, quantization, alternative attention kernels, or enlarged context windows. In the literature, the phrase spans several distinct but related ideas: constant-bit attention mechanisms such as Bitformer’s $T$-bit XOR–popcount attention [2311.13502]; constant 1-bit or few-bit parameterizations such as BitNet, BMT, and related quantized Transformer variants [2310.11453; 2302.04907]; constant-bit post-training or integer-only inference schemes [2208.07339; 2310.16836; 2303.13601; 2403.06082]; and formal results showing that constant bit-size transformers remain computationally universal when the context window grows with problem space complexity [2506.12027]. A separate but adjacent line of work studies fixed-size recurrent state in Transformer blocks, exemplified by Tensor Memory, where memory size is constant even as sequence length increases [2605.27686]. Taken together, these works characterize constant-bit-size design as a trade-off surface between precision, memory, compute, context length, and serial reasoning.

## 1. Conceptual scope and formal definitions

The most explicit formalization appears in "Constant Bit-size Transformers Are Turing Complete" [2506.12027]. There, a decoder-only transformer $\mathrm{TF}_\theta$ is specified by an embedding dimension $d$, a number $H$ of attention heads, feed-forward layers of width $O(d)$, and real-valued parameters $\theta$, each stored in $p$ bits, with total bit-size
\[
\mathrm{BitSize}(\mathrm{TF}) \;=\; p \times |\theta|\,.
\]
The model is *constant bit-size* if both $p$ and $|\theta|$ are fixed as the input length $n$ grows [2506.12027].

That paper also defines the context window length $s(n)$ by requiring each next token to depend only on the last $s$ tokens, and introduces the class $\mathrm{WINDOW}[s(n)]$ of decision problems solvable by a constant bit-size transformer with context window $O(s(n))$ [2506.12027]. This formalization separates three resources that are often conflated in practical discussions: parameter count, numerical precision, and context window.

In empirical systems papers, “constant bit-size” is used more loosely. Bitformer uses exactly $T$ binary bits per original float entry for $Q_b,K_b$, with $V_f$ remaining float [2311.13502]. BitNet trains 1-bit weights from scratch via BitLinear, while keeping gradients and optimizer states in FP32 and using 8-bit activations in experiments [2310.11453]. Binarized Neural Machine Translation distinguishes one-bit weights from one-bit activations and reports that a one-bit weight-only Transformer can achieve the same quality as a float one, while being 16x smaller in size [2302.04907]. Scaled Quantization for the Vision Transformer fixes $P=8$ bits for every tensor together with a 5-bit scale exponent, and carries every operator in integer form without intermediate floating-point computations [2303.13601].

A useful distinction is therefore between constant-bit *parameters*, constant-bit *attention operators*, constant-bit *inference datapaths*, and constant-size *state*. This suggests that the field is not organized around a single architecture class, but around a family of constraints imposed on Transformer computation.

## 2. Constant-bit attention mechanisms

Bitformer is an archetypal constant-bit attention design. Starting from the usual $Q_f=XW_q$, $K_f=XW_k$, and $V_f=XW_v$, it replaces floating-point $Q_f,K_f$ with $T$-step binary sequences $Q_b,K_b\in\{0,1\}^{T\times n\times d}$ using a Time-Integrate-and-Fire converter [2311.13502]. After pre-normalization to $[0,1]$, the binary sequences are generated by
\[
H_q^{t}[i,j] = V_q^{t-1}[i,j] + \hat Q_f[i,j],
\]
\[
Q_b^{t}[i,j] = \Theta(H_q^{t}[i,j]-V_{th}),
\]
\[
V_q^{t}[i,j] = H_q^{t}[i,j](1-Q_b^{t}[i,j]) + (H_q^{t}[i,j]-V_{th})Q_b^{t}[i,j],
\]
and analogously for $K_b$ [2311.13502]. The attention score at time step $t$ is computed by a Hamming-distance kernel via XOR plus popcount,
\[
(A_t)_{i,j} = \sum_{\ell=1}^d \mathrm{XOR}(Q_b^t[i,\ell],K_b^t[j,\ell]),
\]
and the final attention matrix is
\[
A=(\sum_{t=1}^T A_t + 1)^{-1},
\]
followed by $Y=A\,V_f$ [2311.13502].

The central computational claim is a reduction from $O(n^2d)$ float-MAC complexity to an effective $O(n^2T)$ bitwise complexity, since each Hamming distance on $d$ bits can be implemented with a single word-parallel popcount and $T\ll d$ [2311.13502]. The bit precision is constant because each $Q_b,K_b$ element is a single binary bit and exactly $T$ bits are used per original float entry [2311.13502].

BinaryAttention, proposed for vision and diffusion transformers, is a more aggressive one-bit formulation of query–key attention [2603.09582]. It retains only the sign of queries and keys, replacing floating dot products with bit-wise operations. With normalized $q_i,k_j$, the paper rewrites sign-based similarity through Hamming geometry:
\[
s_i^T t_j = d - 2\|s_i-t_j\|_H,
\]
so that the exponentiated similarity becomes a Hamming-distance kernel [2603.09582]. A Gaussian result is also given:
\[
E[\mathrm{sign}(q)\,\mathrm{sign}(k)^T] = (2/\pi)\,\arcsin(C),
\]
where $C = D_q^{-1/2}\Sigma_{qk}D_k^{-1/2}$ [2603.09582]. In practice the logits are
\[
S'_{ij}=(\mu_q\mu_k/\sqrt d)\,\mathrm{sign}(q_i)^T\mathrm{sign}(k_j),
\]
augmented by a trainable bias $B_{ij}$ [2603.09582].

These approaches differ in representational granularity. Bitformer uses a fixed chain of $T$ binary events to approximate real-valued attention structure [2311.13502], whereas BinaryAttention uses a direct one-bit sign representation plus bias correction [2603.09582]. This suggests two distinct constant-bit attention regimes: multi-bit temporal coding and single-bit sign coding.

## 3. One-bit and few-bit parameterizations

BitNet introduces BitLinear as a drop-in replacement of $\mathrm{nn.Linear}$ to train 1-bit weights from scratch [2310.11453]. Given a real latent weight matrix $W$, it first centers and binarizes:
\[
\alpha=\frac{1}{nm}\sum_{i,j} W_{ij},\qquad
\widetilde W=\mathrm{Sign}(W-\alpha),
\]
with scaling factor
\[
\beta=\frac{1}{nm}\|W\|_1.
\]
Activations are quantized to $b$ bits using absmax after a sub-LayerNorm:
\[
\gamma=\|x\|_\infty,\qquad
\widetilde x=\mathrm{Clip}(x/\gamma\times 2^{b-1},-2^{b-1}+\epsilon,2^{b-1}-\epsilon).
\]
The forward pass is
\[
y=\widetilde W\widetilde x,\qquad
y\leftarrow y\,\frac{\beta\gamma}{2^{b-1}},
\]
and all non-differentiable steps use the Straight-Through Estimator [2310.11453]. The method maintains a high-precision latent copy of each weight, uses a substantially larger peak learning rate, and introduces group quantization for model parallelism to avoid cross-GPU all-reduces of $\alpha,\beta,\gamma$ [2310.11453].

BitNet’s empirical results frame constant 1-bit weights as a scaling-law question rather than only a compression question. With fixed training tokens, loss follows
\[
L(N)=aN^b+c,
\]
with exponents almost identical to FP16 Transformers from 0.125B to 6.7B parameters, and the gap to FP16 shrinks as $N$ grows [2310.11453]. Memory is reduced by 16× relative to FP16 weights and 8× relative to 8-bit integer weights, while activations remain at 8 bit in the reported experiments [2310.11453].

Binarized Neural Machine Translation studies one-bit Transformers in the translation setting and identifies a failure mode absent from many high-level summaries: inflated dot-product variance under one-bit weights and activations [2302.04907]. For inner dimension $D$, after binarization the variance becomes
\[
\mathrm{Var}_{\mathrm{bin}}(A_b\cdot W_b)=D\cdot(B^4/16),
\]
which for typical $D=4096$ and $B\in[1,3]$ yields at least 256× inflation [2302.04907]. To stabilize training, the model adds extra LayerNorms, inserts a LayerNorm between ReLU and the second binarized FFN layer, and places an extra residual around the binarized output projection in attention [2302.04907]. On WMT17 En–De, the base float 6L6L Transformer has validation loss 1.39 and BLEU 26.35; weight-only binarization gives validation loss 1.38 and BLEU 25.93, while a stable 1-bit FFN gives validation loss 1.40 and BLEU 25.44 [2302.04907].

Related application-specific evidence appears in BitParT for top tagging [2508.07431]. There, FFNs and classifier layers are converted to binary-weight variants while attention, LayerNorm, and physics-informed modules remain full precision. On the Top Quark Tagging Reference Dataset, ParT achieves 94.00% accuracy and 0.9862 AUC, while BitParT achieves 93.99% accuracy and 0.9856 AUC, with an estimated total footprint of $\simeq 1.68$ MB versus $\simeq 4.9$ MB for full fp32 [2508.07431].

Sparse Binary Transformers for Multivariate Time Series Modeling combine one-bit retained weights with sparsity masks [2308.04637]. The effective weight is
\[
W_{\rm bin}=\alpha(B\odot M)\in\{-\alpha,0,+\alpha\}^{i\times j},
\]
with storage
\[
S_{\rm sbt}=pN\cdot 1 + 32 \quad\text{bits}
\]
for a layer with $N$ weights and keep fraction $p$ [2308.04637]. The paper reports up to a 53× reduction in stored bits and up to a 10.5× reduction in FLOPs [2308.04637].

## 4. Integer-only and post-training constant-bit inference

A separate tradition keeps training unchanged and imposes a fixed low-bit inference representation after the fact. LLM.int8() uses vector-wise 8-bit quantization for feed-forward and attention projection layers, combined with a mixed-precision decomposition for emergent outlier dimensions [2208.07339]. For each row $i$ of $X$ and column $j$ of $W$, the method uses separate abs-max scales
\[
c_x[i]=127/\|X[i,\cdot]\|_\infty,\qquad
c_w[j]=127/\|W[\cdot,j]\|_\infty,
\]
with dequantized output
\[
C_{16}[i,j]\approx (1/c_x[i])(1/c_w[j])\,C_{i32}[i,j].
\]
Outlier feature dimensions are isolated into a 16-bit multiplication if their absolute values exceed threshold $\alpha=6.0$ in at least 25% of layers and at least 6% of positions, usually with $|O|\le 7$ [2208.07339]. This cuts memory by roughly 50%, and on a 13B model the reported C4 validation perplexity is 12.45 for FP32, 16.48 for vector-wise int8 abs-max alone, and 12.45 for LLM.int8() [2208.07339].

LLM-FP4 quantizes both weights and activations to 4-bit floating-point values in a post-training manner [2310.16836]. It searches over FP4 formats E3M0, E2M1, and E1M2, where
\[
1+e+m=4.
\]
The clipping range is determined by
\[
Q_{\max}=(2-2^{-m})2^{2^e-b-1},\qquad Q_{\min}=-Q_{\max},
\]
and the method introduces a pre-shifted exponent-bias reparameterization to absorb per-channel activation scaling into weights [2310.16836]. On LLaMA-13B, the reported average score across BoolQ, PIQA, HellaSwag, WinoGrande, ARC-e/c is 68.9 for full precision and 63.1 for the 4-bit weight/activation model, outperforming prior PTQ baselines by 12.7 points [2310.16836].

FrameQuant targets almost two-bit quantization using Fusion Frames [2403.06082]. Rather than quantize $\Theta\in\mathbb R^{d\times d}$ directly, it quantizes
\[
D=P_{\rm out}^T\Theta P_{\rm in},
\]
after clipping at $\pm 2\sigma$ and solving a GPTQ-style reconstruction problem
\[
\min_{\hat D}\ \|(D-\hat D)C_{\rm prev}\|_F^2
=
\mathrm{trace}[(D-\hat D)^T H (D-\hat D)].
\]
The paper reports, for example, that a 1024×1024 FP32 weight matrix of 4 MB becomes $\sim 0.3$ MB under FrameQuant(2 bits, $r=1.1$), a 13× storage reduction [2403.06082].

Scaled Quantization for the Vision Transformer is the most explicit integer-only constant-datapath scheme [2303.13601]. Every real tensor value is represented as
\[
F \approx \mathrm{INT}\cdot 2^S,
\]
with $P=8$ signed integer bits and a 5-bit signed scale exponent shared across the tensor [2303.13601]. Multiply, add, softmax, layer normalization, inverse square-root, and GELU are all rewritten in integer form using ScaleMultiplication, ScaleSummation, ScaleDivision, truncated Maclaurin series for $e^x$, and Newton’s method for $1/\sqrt x$ [2303.13601]. This is a purely post-training quantization method with no retraining, no mixed precision, and no float intermediates [2303.13601].

These schemes show that “constant bit-size” at inference often refers less to a universal representational theorem than to a hardware contract: each tensor or operator carries a fixed-width representation across layers.

## 5. Computational expressiveness and Turing completeness

The theoretical centerpiece of the subject is the result that constant bit-size transformers are Turing complete [2506.12027]. The main simulation theorem states that for every single-tape Turing machine $M$ running in time $t(n)$ and space $s(n)\ge n$, there exists a transformer of constant bit-size and context window
\[
s(n)=O(\max\{n,\text{space used by }M\})
\]
that produces the same final accept/reject bit, possibly after $O(t(n)\cdot s(n))$ intermediate chain-of-thought steps [2506.12027]. The exact characterization is
\[
\mathrm{WINDOW}[s(n)] = \mathrm{SPACE}[s(n)].
\]
In particular, for polynomial $s(n)=n^k$, the paper recovers $\mathrm{WINDOW}[\mathrm{poly}(n)] = \mathrm{PSPACE}$ [2506.12027].

The proof proceeds by simulating a Turing machine with a Post machine and then a Post machine with a constant-size transformer [2506.12027]. The vocabulary is taken as $V=\Sigma\times Q$, each memory position stores a token $(\sigma_j,q_j)$, relative positional encodings mark the oldest and newest positions in the context window, one hardmax attention head selects the first or last token in the window, and a small constant-size ReLU network implements the finite transition map $\delta:Q\times\Sigma\to Q\times\Sigma$ [2506.12027]. The crucial point is that no parameter count or precision bits grow with $n$; only the context window does [2506.12027].

This result improves earlier universality arguments that required scaling embedding dimension or numerical precision with input length. The paper explicitly contrasts itself with previous works that required such growth and argues instead that only context window scaling is necessary, at the rate of the task’s space complexity rather than its time complexity [2506.12027].

A later result, "Efficient Turing Machine Simulation with Transformers" [2512.00003], refines this efficiency picture. For a $(t(n),s(n))$-bounded $k$-tape TM and fixed head-layer product $K=H\cdot L$, it proves the existence of a constant-bit Transformer with context window $S=O(s(n))$ and CoT overhead per TM step
\[
\Delta=O(s(n)^{6k/K}),
\]
so that by choosing $K\gg k$, the exponent $c=6k/K$ can be made arbitrarily small [2512.00003]. The construction uses synchronous multi-queue TMs as an intermediate model and sparse attention with fixed geometric offsets [2512.00003]. This suggests that the original $O(t(n)\cdot s(n))$ chain-of-thought overhead is not intrinsic to constant-bit universality, though the later result still relies on a growing context window.

## 6. Chain of thought, serial computation, and constant precision

The relationship between constant-bit precision and chain of thought is clarified by "Chain of Thought Empowers Transformers to Solve Inherently Serial Problems" [2402.12875]. That paper considers constant-depth decoder-only transformers with constant-bit floating-point precision, where all operations are rounded into a fixed floating-point set $F_{e,s}$ with constant exponent and mantissa bits [2402.12875].

Without CoT, the main upper bound is that constant-depth, constant-precision transformers decide only $\mathsf{AC}^0$ languages:
\[
T[\mathrm{poly}(n)][1][1] \subseteq \mathrm{AC}^0,
\]
with the proof relying on the fact that finite-precision iterated summation can be compiled into constant-depth Boolean circuits [2402.12875]. With $T$ steps of CoT, however, the paper proves
\[
\mathrm{SIZE}[T(n)] \subseteq \mathrm{CoT}[T(n)][\Theta(\log n)][1],
\]
and in particular
\[
\mathrm{P}/\mathrm{poly} = \mathrm{CoT}[\mathrm{poly}(n)][\log n][1]
\]
for constant-depth transformers with constant-bit precision and $O(\log n)$ embedding size [2402.12875].

The empirical section reinforces the expressiveness argument. On modular addition, permutation-group composition, iterated squaring, and circuit value problems, CoT greatly improves low-depth transformers on tasks that are hard for parallel computation [2402.12875]. The reported excerpt gives, for instance, S$_5$ accuracy of 20% without CoT versus 92% with CoT for depth $L=1$, and 10% versus 98% on CVP for depth $L=2$ [2402.12875].

This addresses a common misconception. Constant-bit precision by itself does not imply weak computation in the presence of autoregressive generation. Rather, finite precision sharply limits the computational class of a single forward pass, but CoT restores serial depth across steps [2402.12875; 2506.12027]. A plausible implication is that practical reasoning improvements from CoT can be interpreted as an externalization of intermediate state into the token stream, compensating for the bounded serial depth of shallow constant-precision computation.

## 7. Hardware consequences, empirical trade-offs, and adjacent fixed-state memory

Across the low-bit literature, the main engineering motivation is the replacement of float MACs by cheaper bitwise or integer operations. Bitformer reports on Xilinx ZCU104: BRAM increased by 3.4%, latency decreased by 14.5%, DSP usage decreased by 70.4%, FF decreased by 40.9%, and LUT decreased by 36.4% [2311.13502]. Using 45 nm CMOS energy numbers, it also cites MAC at 4.6 pJ and binary AC at 0.9 pJ, with Bitformer estimated at approximately one quarter the energy of a full-precision Transformer for comparable accuracy [2311.13502]. On ImageNet-1K with 224×224 input, Bitformer ($T=8$) attains 80.2% top-1 at 3.1 G(S)OPs versus Swin-T at 80.9% top-1 and 4.5 GFLOPs [2311.13502].

BitNet’s energy model at 7 nm gives, for a 6.7B model with batch length 512, FP16 matrix-multiply layers at $E_{\rm mul}=1.14$ J and $E_{\rm add}=0.54$ J versus BitNet at $0.02$ J and $0.04$ J, corresponding to approximately 200× less multiply energy and approximately 30× less add energy for the main matrix-multiply layers [2310.11453].

BinaryAttention reports a custom CUDA kernel that is 2.0×–2.2× faster than FlashAttention2 on A100 GPUs at head-size 128 across sequence lengths from 256 to 4096, and end-to-end ViT throughput at 1024×1024 input resolution that is 1.5× that of FlashAttention2 [2603.09582]. It also quantifies overheads: sign quantization of $Q,K$ takes approximately 2.8 ms, quantization of $V$ to u8 approximately 1.9 ms, around 5% of kernel time [2603.09582].

The quality side of the trade-off is heterogeneous. Bitformer reports average text-classification accuracy of 90.2% for Bitformer ($T=8$) versus 89.0% for a float32 Transformer on THUCNews and IMDB; BLEU 25.0 versus 25.8 on WMT18 Zh–En; and CIFAR10 accuracy improving from 69.3% at $T=2$ to 95.9% at $T=8$ and 96.0% at $T=16$ [2311.13502]. BMT reports that one-bit activations incur varying degrees of quality drop, though architectural changes mitigate this [2302.04907]. LLM.int8() reports no degradation on zero-shot evaluation up to 175B parameters [2208.07339], while LLM-FP4 reports larger but controlled loss at 4-bit weight/activation precision [2310.16836].

An adjacent development is fixed-size recurrent state. Tensor Memory augments Transformer blocks with a fixed-size recurrent 3D memory tensor $h_t,c_t\in\mathbb R^{C\times D\times H\times W}$, updated by Gaussian soft writes, a factorized 3D operator, and ConvLSTM-style gating [2605.27686]. The memory size is constant because the same tensor is reused at every step; storage remains $2CDHW$ floats regardless of sequence length [2605.27686]. This is not a quantization method, but it is a constant-size-state analogue to constant-bit-size parameterization. A plausible implication is that the broader research program is increasingly concerned with decoupling some notion of model state from input length, whether through finite precision, bounded parameter bit-size, or fixed recurrent memory.

The principal open limitations stated in the literature recur across papers. Very low $T$ in Bitformer loses expressive power for complex tasks, and increasing $T$ too far erodes the memory–compute advantage [2311.13502]. BitNet still uses 8-bit activations and leaves full 1-bit W1A1 to future work [2310.11453]. BinaryAttention quantizes only $Q,K$ to 1 bit and leaves $P,V$ at 8 bit [2603.09582]. BMT shows that naïve binarization destabilizes variance and requires extra normalization and skips [2302.04907]. Theoretical universality results require sufficiently long context windows and, in their baseline form, can incur substantial chain-of-thought overhead [2506.12027].

In aggregate, constant bit-size transformers should not be understood as a single settled model family. They are better described as a research area organized around fixed-width computation in Transformer architectures, with three major conclusions already established. First, low-bit and even one-bit Transformer components can remain competitive on substantial empirical workloads when accompanied by scaling, normalization, bias correction, or selective mixed precision [2311.13502; 2310.11453; 2302.04907; 2603.09582]. Second, fixed precision does not preclude strong computational expressiveness once context length and autoregressive computation are admitted [2506.12027; 2402.12875]. Third, the practical frontier lies in hardware–algorithm co-design: bitwise kernels, sparse or geometric-offset attention, integer-only softmax and normalization, and selective preservation of full precision in the most sensitive substructures [2311.13502; 2303.13601; 2512.00003].

Source: https://www.emergentmind.com/topics/constant-bit-size-transformers