Modular-Addition Transformers

• Modular-addition transformers are architectures designed to compute sums modulo a prime using continuous Fourier-based representations.
• They offer a testbed for studying algorithmic generalization and grokking with applications ranging from cryptographic settings to mechanistic interpretability.
• Recent research shows these models learn nontrivial sine/cosine circuits that scale to high-dimensional modular arithmetic problems in cryptography.

Searching arXiv for papers on modular-addition transformers and related mechanistic interpretations. to=arxiv_search 天天中彩票腾讯json_object 新天天彩票 unfortunately single_search queries not supported? to=arxiv_search 天天中彩票充值ి{"query":"modular addition transformers mechanistic interpretability", "max_results": 10} to=arxiv_search 天天中彩票投注_json 彩经彩票{"query":"Teaching Transformers Modular Arithmetic at Scale (Saxena et al., 2024)", "max_results": 5} to=arxiv_search 天天中彩票被{"query":"Teaching Transformers Modular Arithmetic at Scale", "max_results": 5} Modular-addition transformers are transformer architectures trained to compute addition in a finite cyclic group, typically either the two-input map $(a,b)\mapsto (a+b)\bmod p$ or the more general $N$-input map $(x_1,\dots,x_N)\mapsto \sum_{i=1}^N x_i \bmod q$. The topic has become a standard testbed for algorithmic generalization, grokking, and mechanistic interpretability because the target function is algebraically simple while the learned internal solutions are nontrivial. Recent work spans stylized one-layer transformers analyzed through Fourier circuits, mechanistic accounts of grokking, and scaled encoder-only transformers that reach regimes such as $N=256$ and $q=3329$, which are directly relevant to lattice-based cryptography (Li et al., 2024, Mohamadi et al., 2024, Saxena et al., 2024).

1. Formal task definitions and problem regimes

The canonical modular-addition task fixes a prime modulus $p$ and inputs $a_1,\dots,a_k\in\mathbb Z_p$, with target

$S=(a_1+\cdots+a_k)\bmod p\in\mathbb Z_p.$

A standard dataset formulation is

$$D_p=\{((a_1,\dots,a_k),y): y=(a_1+\cdots+a_k)\bmod p\},$$

which is used to study both one-hidden-layer neural networks and one-layer transformers (Li et al., 2024).

The grokking literature also distinguishes regression and classification variants. In the regression version, each example is a one-hot triple $x=(e_a,e_b,e_c)\in\{0,1\}^{3p}$ with target $N$0 under squared loss. In the classification version, inputs are pairs $N$1 and targets are one-hot vectors $N$2 under cross-entropy (Mohamadi et al., 2024).

A separate line of work reformulates the task at much larger scale as

$N$3

with $N$4 up to $N$5 and prime moduli $N$6 up to $N$7. That regime is motivated by cryptographic settings in which prior work on machine learning had only handled $N$8 and $N$9, whereas systems such as CRYSTALS-Kyber involve $(x_1,\dots,x_N)\mapsto \sum_{i=1}^N x_i \bmod q$0 and $(x_1,\dots,x_N)\mapsto \sum_{i=1}^N x_i \bmod q$1 (Saxena et al., 2024).

2. Fourier structure and learned computational circuits

A central empirical and theoretical result is that modular-addition transformers organize their internal computation around Fourier features. In the one-layer theoretical construction, each hidden unit or attention head aligns with exactly one frequency $(x_1,\dots,x_N)\mapsto \sum_{i=1}^N x_i \bmod q$2, and the final linear layer recombines the resulting modes through a learned inverse DFT to recover the one-hot encoding of the sum modulo $(x_1,\dots,x_N)\mapsto \sum_{i=1}^N x_i \bmod q$3 (Li et al., 2024).

In learned depth-1 transformer circuits, the embedding matrix is approximately organized into sinusoidal blocks. For a token $(x_1,\dots,x_N)\mapsto \sum_{i=1}^N x_i \bmod q$4, the learned embedding can be analyzed as a concatenation of coordinates of the form $(x_1,\dots,x_N)\mapsto \sum_{i=1}^N x_i \bmod q$5, where $(x_1,\dots,x_N)\mapsto \sum_{i=1}^N x_i \bmod q$6. This representation makes the addition law compatible with the trigonometric identities

$(x_1,\dots,x_N)\mapsto \sum_{i=1}^N x_i \bmod q$7

so that attention and MLP components can implement addition by moving between Fourier channels and their pairwise interactions (Furuta et al., 2024).

For one-layer transformers, the attention matrices themselves admit a Fourier-factorized description. A head’s low-rank key-query product can be written as an outer product of cosines,

$(x_1,\dots,x_N)\mapsto \sum_{i=1}^N x_i \bmod q$8

which, after multiplication by one-hot inputs, realizes a frequency mode of the sum. The softmax then converts these scores into a peaked distribution, while the value stage carries the corresponding cosine feature of candidate outputs (Li et al., 2024).

This Fourier mechanism is often described as a “clock” algorithm. A common misconception is that transformers solving modular addition learn a digital carry-ripple circuit. In the one-layer four-head setting studied for full-precision, binary, and ternary networks, none of the models learn such a discrete digital circuit; instead, all three encode continuous sine/cosine channels at learned frequencies and use attention to sum those channels across the two inputs (Li, 2024).

3. Optimization, grokking, and sample-complexity structure

The modular-addition benchmark is also central to the theory of grokking. Early in training, when the kernel regime approximately holds, permutation-equivariant models behave like their NTK linearization. In that regime, no permutation-equivariant model can achieve small population error on modular addition unless it sees at least a constant fraction of all possible data points. For regression, if a kernel method is equivariant under permutations of $(x_1,\dots,x_N)\mapsto \sum_{i=1}^N x_i \bmod q$9 and the number of training triples satisfies $N=256$0, then the expected population loss obeys

$N=256$1

For classification, any permutation-equivariant kernel on $N=256$2 versus one-hot $N=256$3 needs $N=256$4 examples to beat chance (Mohamadi et al., 2024).

Late in training, the picture changes. The same work shows that models eventually escape the kernel regime, and once they do, bounded $N=256$5 solutions can generalize with substantially fewer examples. For regression, a width-$N=256$6 interpolator with $N=256$7 satisfies

$N=256$8

so $N=256$9 suffices. For classification, a width-$q=3329$0 zero-training-error solution with $q=3329$1-normalized margin within a constant factor of the maximum satisfies an error bound leading to $q=3329$2 for near-perfect generalization. Empirically, train loss reaches zero while test performance remains poor until the empirical NTK changes substantially; only after leaving the lazy regime does feature learning emerge and test accuracy jump (Mohamadi et al., 2024).

A complementary max-margin analysis gives an explicit width requirement for Fourier solutions. In the one-layer setting, if $q=3329$3 denotes width, then achieving the maximum normalized $q=3329$4 margin on $q=3329$5 requires

$q=3329$6

with each hidden neuron living on exactly one nonzero frequency. This links the learned Fourier decomposition to the implicit bias of weight-decay training at small $q=3329$7, for which global minimizers converge to a solution achieving the maximum normalized margin $q=3329$8 (Li et al., 2024).

4. Scaling to large $q=3329$9 and cryptographic moduli

The most substantial departure from the classic small-$p$0 toy setting is the encoder-only transformer training pipeline introduced for large-scale modular arithmetic. That work proposes three changes: more diverse training data, an angular embedding, and a custom loss function. The goal is to learn modular sums for values of $p$1 up to $p$2 and prime moduli $p$3 up to $p$4 (Saxena et al., 2024).

The training distribution is no longer uniform over $p$5 alone. Instead, it interleaves sparsity sampling and tail sampling. Sparsity sampling chooses the number of nonzero positions using

$p$6

creating a curriculum of easier, sparser sums. Tail sampling targets rare wrap-around cases by drawing examples whose pre-modulo average lies in the tails of the uniform sum-distribution; only a small fraction of training examples, $p$7, are drawn from this distribution (Saxena et al., 2024).

The embedding is explicitly modular-aware. Each integer $p$8 is mapped to an angle

$p$9

so that adjacency between $a_1,\dots,a_k\in\mathbb Z_p$0 and $a_1,\dots,a_k\in\mathbb Z_p$1 is preserved geometrically. The loss combines a mean-squared term with a regularizer that both blows up at the origin and encourages predictions to remain near the unit circle: $a_1,\dots,a_k\in\mathbb Z_p$2 Once $a_1,\dots,a_k\in\mathbb Z_p$3, the loss is minimized at both $a_1,\dots,a_k\in\mathbb Z_p$4 and $a_1,\dots,a_k\in\mathbb Z_p$5, so wrapping becomes a loss minimum rather than a discontinuity (Saxena et al., 2024).

The model itself is an encoder-only transformer with $a_1,\dots,a_k\in\mathbb Z_p$6 layers, hidden dimension $a_1,\dots,a_k\in\mathbb Z_p$7, and $a_1,\dots,a_k\in\mathbb Z_p$8 attention heads, with positional encodings added to the input angular embeddings. The optimizer is Adam with $a_1,\dots,a_k\in\mathbb Z_p$9, a $S=(a_1+\cdots+a_k)\bmod p\in\mathbb Z_p.$0 k-step linear warmup, and cosine learning-rate decay. Training uses batch size $S=(a_1+\cdots+a_k)\bmod p\in\mathbb Z_p.$1 per GPU on $S=(a_1+\cdots+a_k)\bmod p\in\mathbb Z_p.$2 V100 GPUs, with $S=(a_1+\cdots+a_k)\bmod p\in\mathbb Z_p.$3 examples per epoch per GPU for $S=(a_1+\cdots+a_k)\bmod p\in\mathbb Z_p.$4 epochs, and no regularization beyond the custom loss (Saxena et al., 2024).

On a held-out uniform test set of $S=(a_1+\cdots+a_k)\bmod p\in\mathbb Z_p.$5 k examples, the reported performance is: for $S=(a_1+\cdots+a_k)\bmod p\in\mathbb Z_p.$6, MSE $S=(a_1+\cdots+a_k)\bmod p\in\mathbb Z_p.$7 and $S=(a_1+\cdots+a_k)\bmod p\in\mathbb Z_p.$8 exact accuracy; for $S=(a_1+\cdots+a_k)\bmod p\in\mathbb Z_p.$9, MSE $$D_p=\{((a_1,\dots,a_k),y): y=(a_1+\cdots+a_k)\bmod p\},$$0 and $$D_p=\{((a_1,\dots,a_k),y): y=(a_1+\cdots+a_k)\bmod p\},$$1 exact; for $$D_p=\{((a_1,\dots,a_k),y): y=(a_1+\cdots+a_k)\bmod p\},$$2, MSE $$D_p=\{((a_1,\dots,a_k),y): y=(a_1+\cdots+a_k)\bmod p\},$$3 and $$D_p=\{((a_1,\dots,a_k),y): y=(a_1+\cdots+a_k)\bmod p\},$$4 exact; for $$D_p=\{((a_1,\dots,a_k),y): y=(a_1+\cdots+a_k)\bmod p\},$$5, MSE $$D_p=\{((a_1,\dots,a_k),y): y=(a_1+\cdots+a_k)\bmod p\},$$6 and $$D_p=\{((a_1,\dots,a_k),y): y=(a_1+\cdots+a_k)\bmod p\},$$7 exact; and for $$D_p=\{((a_1,\dots,a_k),y): y=(a_1+\cdots+a_k)\bmod p\},$$8, MSE $$D_p=\{((a_1,\dots,a_k),y): y=(a_1+\cdots+a_k)\bmod p\},$$9 and $x=(e_a,e_b,e_c)\in\{0,1\}^{3p}$0 exact, while $x=(e_a,e_b,e_c)\in\{0,1\}^{3p}$1 accuracy remains $x=(e_a,e_b,e_c)\in\{0,1\}^{3p}$2. Baseline MLPs or transformers trained on uniform data alone fail, with exact accuracy approximately $x=(e_a,e_b,e_c)\in\{0,1\}^{3p}$3 even on the $x=(e_a,e_b,e_c)\in\{0,1\}^{3p}$4 problem. The same approach retains $x=(e_a,e_b,e_c)\in\{0,1\}^{3p}$5 accuracy of approximately $x=(e_a,e_b,e_c)\in\{0,1\}^{3p}$6 at $x=(e_a,e_b,e_c)\in\{0,1\}^{3p}$7 (Saxena et al., 2024).

5. Mechanistic interpretability, compression, and representational geometry

Mechanistic work on modular-addition transformers has progressively moved from neuron-by-neuron inspection toward analytic and geometric descriptions of full circuits. In one-layer “pizza” transformers, the MLP can be rigorously interpreted as a quadrature scheme. Under an infinite-width lens, the ReLU MLP

$x=(e_a,e_b,e_c)\in\{0,1\}^{3p}$8

is approximated by an integral kernel, and each neuron corresponds to a rectangle in a numerical-integration rule for a trigonometric identity. The resulting Riemann-sum error satisfies

$x=(e_a,e_b,e_c)\in\{0,1\}^{3p}$9

and, for roughly uniform spacing, $N$00. In the trained pizza MLP analyzed in that work, $N$01 and $N$02 yield $N$03, while the relative error is approximately $N$04, providing a non-vacuous proof in $N$05 time (Yip et al., 2024).

A second development replaces local circuit sketches with manifold-level descriptions. Earlier interpretations had associated trainable-attention models with “Clock” circuits and uniform-attention models with “Pizza” circuits. A later geometric-topological analysis argues that this is not the case: both architectures implement the same algorithm via topologically and geometrically equivalent representations. The first-layer frequency-specific manifolds are modeled either as a filled $N$06-D disc or as a discrete torus $N$07, and deeper layers collapse these structures to the circle of correct-sum logits. Across $N$08 independent seeds of each architecture, principal-component geometry, phase-alignment distributions, MMD tests, and persistent homology are used to argue that the learned circuits are equivalent at the representation level rather than fundamentally distinct (Moisescu-Pareja et al., 31 Dec 2025).

A further unification is the “approximate Chinese Remainder Theorem” interpretation. In that account, neurons activate exclusively on approximate cosets, frequency clusters behave like modular subsystems, and the full network aggregates a small set of frequencies to isolate the correct output. The theory predicts that universally learned solutions in deep networks with trainable embeddings or more than one hidden layer require only $N$09 features, and the paper reports empirical confirmation across MLPs and transformers (McCracken et al., 23 May 2025).

6. Architectural variants, robustness, and extensions beyond plain addition

The modular-addition transformer literature also serves as a comparative laboratory for architecture and representation. In binary, ternary, and full-precision one-layer four-head transformers trained on modulus $N$10, all models learn similar algorithms rather than qualitatively different ones. Binary BitNet and ternary BitNet $N$11 approximate the same Fourier “clock” mechanism, and the study concludes that quantization does not make the learned solution fundamentally more interpretable; binary networks in fact exhibit a higher Fourier noise floor than full-precision models (Li, 2024).

A distinct extension studies text-based modular arithmetic at the character level. For $N$12, a two-layer transformer with learned absolute positional embeddings can achieve high in-distribution accuracy yet fail catastrophically under position shift or out-of-distribution natural-language templates. The identified failure mode is that a baseline model is near-perfect at one absolute position but collapses when the same arithmetic expression is shifted. A training recipe combining explicit expression boundary markers, a steps-based position curriculum, diverse template mixtures, and multi-variant consistency regularization substantially improves robustness while maintaining high in-distribution accuracy; an ALiBi-style ablation fails to learn the task under that setup (Yudin, 7 Jan 2026).

Modular addition also functions as a source task for transfer and for comparison with other modular operators. In modular polynomials and related arithmetic, Fourier Frequency Sparsity and Fourier Coefficient Ratio are proposed as progress measures. Addition is characterized by a sharp drop in Fourier Frequency Sparsity while Fourier Coefficient Ratio stays approximately $N$13, whereas other operators exhibit different signatures. Transfer from grokked addition is limited rather than universal: pre-grokked addition models help on linear expressions such as $N$14, but transferability is reported only for specific combinations, and challenging non-factorizable polynomials do not exhibit equally clear patterns (Furuta et al., 2024).

At larger scale, the same large-$N$15 training pipeline used for modular addition extends to other modular functions $N$16, including

$N$17

for which the reported exact accuracy exceeds $N$18 on $N$19. This suggests a broader role for modular-aware embeddings, losses, and training distributions in neural computation over finite rings, including tasks motivated by Learning With Errors and related cryptosystems (Saxena et al., 2024).