Four-Digit Addition Analysis

• Four-digit addition (FDA) is a structured operation on 4-digit vectors where standard addition computes sums with carry using base-dependent digit interpretation.
• Its algebraic formulation allows various group laws, ranging from full carry to carry-free operations, by partitioning digit vectors into contiguous blocks.
• Transformer studies of FDA reveal digit-wise subroutines and carry propagation mechanisms achieving over 95% accuracy, highlighting both computational strengths and edge-case challenges.

Four-digit addition (FDA) denotes the addition of two 4-digit quantities, most commonly in base 10, but it also admits a more general formalization as an abelian-group law on the digit-vector space $A_b^4$. In algebraic treatments, FDA distinguishes full integer addition with carry from coordinatewise modular addition and mixed blockwise operations; in mechanistic interpretability, it serves as a clear probe of whether Transformers encode carries, digit structure, and output selection algorithmically rather than through superficial pattern matching (Hellekalek, 2012, Quirke et al., 2023, Yan, 9 Jun 2025).

1. Formal definition and representational conventions

Let $b \ge 2$ be an integer base and let

$A_b = \{0,1,\dots,b-1\}.$

The set of length-4 digit vectors is

$A_b^4 = A_b \times A_b \times A_b \times A_b.$

An element is written as

$x=(x_0,x_1,x_2,x_3), \qquad x_i \in A_b,$

with the least significant digit first. The corresponding integer interpretation is given by

$$\operatorname{int}_b(x_0,x_1,x_2,x_3)=x_0+x_1b+x_2b^2+x_3b^3,$$

and $\operatorname{dig}_b$ denotes the inverse map from residues modulo $b^4$ back to 4-digit vectors. Standard FDA is then

$$x+y=\operatorname{dig}_b\bigl(\operatorname{int}_b(x)+\operatorname{int}_b(y)\pmod{b^4}\bigr),$$

which is addition with carry across all four positions. The contrasting operation is coordinatewise addition without carry,

$$(x_0,x_1,x_2,x_3)+(y_0,y_1,y_2,y_3)=((x_0+y_0)\bmod b,\dots,(x_3+y_3)\bmod b).$$

For prime base $b \ge 2$0, these are the two basic types of digit-vector addition: addition with carry and addition without carry, also called XOR-addition in the binary case (Hellekalek, 2012).

2. Algebraic classification of four-digit additions

The algebraic theory of FDA is not limited to the usual schoolbook operation. For prime base $b \ge 2$1, every abelian group of order $b \ge 2$2 is isomorphic to a product

$b \ge 2$3

To realize such a group law on $b \ge 2$4, one chooses a composition of $b \ge 2$5, cuts the digit vector into contiguous blocks of those lengths, and performs integer addition with carry inside each block. A block of length $b \ge 2$6 is simply addition without carry. The eight compositions of $b \ge 2$7 are

$b \ge 2$8

Because the number of compositions of $b \ge 2$9 is $A_b = \{0,1,\dots,b-1\}.$0, there are at least

$A_b = \{0,1,\dots,b-1\}.$1

different additions on $A_b = \{0,1,\dots,b-1\}.$2 arising from compositions alone. Standard FDA corresponds to the single-block composition $A_b = \{0,1,\dots,b-1\}.$3, hence to $A_b = \{0,1,\dots,b-1\}.$4; fully carry-free FDA corresponds to $A_b = \{0,1,\dots,b-1\}.$5, hence to $A_b = \{0,1,\dots,b-1\}.$6. The intermediate compositions, such as $A_b = \{0,1,\dots,b-1\}.$7, permit carries within blocks but block them across boundaries. The same paper further shows that, for prime base $A_b = \{0,1,\dots,b-1\}.$8, compositions together with automorphisms of the residue groups generate

$A_b = \{0,1,\dots,b-1\}.$9

different additions on $A_b^4 = A_b \times A_b \times A_b \times A_b.$0 (Hellekalek, 2012).

3. FDA as an algorithm in one-layer Transformers

Mechanistic work on a one-layer Transformer trained for $A_b^4 = A_b \times A_b \times A_b \times A_b.$1-digit integer addition characterizes addition as a composition of local subtasks rather than as a monolithic operation. The paper explicitly trains and analyzes 5-, 10-, and 15-digit models, and states that for a 4-digit addition task the same algorithm would apply. In the 4-digit specialization, the model receives

$A_b^4 = A_b \times A_b \times A_b \times A_b.$2

and predicts a 5-digit answer

$A_b^4 = A_b \times A_b \times A_b \times A_b.$3

The discovered subroutines are BaseAdd, MakeCarry1, UseSum9, UseCarry1, and UseSum9Cascade. In summarized form, the current answer digit is computed as

$A_b^4 = A_b \times A_b \times A_b \times A_b.$4

where the effective carry depends on whether the previous column produced a carry and, in rare cases, whether a carry must propagate through a column whose digit pair sums to $A_b^4 = A_b \times A_b \times A_b \times A_b.$5. With three heads, the roles separate cleanly: one head attends to the current digit pair and computes BaseAdd, one attends to the previous digit pair and computes MakeCarry1 and UseSum9, and one attends to the previous-previous digit pair and computes an additional carry signal used in cascade logic. The MLP then combines these features as a $A_b^4 = A_b \times A_b \times A_b \times A_b.$6 key space, yielding a 60-way key-value mechanism for digit prediction (Quirke et al., 2023).

4. Carry propagation, parallel digit streams, and edge cases

The one-layer Transformer does not implement a human-style sequential right-to-left routine with a persistent carry register. Instead, it uses parallel digit-specific streams: each answer digit is computed in a dedicated row after the full question has been revealed, and the necessary information is re-read from the input by attention. This gives a time-dense circuit in which current, previous, and previous-previous digit pairs suffice for most cases. The boundary digits are treated differently: the units digit requires only BaseAdd, while the most significant answer digit is learned fastest because, in $A_b^4 = A_b \times A_b \times A_b \times A_b.$7-digit plus $A_b^4 = A_b \times A_b \times A_b \times A_b.$8-digit addition, it is constrained to be $A_b^4 = A_b \times A_b \times A_b \times A_b.$9 or $x=(x_0,x_1,x_2,x_3), \qquad x_i \in A_b,$0 (Quirke et al., 2023).

The principal failure mode is cascading UseSum9. Simple sum-to-9 cases are handled correctly, but cases in which a carry must propagate through multiple consecutive 9-columns remain weak and error-prone. Examples include $x=(x_0,x_1,x_2,x_3), \qquad x_i \in A_b,$1 and, in the four-digit regime, extreme patterns such as $x=(x_0,x_1,x_2,x_3), \qquad x_i \in A_b,$2. The paper attributes this to the fixed-depth, left-to-right, one-layer architecture: the model can look back two digit positions reliably, but longer carry chains are structurally difficult. For FDA, this implies that typical inputs are tractable, while the hardest cases are those in which carries traverse several adjacent columns whose local sums equal $x=(x_0,x_1,x_2,x_3), \qquad x_i \in A_b,$3 (Quirke et al., 2023).

5. Layer-wise FDA trajectories in LLaMA-3-8B-Instruct

A separate line of work probes LLaMA-3-8B-Instruct on multi-digit addition using the prompt format "Calculate: num1 + num2 = ", always reading the hidden state of the trailing space before the first answer token is generated. The model has 32 transformer layers, achieves overall arithmetic accuracy of 97.29%, and attains 95.51% accuracy on four-digit addition. The central result is a four-stage trajectory in the forward pass. First, formula-structure representations become linearly decodable: probes trained only on 2-digit operands decode the expression type of 4-digit additions with nearly 100% accuracy by about layer 14, even though 4-digit problems are out of distribution for probe training. Second, core computational features emerge: in three-digit settings, carry detection begins a reliable upward trend around layer 14 and is near 100% by layer 19, while exact sum-class prediction rises sharply around layer 16 and is approximately 100% by layer 19 and above. Third, numerical abstractions of the result become clearer: for three-digit sums, hundreds, tens, and units digits become increasingly decodable from about layers 19–25, and by layer 28 and beyond all three positions are decoded with high, stable accuracies, near perfect in practice. Fourth, final content organization and token selection occur near the output: in logit-lens analysis of 3-digit$x=(x_0,x_1,x_2,x_3), \qquad x_i \in A_b,$43-digit$x=(x_0,x_1,x_2,x_3), \qquad x_i \in A_b,$53-digit problems, the earliest layer at which the correct first result token becomes top-1 always lies between layers 23 and 31, with a peak at layer 30 where about 45.4% of examples first become correct (Yan, 9 Jun 2025).

For FDA specifically, two quantitative observations are explicit: four-digit answer accuracy is 95.51%, and formula-type decodability for 4-digit expressions is nearly perfect by layer 14. The paper does not train separate probes for a 4-digit result’s thousands place. This suggests, rather than directly proves, that the digit-wise abstraction observed for three-digit outputs extends to thousands, hundreds, tens, and ones in four-digit sums. The same caution applies to carry structure beyond the hundreds place: the three-digit carry probes show coherent positional carry signals, and a plausible implication is that four-digit carries are represented by the same mechanism repeated over one additional position (Yan, 9 Jun 2025).

6. Generalization, limitations, and scientific significance

FDA occupies an unusual position: it is simultaneously elementary arithmetic, a finite-group classification problem, and a mechanistic benchmark for sequence models. In LLaMA-3-8B-Instruct, accuracy remains above 93% for 1- to 6-digit addition except for a slight drop at 5 digits, and the structural probes trained on 2-digit data generalize almost perfectly to 4-digit expressions. In the same study, a hundreds-digit probe trained on addition generalizes to multiplication with peak accuracy above $x=(x_0,x_1,x_2,x_3), \qquad x_i \in A_b,$6 and to subtraction with peak accuracy around $x=(x_0,x_1,x_2,x_3), \qquad x_i \in A_b,$7–$x=(x_0,x_1,x_2,x_3), \qquad x_i \in A_b,$8, indicating partially operation-general digit representations. These observations support the paper’s interpretation of a hierarchical process that favors internal computation over rote memorization, while also underscoring that probe and logit-lens evidence is correlational rather than causal (Yan, 9 Jun 2025).

The limitations are equally clear. The LLaMA study restricts itself to the single prompt format "Calculate: a+b = " and does not test alternative phrasings, vertical layouts, or spacing variations. Its explicit carry and digit probes are conducted on three-digit outputs, not four-digit outputs. The one-layer Transformer study does not explicitly train a 4-digit model, although it states that the same algorithm would apply, and its failure on cascading UseSum9 cases shows that correct local behavior does not guarantee exact handling of all edge cases. The algebraic classification, for its part, clarifies that “four-digit addition” is not uniquely defined once one abstracts away from standard integer interpretation: only the single-block composition $x=(x_0,x_1,x_2,x_3), \qquad x_i \in A_b,$9 yields ordinary integer addition modulo $$\operatorname{int}_b(x_0,x_1,x_2,x_3)=x_0+x_1b+x_2b^2+x_3b^3,$$0; all other group laws on $$\operatorname{int}_b(x_0,x_1,x_2,x_3)=x_0+x_1b+x_2b^2+x_3b^3,$$1 alter carry propagation by construction (Hellekalek, 2012, Quirke et al., 2023, Yan, 9 Jun 2025).

Taken together, these results position FDA as a compact but technically rich object of study. In algebra, it exemplifies how carry structure determines the group law on digit vectors. In mechanistic interpretability, it exposes whether a model has learned structure recognition, carry computation, digit-wise abstraction, and late-stage token commitment. In both settings, the central issue is the same: whether the apparent simplicity of four-digit sums conceals a genuinely structured computation.