Digit-Position-Specific Circuits in Transformers

• Digit-position-specific circuits are modular arithmetic subgraphs in transformer MLPs that compute separate digit outputs (units, tens, hundreds) using position-aware activations.
• The identification employs Fisher scoring on neuron activations to isolate neurons selective for specific digit-pair tasks, confirming their role through robust intervention tests.
• Interventional studies show that manipulating these circuits shifts only the corresponding output digit, evidencing a compositional and parallel approach to arithmetic computation.

Searching arXiv for the specific papers and closely related mechanistic-interpretability work on position-aware circuit discovery and arithmetic circuits. Digit-position-specific circuits are position-localized arithmetic subcircuits in decoder-only transformers: subsets of MLP neurons, distributed across a contiguous layer range, that are highly selective for computing a single output digit and that causally control that digit of the model’s prediction. In the reported 3-digit addition and subtraction setting, the units, tens, and hundreds places are implemented by three largely independent subroutines, each corresponding to a modular arithmetic subtask such as $(a_k+b_k)\bmod 10$ for digit position $k$ (Baeumel et al., 4 Aug 2025). Within mechanistic interpretability, these circuits exemplify a broader shift from position-invariant analyses toward explicitly position-aware accounts of computation, in which the same model component can have materially different functional roles at different positions (Haklay et al., 7 Feb 2025).

1. Definition and mathematical formulation

For a 3-digit integer $A$, the decomposition used in the arithmetic analysis is

$$A = a_2\cdot 10^2 + a_1\cdot 10^1 + a_0\cdot 10^0,\qquad a_k\in\{0,\dots,9\}.$$

Given two such integers $A$ and $B$, their sum $S=A+B$ decomposes into digits $s_k$. In the absence of carries,

$s_k = (a_k+b_k)\bmod 10,\qquad k\in\{0,1,2\}.$

More generally,

$S \bmod 10^k = \sum_{j=0}^{k-1} s_j\cdot 10^j,$

so the units digit is $k$0, the tens digit is $k$1, and the hundreds digit is $k$2 (Baeumel et al., 4 Aug 2025).

A digit-position-specific circuit is defined as a subset of MLP neurons spread across a contiguous range of layers $k$3 such that the neurons are highly selective for the subtask “compute $k$4” at digit position $k$5, and causally control exactly that digit of the output. In the reported formulation, there is one circuit for the units place $k$6, one for the tens place $k$7, and one for the hundreds place $k$8 (Baeumel et al., 4 Aug 2025).

The central empirical claim is not merely that digit information is represented, but that the circuits run in parallel and independently. Intervening on the tens circuit shifts only the predicted tens digit, leaving units and hundreds untouched. This distinguishes the proposed mechanism from a monolithic arithmetic module and from explanations based purely on whole-number memorization (Baeumel et al., 4 Aug 2025).

2. Identification by supervised feature importance

The reported identification procedure begins from datasets $k$9 and $A$0 of 1,000 prompts each, all chosen so that no carry occurs in any digit. For each MLP neuron $A$1 in each layer $A$2, the activation $A$3 is collected on every prompt $A$4, where $A$5. Prompts are then grouped by their digit-pair class at position $A$6, with classes $A$7. For example, for “347+231” the unit-digit class is $A$8 (Baeumel et al., 4 Aug 2025).

Let $A$9 denote the subset of prompts with class $$A = a_2\cdot 10^2 + a_1\cdot 10^1 + a_0\cdot 10^0,\qquad a_k\in\{0,\dots,9\}.$$0 at digit position $$A = a_2\cdot 10^2 + a_1\cdot 10^1 + a_0\cdot 10^0,\qquad a_k\in\{0,\dots,9\}.$$1. The quantities used for neuron scoring are

$$A = a_2\cdot 10^2 + a_1\cdot 10^1 + a_0\cdot 10^0,\qquad a_k\in\{0,\dots,9\}.$$2

$$A = a_2\cdot 10^2 + a_1\cdot 10^1 + a_0\cdot 10^0,\qquad a_k\in\{0,\dots,9\}.$$3

and

$$A = a_2\cdot 10^2 + a_1\cdot 10^1 + a_0\cdot 10^0,\qquad a_k\in\{0,\dots,9\}.$$4

The Fisher Score for neuron $$A = a_2\cdot 10^2 + a_1\cdot 10^1 + a_0\cdot 10^0,\qquad a_k\in\{0,\dots,9\}.$$5 at digit position $$A = a_2\cdot 10^2 + a_1\cdot 10^1 + a_0\cdot 10^0,\qquad a_k\in\{0,\dots,9\}.$$6 is then

$$A = a_2\cdot 10^2 + a_1\cdot 10^1 + a_0\cdot 10^0,\qquad a_k\in\{0,\dots,9\}.$$7

This score is large when a neuron’s activation varies strongly across different values of the $$A = a_2\cdot 10^2 + a_1\cdot 10^1 + a_0\cdot 10^0,\qquad a_k\in\{0,\dots,9\}.$$8-th digit-pair while remaining internally consistent within each class (Baeumel et al., 4 Aug 2025).

A threshold $$A = a_2\cdot 10^2 + a_1\cdot 10^1 + a_0\cdot 10^0,\qquad a_k\in\{0,\dots,9\}.$$9, described as a hyperparameter in $A$0, defines the candidate circuit

$A$1

In the reported LLaMA3-8B setting, $A$2 for units, $A$3 for tens, and $A$4 for hundreds yielded a compact yet complete circuit (Baeumel et al., 4 Aug 2025).

This procedure operationalizes digit-position specificity at the neuron level. Rather than searching for generic “arithmetic neurons,” it isolates neurons whose variance structure is aligned with a particular digit-pair subproblem. A plausible implication is that the relevant modularity is statistical before it is causal: the Fisher-based screening identifies candidate subcircuits whose selectivity can then be tested interventionally.

3. Causal verification by interchange interventions

The causal test uses paired prompts, denoted “base” and “source,” that share two of the three digit-pair subtasks and differ in the remaining one. The example provided is:

• Base: “347 + 231 = 578” with subtasks $A$5hundreds: $A$6, tens: $A$7, units: $A$8
• Source: “261 + 512 = 773” with subtasks $A$9hundreds: $B$0, tens: $B$1, units: $B$2 (Baeumel et al., 4 Aug 2025)

At each layer $B$3, the activations of neurons $B$4 in the base prompt are replaced with their corresponding activations from the source prompt. After the forward pass, the model’s distribution is examined over all eight digit-wise variants of the result, including $B$5 (base-only), $B$6 (hundreds from source), $B$7 (tens from source), and $B$8 (units from source). If $B$9 implements digit position $S=A+B$0, probability mass should shift primarily from $S=A+B$1 to the single variant that swaps exactly digit $S=A+B$2 (Baeumel et al., 4 Aug 2025).

The reported intervention effects are strongly digit-selective. For the hundreds circuit of LLaMA3-8B with $S=A+B$3, intervening increased $S=A+B$4 by $S=A+B$5 percentage points while leaving $S=A+B$6 and $S=A+B$7 essentially unchanged. For the unit circuit with $S=A+B$8, intervening increased $S=A+B$9 by $s_k$0 points and drove $s_k$1 down by $s_k$2 points, with negligible effects on tens or hundreds. The “flip rate,” defined as the fraction of base examples for which $s_k$3 after intervention, reached $s_k$4 for units, $s_k$5 for tens, and $s_k$6 for hundreds in LLaMA3-8B (Baeumel et al., 4 Aug 2025).

The same section reports an Olmo 2 7B result for the tens circuit with $s_k$7:

$s_k$8

with $s_k$9 pp on all other variants. The paper interprets these patterns as showing a one-to-one causal link between a targeted digit-circuit intervention and the corresponding output digit (Baeumel et al., 4 Aug 2025).

These interventions are significant because they move beyond correlation. The reported selectivity is not merely that circuit neurons carry information about a digit position, but that transplanting those activations shifts precisely that output coordinate.

4. Compositional organization across layers

Once candidate circuits are identified, their layer-wise distribution provides a second level of structure. In multi-digit tokenization models, operand information is reported to first reach the final token’s residual stream in mid-layers, at approximately $s_k = (a_k+b_k)\bmod 10,\qquad k\in\{0,1,2\}.$0–$s_k = (a_k+b_k)\bmod 10,\qquad k\in\{0,1,2\}.$1 in LLaMA3-8B. Searching MLP layers from $s_k = (a_k+b_k)\bmod 10,\qquad k\in\{0,1,2\}.$2, roughly $s_k = (a_k+b_k)\bmod 10,\qquad k\in\{0,1,2\}.$3 of neurons per layer exceed $s_k = (a_k+b_k)\bmod 10,\qquad k\in\{0,1,2\}.$4, indicating wide circuits rather than sparse single-neuron pathways (Baeumel et al., 4 Aug 2025).

The reported circuits are also persistent. Tens- and hundreds-digit circuits extend deep into the network, while the tens circuit slightly shrinks after $s_k = (a_k+b_k)\bmod 10,\qquad k\in\{0,1,2\}.$5, which the paper interprets as suggesting that the model begins to combine digit-level subtasks after that point. This suggests a staged computation in which local digit subroutines are maintained over several layers before later aggregation (Baeumel et al., 4 Aug 2025).

Distinctiveness is quantified through overlap. For $s_k = (a_k+b_k)\bmod 10,\qquad k\in\{0,1,2\}.$6, the observed overlap between any two digit circuits in a given layer is less than $s_k = (a_k+b_k)\bmod 10,\qquad k\in\{0,1,2\}.$7, which is presented as evidence for modularity. Sufficiency is assessed by training an LDA classifier on activations from only the circuit neurons: classification accuracy remains within 1–3 points of an LDA trained on the full MLP activation. The reported conclusion is that the circuit neurons are not only necessary but also largely sufficient for distinguishing the 100 possible digit-pairs (Baeumel et al., 4 Aug 2025).

A qualitative study of the top-20 neurons by Fisher Score per circuit found intelligible digit-level heuristics embedded inside the larger modules. The examples given include a unit-circuit neuron that fires strongly whenever $s_k = (a_k+b_k)\bmod 10,\qquad k\in\{0,1,2\}.$8, another neuron that fires whenever the hundreds digit of the result is in $s_k = (a_k+b_k)\bmod 10,\qquad k\in\{0,1,2\}.$9, and another whenever the second operand’s tens digit equals $S \bmod 10^k = \sum_{j=0}^{k-1} s_j\cdot 10^j,$0. The paper emphasizes that these heuristic neurons reside in the circuit corresponding to their digit (Baeumel et al., 4 Aug 2025).

5. Generality, robustness, and relation to position-aware circuit discovery

The arithmetic results are reported across model families and tokenization schemes. LLaMA3-8B, LLaMA3-70B, Olmo 2 7B, and Gemma 2 9B all exhibit the same three circuits, with consistent effect sizes and flip rates. The only reported difference is that single-digit tokenization models such as Gemma generate one digit per forward pass and therefore show a dominant circuit for the next digit and a smaller circuit for lookahead of the carry (Baeumel et al., 4 Aug 2025).

The same work argues that digit-wise computation is robust to tokenization. LLaMA models may encode “347” as one token, yet they still perform addition digit by digit; Gemma tokenizes each digit separately and also exhibits the same modular structure. Carry perturbations provide a further robustness test: even when the source prompt introduces a nonzero carry bit, intervention on $S \bmod 10^k = \sum_{j=0}^{k-1} s_j\cdot 10^j,$1 still flips only the units digit with minimal effect on the tens or predicate carry-adjusted values. The paper interprets this as suggesting that carry logic is handled elsewhere and that digit circuits operate locally (Baeumel et al., 4 Aug 2025).

This locality connects naturally to position-aware circuit discovery more broadly. Haklay et al. define a circuit as a minimal subgraph of a model’s computation graph that executes a specific task, and identify a limitation of prior automatic methods: they aggregate importance across token positions, assuming position invariance. Their analysis shows that such aggregation can cause cancellation and can overestimate broadly weak edges relative to narrowly strong ones. Position-aware edge attribution patching instead scores attention sub-edges separately for each position pair, and dataset schemas permit analogous discovery on variable-length prompts (Haklay et al., 7 Feb 2025).

The broader relevance to digit-position-specific circuits is methodological. Position-aware discovery explicitly distinguishes the same model component at different input positions, enabling circuits that reflect interactions such as “digit-4 attends to digit-1” in greater-than comparison. In the reported Greater-Than case, a non-positional circuit required approximately 1,000 edges to reach $S \bmod 10^k = \sum_{j=0}^{k-1} s_j\cdot 10^j,$2, whereas the position-aware method found a 50-edge circuit. In GPT2-small IOI, aggregating across positions changed the top 1% ranked edges by 17%, compared with 4% in a control condition (Haklay et al., 7 Feb 2025). This suggests that digit-position-specific arithmetic circuits belong to a wider family of mechanisms whose explanatory precision depends on retaining positional structure.

6. Interpretation, misconceptions, and open directions

A common misconception is that arithmetic behavior in LLMs should be explained either by memorization of entire answers or by a single diffuse “adder” distributed without clear internal boundaries. The reported evidence instead supports a compositional account: for 3-digit addition and subtraction, the model implements three modular routines that compute digit-local subtasks in parallel, and targeted interventions flip exactly the intended output digit (Baeumel et al., 4 Aug 2025).

The interpretive significance for mechanistic interpretability is explicit in the source material. The arithmetic study presents digit-position-specific circuits as a clearer target than a search over many loosely defined heuristics. Because the circuits are described as causally necessary for digit outputs, the paper proposes that they could serve as targets for model editing and robustness interventions, for example by fine-tuning only $S \bmod 10^k = \sum_{j=0}^{k-1} s_j\cdot 10^j,$3 neurons to correct systematic errors. It also identifies multiplication, division, and multi-digit carries as natural extensions of the same methodology, combining Fisher-based neuron selection with targeted interventions (Baeumel et al., 4 Aug 2025).

The position-aware circuit literature suggests a further methodological caution. If positional distinctions are collapsed too early, minimal mechanisms can appear larger, less faithful, or less precise than they are. For arithmetic, this matters because the object of explanation is not merely “number processing,” but computation tied to specific digit positions and, potentially, to specific cross-position interactions. A plausible implication is that future work on arithmetic circuits will need to combine neuron-level digit selectivity with graph-level position awareness in order to separate local digit computation from carry-management pathways (Haklay et al., 7 Feb 2025).

Taken together, these results characterize digit-position-specific circuits as modular arithmetic subgraphs with both statistical selectivity and causal efficacy. They provide evidence that symbolic-looking behavior in LLMs can be decomposed into position-local computations rather than treated as an undifferentiated capability (Baeumel et al., 4 Aug 2025).