Repeated-Token Counting Reveals a Dissociation Between Representations and Outputs

Abstract: LLMs fail at counting repeated tokens despite strong performance on broader reasoning benchmarks. These failures are commonly attributed to limitations in internal count tracking. We show this attribution is wrong. Linear probes on the residual stream decode the correct count with near-perfect accuracy at every post-embedding layer, across all model depths. This holds even at the exact layers where the wrong answer crystallizes while the model simultaneously outputs an incorrect count. Attention patterns show no evidence of collapse over repeated tokens and tokenization artifacts account for none of the failure. Instead, a format-triggered multi-layer perceptron (MLP) block overwrites the correctly-encoded count with a fixed wrong answer at roughly 88--93,% network depth. This prior fires for repeated word-tokens in space-separated list format and is absent for repeated digit-tokens. It is suppressed by comma-separated delimiters in larger models but persists in smaller ones. The finding holds across Llama-3.2 (1B and 3B) and Qwen2.5 (1.5B, 3B and 7B) at consistent relative depth. Counting failure is a failure of routing not of representation and the two require different interventions.

• The paper demonstrates that LLMs correctly encode token counts internally while misrouting this information due to format-specific MLP block errors.
• It uses mechanistic interpretability methods like linear probes and residual stream decomposition to pinpoint deterministic failure layers in the network.
• Findings suggest that addressing counting errors requires circuit-level interventions rather than improvements in the internal representation.

Dissociation Between Internal Representations and Output Routing in Repeated-Token Counting Tasks

Background and Motivation

This paper systematically investigates systematic failures of LLMs when tasked with counting repeated tokens, a simple yet foundational list-processing operation. Despite robust performance on reasoning benchmarks and extensive prior attribution of counting failures to deficiencies in internal representation or sequence-length estimation [Hasani et al., 2026; Zhang et al., 2024], the present work demonstrates that these attribution schemas are incomplete. Using mechanistic interpretability (linear probes, logit lens, residual stream decomposition), the authors provide empirical evidence that models encode the correct count internally throughout the network’s depth. The source of behavioral failure is in how this information is routed to the output.

Experimental Framework

The experiments encompass five instruction-tuned, decoder-only models: Llama-3.2-1B, Llama-3.2-3B, Qwen2.5-1.5B, Qwen2.5-3B, and Qwen2.5-7B, spanning two architectures and three scales. Deterministic evaluation via greedy decoding enables fine-grained tracing of behavioral priors. The repeated-token counting task and control variants rigorously test counting across surface formats (space vs. comma separation), token types (alphabetic vs. digit), and anomaly insertion (single intruder at variable positions). All payloads are verified for unambiguous token count (ruling out tokenization artifacts).

Core Findings

Internal Representation of Count

Across all tested models, the correct count is linearly decodable from the residual stream after the first transformer block and remains available at every subsequent layer. Ridge regression probes yield MAE $<0.25$ and $R^2>0.99$ for repeated-token cases across Llama and Qwen variants; unique-token prompts are linearly encoded with slightly higher error.

Behavioral Failure Mechanism

For models exhibiting counting failure (Llama-3.2-1B, Llama-3.2-3B, Qwen2.5-1.5B), the wrong output emerges at a deterministic layer—an MLP block at 79–93% of network depth. This MLP is format-triggered: upon recognizing a repeated word-token list in space-separated format, it overwrites the count representation with a fixed, incorrect value. The prior is not engaged for digit-tokens; these models correctly count repeated digits, pinpointing a task-format-specific circuit.

Format sensitivity is both model- and scale-dependent. In some cases (Llama-3.2-3B, Qwen2.5-1.5B), comma separation suppresses the prior and enables correct output. In Llama-3.2-1B, the MLP prior persists across formats, indicating incomplete format recognition in smaller scale models.

Attention and Routing

Attention mechanisms do not collapse or concentrate abnormally on repeated tokens; entropy and uniformity statistics are nearly identical across conditions. The behavioral attractor arises not from a loss of representational diversity in attention, but downstream routing.

In larger Qwen models that solve counting (Qwen2.5-3B, Qwen2.5-7B), internal attention correctly detects anomalies (over-attending banana in single-intruder cases) but fails to propagate this detection to the output—demonstrating routing failure at the attention-to-output pathway rather than the residual stream.

Distribution and Generalization of the Prior

The MLP prior does not generalize beyond repeated word-token lists. Paraphrase and symbol substitution experiments reveal that format and symbol specificity are key; digit-token lists (e.g., "1 1 1 ...") bypass the prior entirely, whereas word-token lists ("apple apple ...") trigger deterministic errors.

Ablation studies confirm the causal role of identified MLP blocks but show distributed priors: disabling a single MLP writer simply unmasks downstream writers and does not restore correct count output.

Scale Thresholds and Model Family Regularities

Counting failures and prior activation thresholds are model-specific. The lock-in depth for the fixed wrong answer is consistent (78–93% network depth) across Llama and Qwen families, indicating architectural regularity. Qwen2.5-3B and Qwen2.5-7B—which solve counting—nonetheless fail at anomaly detection, with correction only possible for multiple intruders, consistent with a scale-dependent recency bias in attending to the tail of the sequence.

Numerical Results and Contradictory Claims

• Linear probes achieve $R^2>0.99$ from L01 for repeated-token prompts in every model, even at layers where output is incorrect.
• Behavioral failure is deterministic: all seeds yield identical outputs under greedy decoding across all tested models.
• MLP block lock-in of wrong answers occurs at $78$–$93\%$ depth in failing models.
• Format-specific corrections: comma separation rescues behavioral output in some models, but not universally; format triggers are model- and scale-specific.
• Digit-tokens are correctly counted where word-tokens are not, despite identical task framing.

The claim that LLMs fail counting due to internal representation limitations is directly contradicted: the encoded count is provably present in model internals.

Implications and Future Directions

This work reframes behavioral failure in LLMs as predominantly a routing issue rather than a representational deficit. This has immediate implications:

• Intervention strategy: Correcting counting failures demands circuit-level identification and modification of format-dependent MLP writers, not architectural overhauls or training of internal representations.
• Generalization limits: Because routing failures are circuit-specific, evaluation of LLM reliability must include tests for format triggers, token-type switches, and paraphrase variation.
• Mechanistic interpretability: This methodology provides a scalable template for locating and correcting subtle routing failures for other reasoning tasks.
• Prompt engineering: Surface format and delimiter selection can act as adversarial attacks or repairs on behavioral priors, demanding further study of prompt variants.
• Theory: The dissociation between internal representation and behavioral output necessitates a revision of how LLMs are evaluated and debugged in AI safety and reliability contexts.

Long-term, these findings suggest that as models scale and architecture complexity increases, the locus of routing failures may shift—from residual stream overwrite (mid-scale) to suppressed attention-to-output pathways (larger-scale). This spectrum requires dynamic interpretability and circuit-intervention tools for robust AI deployment.

Conclusion

The paper establishes that repeated-token counting failures in LLMs are not attributable to deficient internal representation. Rather, task-specific, format-triggered MLP blocks overwrite the correctly encoded count, resulting in deterministic incorrect outputs. Mechanistic interpretability reveals that the correct value is encoded at all layers, and behavioral failure is due to misrouting at specific network depths. This dissociation, replicated across architectures, formats, and scales, invites a paradigm shift from representational analysis to circuit-level debugging and modification for robust LLM reasoning (2605.09239).