Counting as a minimal probe of language model reliability

Abstract: LLMs perform strongly on benchmarks in mathematical reasoning, coding and document analysis, suggesting a broad ability to follow instructions. However, it remains unclear whether such success reflects general logical competence, repeated application of learned procedures, or pattern matching that mimics rule execution. We investigate this question by introducing Stable Counting Capacity, an assay in which models count repeated symbols until failure. The assay removes knowledge dependencies, semantics and ambiguity from evaluation, avoids lexical and tokenization confounds, and provides a direct measure of procedural reliability beyond standard knowledge-based benchmarks. Here we show, across more than 100 model variants, that stable counting capacity remains far below advertised context limits. Model behavior is consistent neither with open-ended logic nor with stable application of a learned rule, but instead with use of a finite set of count-like internal states, analogous to counting on fingers. Once this resource is exhausted, the appearance of rule following disappears and exact execution collapses into guessing, even with additional test-time compute. These findings show that fluent performance in current LLMs does not guarantee general, reliable rule following.

• The paper introduces Stable Counting Capacity (SCC) to diagnose LLMs’ ability to maintain procedural state with a simple counting rule.
• It demonstrates that LLMs exhibit an abrupt failure in counting beyond a discrete limit, regardless of additional reasoning or token length.
• Analysis reveals a latent count-tracking trajectory within transformers that collapses at a critical capacity, impacting multi-step tasks.

Counting as a Minimal Probe of LLM Reliability

Motivation and Problem Statement

Despite substantial advances in LLMs, their abilities for robust, general procedural rule-following remain ambiguous. Standard knowledge- and task-based benchmarks predominantly measure a mixture of reasoning, factual retrieval, and tool usage, but can neither directly probe the reliability and generalization of elementary rule execution nor reliably distinguish genuine algorithmic competence from pattern matching and memorization. The authors propose "Stable Counting Capacity" (SCC) as a purely mechanical benchmark—a minimal, knowledge-independent assay—to diagnose LLMs’ ability to preserve a procedural state across unbounded sequence lengths by performing a simple counting rule.

Figure 1: SCC benchmark isolates mechanical rule-following by incrementally probing counting over item sequences across diverse LLMs, revealing a uniformly restricted counting capacity compared to nominal context windows.

Stable Counting Capacity Assay Design

The SCC benchmark radically reduces confounds by removing knowledge requirements, semantic structure, and tokenization irregularities. The probe consists of presenting a sequence of identical items (e.g., "a, a, a, ...") of variable length and prompting the model to return the exact count. This process iteratively extends the sequence length until the model cannot stably and accurately return the correct count. The precise locus of failure, the "counting capacity" (CC), quantifies model limits in procedural tracking. SCC further avoids JSON or templated schemas, and uses an adaptive randomized ladder for sequence length selection—rendering guesswork insufficient and eliminating semantic cues.

Empirically, all 126 evaluated LLM variants—including proprietary and open-weight models with nominal context windows spanning tens of thousands of tokens—exhibit an abrupt failure at a discrete CC far below their advertised context length.

Failure Dynamics at the Counting Boundary

Models demonstrate perfect counting up to the CC, after which outputs collapse into attractors: salient, rounded numbers (e.g., 500, 1000) or other plausible guesses, entirely decoupled from the real count. There is no evidence of a smooth, continuous degradation. Instead, the error profile resembles an internal, finite-state automaton exhaustively consuming its capacity and then defaulting to routine or overlearned values.

Figure 2: Model predictions transition abruptly from accurate count tracking to wild, attractor-driven guesses upon surpassing the counting capacity.

Variations in the counted character and delimiter shift CC across models, even when token counts are held constant, exposing that the underlying procedural state is not fully abstract—it is entangled with surface form and syntactic features. Additionally, introducing hierarchical/nested counting, which increases the complexity of the required state but remains rule-based, yields identical abrupt collapse dynamics and bounded capacity.

Failure is robust to changes in test-time computation, output length, or the use of reasoning-augmented decoding: increasing the model’s output tokens or prompting for chain-of-thought reasoning does not expand CC.

Resource Competition and Interference Effects

A cross-analysis of token usage—both in base and reasoning-augmented model variants—demonstrates that increased token consumption or allocation to elaborate reasoning is not only ineffectual for raising CC but can cause detrimental interference with the procedural tracking required for counting.

Figure 3: Neither increased inference-time computation nor additional reasoning reliably enhance procedural state maintenance; complex tasks actively compete with and impair counting accuracy.

Matched dual-task experiments pairing counting with benchmark-style questions (reasoning, coding, math, factual recall) indicate that complex subtasks induce much greater counting errors than token-length-matched controls or dual counting tasks. This shows complex processing hijacks the same finite set of internal resources responsible for procedural state tracking in the transformer.

Mechanistic Insights Into Procedural State Encoding

Mechanistic investigation using open-weight models (Gemma 3 27B-it, Qwen 3.5 35B-A3B) reveals a linearly readable, count-related latent direction in the transformer’s residual stream, which robustly tracks the count across layers strictly within the CC. This linear manifold collapses exactly at the transition to failure, where model outputs revert to attractors.

Figure 4: Residual stream probes reveal an explicit, finite count-tracking trajectory in the hidden state across layers. Collapse of this manifold corresponds to loss of accurate counting.

Sparse autoencoder (SAE) analysis implicates a coalition of distributed features—no single accumulator neuron or feature dominates. Activation patching experiments across layers demonstrate that causal control over the output count transitions from intermediate to late layers, in line with an architecture that first constructs per-token progress and then condenses state before output.

Patching these trajectories in the middle layers is more effective at manipulating count outputs than intervention at initial or final tokens. Attempts to recover lost counts by injecting the “right” progress coordinate fail once the manifold collapses, supporting the hypothesis that the real limitation is not retrieval of information but catastrophic loss of a usable representation.

Alignment with Standard Benchmarks

Correlations between SCC and established benchmarks—GPQA Diamond (knowledge), SWE-bench Verified (coding), and ARC-AGI-2 (abstract reasoning)—are weak to moderate. Notably, improvement on fixed-format tasks such as ARC-AGI-2 is tightly linked to CC only for models trained after exposure to the benchmark, underscoring the risk of overfitting to task form and masking underlying deficits in genuine procedural reliability.

Figure 5: Weak-to-moderate correlation between CC and knowledge/reasoning benchmarks, demonstrating that standard leaderboards can obscure core procedural failure modes. Performance on ARC-AGI-2 becomes closely linked to CC only following benchmark-specific exposure.

High scores on knowledge- or reasoning-heavy leaderboards are thus not evidence for open-ended rule execution, and increases in CC correlate with benchmark gains only when models receive explicit, repeated exposure during (pre)training or fine-tuning.

Theoretical and Practical Implications

The authors’ findings converge with transformer theory: procedural variable tracking, especially counting and formal language generalization, is inherently limited in vanilla transformers [hahn2020selfattention; deletang2023chomsky]. The empirical evidence, however, strikes a more severe note: deployed LLMs cannot reliably carry a simple procedural state for the length of their context, and failure is abrupt, unpredictable, and undetectable from the output until it occurs. Memory and state-extension methods—such as adding tokens, chain-of-thought, or structural prompts—do not fundamentally address the limitation.

This brittle state maintenance has direct consequences for agentic tasks, long-form code generation, planning, and multi-step tool use—domains that depend on persistent, trustworthy variable tracking over long contexts. Without architectural innovations (e.g., explicit recurrence [dai2019transformerxl], external memory modules [borgeaud2022retro], or verifiable execution traces), further parameter scaling or benchmark tuning will yield diminishing returns on true mechanistic reliability.

Conclusion

This work provides a systematic, mechanistically-grounded assay for evaluating rule-following and state-tracking in LLMs, divorced from confounds present in existing knowledge- and reasoning-based leaderboards. SCC robustly reveals a finite, model-dependent procedural capacity across state-of-the-art models, and characterizes both behavioral and internal collapse dynamics. Enhancing LLMs' ability to execute simple rules over unbounded horizons will require not only improved training protocols but also architectural solutions explicitly designed for persistent state and procedural invariance.