Stable Counting Capacity in Language Models

• Stable Counting Capacity is a precise assay that measures a model’s ability to reliably update an internal tally across repeated counting tasks with controlled inputs.
• The methodology strips away semantic and tokenization confounds by using homogeneous sequences and exact-integer outputs to test procedural state maintenance.
• Empirical results reveal that even models with large nominal context windows exhibit abrupt failures, highlighting a fundamental limitation in internal state tracking.

Stable Counting Capacity (SCC), also termed the “counting capacity” (CC), is an assay introduced to measure how far a LLM can stably execute the rule “count the number of repeated items” before exact rule-following collapses (Dai et al., 3 May 2026). The assay is intentionally mechanical: it removes knowledge dependencies, semantics, and ambiguity, uses exact-integer outputs, and avoids lexical and tokenization confounds so that evaluation targets procedural state maintenance rather than factual recall or task familiarity. In the reported study, SCC is evaluated across 126 model variants, and every evaluated model exhibits a finite capacity far below nominal context limits, with failure characterized by abrupt collapse from exact counting to large, attractor-like guesses rather than gradual numerical drift.

1. Definition and conceptual role

Stable Counting Capacity is defined as the largest sequence length for which a model remains stable under a randomized counting evaluation. Here, “stable” does not mean occasional correctness on isolated instances; it means reliable exact counting across a randomized band of target lengths around a tier. The central claim is that this probes a minimal form of procedural reliability: the ability to maintain and update an internal tally while repeatedly applying a single rule.

The assay is designed as a minimal probe because counting repeated identical items strips away many confounds present in standard benchmarks. No factual knowledge is required. No semantic interpretation is required. Scoring is ambiguity-free because the desired output is an integer. The counted unit is the item rather than the token, and controlled syntax variations are used to check that the result is not an artifact of a particular tokenizer or output schema. This makes SCC a test of whether a model can preserve a rule-defined internal state over many steps, which the paper treats as a prerequisite for broader multi-step behaviors such as constraint tracking, variable maintenance, and exact intermediate-state preservation.

A recurring misconception in language-model evaluation is that strong performance on mathematical reasoning, coding, or long-context benchmarks implies reliable general rule following. The reported SCC results argue against that inference: fluent task performance and large advertised context windows do not guarantee stable execution of even the simplest exact procedure once the required internal tracking resource is exhausted.

2. Formalization and evaluation protocol

Let $s$ be a single symbol and $s^n$ the homogeneous sequence of $n$ items. For a model $f$ that returns an integer from a counting prompt, the per-instance accuracy indicator is defined by

$I(n)=1[f(s^n)=n],$

and the accuracy curve is

$A(n)=P[I(n)=1].$

The operational criterion for stability is based on normalized mean absolute error at a tier length $L$: $$\mathrm{nMAE}(L)=\frac{1}{K}\sum_{k=1}^{K}\frac{|\hat{x}_k-x_k|}{L},$$ where $K=16$ targets $x_k$ are sampled by $s^n$0, $s^n$1, and $s^n$2 is the parsed integer from the model’s output. A tier $s^n$3 is stable if

$s^n$4

Stable Counting Capacity is then

$s^n$5

The evaluation uses an adaptive randomized ladder. It initializes at $s^n$6, samples $s^n$7 targets in the randomized band $s^n$8–$s^n$9, and classifies the tier as stable or unstable according to the $n$0 threshold. If stable, evaluation expands upward; otherwise, a binary search refines the boundary between the highest verified stable tier and the lowest failed tier. If a model fails already at $n$1, its result is recorded categorically as $n$2.

Scoring parses the last valid integer in the model output; if none is present, the instance is classified as failure. Exact-match integer scoring is used against the true item count. The paper notes that the randomized tier design bounds false positives from guessing strategies at approximately $n$3, because guesses cannot consistently satisfy the tier-wide randomized threshold (Dai et al., 3 May 2026).

3. Assay construction and confound control

The baseline stimuli are homogeneous sequences of identical items, using repeated lowercase characters separated by “, ”. Prompts contain no changing symbols, semantic landmarks, or external memory aids. The counted unit is the item, not the token. The main assay also avoids JSON or constrained grammars, so parser or constrained-decoding behavior is not folded into the measurement.

A hierarchical variant generalizes beyond simple repeated-token counting. Instead of $n$4, the input consists of records containing a KEY, a deeply nested PATH field, and a SIDE field of distractors; the task is to count records whose KEY matches the deepest PATH token, strictly applying an equality rule. This turns the assay into a broader probe of structural rule tracking rather than mere repetition length.

Several confound checks are built into the design. Tokenization lengths are verified across tokenizers, but results are interpreted at the item level, not the token level. Syntax variations, including swapping the counted symbol or delimiter, are explicitly evaluated. Randomization within tiers prevents coarse magnitude guessing from passing the stability threshold. Tool use is disabled throughout, and served models are evaluated under documented or default decoding configurations without forcing numerical grammars.

This design supports a narrow but technically precise interpretation of SCC. It does not ask whether a model “understands counting” in a semantic sense. Rather, it asks whether the model can maintain the state needed to execute a repeated exact update reliably across prompts, lengths, and surface variations.

4. Empirical profile across model families

Across 126 variants spanning proprietary systems and open-weight transformers, including instruction-tuned and reasoning-augmented models, every evaluated model has a finite CC far below its nominal context limit (Dai et al., 3 May 2026). Capacities span a broad range, and newer models often support larger CCs, but the paper emphasizes that these capacities remain orders of magnitude below advertised context sizes.

The reported failure dynamics are sharp. Within the stable region, outputs lie perfectly on the diagonal of true versus predicted count. Near the boundary, accuracy collapses abruptly to large errors rather than degrading through small off-by-one mistakes. Predictions cluster at salient rounded integers such as $n$5, $n$6, and $n$7, producing horizontal attractor bands across true counts. In the baseline assay, Gemma 3 27B-it is reported to count correctly up to 27 items before the first error; beyond CC, outputs jump to rounded guesses such as $n$8 or $n$9.

The same pattern appears in the hierarchical rule-tracking setting. Even there, the best model counted 416 valid matches before collapse, which the paper presents as evidence that the phenomenon is not specific to repeated-character inputs but extends to deeper structural state maintenance.

Instruction-following also degrades at failure. Across trials, 5% of outputs—501 out of 9797—lacked a valid single number, including blank outputs, prompt echoes, code-format artifacts, and spurious reasoning traces. The paper interprets this as procedural collapse affecting not only the tally itself but also output-format control.

Added test-time compute does not reliably repair failure. Average total token consumption at the CC boundary lies on an empirical efficiency frontier of about two consumed tokens per true count. Reasoning variants consume dramatically more tokens than matched base models but show small or negative changes in CC. A related dual-task experiment with gpt-5.4-mini shows that simultaneously performing counting and benchmark-like tasks such as BBH, CRUXEval-O, MATH-500, and MMLU-Pro increases counting error more than length-matched irrelevant-code controls or a secondary independent counting control, indicating competition for a limited internal tracking resource.

The paper also reports syntax sensitivity: swapping the counted character or delimiter can cause significant CC shifts even when input-token counts are similar. This argues against a fully abstract, syntax-invariant counter shared across surface forms.

5. Behavioral and mechanistic interpretation

The behavioral signature motivates the paper’s central interpretation: models behave as if they possess a finite set of count-like internal states, analogous to “counting on fingers,” rather than an open-ended exact counting mechanism. Once that resource is exhausted, exact rule following disappears and outputs revert to guessing.

Mechanistic probing on Gemma 3 27B-it supports this picture. A linearly readable count-related coordinate appears in the residual stream during successful counting, with projections at layers 16, 31, 40, and 53 tracking the true count linearly across the stable regime. This linear structure disappears at the behavioral failure boundary. Teacher-forced minimum logit margins for the correct integer also decay sharply near the boundary and can become negative. With $f$0 the residual at the final token and $f$1 the correct token, the paper defines

$f$2

Within CC, $f$3; near and beyond CC, $f$4 or becomes negative, indicating that the model ceases to prefer the correct count even under the correct prefix.

Sparse autoencoder analysis using Gemmascope 2 finds count-correlated features that are structured and non-monotonic, suggesting that the relevant representation is a coalition of features rather than a single accumulator. Changing the counted symbol or delimiter reorganizes this coalition, which is consistent with the observed syntax sensitivity.

Activation patching indicates distributed causal control. Final-token patching alters the count only in late layers, around layer 51 of 62. Full-sequence patching, with donor states interpolated to recipient length, has its strongest effect in middle layers, around layer 31, and is more influential overall. The paper interprets this as evidence that per-token progress trajectories are constructed in intermediate layers and later transferred into the final prompt state for decoding. Attempts to rescue failed sequences by clamping scalar progress coordinates did not reliably recover exact counts, implying that the causal representation is richer than a single scalar variable (Dai et al., 3 May 2026).

6. Implications, benchmark relations, and limitations

SCC is presented as a minimal but information-rich assay of procedural reliability. Its main implication is that current LLMs can appear to follow rules while not maintaining the persistent internal variables those rules require. This matters for agentic planning, repository-scale coding, long-form generation, and multi-step tool use, all of which depend on stable tracking of constraints, variables, commitments, and intermediate results. The paper’s formulation is that such capabilities may be locally reliable yet globally brittle.

Cross-benchmark comparisons reinforce this point. SCC correlates only weakly to moderately with conventional benchmark scores such as GPQA Diamond and SWE-bench Verified, suggesting that those leaderboards are largely blind to procedural reliability. For ARC-AGI-2, newer models trained after the benchmark’s release show a strong linear correlation between $f$5 and ARC-AGI-2 score. The paper interprets this as consistent with task-format familiarity rather than fundamentally improved state maintenance.

The work does not claim that homogeneous counting exhausts the space of procedures. It explicitly identifies the task as artificial. The argument is instead that a probe this stripped-down should be easy for any system with robust general rule execution, so failure on it reveals a fundamental limitation rather than mere benchmark mismatch. A plausible implication is that improvements in benchmark performance alone may not resolve this limitation unless architectures or training objectives change to support explicit persistent state.

Suggested improvement paths include architectural support for persistent variables and recurrence, such as Transformer-XL and recurrent memory transformers; external memory or retrieval with verifiable execution traces; explicit counters or structured state modules; and training objectives that reward exact state preservation. Reported limitations include the intentional artificiality of homogeneous counting, the possibility of hidden preprocessing in proprietary models, and the fact that mechanistic analysis is limited to open-weight systems.

Reproducibility materials include inference logs for 126 models, token-usage metadata, parsed outputs, dual-task measurements, and SAE feature activations, together with code and protocol details, at the repository specified by the paper (Dai et al., 3 May 2026).