Numeracy in Large Language Models: Fundamental Limitations and Paths to Improvement

Abstract: LLMs achieve strong results on mathematical reasoning benchmarks yet remain unreliable on elementary numerical tasks, including magnitude comparison, large-integer arithmetic, fractions, and scientific notation. This survey examines basic numerical understanding as a capability distinct from high-level mathematical reasoning. We propose the Numerical Grounding Framework (NGF), which decomposes numeracy into Representational Grounding (RG), mapping numeral forms to value, magnitude, and equivalent representations, and Procedural Grounding (PG), executing arithmetic operations in accordance with their mathematical definitions. Using NGF, we organize recent diagnostic benchmarks, failure modes, structural explanations, and mitigation strategies. We review evidence concerning tokenization, positional encoding, embedding geometry, and pretraining-data distribution. We also apply NGF in a coordinated evaluation of three frontier model families across Number Cookbook, NumericBench, and GSM-Symbolic, comparing atomic, contextual, and reasoning-assisted numeracy. Architectural interventions such as digit-aware tokenization and Abacus Embeddings can improve models trained from scratch but are generally unavailable to users of pretrained systems, for whom supervised fine-tuning, reasoning scaffolds, and external tools are more practical. We conclude with deployment recommendations and research directions for more reliable numerical behavior in foundation models.

• The paper introduces the Numerical Grounding Framework, separating representational grounding from procedural grounding and showing average accuracy gaps of about 0.19 in-domain and 0.27 out-of-domain.
• The evaluation finds that extended reasoning improves out-of-domain procedural grounding by 0.287 versus 0.093 for representational grounding, but can reduce in-domain accuracy while using 21.3 times more tokens.
• The paper concludes that pretrained models benefit most from synthetic-data fine-tuning, process rewards, and tool use, while durable fixes such as digit-level tokenization and Abacus Embeddings generally require changes during pretraining.

Overview and thesis

This survey by Aoxin Ni addresses a persistent contradiction in LLM capabilities: frontier models achieve near-saturated performance on competitive mathematical reasoning benchmarks such as GSM8K and MATH, yet fail on elementary numerical tasks—magnitude comparison, addition of large integers, fraction arithmetic, and scientific-notation operations. The paper's central thesis is that these failures are not isolated glitches but symptoms of insufficient numerical grounding, formalized in an original two-part construct, the Numerical Grounding Framework (NGF). NGF decomposes numeracy into Representational Grounding (RG)—the faithful mapping of numeral surface forms to value, magnitude, and format-equivalent internal representations—and Procedural Grounding (PG)—the faithful execution of arithmetic procedures consistent with their mathematical definitions (2608.13129). The framework is explicitly grounded in Harnad's symbol grounding problem and Dehaene's dual-system theory of human number sense, with RG analogous to the approximate number system and PG to exact symbolic computation.

The survey's distinguishing contributions are: (1) the NGF framework itself; (2) systematic coverage of the 2024–2025 generation of diagnostic benchmarks; (3) a taxonomy of failure modes mapped to grounding dimensions; (4) a coordinated empirical evaluation across three benchmark suites testing four NGF-derived predictions; and (5) a mitigation analysis organized around what the author terms the Pretrained-Model Constraint.

The evaluation landscape

The paper argues that legacy benchmarks have become saturated and diagnostically uninformative: frontier models exceed 95% on GSM8K while remaining fragile under modest perturbation. A newer generation of targeted benchmarks emerged between 2024 and 2025:

Benchmark	Focus	Primary grounding dimension
NumericBench	In-context numeracy within natural-language passages	RG
Number Cookbook	17 atomic tasks across integer/float/fraction/scientific formats	RG + PG
GSM-Symbolic / GSM-NoOp	Parametric variation and irrelevant distractors	RG
GSM-Ranges	Scale-invariance across operand magnitudes	PG
MGSM	Multilingual word problems across ten languages	RG

The author concedes several gaps in this landscape: benchmarks are almost entirely English-centric, atomic-task performance does not reliably predict contextual performance, and no benchmark covers the full distributional range of real-world numerical formats.

Failure modes

Four systematic failure classes are identified, each mapped to a grounding dimension:

Fragility (RG): The GSM-Symbolic study shows that merely substituting numerical values in word problems causes significant accuracy drops, indicating reliance on memorized solution templates. Adding irrelevant numerical distractors degrades performance by up to 65% in some models—a striking figure that directly contradicts any claim that current models "reason" about quantities rather than pattern-match over them.

Tokenization artifacts (RG): The canonical example is models asserting $9.11 > 9.9$. BPE tokenizers segment numbers by corpus co-occurrence frequency rather than mathematical structure, producing non-monotonic surface-to-value mappings. The decimal $9.11$ may tokenize as $[9, ., 11]$ while $9.9$ tokenizes as $[9, ., 9]$, inviting comparison of incomparable suffix tokens. Tokenization is also context-dependent, preventing any stable place-value schema from being learned.

Length generalization failure (PG): Models trained on operands up to $N$ digits fail catastrophically at $N{+}1$ digits, a robust pattern documented across architectures and traced primarily to positional encoding limitations (RoPE's out-of-distribution behavior; ALiBi's recency bias actively suppressing the long-range dependencies required for carry propagation).

Algorithmic asymmetry (PG): Subtraction exhibits "sign blindness"—models produce correct magnitudes but omit negative signs when $A < B$—and division remains the hardest elementary operation without chain-of-thought scaffolding.

Root causes

The survey identifies four structural sources of innumeracy, each damaging a specific grounding dimension:

1. BPE tokenization (damages RG): frequency-driven merging is mathematically destructive for numerals, producing inconsistent segmentation, obscured digit positions, and context-dependent token boundaries.
2. Positional encoding (damages PG): RoPE fails on unseen positional distances; ALiBi's distance penalty suppresses exactly the long-range attention that carry propagation requires.
3. Embedding discontinuity (damages RG): token embeddings for successive integers show no guaranteed geometric ordering; PCA projections via the xVal analysis do not form monotone sequences, though probing studies show magnitude is partially decodable from deeper layers.
4. Pretraining data distribution (damages both): numerical sequences are sparse and skewed toward round numbers and small integers; arithmetic accuracy correlates with operand frequency more strongly than with model scale—a claim that runs against naive scaling narratives.

Empirical evaluation

The coordinated evaluation tests four predictions derived from NGF—RG–PG dissociation, reasoning compensation, tokenizer-specific RG profiles, and primitive–contextual transfer—across GPT-5.4, Claude Opus 4.6, and Gemini 3 (evaluated at MINIMAL and HIGH thinking budgets).

Dissociation: Every configuration shows higher RG than PG accuracy, with an average gap of approximately 0.19 in domain and 0.27 out of domain. Model rankings vary by dimension rather than shifting uniformly, supporting the claim that numerical competence is not monolithic.

Reasoning-budget tradeoff: Extended reasoning disproportionately improves out-of-domain PG (+0.287) relative to RG (+0.093), exactly as NGF predicts if chain-of-thought supplies procedural scaffolding without altering input representation. Notably, HIGH thinking slightly hurts in-domain PG—an empirical signature of the "overthinking" pathology—and consumes 21.3× as many tokens per request (6,247 vs. 293). Gemini MINIMAL achieves 92.5% in-domain exact-match accuracy versus 86.4% for HIGH, so the extra budget buys out-of-domain robustness at substantial cost and in-domain regression.

Tokenizer-specific blind spots: Different models maintain ≥90% accuracy to different digit lengths on different tasks—Claude on float comparison, Gemini on digit counting and length, GPT on integer comparison—consistent with tokenizer-specific surface-form exposure rather than a single global numeracy score.

Primitive–contextual dissociation: Rankings invert across benchmarks. GPT-5.4 trails on Number Cookbook exact-match accuracy (76.6% in-domain) yet leads on NumericBench contextual arithmetic (84.8%) and GSM-Symbolic (96.8% main split), while Gemini MINIMAL shows the opposite profile—strong primitives (92.5%) but near-collapse on GSM-Symbolic without extended reasoning (26.4%, falling to 10.5% with an added clause). This partial decoupling leads the author to propose extending NGF with a third axis: contextual deployment—the ability to activate grounding inside natural language and identify causally relevant quantities. This is an honest acknowledgment that the original two-part framework does not fully account for observed behavior.

Mitigation strategies and the Pretrained-Model Constraint

The most consequential practical finding is the Pretrained-Model Constraint: architectural interventions that dramatically improve scratch-trained models are frequently inapplicable to already-pretrained LLMs. LEFT (Little-Endian fine-tuning) achieves near-perfect addition accuracy but requires scratch training; Abacus Embeddings enable $5\times$ length generalization (20-digit training generalizing to 100-digit inputs) but require architectural modification from the start; xVal requires replacing all number-token embeddings and output heads at pretraining cost; digit-level tokenization must be fixed during vocabulary construction (as Llama 3 did with its fixed three-digit scheme).

For practitioners working with existing pretrained models, the viable interventions are narrower:

Strategy	Pretrained-compatible	Grounding	Key limitation
Digit/fixed-span tokenization	No	RG	Scratch-train required
xVal	No	RG	Poor for exact digit tasks
Format-sensitive SFT	Yes	RG	Weak cross-format generalization
LEFT	No	PG	Addition only
Abacus Embeddings	No	PG	Scratch-train required
Process reward models	Yes	PG	Needs step-labeled data
SFT on synthetic data	Yes	Both	Doesn't fix tokenization artifacts
CoT / reasoning models	Yes	Both	~18× token cost; overthinking
Tool use (PAL)	Yes	Both	Relocates RG burden to equation formulation
Self-consistency	Yes	PG	Additional inference compute

Supervised fine-tuning on diverse synthetic numerical examples and inference-time scaffolding emerge as the most consistently effective approaches for pretrained models. For safety-critical applications (finance, medicine, engineering), the recommendation is unambiguous: delegate computation to deterministic tools via program-aided generation, with the caveat that tool use guarantees PG only by externalization—it does not eliminate RG requirements, since the model must still extract quantities and formulate correct equations.

Limitations and open questions

Several limitations are conceded or evident. The survey follows a structured narrative review methodology rather than a standardized meta-analytic protocol, appropriate for a fast-moving field but less reproducible. The empirical evaluation covers three model families at a single point in time with temperature 0, and the Gemini thinking-budget findings may not generalize to other reasoning-model implementations. The proposed third NGF axis (contextual deployment) is motivated but not formally defined or separately measured. Open questions include: whether Abacus-style inductive biases can be integrated into large-scale pretraining; whether a "number-aware tokenizer" can be adopted without full retraining; how cross-lingual and cross-format numeracy (beyond MGSM's word-problem scope) should be diagnosed; and whether process-reward training can deliver PG gains without the overthinking pathology of current RL-trained reasoners.

Conclusion

This survey makes a disciplined case that LLM numeracy failures are structural rather than stochastic, and that they decompose along representational and procedural lines in ways that are empirically measurable and predictively useful. Its strongest claims—that scaling alone cannot fix tokenization-induced representational failures, that reasoning compensates PG preferentially at significant computational cost, and that the most structurally sound fixes are locked behind the pretrained-model constraint—are supported by both the surveyed literature and the paper's own coordinated evaluation. The practical upshot is a two-tier intervention landscape: scratch-trained models can adopt digit-level tokenization, place-value embeddings, and continuous number encodings, while deployed models must rely on fine-tuning, process rewards, and tool-augmented inference. Until tokenization, positional encoding, and data composition are addressed jointly at the pretraining stage, LLMs will remain powerful reasoning scaffolds but unreliable calculators.