SenseMath: Do LLMs Have Number Sense? Evaluating Shortcut Use, Judgment, and Generation

Abstract: LLMs often default to step-by-step computation even when efficient numerical shortcuts are available. This raises a basic question: do they exhibit number sense in a human-like behavioral sense, i.e., the ability to recognize numerical structure, apply shortcuts when appropriate, and avoid them when they are not? We introduce SenseMath, a controlled benchmark for evaluating structure-sensitive numerical reasoning in LLMs. SenseMath contains 4,800 items spanning eight shortcut categories and four digit scales, with matched strong-shortcut, weak-shortcut, and control variants. It supports three evaluation settings of increasing cognitive demand: Shortcut Use (whether models can apply shortcuts on shortcut-amenable problems); Applicability Judgment (whether they can recognize when a shortcut is appropriate or misleading); and Problem Generation (whether they can generate new problem items that correctly admit a given type of shortcut). Our evaluation across five LLMs, ranging from GPT-4o-mini to Llama-3.1-8B, shows a consistent pattern: when explicitly prompted, models readily adopt shortcut strategies and achieve substantial accuracy gains on shortcut-amenable items (up to 15%), yet under standard chain-of-thought prompting they spontaneously employ such strategies in fewer than 40% of cases, even when they demonstrably possess the requisite capability. Moreover, this competence is confined to the Use level; models systematically over-generalise shortcuts to problems where they do not apply, and fail to generate valid shortcut-bearing problems from scratch. Together, these results suggest that current LLMs exhibit procedural shortcut fluency without the structural understanding of when and why shortcuts work that underlies human number sense.

• The paper introduces SenseMath, a controlled benchmark that isolates LLMs' use of shortcut strategies in numerical reasoning.
• Evaluation shows that explicit number-sense prompts boost shortcut use, with large models achieving up to 15 percentage point accuracy gains.
• Models struggle with applicability judgment and problem generation, revealing a gap between procedural fluency and genuine structural understanding.

SenseMath: Structure-Sensitive Evaluation of Number Sense in LLMs

Overview and Motivation

The paper "SenseMath: Do LLMs Have Number Sense? Evaluating Shortcut Use, Judgment, and Generation" (2604.01988) addresses the gap between high arithmetic performance in LLMs and their structural understanding of numbers. While SOTA models approach human-level accuracy on mathematical benchmarks via protocols such as CoT prompting and RL fine-tuning, there is little direct evidence that these models possess number sense in the behavioral sense emphasized by mathematics education research: flexible recognition and exploitation of numerical structure, selective shortcut use, and context-sensitive judgment about when and why shortcuts apply.

To rigorously audit these capabilities, the authors present SenseMath, a controlled benchmark over 4,800 structured items spanning 8 categories of number-sense shortcuts and 4 digit scales. SenseMath leverages a matched-variant design—with strong-shortcut, weak-shortcut, and control items per template—to causally attribute model performance to strategy selection rather than surface difficulty. Three evaluation paradigms are formulated: shortcut execution (Apply), applicability judgment (Analyze), and problem generation (Create).

Figure 1: SenseMath benchmark schematic, illustrating matched item variants for isolating selective shortcut exploitation across prompting conditions.

Benchmark Construction and Protocols

The SenseMath generation pipeline instantiates 8 shortcut categories covering core number-sense strategies at problem, reasoning, and option levels. Examples include structural decompositions (e.g., $(100-2)\times 37$), magnitude estimation, relative-benchmark comparisons, cancellation, compatible number rounding, landmark anchoring (e.g., mapping to 50%), algebraic equation identities, and option-level elimination. Each template is scaled over 2, 4, 8, and 16-digit operand settings to orthogonally modulate intrinsic cognitive load without confounding shortcut structure.

For both, item-level and category-level manipulations are tightly controlled: each question appears in strong, weak, and control variants sharing surface form but differing in shortcut availability and efficiency (see Appendix examples). All distractors are calibrated to preclude trivial elimination, and answer correctness is enforced programmatically.

Evaluation is positioned at three increasing levels of cognitive demand:

• Shortcut Use: Can models employ shortcuts spontaneously under standard (CoT) versus number-sense (NS) prompting? Is capability latent but untriggered, or fundamentally lacking?
• Applicability Judgment: Can models distinguish when a shortcut is genuinely available (YES/NO), or identify if a worked solution deploys such a shortcut (SHORTCUT/COMPUTATION)?
• Problem Generation: Can models generate novel items that admit a specific shortcut (and construct matched controls), satisfying structural constraints as verified by deterministic checklists?

Experimental Results and Analysis

Shortcut Use: Latent Capability, Prompt-Dependent Activation

Across all tested LLMs (GPT-4.1-mini, GPT-4o-mini, Qwen3-30B/8B, Llama-3.1-8B), shortcut execution under default CoT prompting is suppressed, with shortcut strategies expressed in fewer than 40% of responses on shortcut-amenable items at moderate digit scales ($d=4$), despite being computationally optimal. Explicit NS prompting (category-agnostic cues for "mathematical intuition and ease") activates shortcut reasoning robustly—from 68% to 86% usage, driving up to 15pp accuracy gains for large models. The delta is strongest for capable models as digit scale increases, demonstrating that shortcut knowledge is present but often dormant under standard prompting. In contrast, 8B-parameter models do not exhibit meaningful NS benefits, implying a model size or data threshold for acquired number-sense strategies.

Figure 2: Shortcut usage rate on strong shortcut items at $d=4$: NS prompt sharply increases exploitation, suggesting latent but unexploited number-sense competencies.

Figure 3: Average accuracy gain from NS prompting over CoT, broken out by strategy variant (strong/weak/control); large models benefit most where shortcuts are structurally valid.

Applicability: Overgeneralization and Lack of Contextual Judgment

When tasked with judging shortcut applicability (J1), LLMs display systematic overgeneralization. Most models output YES for almost all items, failing to reject control items where no shortcut applies (0–21% correct control rejections depending on model). Conversely, in the classification task (J2), where the solution is given, accuracy surges (70–91%), indicating that LLMs can recognize the presence of a shortcut post-hoc but cannot reliably judge applicability ex-ante. This dissociation demonstrates a gap between procedural facility and declarative, context-sensitive understanding.

Problem Generation: Superficial, Surface-Based Outputs

Problem-generation results reveal that even the strongest models very rarely construct valid, structurally-sound shortcut-amenable items when prompted, passing all structural, variant, and correctness checks at only 2–24% rates. Most generations succeed in passing 4–5 out of 6 constraints (e.g., surface form, matching, distractor calibration), but fail critical shortcut-existence or answer-correctness checks. Models reproduce superficial patterns of shortcut-friendly items (e.g., round numbers) without ensuring the actual structure enables a genuine shortcut.

Figure 4: Distribution of the number of verification checks passed by generated item pairs: majority pass several surface constraints, but shortcut-existence is the persistent bottleneck.

Per-Category and Scale Analyses

Performance trends show substantial variation across shortcut types. Relative distance (RD) and option elimination (OE) yield the largest NS gains for advanced models, whereas simple magnitude estimation can sometimes degrade under NS, due to erroneous or spurious rounding. The benefit from NS prompting is also positively correlated with digit scale: as the raw arithmetic becomes more complex, models increasingly default to structural solutions—but only when suitably prompted.

Figure 5: Normalized NS improvement per shortcut category at $d=8$; meta-reasoning and relational strategies see the largest increases for capable models, exposing differential strategy-specific benefits.

Theoretical and Practical Implications

These results rigorously establish that current LLMs possess procedural shortcut fluency absent genuine number-sense: they can execute explicitly-cued shortcut strategies but lack the higher-order conceptual insight to gauge validity, avoid overgeneralization, or synthesize novel structurally-grounded items. The observed Apply $\gg$ Analyze $>$ Create gradient mirrors classic developmental dissociations in human cognition, where procedural mastery precedes structural insight [mcintosh1992proposed, sowder1992making, rittle2001developing].

For practical deployment, especially in educational settings (e.g., as AI math tutors), this lack of structural understanding raises significant trust and generalization concerns. Overgeneralization can produce confidently incorrect advice, while failure to generate meaningful practice problems impedes adaptive learning. Improvements from post-training using NS-cued, shortcut-preferred policies are modest: models fine-tuned with DPO on NS solutions show small in-domain gains but no transformative expansion of number-sense abilities.

Comparison to Related Work

Previous evaluations (e.g., GSM8K, MATH, MathBench, MathVista) emphasize final answer accuracy and sometimes require multi-turn reasoning but do not diagnose whether solutions exploit structure-sensitive strategies or reflect true, context-aware number sense [wei2022chain, cobbe2021training, hendrycks2021measuring]. Efforts probing LLMs' "number sense" typically examine narrow, perception-level facets (e.g., magnitude comparison, digit manipulation, heuristic search) [yang2024number, rahman2025fragile, li2025numeracy, nikankin2024arithmetic]. SenseMath offers a systematic, behaviorally principled probe of compositional and flexible numerical reasoning at multiple cognitive levels.

Directions for Future Research

SenseMath highlights major directions for the advancement of LLM numerical reasoning:

• Strategy-Conditioned Training: Fine-grained, reasoning-level supervision (e.g., with structurally-annotated proof chains, counterfactuals) to align LLMs' strategy selection mechanisms.
• Meta-Reasoning Architectures: Integration of modules capable of explicit applicability checks, counterexample reasoning, or meta-cognitive reflection analogous to human number sense development.
• Evaluation Frameworks for Deeper Skills: Extend beyond SenseMath to domains where mathematical structure, abstraction, and selective generalization are critical (e.g., geometry, combinatorics, algorithmic pattern discovery).

Conclusion

SenseMath provides compelling evidence that current LLMs' arithmetic prowess relies primarily on procedural and heuristic fluency, not on the flexible, structure-sensitive number sense characteristic of robust human reasoning. Strategy use can be triggered via explicit prompting, but contextual judgment and generative competence remain severely limited. These findings caution against equating high answer accuracy or CoT trace plausibility with genuine mathematical understanding, especially for applications demanding adaptive and trustworthy instructional support.