Can LLMs subtract numbers? (2511.02795v1)

Abstract: We present a systematic study of subtraction in LLMs. While prior benchmarks emphasize addition and multiplication, subtraction has received comparatively little attention despite being structurally distinct as a non-commutative operation. We evaluate eight pretrained LLMs spanning four families on addition and subtraction problems. Our experiments reveal that subtraction accuracy lags behind addition by a wide margin. We find that the errors for ($a-b$) are concentrated in cases where ($a<b$). In such cases, LLMs frequently produce the correct magnitude but omit the negative sign. Probing analyses show that LLMs internally encode whether results should be negative, yet this information is often not reflected in generated outputs. We further test well-known techniques such as few-shot learning and instruction-tuning to see if they can improve the LLMs' performance. Our results suggest that while few-shot prompting yields modest gains, the instruction-tuned models achieve near-perfect accuracies in generating the negative sign. Together, these findings provide a clearer characterization of the limitations and recoverability of LLMs' arithmetic capabilities in subtraction.

• The paper evidences that LLMs show significantly lower accuracy on subtraction, particularly when handling negative results.
• It employs synthetic datasets and both zero-shot and few-shot settings to systematically benchmark eight diverse LLM families.
• Instruction tuning is shown to markedly improve subtraction performance by addressing the error of omitting the negative sign.

Subtraction in LLMs: Systematic Evaluation and Error Analysis

Motivation and Background

This paper addresses a notable gap in the evaluation of LLMs' arithmetic capabilities by focusing on subtraction, a non-commutative operation that has received limited attention compared to addition and multiplication. The authors argue that subtraction presents unique challenges for LLMs due to its reliance on operand order and the necessity for robust positional representations, especially in cases requiring borrowing. The paper systematically benchmarks eight open-source LLMs from four families (Gemma-2, Qwen3, OLMo-2, Llama-3) on both addition and subtraction tasks, with a particular emphasis on the structural and representational demands of subtraction.

Experimental Design

The evaluation protocol involves generating synthetic datasets of integer arithmetic problems, with operands sampled uniformly from the range of single-token numbers for each model's tokenizer. The datasets are balanced for $a > b$ and $a < b$ cases, and multiple prompt variants are used to control for spurious correlations. Both zero-shot and n-shot (few-shot) settings are considered, and inference is performed using greedy decoding for pretrained models and recommended sampling strategies for instruction-tuned variants. The paper also extends the analysis to multi-token numbers and probes the internal representations of LLMs to assess their encoding of sign information.

Key Findings

Subtraction vs. Addition Performance

Across all evaluated LLMs, subtraction accuracy is consistently and substantially lower than addition accuracy under matched complexity. For example, Qwen3-8B and OLMo-2-32B achieve near-perfect scores on addition but only approximately 57% and 56% on subtraction, respectively. Smaller models perform poorly on both tasks, but the performance gap between addition and subtraction is pronounced in larger models.

Asymmetry in Operand Order

A strong asymmetry is observed in subtraction performance: LLMs are highly accurate when $a > b$ (result is positive) but their accuracy collapses when $a < b$ (result is negative). In several cases, accuracy drops below 5% for negative results, even in models that are otherwise highly competent at addition and positive subtraction.

Error Analysis: Omission of Negative Sign

The dominant failure mode for $a < b$ is the omission of the negative sign. When accuracy is measured without regard to sign, scores increase dramatically, often exceeding 90%. This indicates that LLMs frequently compute the correct magnitude but fail to output the negative sign, suggesting a disconnect between internal representation and output generation.

Probing Internal Representations

Linear probes trained on the final layer activations of representative LLMs (Gemma-2 9B, Llama-3.1-8B, Qwen3-8B) achieve near-perfect accuracy in predicting whether the result should be positive or negative. This demonstrates that LLMs internally encode the sign information, but this knowledge is not reliably surfaced in the generated output.

Few-shot and Instruction-tuned Improvements

Few-shot prompting yields modest and inconsistent improvements in subtraction accuracy for pretrained LLMs, with some models showing moderate gains and others remaining unstable. In contrast, instruction-tuned LLMs achieve near-perfect accuracy on subtraction, including cases where $a < b$. The authors attribute this improvement to exposure to subtraction data during instruction fine-tuning, as confirmed by analysis of the OLMo-2 instruction-tuning dataset.

Implications and Future Directions

The findings highlight subtraction as an inherently more challenging task for LLMs than addition, primarily due to the non-commutative nature of the operation and the requirement to generate negative results. The systematic omission of the negative sign, despite correct internal encoding, points to a representational-generation mismatch that warrants further investigation. Instruction tuning is shown to be highly effective in bridging this gap, suggesting that targeted data augmentation and fine-tuning strategies can substantially improve LLMs' arithmetic capabilities.

From a practical perspective, these results underscore the importance of including subtraction (and negative-result cases) as a standard diagnostic in LLM evaluation benchmarks. Theoretical implications include the need to better understand the mechanisms by which LLMs transfer internal representations to output, particularly for tasks involving sign and magnitude. Future research should explore architectural modifications, decoding strategies, and training objectives that explicitly address this mismatch. Additionally, the extension of these findings to multi-token numbers and more complex arithmetic operations remains an open area for further paper.

Conclusion

This paper provides a comprehensive analysis of subtraction in LLMs, revealing a persistent challenge in generating negative results despite correct internal computation. The observed representational-generation disconnect and the efficacy of instruction tuning have significant implications for both the evaluation and improvement of LLMs' numerical reasoning abilities. Subtraction should be prioritized as a diagnostic tool in future research, and targeted fine-tuning approaches offer a promising path toward closing the gap in arithmetic competence.