Nothing from Something: Can a Language Model Discover 0?

Abstract: AI systems based on artificial neural networks are being developed with aspirations of pushing the boundary of human mathematical knowledge. A key question for these systems is how much they can reach beyond their training data. Mathematical discovery requires a strong form of out of distribution generalization; the ability to hypothesize genuinely new - and potentially logically more powerful - mathematical structures. It has been hypothesized that language abilities support such generalizations in human cognition. In this work, we use simple arithmetic as a case study for examining how modern AI models could expand their mathematical horizons, evaluating whether these models can independently discover the concept of "zero". We show that We show that (1) LLMs of a GPT-2 size are unable to perform this generalization at test time regardless of language pretraining, but (2) models can improve substantially after training on tens or hundreds of examples of zero. Additionally, we find that language pretraining reduces the number of required examples by approximately $50\%$, showing that language abilities can scaffold mathematical discovery in neural models.

• The paper demonstrates that transformer models trained solely on nonzero arithmetic data cannot zero-shot infer the concept of zero.
• It employs controlled experiments with various GPT-2 scales and pretraining regimes to reveal that language pretraining significantly enhances sample efficiency in few-shot settings.
• The findings highlight that zero—and similarly the carry digit—pose unique challenges, emphasizing a fundamental limitation in current neural architectures for mathematical abstraction.

Out-of-Distribution Generalization in Arithmetic: Can LLMs Discover Zero?

Introduction

The paper "Nothing from Something: Can a LLM Discover 0?" (2606.17289) undertakes a rigorous investigation of the capability of transformer-based LLMs—specifically the GPT-2 architecture—to generalize to mathematical concepts not explicitly represented in their training data, using the concept of zero as a case study. The research is motivated by both historical and cognitive-scientific perspectives: in human mathematics, the invention of zero marks a qualitative leap, and in child development, the acquisition of zero is both delayed and distinctive compared to the learning of other numerals.

The central question posed is whether a neural model trained exclusively on positive single-digit arithmetic (excluding zero in both operands and results except as a digit in multi-digit answers) can, at test time, infer the function or even the existence of zero. Furthermore, the work assesses whether language pretraining scaffolds such conceptual leaps, drawing on theories from cognitive science regarding language as a facilitator of mathematical abstraction.

Figure 1: Example train and test data for arithmetic experiments. Examples that contain zero (except in the ones place) are held out and tokenization is per-digit.

Experimental Design

The authors deploy GPT-2 variants of two scales—an 800K parameter "small" transformer and a 124M parameter "GPT-2 size" model. Pretraining regimes include: (1) OpenWebText (filtered to exclude all numbers and arithmetic symbols), (2) OpenWebText (unfiltered), and (3) no pretraining. Arithmetic problems consist of two to four single-digit operands combined by addition and subtraction, tokenized per digit and symbol, with the presence of zero systematically excluded from all but the ones-place in multi-digit results.

LLMs are further exposed to either a filtered or unfiltered corpus, and arithmetic finetuning is performed with a manually constructed dataset excluding zero, with distinct held-out test sets featuring zero. Few-shot regimens are also explored, where varying numbers of zero-inclusive examples are allowed during training to measure sample efficiency and transfer.

Results: Zero-Shot and Few-Shot Generalization

Zero-Shot Setting

All models, regardless of pretraining corpus and scale, robustly fail to extrapolate the concept or use of zero from training solely on other digits and arithmetic operations. Test set accuracy remains at chance, with no sign of improvement over the course of training. This result holds true across all model sizes and for both filtered and unfiltered pretraining.

Figure 2: Language pretraining curves and model perplexity on arithmetic train data after language pretraining. The removal of text containing numbers results in a far greater perplexity for arithmetic problems, confirming that the model has little exposure to the relevant concepts prior to the training we provide on arithmetic.

Figure 3: Comparison of model generalization to zero at test time, across training regimes. The training and validation sets consist of arithmetic problems that do not contain zero, while the test set consists entirely of arithmetic problems that do. All models were trained on the same data, but varied in size and whether they were pretrained on text that was filtered to remove mentions of numbers or arithmetic symbols.

Language Pretraining as Scaffold in Few-Shot Regime

When zero-inclusive examples are introduced ("few-shot learning"), both pretrained and untrained models show substantial improvement in test accuracy as the number of such examples increases. Notably, language-pretrained models demonstrate approximately twice the sample efficiency: a reduction of ~50% in the number of examples needed to attain equivalent generalization performance compared to untrained models. Quantitatively, with as few as 64 examples of zero (~0.64% of the training data), the pretrained model exceeds 60% accuracy, whereas the untrained model lags behind.

Figure 4: Model generalization to zero at test time in few-shot regime. Test accuracy is averaged over the last 1000 steps of each training run; mean and standard error are reported over five random seeds. Accuracy on problems involving zero increases rapidly with more examples, and faster for the model with language pretraining.

Is Zero a Special Case?

To interrogate whether the difficulty is specific to zero or reflects a generic property of learning under arithmetic extrapolation, the study extends the "hold-out digit" protocol to all digits. Remarkably, the difficulty in zero-shot generalization is highly localized to zero (and, in base-10, to nine, presumably as the carry digit). In contrast, holding out other digits during training leads to much higher test set accuracy, suggesting that the conceptual leap required to "invent" zero (or the carry digit) is qualitatively distinct.

Figure 5: Final test accuracy on holdout digits 0-9. Zero and nine (carry digit) are hardest to generalize to at test time.

This finding generalizes to arithmetic in base-8, where both zero and the carry digit (seven) are special cases in terms of test-time error rates.

Figure 6: Final test accuracy on holdout digits 0-7, in the base-8 arithmetic regime. Zero and seven (carry digit) are hardest to generalize to at test time.

Cosine similarity analyses of the learned token embeddings further support an interpolation-driven hypothesis: digits that inhabit the "middle" of the digit range have more similar neighbors, rendering them easier to interpolate to; by contrast, zero and the carry digits are edge cases with fewer high-similarity neighbors in representation space.

Figure 7: Number of digits with cosine similarity $\geq 0.65$ with holdout digits 0-9. Digits that fall in the middle of the range have more "near neighbors".

Figure 8: Number of digits with cosine similarity $\geq 0.65$ with holdout digits 0-7, in the base 8 arithmetic regime. Digits that fall in the middle of the range have more "near neighbors".

Theoretical and Practical Implications

The research demonstrates that transformer-based LMs, even with moderate scale and pretraining, lack the capacity for non-trivial out-of-distribution generalization, specifically the formation of novel mathematical abstractions such as zero, in the absence of explicit data. This suggests sharp boundaries on the mathematical inventiveness of current LMs under standard pretraining and instruction regimes, especially in the absence of sample diversity or model inductive bias toward abstraction.

However, the consistent and strong effect of language pretraining on sample efficiency for few-shot learning suggests that language modeling does induce latent structure conducive to faster generalization, even for mathematical concepts not seen during pretraining. This aligns closely with cognitive science hypotheses regarding language as a scaffold for abstract concept acquisition.

For both practical deployment in mathematical domains and for the evaluation of neural models' capacity for scientific discovery, the findings underscore the need for either systematic data exposure, specialized training procedures (e.g., chain-of-thought, reasoning traces), or architectural inductive biases not present in current transformer models.

Future Directions

Several avenues for future research are underscored:

• Scaling Laws: Investigate larger-scale models and pretraining corpora to assess whether increased capacity or training diversity can induce genuine conceptual generalization.
• Prompting and Scratchpad Methods: Evaluation of chain-of-thought and intermediate-step prompting to facilitate extrapolation absent direct supervision.
• Explicit Inductive Bias: Exploration of model architectures or learning objectives explicitly designed to support abstraction and systematicity beyond compositional generalization.
• Mechanistic Interpretability: Identification of specific representations and attention/circuitry pathways within models that facilitate (or constrain) the abstraction of zero-like concepts.

Conclusion

The study establishes that, in arithmetic, "zero" represents a boundary case for neural LLMs: without explicit supervision, transformer LMs cannot discover zero, in stark contrast to their documented capacity for in-distribution generalization and interpolation. Language pretraining does not enable zero-shot conceptual leap, but confers significant sample efficiency advantages in the few-shot regime. For building models that participate meaningfully in mathematical discovery, the challenge of concept invention—here, exemplified by zero—remains fundamentally unsolved in transformer-based neural architectures.