Arithmetic Pedagogy for Language Models

Abstract: We investigate whether methods of human mathematics pedagogy can guide the training of LLMs toward arithmetic reasoning. Building on the GASING method -- an Indonesian pedagogy that solves basic arithmetic through a left-to-right procedure aligned with the causal order of token generation -- we operationalize each operation as a computational procedure whose execution trace is serialized into natural-language Chain-of-Thought (CoT) supervision. A small GPT-2 decoder (86M parameters) with a syllabic-agglutinative TOBA tokenizer for Indonesian is trained from scratch on this data using only a next-token prediction objective, without reinforcement learning or reward-based optimization. Monitoring training reveals three distinct learning phases, and mechanistic analyses -- attention-masking interventions on the CoT information graph, residual-stream probing, and logit-lens inspection -- show that the model first internalizes a procedural pathway and subsequently develops an associative, ``mental-arithmetic'' capacity that retrieves intermediate results without explicit step-by-step computation. The trained model reaches over 80% accuracy on held-out problems and attains competitive performance against substantially larger LLMs, indicating that targeted, pedagogically grounded training can yield strong and economical arithmetic capability at small scale.

• The paper establishes that applying GASING pedagogy significantly improves arithmetic accuracy in small-scale language models.
• It leverages serialized chain-of-thought supervision and attention masking to reveal distinct learning phases with performance rising above 80%.
• Results compare favorably with larger LMs, demonstrating that curriculum-based training can drive efficient numerical reasoning.

Arithmetic Pedagogy for LLMs: Analysis and Implications

Background and Motivation

The paper "Arithmetic Pedagogy for LLMs" (2606.05106) systematically investigates the role of human mathematics pedagogy, specifically the GASING method, in guiding the training of LLMs to acquire arithmetic reasoning. Conventional LLMs demonstrate limited mathematical capability at small parameter scales, with improvements attributed largely to increased data, model size, and computational resources. While prior studies emphasize statistical learning and implicit emergence of arithmetic abilities, this work foregrounds the pedagogical dimension: aligning LLM training with structured procedures proven effective in human arithmetic education.

Methodology

The study leverages the GASING pedagogy, a left-to-right arithmetic framework, and operationalizes its computational processes as serialized execution traces integrated as Chain-of-Thought (CoT) supervision. These traces are formatted in natural language and paired with arithmetic questions for training a GPT-2 decoder (86M parameters), employing the syllabic-agglutinative TOBA tokenizer tailored for Indonesian. The model is trained exclusively on next-token prediction, absent any reinforcement or reward-based corrections.

A dataset of 90,000 instructional samples covers basic arithmetic operations (addition, subtraction, multiplication, division) within three-digit base-10 integers, with systematic constraints to maximize diversity and avoid trivial memorization. The training regime is monitored with periodic checkpoints, evaluating performance on 10,000 held-out arithmetic problems.

Learning Phases and Model Mechanisms

LLM training reveals three distinct learning phases, evidenced by the evolution of cross-entropy loss and arithmetic accuracy:

Figure 1: Model's arithmetic accuracy and cross-entropy loss trajectory, indicating distinct learning phases during training.

The early phase is characterized by rapid loss reduction, predominantly attributed to learning linguistic and syntactic regularities. The mid-phase features a transition as accuracy surges from sub-10% to approximately 60%, coinciding with improvements in critical token prediction and internalization of procedural arithmetic structure. In the final phase, accuracy stabilizes above 80%, with incremental refinements in handling numerically complex operations.

Mechanistic interpretability interventions are applied to elucidate the emergence of arithmetic reasoning. Attention suppression along CoT traces enables measurement of information contrast values, quantifying the impact of blocking ideal computational dependencies versus irrelevant spans:

Figure 2: Information contrast metric ($\Gamma$) for attention suppression and occurrence of computation errors in CoT sequences across training checkpoints.

Positive information contrast indicates alignment between internalized reasoning pathways and the GASING procedural framework. Concurrently, analysis via residual-stream probing and logit-lens techniques confirms a sharpening of correct intermediate result representations:

Figure 3: Classifier accuracy and logit margin analysis in the residual stream, revealing increased separation and preference for correct digits at later model layers.

Intermediate computation errors decline through the mid-phase, highlighting the model's acquisition of associative, mental-arithmetic-like recall. The model's hidden states increasingly surface correct solution candidates without explicit procedural rollout, signifying a shift from algorithmic execution toward retrieval-based, memory-driven computation.

Arithmetic Competency Evaluation

Disaggregated performance metrics show heterogeneous learning rates across arithmetic operations. Addition and subtraction are mastered early, whereas multiplication and division, due to increased computational depth and dependency on prior results, require more extensive training:

Figure 4: Operation-specific computation accuracy showing distinct trajectories for addition, subtraction, multiplication, and division.

Comparisons with other LLMs spanning much larger parameter regimes confirm that the GASING-trained GPT-2 achieves competitive—sometimes superior—performance on basic arithmetic tasks. Notably, the model's accuracy exceeds several baseline LLMs whose parameter counts are orders of magnitude higher:

Figure 5: Benchmarking arithmetic capability across a selection of LLMs.

The model demonstrates efficient arithmetic learning relative to model size:

Figure 6: Parameters versus accuracy among Transformer-based LMs, with GASING model achieving strong results at markedly smaller scale.

Practical and Theoretical Implications

The findings robustly support the hypothesis that targeted, conceptually aligned pedagogy in the training corpus can substantially enhance arithmetic reasoning and generalization in LLMs, even at small scales. Adaptations of GASING principles directly address the causal mismatch between token inference and conventional format in arithmetic—especially carry and borrow mechanics in addition and subtraction—which prior works identified as a hindrance (Lee et al., 2023).

Mechanistically, the interplay of CoT supervision, information routing, and attention masking yields interpretability for the internal structure of learned arithmetic procedures, providing a concrete pathway for disentangling emergent abilities in LMs. The emergence of associative recall mirrors human cognitive processes for mental arithmetic, suggesting that linguistic and procedural scaffolding can drive abstraction and compositional generalization in neural architectures.

Practically, pedagogically grounded training reduces the need for prohibitively large model scales to achieve competent numerical reasoning. This opens the possibility of efficient, specialized LM deployment (e.g., for educational tools, low-resource environments) where parameter economy and interpretability are paramount.

Future Directions

Avenues for future research include the transferability of pedagogical approaches across different language families and numeral systems, integration with more complex mathematical domains, and exploration of hybrid approaches combining explicit program traces and CoT to maximize both generalization and interpretability. There is notable potential for leveraging structured curriculum and exemplars in other reasoning-intensive domains, such as symbolic logic or algorithmic tasks.

The emergence dynamics observed in small-scale settings could inform the design of curricula for progressively larger LLMs, with implications for controllable skill acquisition and targeted augmentation of reasoning faculties.

Conclusion

The study rigorously demonstrates that the application of human arithmetic pedagogy—specifically the GASING method—enables small-scale LLMs to internalize arithmetic reasoning, achieve strong accuracy, and rival much larger LLMs through targeted, data-efficient training. The alignment of procedural computation with causal token inference, combined with mechanistic interpretability, advances both understanding and engineering of cognitive capacities in LLMs. These results underscore the value of integrating pedagogical insights into AI system design and training, shaping directions for efficient, transparent, and robust arithmetic reasoning in future models.