Programming by Backprop: LLMs Acquire Reusable Algorithmic Abstractions During Code Training (2506.18777v1)

Abstract: Training LLMs on source code significantly enhances their general-purpose reasoning abilities, but the mechanisms underlying this generalisation are poorly understood. In this paper, we propose Programming by Backprop (PBB) as a potential driver of this effect - teaching a model to evaluate a program for inputs by training on its source code alone, without ever seeing I/O examples. To explore this idea, we finetune LLMs on two sets of programs representing simple maths problems and algorithms: one with source code and I/O examples (w/ IO), the other with source code only (w/o IO). We find evidence that LLMs have some ability to evaluate w/o IO programs for inputs in a range of experimental settings, and make several observations. Firstly, PBB works significantly better when programs are provided as code rather than semantically equivalent language descriptions. Secondly, LLMs can produce outputs for w/o IO programs directly, by implicitly evaluating the program within the forward pass, and more reliably when stepping through the program in-context via chain-of-thought. We further show that PBB leads to more robust evaluation of programs across inputs than training on I/O pairs drawn from a distribution that mirrors naturally occurring data. Our findings suggest a mechanism for enhanced reasoning through code training: it allows LLMs to internalise reusable algorithmic abstractions. Significant scope remains for future work to enable LLMs to more effectively learn from symbolic procedures, and progress in this direction opens other avenues like model alignment by training on formal constitutional principles.

Programming by Backprop: Insights into Algorithmic Abstractions

The paper "Programming by Backprop: LLMs Acquire Reusable Algorithmic Abstractions During Code Training" explores the phenomenon of LLMs enhancing their reasoning abilities through exposure to source code. The authors introduce the concept of Programming by Backprop (PBB) as a mechanism explaining this enhancement, where models learn to evaluate programs based solely on their code without direct input-output (I/O) examples during training. The paper examines this effect across various experimental settings and aims to clarify how post-training finetuning with code allows LLMs to internalize reusable computational abstractions.

Key Findings and Methodologies

Through a series of finely controlled experiments, the authors demonstrate that LLMs are capable of evaluating programs for inputs, even when the programs have been presented only as source code. The experimental methodology involves finetuning LLMs on two distinct sets of programs: one that includes both source code and I/O examples, and another that includes only source code. Across datasets including random arithmetic problems, Leetcode algorithms, and custom ciphers, several findings emerged:

1. Effectiveness of Code over Natural Descriptions: Training models on explicit code results in better performance compared to semantically equivalent natural language descriptions. This suggests the structured syntax of programming languages facilitates learning procedural abstractions more effectively.
2. Implicit Execution and Chain-of-Thought: LLMs can produce outputs for programs seen only as source code directly through implicit execution. Utilizing chain-of-thought reasoning improves this capability, indicating that explicit sequential reasoning aids in managing complex operations within models.
3. Stage-Based Training: Proactive-PBB involves a two-stage approach where initial finetuning on programs with both code and paired I/O examples enhances subsequent learning from code-only programs. Retroactive-PBB relies on reinforcement learning (RL) for improved outcomes, demonstrating that RL paradigms may better equip models to generalize algorithmic reasoning.
4. Generalization Across Domains: The ability to evaluate programs trained only as source code transfers across different algorithmic domains, suggesting that models can apply learned abstractions in diverse contexts. Code-trained models exhibit more uniform generalization across input variations than those trained solely on I/O pairs.

Practical and Theoretical Implications

The ability of LLMs to internalize and generalize algorithmic knowledge across tasks represents a significant step forward in understanding model reasoning capabilities. Practically, Programming by Backprop can inform strategies for training LLMs efficiently, reducing reliance on task-specific demonstrations and facilitating scalable model development. Theoretically, these findings elucidate potential pathways for improving model safety and interpretability by predicting model generalization patterns more accurately.

The paper advances the discourse on the internalization of procedural knowledge and opens avenues for further research on leveraging symbolic procedures in model training, aligning model behaviors with formal principles, and overcoming dataset biases effectively.

Conclusion and Future Directions

This paper presents a compelling narrative on the importance of code exposure in augmenting the reasoning capabilities of LLMs. While posing the question of whether similar phenomena occur during large-scale pretraining, it affirms the significance of exploring how models can be programmed to internalize useful computational abstractions efficiently. Future research could explore understanding the emergence of algorithmic reasoning during pretraining and explore innovative means of algorithm distillation. The exploration of model alignment using constitutional principles presented via symbolic code also represents a promising future direction.

In summary, "Programming by Backprop" provides substantial insights into transformational learning processes within LLMs, suggesting significant potential for enhancing the flexibility and robustness of AI systems through code-based training methodologies.