How Much Backtracking is Enough? Exploring the Interplay of SFT and RL in Enhancing LLM Reasoning (2505.24273v1)

Abstract: Recent breakthroughs in LLMs have effectively improved their reasoning abilities, particularly on mathematical and logical problems that have verifiable answers, through techniques such as supervised finetuning (SFT) and reinforcement learning (RL). Prior research indicates that RL effectively internalizes search strategies, enabling long chain-of-thought (CoT) reasoning, with backtracking emerging naturally as a learned capability. However, the precise benefits of backtracking, specifically, how significantly it contributes to reasoning improvements and the optimal extent of its use, remain poorly understood. In this work, we systematically investigate the dynamics between SFT and RL on eight reasoning tasks: Countdown, Sudoku, Arc 1D, Geometry, Color Cube Rotation, List Functions, Zebra Puzzles, and Self Reference. Our findings highlight that short CoT sequences used in SFT as a warm-up do have moderate contribution to RL training, compared with cold-start RL; however such contribution diminishes when tasks become increasingly difficult. Motivated by this observation, we construct synthetic datasets varying systematically in the number of backtracking steps and conduct controlled experiments to isolate the influence of either the correctness (content) or the structure (i.e., backtrack frequency). We find that (1) longer CoT with backtracks generally induce better and more stable RL training, (2) more challenging problems with larger search space tend to need higher numbers of backtracks during the SFT stage. Additionally, we demonstrate through experiments on distilled data that RL training is largely unaffected by the correctness of long CoT sequences, suggesting that RL prioritizes structural patterns over content correctness. Collectively, our results offer practical insights into designing optimal training strategies to effectively scale reasoning in LLMs.

Interplay of SFT and RL in Enhancing LLM Reasoning

This paper explores the dynamics between supervised fine-tuning (SFT) and reinforcement learning (RL) in improving the reasoning capabilities of LLMs, focusing on the role of backtracking within reasoning tasks. The researchers present a systematic analysis across eight varied reasoning tasks, including Countdown, Sudoku, Arc 1D, and others, to understand the influence of backtracking on reasoning performance and training efficiency.

The paper highlights several key findings from controlled experiments conducted using synthetic datasets tailored with varying amounts of backtracking steps. These datasets help isolate the effects of reasoning structure, embodied by backtracking frequency, from the correctness of reasoning content. The authors demonstrate that for complex reasoning problems requiring expansive search spaces, an increased number of backtracks significantly enhances RL training and reasoning performance. This implies that backtracking is beneficial in navigating larger solution spaces and stabilizing training outcomes.

In contrast, for simpler tasks or smaller search spaces, shorter chain-of-thought sequences without substantial backtracking suffice for optimal RL training. Interestingly, the paper reports that the correctness of reasoning sequences during RL initialization has minimal effect on eventual performance, suggesting RL's primary focus on structural reasoning patterns rather than content accuracy.

The implications of this research are profound for developing LLMs capable of handling varied reasoning tasks. The results indicate that constructing training strategies with a balanced mix of backtracking tailored to problem complexity could lead to more effective reasoning models. Additionally, the findings challenge the conventional emphasis on strict content accuracy during the RL phase, emphasizing structural learning patterns.

Looking forward, the insights from this paper could inform the design of LLMs optimized for reasoning tasks, especially in domains where verifiable answers are critical. By leveraging structured reasoning patterns as shown, future AI systems may achieve stable and explainable reasoning across diverse application areas. Moreover, these findings reinforce the value of scaling inference-time compute rather than focusing solely on model parameters or pretraining corpus size, thus optimizing the training processes for emergent reasoning capabilities in AI.

The paper contributes to the ongoing exploration of RL strategies in AI model training, suggesting nuanced approaches to data selection and process configuration. Researchers must continue to investigate the interplay between reasoning structures and behavior amplification to push the boundaries of AI's reasoning abilities further.