RASFT: Rollout-Adaptive Supervised Fine-Tuning for Reasoning

Abstract: Supervised fine-tuning (SFT) is a prevailing method for adapting LLMs to reasoning tasks by imitating offline expert demonstrations, often treating a single expert trajectory as the target behavior. However, reasoning is not simple path imitation: rigidly following one demonstrated solution may overfit to surface forms and suppress the model's own reasoning distribution. We propose Rollout-Adaptive Supervised Fine-Tuning (RASFT), a policy-aware SFT framework that calibrates expert supervision according to problem-level solvability estimated from verified on-policy rollouts. For each problem, RASFT strengthens expert guidance when the current policy struggles, while relaxing rigid imitation and incorporating correct self-generated trajectories when the model already exhibits reliable reasoning behavior. To preserve useful reasoning priors, RASFT further introduces a clipped inverse ratio between the frozen reference model and the current policy to constrain excessive policy drift. Experiments across multiple models on six mathematical reasoning benchmarks and two code reasoning benchmarks show that RASFT achieves better overall performance than SFT, SFT variants, and representative RL methods. The code is available at https://github.com/zjd1sq/RASFT.

• The paper introduces RASFT, a rollout-adaptive fine-tuning framework that adjusts expert supervision using a solvability metric based on on-policy rollouts.
• It integrates verified on-policy outputs with an inverse importance sampling ratio to maintain stability and prevent overfitting in reasoning tasks.
• Empirical results show significant performance gains and stability improvements over standard SFT and RL approaches in math and code reasoning benchmarks.

Rollout-Adaptive Supervised Fine-Tuning for Reasoning: An Overview of RASFT

Motivation and Limitations of Existing SFT Paradigms

Supervised fine-tuning (SFT) is the de facto approach for adapting LLMs to reasoning tasks by imitating expert (often single-path) demonstrations. However, in tasks such as mathematical or code reasoning, a problem can admit multiple valid solution paths. Overly rigid imitation of a single demonstration can suppress the model's intrinsic reasoning distribution, causing overfitting to demonstration idiosyncrasies and underutilization of pretrained reasoning skills. Recent SFT variants (e.g., DFT, ASFT, ProFiT) introduce rescaling or reference-anchoring at the token or demonstration level to mitigate this, but their adaptivity remains predominantly internal to the offline data and does not engage with the evolving capabilities of the current model policy.

RASFT Framework: Rollout-Based Policy-Aware Supervised Fine-Tuning

RASFT proposes a shift from static, demonstration-centric SFT to a rollout-adaptive, policy-aware regime. For each training instance, the method computes a problem-level solvability score via verified on-policy rollouts—sampling outputs from the current policy, vetting them with task-specific verifiers (e.g., final-answer matching, code execution), and recording the fraction of correct samples. This solvability index modulates the weight of expert supervision: stronger for problems where the policy underperforms, and attenuated when reliable reasoning emerges from the model itself.

In the RASFT pipeline, the candidate target set for a prompt consists of the reference expert solution plus all correct on-policy rollouts. The model is updated via a weighted loss, where the expert and rollout trajectories are adaptively balanced using the solvability metric. To prevent excessive policy drift from the pretrained initialization, a sequence-level inverse importance sampling ratio between the current and frozen reference policy acts as a conservative regularizer.

Figure 1: RASFT pipeline: (a) sequential sampling of on-policy rollouts and their verification, (b) roll-out-derived solvability calibrating expert/rollout weights, (c) training objective integrating pool-normalized weights, an inverse ratio, and token probabilities.

Numerical Results: Performance, Transferability, and Ablations

RASFT demonstrates universality and robustness across model scales (Qwen2.5-Math-1.5B/7B, Qwen2.5-Coder-3B/7B, Llama-3.2-3B) and benchmarks (six mathematical and two code reasoning tasks). Absolute numerical gains over baselines are significant, especially for under-optimized base models (Qwen2.5-Math-1.5B: +2.72 average points over best SFT variants, Llama-3.2-3B code: +6.70).

Ablation studies indicate three orthogonal contributors to RASFT’s efficacy:

• Difficulty-adaptive weighting: Removing the solvability-driven modulation yields a mean drop of 2.19 points, highlighting the necessity of policy-aware calibration.
• On-policy rollouts: Omitting model-generated candidates eliminates the online feedback loop, resulting in marked regressions, especially on harder benchmarks.
• Inverse policy ratio regularization: Eliminating the sequence-level ratio leads to overfitting, with reduced average accuracy (27.30% vs. 27.72%).

RASFT is compatible with various SFT variants (including ProFiT and DFT) and consistently improves their scores, supporting its transferability.

RASFT vs Reinforcement Learning Methods

RASFT outperforms competitive RL-based methods such as GRPO and LUFFY, especially on hard tasks where RL-based policies struggle with exploration cold-start. By adaptively scaling expert guidance dependent on real-time solvability, RASFT avoids stagnation common in pure on-policy RL and achieves higher mean accuracy and greater training stability.

Training Dynamics and Sensitivity Analysis

RASFT maintains stable optimization trajectories (Figure 2), with a smooth decrease in training loss and modest parameter-update densities compared to other strong SFT baselines, indicating efficient control over catastrophic drift and overfitting.

Figure 2: Training loss and parameter-update density across SFT variants; RASFT preserves stability and a lower update density, consistent with effective regularization.

The performance is non-monotonic w.r.t. the number of rollouts: moderate $K$ (e.g., $K=5$) outperforms both low and excessive rollout regimes, attributed to the trade-off between coverage of the model’s solution space and preservation of useful expert priors.

Practical and Theoretical Implications

Practically, RASFT is amenable to domains with automated verification (math, code), and experimentally exhibits consistent improvement over both standard SFT and representative RL paradigms. Theoretically, it embodies a hybrid regularization regime—constrained policy adaptation leveraging real-time estimates of policy uncertainty and competence, bridging the gaps between static demonstration learning and high-variance RL objectives.

The inverse importance sampling ratio is particularly notable for stabilizing learning and preventing loss of pretrained competencies, a major concern in large-scale transfer and continual learning. The integration of on-policy correct rollouts into the candidate supervision pool fosters exploration without incurring reward model bias or drift.

Future Prospects and AI Development

RASFT’s framework opens several directions: extending rollout-adaptive supervision to open-ended reasoning tasks lacking strict ground-truth verification, integrating more sophisticated verifiers (e.g., LLM-based judges, process verifiers), and scaling to larger corpora and models. Its policy-aware balancing mechanism could inspire further objective refinement in domains where preference modeling or reward learning is ambiguous or costly.

Conclusion

RASFT advances SFT for reasoning in LLMs by shifting from uniform trajectory imitation to adaptive, policy-aware supervision guided by verified rollouts and constrained by reference-relative ratios. Extensive empirical evaluation demonstrates RASFT’s superiority over both SFT variants and RL-based methods across multiple reasoning domains and models. The approach elucidates the importance of dynamically calibrating expert guidance relative to model policy competence, suggesting a promising path for scalable and stable post-training of LLMs in complex reasoning applications.

Reference: "RASFT: Rollout-Adaptive Supervised Fine-Tuning for Reasoning" (2606.07006)