---
title: 'Iterative RL/SFT: Bridging Supervised & RL'
url: https://www.emergentmind.com/topics/iterative-reinforcement-learning-sft
type: topic
---

# Iterative RL/SFT: Bridging Supervised & RL

Iterative Reinforcement Learning/SFT encompasses algorithmic frameworks and methodologies that alternate or dynamically combine supervised fine-tuning (SFT) with reinforcement learning (RL) or preference optimization, often in multiple rounds, to improve large (multimodal) language model performance, alignment, and robustness across reasoning, safety, and capability domains. Recent research demonstrates a spectrum of designs—ranging from explicit alternating SFT and RL phases, adaptive interleaving, plug-and-play insertion of one paradigm into another, and meta-iterative defense–attack loops—each tailored for specific data regimes, supervision qualities, and downstream tasks. The iterative aspect is leveraged both to refine training signals (e.g., through label improvement or RL-driven trajectory generation) and to mitigate shortcomings such as overfitting, catastrophic forgetting, reward hacking, or limited exploration.

## 1. Core Mechanisms in Iterative RL/SFT

Iterative RL/SFT strategically alternates between supervised updating and reward-driven optimization, typically to combine their respective advantages and compensate for their individual limitations. In the canonical setting, SFT initializes the model on human-annotated demonstrations, ensuring fidelity to grounded responses, while RL (or a proxy such as preference optimization) exploits reward signals to push performance on complex, high-reward, or dynamically-defined objectives. Sophisticated variants extend this paradigm:

- **Supervised Iterative Learning from Human Feedback (SuperHF)** iteratively generates candidate completions, filters them using a reward model, and retrains via SFT with a KL-divergence prior to prevent distributional collapse. This unifies the RLHF pipeline but stabilizes convergence and mitigates reward hacking through KL regularization and empirical surrogate posteriors [2310.16763].
- **Inverse Reinforcement Learning in SFT** treats SFT as a bilevel problem, jointly learning a reward function and a policy maximizing regularized expected reward; the updates contrast expert demonstrations with synthetic model samples, unifying self-play and IRL approaches [2405.17888].
- **Plug-and-Play Frameworks** such as MIFO decouple RL and SFT intervals, storing challenging RL examples in a buffer to be used for SFT on uncertain/high-entropy tokens, while freezing RL-critical parameters to prevent catastrophic forgetting [2510.04454].
- **Step-Adaptive Integration (SASR)** guides the SFT–RL trade-off at each optimization step by monitoring gradient norms and KL divergence from initial data, setting the adaptive schedule for RL and SFT loss terms [2505.13026].

These frameworks can operate in (i) explicit, fixed-phase alternations; (ii) dynamically scheduled or interleaved updates; (iii) meta-iterative loops where outputs or weaknesses generated at each round refine the training signal for subsequent rounds.

## 2. Reward Learning, Data Refinement, and Preference Optimization

A central innovation in modern iterative RL/SFT is the elevation of reward learning or data refinement as online or iterative processes, addressing problems of limited or unreliable supervision:

- **Iterative Label Refinement (ILR)** replaces error-prone SFT training examples with high-quality model-generated alternatives selected through human (or simulated) preferences. The process iterates, cross-labeling across data splits, updating only a controlled fraction $\alpha$ of labels at each round [2501.07886]. This method, compared to RLHF (e.g., DPO), is particularly robust to weak supervision, avoids overoptimization under noisy feedback, and permits large effective model updates via improved data.
- **Direct Preference Optimization**: Methods such as DPO allow the policy to be updated in closed-form with respect to win–lose pairs, and are employed in step-wise iterative settings to calibrate tool-use segments, multi-modal generation, or hybrid visual-text tasks [2501.09766, 2504.02949].
- **Reward Models and Filtering:** Reward models can be iteratively retrained or serve as filtering agents (as in SuperHF or iTool), defining surrogate training distributions, preventing mode collapse, and enabling interpretable trade-offs (e.g., between high reward and diversity measured via METEOR similarity or entropy penalties).

Empirical results indicate that iterative data refinement and reward-informed filtering perform favorably under unreliable supervision and lead to greater robustness and generalization in complex domains (e.g., mathematical reasoning, code synthesis, safe instruction following).

## 3. Stability, Forgetting, and Data Efficiency

Iterative RL/SFT often targets improved stability, minimal catastrophic forgetting, and efficiency with respect to both data and computation:

- **Catastrophic Forgetting** is explicitly mitigated via entropy-aware loss calculation (focusing SFT on high-uncertainty tokens) and parameter freezing for weights most affected by RL updates [2510.04454]. This approach achieves state-of-the-art reasoning performance with only 1.5% of previous SFT and 20.4% of RL data usage.
- **Online vs. Offline Iterative RL**: RoiRL demonstrates that weighted log-likelihood optimization using majority-vote self-labels in an offline iterative loop achieves greater stability and training speed than online RL methods, eliminating the need for a reference model and enabling up to 2.5× faster training [2510.02892].
- **Task/Knowledge Retention Trade-off**: Research shows SFT accelerates rapid adaptation to novel tasks but may cause severe forgetting, while RFT enables slower but more stable task learning without interfering with prior knowledge. Periodic SFT on RFT-generated correct rollouts can combine rapid acquisition with preserved performance on previous tasks [2506.23508].

The empirical and theoretical consensus is that data distribution and the update schedule, rather than sole algorithmic differences, play the critical role in balancing forgetting and sample efficiency.

## 4. Evaluation, Generalization, and Proxy Metrics

Rigorous evaluation protocols and proxy metrics for iterative RL/SFT pipeline tuning are increasingly necessary to ensure progress is both real and transferable to downstream RL or alignment goals:

- **SFT Score Limitations**: High supervised fine-tuning scores alone do not reliably predict subsequent RL stage gains. Overfitting to homogeneous or short examples can lead to misleading SFT metrics and poor RL outcomes [2510.01624].
- **Alternative Predictors**: Generalization loss on held-out validation (as it starts to rise) and Pass@$k$ metrics on large batches (indicating the presence of at least one correct among many samples) are shown to be far better predictors of RL potential, improving R² correlations by up to 0.5 over naive SFT-based metrics [2510.01624].
- **Iterative Adversarial Training and Safety**: In security-critical modes, such as the SecTOW framework, adversarial attacker-defender models are iteratively optimized using RL (GRPO) to identify and patch vulnerabilities in model reasoning while monitoring over-refusal and diversity via built-in quality metrics [2507.22037].

These findings underscore the importance of proxy metrics, ablation of training schedules, and careful selection of evaluation protocols as integral components of successful iterative RL/SFT deployment.

## 5. Applications and Extensions: Multimodal, Tool Use, and Domain-Specific Reasoning

Iterative RL/SFT methods have been extended and validated in a range of application contexts:

- **Vision-Language and Multimodal Models**: Alternating SFT and RL phases, as in OpenVLThinker and Metis-RISE, lead to iterative self-improvement in multimodal reasoning tasks. SFT distills chain-of-thought traces (potentially from text-only expert models) while RL (typically using GRPO or mixed reward models) enhances adaptive reasoning and enables transfer across visual and textual domains [2503.17352, 2506.13056].
- **Tool Use and Code Generation**: For advanced tool-driven language model tasks (as in iTool and AutoTriton), step-wise MCTS and RL-driven preference optimization iteratively address local deficiencies and adapt to hard, fragmentary errors, yielding measurable improvements over SFT-only or purely synthetic-data approaches [2501.09766, 2507.05687].
- **Legal and Domain-Specific Reasoning**: Iterative SFT–RL pipelines with domain-specific rewards or assessor/reviser agents support robust, interpretable legal reasoning, outperforming much larger general-purpose models by combining LoRA SFT warmup with domain-informed RL reward structures [2510.10072].

Notably, applications systematically leverage the iterative framework to address context-specific deficits—be it complex multi-step tool use, safety via adversarial training, adaptive domain reasoning, or efficient alignment of multimodal models.

## 6. Theoretical Underpinnings and Mathematical Foundations

Several lines of work present theoretical analysis, convergence guarantees, and closed-form objectives for iterative RL/SFT algorithms:

- **Bilevel and Minimax Formulations**: The IRL-cast SFT exemplifies a bilevel framework where the lower-level RL problem admits a closed-form softmax policy, and the upper-level SFT maximizes demonstration log-likelihood subject to regularized policy alignment [2405.17888].
- **Learning Dynamics**: The effect of fine-tuning updates on the probability landscape (and catastrophic forgetting) is analytically explored using the empirical neural tangent kernel, revealing that reinforcement on on-distribution rollouts preserves prior knowledge via symmetric gradient interactions [2506.23508].
- **Gradient Regularization, KL, and Adaptive Losses**: Theoretical results connect gradient norm with divergence from initial SFT distribution, justifying adaptive schedule designs (as in SASR) and the use of KL penalties to stabilize policy updates and prevent overfitting or collapse [2505.13026, 2310.16763].

These mathematical foundations inform practical algorithm design, providing convergence guarantees, efficiency justifications, and insight into the trade-offs between exploration (RL-driven) and exploitation (SFT-driven).

## 7. Trends, Future Directions, and Open Challenges

The iterative RL/SFT paradigm continues to evolve, and several key trajectories are visible:

- **Cyclic Multi-Round Training**: Forward iterations alternating SFT and RL (potentially in an adaptive or performance-triggered schedule) are proposed to consolidate gains, reactivate reasoning, and enable continual improvement, especially in large-scale domain- or multimodal models [2506.13056].
- **Data-Centric Refinement**: Emphasis is shifting towards iterative dataset improvement (ILR, data cleaning via feedback) as an alternative or supplement to direct iterative model optimization, with special benefits under unreliable or noisy supervision [2501.07886].
- **Algorithm Agnosticism and Meta-Frameworks**: Modular, plug-and-play architectures allow for domain, data type, and supervision noise adaptation, decoupling the iterative pipeline from any one RL or SFT algorithm [2510.04454].
- **Quality Control and Interventional Monitoring**: Systematic use of intermediate validation loss, diversity/entropy measures, and task-appropriate adversarial challenges are emerging as best practices for controlling runaway optimization, catastrophic forgetting, or overrefusal in sensitive applications [2507.22037, 2510.01624].

A plausible implication is that iterative RL/SFT will increasingly serve as a foundational protocol for continual model improvement, robust safety assurance, and unified alignment across emergent domains, as scaling, data availability, and real-world complexity intensify.

Source: https://www.emergentmind.com/topics/iterative-reinforcement-learning-sft