Alternating Reinforcement Fine-Tuning

• Alternating Reinforcement Fine-Tuning is a training strategy that interleaves SFT and RL phases to balance imitation learning with reward-driven optimization.
• It employs methods like gradient blending, curriculum schedules, and selective updates to mitigate instability and catastrophic forgetting.
• Empirical studies show that ARFT improves LLM performance by optimizing reward metrics while managing trade-offs with SFT loss.

Alternating Reinforcement Fine-Tuning (ARFT) refers to a broad set of training schedules in which supervised fine-tuning (SFT) and reinforcement learning (RL) stages are interleaved, typically in the post-training phase of LLMs and related architectures. These procedures exploit the strengths of each paradigm—SFT’s data-driven imitation and RL’s reward-driven optimization—while seeking to overcome their individual limitations. Modern ARFT encompasses a range of frameworks, from deterministic two-stage loops (e.g., SFT→RL→SFT…) to granular schemes where SFT- and RL-inspired gradients are blended within each minibatch or even per token. The empirical and theoretical study of ARFT has revealed deep coupling between the learning signals, raising questions about objective non-decouplability, stability, and optimal schedules.

1. Objectives and Theoretical Foundations

The central challenge motivating ARFT is the mismatch between the objectives of SFT and RL:

• Supervised Fine-Tuning (SFT) minimizes the negative log-likelihood (NLL) of expert demonstrations:

$$L_{\mathrm{SFT}}(\theta) = -\mathbb{E}_{(x, y) \sim D}[\log \pi_\theta(y|x)]$$

where $(x, y)$ are prompt-response pairs from a reference distribution $D$.

• Reinforcement Learning (RL), by contrast, maximizes expected reward over model outputs:

$$J(\theta) = \mathbb{E}_{x \sim q(x), y \sim \pi_\theta(\cdot|x)}[r(x, y)]$$

where $r(x, y)$ is a reward signal derived from human, rule-based, or learned feedback.

Key theoretical work has established that SFT and RL cannot be “decoupled” without permanent, non-recoverable degradation of at least one objective (Niu et al., 12 Jan 2026). Specifically:

• SFT→RL: Conducting RL after achieving SFT optimality always increases the SFT loss by at least a fixed constant whenever any nontrivial reward improvement is obtained.
• RL→SFT: Any subsequent SFT update (constrained by nonzero KL divergence) strictly reduces the RL-achieved reward.

These theorems preclude the existence of a naively alternating loop that preserves the optimality for both SFT and RL objectives, indicating that ARFT must be regarded as a compromise or as a form of joint, rather than decoupled, optimization.

2. Algorithmic Variants of Alternating Training

Multiple algorithmic schedules have emerged, each blending SFT and RL signals at varying granularities:

• Pure Alternation: Cycles of full SFT epochs followed by full RL epochs, or vice versa, sometimes with several alternations (“rounds”).
• Hybrid or Interleaved Schemes: Within each iteration, supervised and reinforcement objectives are combined via gradient blending or schedule-based mixing (Huang et al., 2 Jul 2025).

Example Frameworks

• ARES (Alternating RL and SFT): A two-stage loop optimizing for rationale quality in multimodal chain-of-thought settings. RL, guided by sentence-level AI feedback, is alternated with SFT using teacher-corrected outputs. Empirical runs on ScienceQA and A-OKVQA demonstrate improved solution quality and answer accuracy compared with single-stage or non-alternating methods (Byun et al., 2024).
• Prefix-RFT: Each PPO batch performs a mix of (a) on-policy rollouts for RL and (b) “hybrid” rollouts beginning with an SFT demonstration prefix and continuing with policy sampling. Imitation gradients are applied to prefix tokens, while advantage-weighted RL gradients drive the suffix. Entropy-based clipping and dynamic prefix-length scheduling govern the alternation throughout training (Huang et al., 2 Jul 2025).
• Rubric-ARM: An alternating framework for non-verifiable domains (e.g., creative writing), ARFT alternates between optimizing a rubric generator and a judge via RL, reducing gradient variance and boosting alignment accuracy in pairwise reward modeling tasks (Xu et al., 2 Feb 2026).

3. Empirical Evidence and Performance Effects

ARFT paradigms have been tested across a range of LLMs and tasks, revealing universal signature phenomena:

Schedule	SFT Test Loss	RL Reward Metric	Practical Consequence
SFT→RL	Increases post-RL	Reward rises	SFT fit degrades (“forgetting”)
RL→SFT	Decreases post-SFT	Reward collapses	Alignment lost
Blended/Prefix-RFT	Intermediate	Surpasses pure SFT/RL	Compromise frontier achieved

• On the Qwen3-0.6B and Qwen2.5-Math-7B models, purely alternating SFT and RL leads to “catastrophic forgetting” of previous gains: SFT increases during RL and rewards collapse during SFT (Niu et al., 12 Jan 2026, Huang et al., 2 Jul 2025).
• Prefix-RFT significantly outperforms SFT- or RL-only schedules on both domain-specific (mathematical) and general reasoning tasks, requiring minimal changes to the PPO pipeline (Huang et al., 2 Jul 2025).
• In multi-modal scenarios (e.g., ARES), alternation between RL (using AI feedback) and SFT corrections yields quality improvements judged by external LLMs and boosts in answer accuracy (Byun et al., 2024).
• Rubric-ARM demonstrates that alternation reduces variance and improves labeler alignment and downstream RL policy performance (Xu et al., 2 Feb 2026).

4. Mechanisms for Blending SFT and RL

Alternating strategies vary in their implementation details, but several common mechanisms have emerged:

• Gradient Blending: Jointly optimizing a total loss combining SFT and RL objectives, with token- or sample-dependent interpolation (Huang et al., 2 Jul 2025).
• Selective Update Schedules: Controlling the magnitude or frequency of each update, such as limiting KL divergence or annealing imitation weights over training (Niu et al., 12 Jan 2026, Huang et al., 2 Jul 2025).
• Curriculum Schedules: Dynamically decaying the influence of SFT (e.g., prefix length decay in Prefix-RFT) to shift gradually from imitation toward exploration as model competence grows (Huang et al., 2 Jul 2025).
• Entropy Clipping: Preferring high-entropy (i.e., uncertain) tokens within SFT prefixes to avoid overwhelming RL updates with static supervision (Huang et al., 2 Jul 2025).
• Latent Structure Alternation: In Rubric-ARM, alternation is extended to separate components (rubric generation, judgment) within the reward modeling pipeline, each optimized while the other is held fixed, achieving lower gradient variance and improved sample efficiency (Xu et al., 2 Feb 2026).

5. Limitations, Instabilities, and Practical Guidelines

ARFT is subject to fundamental and practical limitations:

• Non-decoupling: There exists no schedule of SFT and RL alternation that can simultaneously achieve both objectives’ optima except in degenerate, rewardless cases (Niu et al., 12 Jan 2026).
• Forgetting: The mutual interference of objectives leads to immediate loss of previously acquired fit or alignment with each phase switch.
• Data and Feedback Sensitivity: Performance is sensitive to demonstration data quality, RL reward design, and feedback granularity (e.g., sentence-level vs. global judgments) (Byun et al., 2024, Huang et al., 2 Jul 2025).
• Resource Requirements: Some frameworks (e.g., ARES) require costly advanced AI feedback for both RL and SFT stages, limiting scalability (Byun et al., 2024).

Practical mitigations include:

• Combining objectives in a single loss, with adaptive weighting or curriculum-based annealing (Niu et al., 12 Jan 2026, Huang et al., 2 Jul 2025).
• Limiting the strength or number of SFT steps after RL (and vice versa) to control KL shift (Niu et al., 12 Jan 2026).
• Employing fine-grained or stochastic mixing of updates rather than pure alternation (Huang et al., 2 Jul 2025).
• Using feedback-correction or rubric-guided alternation for greater interpretability and stability (Byun et al., 2024, Xu et al., 2 Feb 2026).

6. Broader Impact and Variants Beyond Standard LLM Fine-Tuning

Although ARFT originated in LLM post-training, its methodology has influenced several adjacent domains:

• Reward Modeling: Alternating latent-structure training in rubric-based reward models for flexible, interpretable alignment (Xu et al., 2 Feb 2026).
• Multi-Modal Reasoning: Combining RL and SFT (with AI feedback) enhances chain-of-thought quality and downstream accuracy in text-image contexts (Byun et al., 2024).
• Exploration–Exploitation Frontiers: ARFT frameworks provide a principled approach to balancing demonstration-driven behavior cloning and goal-oriented RL, yielding robust policies that surpass either approach in isolation (Huang et al., 2 Jul 2025).

A plausible implication is that ARFT scheduling ideas may generalize to any composite learning problem where objectives are fundamentally non-aligned, such as multi-agent negotiation (rubric alternation), retrieval-augmented models (alternating retrieval and generation objectives), or curriculum-based RLHF.

7. Directions for Future Work

Emerging lines of investigation include:

• Automatic Schedule Search: Optimizing alternation schedules or blending weights via meta-learning or data-driven adaptive algorithms.
• Fine-Grained Feedback Integration: Incorporating more diverse or contrastive feedback modalities (e.g., latent critique, contrastive preferences) into alternating SFT/RL pipelines (Byun et al., 2024, Xu et al., 2 Feb 2026).
• Progressive Latent Structure: Alternating over deeper levels of abstraction or control (e.g., rubric generator → rubric judge → final model tuning) for improved alignment (Xu et al., 2 Feb 2026).
• Resource Efficiency: Mitigating the cost of advanced feedback (e.g., via distillation or cheap teacher proxies) (Byun et al., 2024).

This suggests that while the inherent coupling between SFT and RL precludes simple alternation as a substitute for joint optimization, alternating reinforcement fine-tuning remains a critical area for advancing the stability, interpretability, and sample efficiency of post-training in LLMs and beyond.