Decoupled Reinforcement Finetuning

• Decoupled reinforcement finetuning is a post-training strategy that separates RL updates along distinct axes, such as parameter subsets, rollout generation, and optimization stages.
• It leverages sparse subnetwork training by updating only 5–30% of model weights, preserving most pretrained parameters to maintain stability and computational efficiency.
• Practical applications across modalities—from diffusion models to hierarchical agents—demonstrate decoupling’s benefits in cost reduction, alignment robustness, and improved rollout efficiency.

Decoupled reinforcement finetuning denotes a family of post-training strategies in which reinforcement-learning updates are separated along one or more axes: the parameters being updated, the timesteps or subproblems being optimized, the policy used to generate rollouts, or the optimization stage itself. In the most direct recent formulation for LLMs, RL fine-tuning is reported to modify only a small subnetwork—typically 5–30% of weights—while leaving most parameters unchanged, and re-training only that subnetwork reproduces the full-model RL solution (Balashov, 23 Jul 2025). Related work uses decoupling to separate denoising windows in diffusion-model RL (Barceló et al., 2024), split behavior-policy rollouts from target-policy optimization in reasoning post-training (Heuillet et al., 13 Aug 2025), replace online multi-turn RL with offline trajectory collection plus importance-weighted supervised updates (Mu et al., 29 May 2026), or isolate planning and execution contexts in hierarchical agents (Liu et al., 14 Dec 2025). An earlier precursor in transfer learning separated representation, forward dynamics, inverse dynamics, and reward function within RL itself (Zhang et al., 2018).

1. Conceptual scope and lineage

The term has no single canonical meaning across the literature. Instead, papers use it to describe several structurally related decompositions. One line isolates a sparse parameter subset that absorbs almost all RL adaptation (Balashov, 23 Jul 2025). A second line separates rollout generation from optimization, either by using a cheaper nested behavior model during training (Heuillet et al., 13 Aug 2025) or by collecting trajectories once under a fixed reference policy and then optimizing with weighted supervised learning (Mu et al., 29 May 2026). A third line decouples internal roles or contexts, as in planner–executor architectures where high-level planning and low-level tool interaction operate in distinct context windows (Liu et al., 14 Dec 2025). A fourth line decouples modules or stages, for example by freezing most of a pretrained diffusion-navigation policy and updating only the higher decoder layers and action head (Sheng et al., 13 Mar 2026), or by splitting image and text encoders across clients and server in federated learning, followed by supervised warm-start and RL-based generalization enhancement (Ma et al., 27 May 2026).

This broader usage extends an older RL intuition. “Decoupling Dynamics and Reward for Transfer Learning” separated the task representation, the forward dynamics, the inverse dynamics, and the reward function, with the explicit goal of transfer across changes in dynamics and reward (Zhang et al., 2018). Contemporary post-training work inherits that modular instinct, but applies it to foundation-model adaptation, rollout efficiency, and alignment stability.

2. Sparse-subnetwork decoupling in LLM RL

A formal account of decoupling via sparse parameter updates is given by the definition of RL-induced parameter update sparsity. Let $\theta_{\mathrm{init}}\in\mathbb{R}^d$ be the pretrained or post-SFT parameter vector and $\theta_{\mathrm{full}}\in\mathbb{R}^d$ the parameter vector after full-model RL fine-tuning. The per-parameter update is

$$\Delta\theta=\theta_{\mathrm{full}}-\theta_{\mathrm{init}}.$$

With a threshold $\tau>0$ to ignore numerical noise, the binary update mask $m\in\{0,1\}^d$ is

$$m_i= \begin{cases} 1,& \text{if } |\Delta\theta_i|>\tau\ 0,& \text{otherwise.} \end{cases}$$

The RL-updated subnetwork is $S=\{i\in\{1,\dots,d\}:m_i=1\}$, and sparsity is the fraction of weights left unchanged,

$$s=\frac{|\{i:|\Delta\theta_i|\le \tau\}|}{d}=1-\frac{|S|}{d}.$$

The reported measurement uses $\tau=10^{-5}$, with update density $1-s=|S|/d$ (Balashov, 23 Jul 2025).

The empirical setup spans multiple model families and RL algorithms. The models and checkpoints include Tulu-3 chat models (8B, 70B), Eurus-2 7B, DeepSeek-Math 7B, Mistral-7B, Llama-3-8B, and MathShepherd-7B. The RL algorithms include PPO, GRPO, ORPO, DPO, SimPO, and PRIME, with sparsity measured by the thresholded mask above; overlap is measured by one-sided overlaps $\theta_{\mathrm{full}}\in\mathbb{R}^d$0 and $\theta_{\mathrm{full}}\in\mathbb{R}^d$1, and matrix-update rank is computed from $\theta_{\mathrm{full}}\in\mathbb{R}^d$2 at tolerance $\theta_{\mathrm{full}}\in\mathbb{R}^d$3 (Balashov, 23 Jul 2025).

Across models and RL algorithms, 70–95% of parameters remain unchanged, so only 5–30% of weights are updated. By contrast, supervised fine-tuning on the same models yields $\theta_{\mathrm{full}}\in\mathbb{R}^d$4–$\theta_{\mathrm{full}}\in\mathbb{R}^d$5. Layer-wise sparsity is described as uniform, with each transformer block having 60–90% unchanged, while LayerNorm weights are nearly 100% untouched. At the same time, the change $\theta_{\mathrm{full}}\in\mathbb{R}^d$6 in each weight matrix is nearly full-rank—99+% of maximum rank—so the phenomenon is not equivalent to deliberately low-rank adaptation such as LoRA. Independent RL runs also share substantial subnetwork overlap far above chance: for different seeds, $\theta_{\mathrm{full}}\in\mathbb{R}^d$7 versus a baseline of approximately 36.7%; for different datasets, $\theta_{\mathrm{full}}\in\mathbb{R}^d$8 and $\theta_{\mathrm{full}}\in\mathbb{R}^d$9 versus baselines of 14.6% and 36.7%; and for different algorithms plus seeds plus data, $$\Delta\theta=\theta_{\mathrm{full}}-\theta_{\mathrm{init}}.$$0 and $$\Delta\theta=\theta_{\mathrm{full}}-\theta_{\mathrm{init}}.$$1 versus baselines of approximately 23.0% and 12.9% (Balashov, 23 Jul 2025).

A further distinction is drawn between transient and stable updates. Many weights move early in RL and revert by convergence; the fraction of transient updates peaks mid-training and then falls to zero, leaving a stable, small subnetwork. This pattern supports the paper’s claim that RL fine-tuning does not broadly reshape the entire model, but instead settles on a limited set of persistent parameter changes (Balashov, 23 Jul 2025).

3. Subnetwork-only training and recovery of full-model solutions

The central constructive claim is that the discovered RL-updated subnetwork is sufficient to reproduce the full RL solution. If $$\Delta\theta=\theta_{\mathrm{full}}-\theta_{\mathrm{init}}.$$2 denotes the RLHF loss, then subnetwork-only training starts from $$\Delta\theta=\theta_{\mathrm{full}}-\theta_{\mathrm{init}}.$$3, computes $$\Delta\theta=\theta_{\mathrm{full}}-\theta_{\mathrm{init}}.$$4, and applies the masked update

$$\Delta\theta=\theta_{\mathrm{full}}-\theta_{\mathrm{init}}.$$5

so that every parameter with $$\Delta\theta=\theta_{\mathrm{full}}-\theta_{\mathrm{init}}.$$6 remains fixed. The practical workflow is explicit: first run a standard RLHF pass to obtain $$\Delta\theta=\theta_{\mathrm{full}}-\theta_{\mathrm{init}}.$$7, then compute $$\Delta\theta=\theta_{\mathrm{full}}-\theta_{\mathrm{init}}.$$8 from the full-model update, and finally re-run RLHF with frozen weights outside $$\Delta\theta=\theta_{\mathrm{full}}-\theta_{\mathrm{init}}.$$9 (Balashov, 23 Jul 2025).

The reported results are unusually strong. For DPO on Tulu-3 8B, $\tau>0$0 matches or slightly exceeds $\tau>0$1 on average reward by $\tau>0$2, while more than 99.93% of weights are identical to $\tau>0$3. For PRIME on Eurus-2 7B math, $\tau>0$4 reaches 72.2% overall accuracy versus 69.8% for $\tau>0$5, with 99.99% parameter match. Training curves converge to the same minima, sometimes with slightly lower loss for $\tau>0$6, and the normalized distance satisfies $\tau>0$7. The paper therefore states that the small RL-updated subnetwork is necessary and sufficient for alignment, and that updates outside $\tau>0$8 are superfluous (Balashov, 23 Jul 2025).

The proposed explanation is distributional rather than architectural in the narrow sense. RLHF typically fine-tunes on data close to the model’s own distribution, often after SFT pre-alignment, so only small behavioral shifts are needed. Algorithms with KL constraints can bound policy divergence per step, but disabling or increasing KL showed little effect on final sparsity; gradient clipping prevents extreme single-step changes but does not produce global sparsity; and dropping clipping led to instability rather than denser updates. Architecturally, LayerNorm parameters seldom change, whereas output biases and a small fraction of attention and feed-forward weights form the stable subnetwork. The paper interprets this as evidence that pretrained models contain partially transferable alignment structure and explicitly reframes sparsity through the lens of the lottery ticket hypothesis (Balashov, 23 Jul 2025).

4. Decoupling patterns across modalities and training pipelines

Several later or parallel systems instantiate decoupling in forms that are not reducible to sparse parameter masks.

Setting	Decoupled elements	Reported outcome
HRF for diffusion models	Denoising windows $\tau>0$9 are selected and updated separately	Better preservation of diversity at uncompromising mean rewards
Adaptive diffusion navigation	Visual encoder and lower decoder frozen; top 5 DiT layers and action head trained	SR 52.0%→58.7%, SPL 0.49→0.54, VRAM usage low (<17 GB)
Nested-ReFT	Layer-skipped nested behavior model generates off-policy rollouts	10–18% rollout-cost reduction with matched or slightly improved accuracy
DRIFT	Offline rollouts decoupled from optimization via importance-weighted SFT	≈2× faster than UFO-5turn and within 10–15% of standard SFT
FedDTL	Client image encoder and server text encoder trained separately; local SFT then RL	Balance between global task adaptation and generalization
CoDA	Planner strategic context isolated from executor workspace	Stable long-context behavior while other baselines degrade

In diffusion-model RL, Hierarchical Reward Fine-tuning treats the denoising process as a hierarchy of windows $m\in\{0,1\}^d$0 and updates one window per epoch, either with a fixed schedule or dynamically through a reward-gap criterion. The paper states that not every denoising step needs to be fine-tuned, adds clipping and a sliding-window regularization scheme, and reports that HRF variants preserve diversity substantially better than DDPO while maintaining competitive rewards; for example, on the aesthetic task, Inception Score changes from 2.07 to 1.58 under DDPO collapse, but to 2.04 under HRF and 1.91 under HRF-D, while Vendi Score remains near the DDPM baseline under HRF and drops sharply under DDPO (Barceló et al., 2024).

For diffusion-based robot navigation, reinforcement-learning fine-tuning is decoupled by freezing the visual encoder and the first $m\in\{0,1\}^d$1 lower DiT layers, and updating only the top $m\in\{0,1\}^d$2 layers plus the action head. This restricts training to approximately 6.8% of parameters, about 11.8M, keeps VRAM usage below 17 GB, and is paired with critic-free GRPO over multi-trajectory samples. On PointGoal in Isaac Sim, the reported improvement is from 52.0% to 58.7% in Success Rate and from 0.49 to 0.54 in SPL on unseen scenes, together with reduced collision frequency; the paper also reports 70% success on 20 real-world trials for the fine-tuned policy versus 50% for the pretrained NavDP (Sheng et al., 13 Mar 2026).

Nested-ReFT decouples the compute cost of the behavior policy from the target policy by sharing weights and randomly skipping transformer layers in the rollout model. The resulting off-policy gradient estimator remains unbiased under the support and bounded-importance assumptions stated in the paper, and empirical results on Qwen2.5-Math-Instruct 1.5B and 7B show that skipping 10–15% of layers with Retrace-$m\in\{0,1\}^d$3 matches or slightly improves reasoning accuracy while cutting rollout cost by 10–18%; a representative 1.5B SVAMP slice reports $m\in\{0,1\}^d$4 tokens/sec and $m\in\{0,1\}^d$5 runtime at 10% skip, and $m\in\{0,1\}^d$6 tokens/sec with $m\in\{0,1\}^d$7 runtime at 15% skip (Heuillet et al., 13 Aug 2025).

DRIFT makes a stronger separation: rollout collection is completely decoupled from optimization. Starting from the KL-regularized RL objective, the paper derives an equivalent importance-weighted supervised-learning objective in which the ideal target distribution is an exponential tilt of a fixed reference policy. It then samples multi-turn trajectories offline, computes prompt-normalized weights $m\in\{0,1\}^d$8, retains terminal turns, and performs weighted SFT. The reported result is that DRIFT-5turn matches or outperforms UFO on most benchmarks, is approximately 2× faster than UFO-5turn, and remains within 10–15% of the cost of standard SFT (Mu et al., 29 May 2026).

FedDTL and CoDA extend decoupling to distributed and hierarchical settings. FedDTL splits a CLIP-like VLM into client-side image encoding and server-side text encoding, applies LoRA in deeper layers, and uses a two-stage local schedule: SFT warm-start followed by GRPO-style RL with clipping and a KL term. Reported averages include 90.95% base accuracy and 82.64% novel accuracy in few-shot Dir(0.1), and 91.64% base with 77.72% novel accuracy in full-data Non-IID (Ma et al., 27 May 2026). CoDA, by contrast, uses a single LLM backbone in two contextually isolated roles—Planner and Executor—and optimizes both through PECO, a trajectory-level GRPO objective masked so that only agent-generated tokens contribute gradients. On seven QA benchmarks at Qwen2.5-3B scale, CoDA-Base improves average EM from 0.380 to 0.407 over the best baseline and remains within $m\in\{0,1\}^d$9 of peak F1 as retrieved context grows, while AutoRefine’s F1 drops 52% from 0.49 to 0.24 (Liu et al., 14 Dec 2025).

5. Theoretical viewpoints, entropy control, and non-decoupling results

Not all decoupling is about parameter subsets or rollout cost. Arbitrary Entropy Policy Optimization explicitly decouples reward maximization from entropy control. Its update is the sum of a standard GRPO gradient and an auxiliary REINFORCE gradient evaluated on temperature-adjusted samples from $$m_i= \begin{cases} 1,& \text{if } |\Delta\theta_i|>\tau\ 0,& \text{otherwise.} \end{cases}$$0, where $$m_i= \begin{cases} 1,& \text{if } |\Delta\theta_i|>\tau\ 0,& \text{otherwise.} \end{cases}$$1 if the measured entropy is below a target $$m_i= \begin{cases} 1,& \text{if } |\Delta\theta_i|>\tau\ 0,& \text{otherwise.} \end{cases}$$2 and $$m_i= \begin{cases} 1,& \text{if } |\Delta\theta_i|>\tau\ 0,& \text{otherwise.} \end{cases}$$3 otherwise. The paper reports that AEPO stabilizes entropy around arbitrary preset levels $$m_i= \begin{cases} 1,& \text{if } |\Delta\theta_i|>\tau\ 0,& \text{otherwise.} \end{cases}$$4, removes entropy collapse in GRPO, and reveals a non-monotonic entropy–performance relation on seven math benchmarks, peaking at average 61.4 for $$m_i= \begin{cases} 1,& \text{if } |\Delta\theta_i|>\tau\ 0,& \text{otherwise.} \end{cases}$$5 after 58.8 at $$m_i= \begin{cases} 1,& \text{if } |\Delta\theta_i|>\tau\ 0,& \text{otherwise.} \end{cases}$$6 and 60.9 at $$m_i= \begin{cases} 1,& \text{if } |\Delta\theta_i|>\tau\ 0,& \text{otherwise.} \end{cases}$$7, then declining to 58.0 at $$m_i= \begin{cases} 1,& \text{if } |\Delta\theta_i|>\tau\ 0,& \text{otherwise.} \end{cases}$$8 (Wang et al., 9 Oct 2025).

A more fundamental caveat is that sequential decoupling of SFT and RL objectives may be impossible under the assumptions of a given post-training setup. “On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-training” proves two claims: under SFT optimality, RL increases SFT loss, and under local RL optimality, SFT lowers RL reward. In its CoLA experiment with Qwen3-0.6B, the held-out cross-entropy loss rises above the base-model loss as soon as GRPO begins in the SFT-then-RL pipeline, while in the RL-then-SFT pipeline the reward falls below the base model, from base approximately 0.385 to final approximately 0.343 after SFT (Niu et al., 12 Jan 2026).

A different response is to decouple RL readiness from the RL algorithm rather than to claim objective-level separability. Behavior Injection identifies two conditions for effective RFT—RL-informative rollout accuracy and data co-influence—and modifies the SFT dataset through BRIDGE augmentation before RL. The paper describes this as a data-centric front end that “decouples the hard work of making a model RL-ready from the RL algorithm itself,” and reports substantially larger RFT–SFT gains than Vanilla, PP-Aug, or RC-Aug. On iGSM, BRIDGE yields $$m_i= \begin{cases} 1,& \text{if } |\Delta\theta_i|>\tau\ 0,& \text{otherwise.} \end{cases}$$9 for Qwen-1.5B and $S=\{i\in\{1,\dots,d\}:m_i=1\}$0 for Qwen-3B in in-distribution/OOD deltas; on PromptBench, it yields $S=\{i\in\{1,\dots,d\}:m_i=1\}$1 for Qwen-1.5B and $S=\{i\in\{1,\dots,d\}:m_i=1\}$2 for Llama-1B (Cen et al., 25 May 2025).

These results suggest that the literature uses “decoupling” in at least two non-equivalent senses. One sense concerns computational or architectural factorization—freezing modules, splitting rollouts from updates, or isolating contexts. The other concerns objective-level separability between SFT and RL. The former is widely supported by empirical systems; the latter is explicitly challenged by the non-decoupling result under its stated assumptions (Niu et al., 12 Jan 2026).

6. Significance, implications, and open problems

Taken together, the papers indicate that decoupling is becoming a general design principle for reinforcement-based post-training, but not a single theorem or recipe. In one regime, it means that RL touches only a sparse, stable parameter subset and can therefore be reproduced by masking gradients outside that subset (Balashov, 23 Jul 2025). In another, it means that rollouts can be generated by cheaper or frozen policies while optimization proceeds elsewhere, as in Nested-ReFT and DRIFT (Heuillet et al., 13 Aug 2025, Mu et al., 29 May 2026). In still others, it means isolating denoising windows, client/server modalities, or planner/executor contexts to reduce interference and preserve diversity or generalization (Barceló et al., 2024, Ma et al., 27 May 2026, Liu et al., 14 Dec 2025).

The practical consequences are already concrete. Reported benefits include reducing the effective number of trainable parameters to approximately 5–30% of model size in LLM RLHF (Balashov, 23 Jul 2025), keeping diffusion-navigation fine-tuning below 17 GB VRAM while improving Success Rate and SPL (Sheng et al., 13 Mar 2026), making multi-turn optimization approximately 2× faster than an online RL baseline while keeping SFT-like simplicity (Mu et al., 29 May 2026), and preserving diversity in diffusion RL where simultaneous fine-tuning of all denoising steps produces mode collapse (Barceló et al., 2024). A plausible implication is that decoupling methods are most useful when they preserve a strong pretrained prior and restrict RL to the components that actually mediate task-specific behavioral change.

Open problems remain explicit in the source literature. The sparse-subnetwork work calls for deeper theoretical study of why pretrained models contain reusable “alignment tickets” and for better interpretability of alignment circuits (Balashov, 23 Jul 2025). DRIFT raises coverage and rollout-refresh questions because fixed offline trajectories cannot discover behaviors absent from the reference policy; its two-stage refresh yields a reported $S=\{i\in\{1,\dots,d\}:m_i=1\}$3 multi@5 gain (Mu et al., 29 May 2026). Nested-ReFT suggests learning the skip mask or adapting the skip ratio per prompt or training stage (Heuillet et al., 13 Aug 2025). The non-decoupling analysis points toward interleaving, unified surrogate losses, adaptive $S=\{i\in\{1,\dots,d\}:m_i=1\}$4, trust-region strategies, reward shaping, and continual-learning-style regularizers rather than pure sequential pipelines (Niu et al., 12 Jan 2026).

Decoupled reinforcement finetuning is therefore best understood not as a claim that RL can always be separated cleanly from the rest of post-training, but as a family of mechanisms for restricting where adaptation occurs, when it occurs, and under which distribution it is computed. The strongest current evidence shows that such restrictions can be surprisingly effective, and in some settings nearly lossless, while the strongest caveat is that objective conflicts between SFT and RL do not disappear merely because the computation has been factorized.