Decoupled Fine-Tuning Approaches

• Decoupled fine-tuning is a strategy that partitions the fine-tuning process into distinct modules or losses to address conflicting adaptation requirements in pretrained models.
• It employs various patterns such as subtask, phase, and optimizer decoupling to isolate different learning signals and improve task-specific performance.
• This methodology enhances accuracy and mitigates catastrophic forgetting while optimizing computational efficiency, albeit with potential increases in training complexity.

Searching arXiv for papers on decoupled fine-tuning and related terminology. Decoupled fine-tuning is a family of adaptation strategies in which fine-tuning is not treated as a single homogeneous optimization problem, but is partitioned into components with different roles, constraints, or inductive biases. In the recent literature, that partition can occur at the level of subtasks, losses, modules, modalities, domains, optimization phases, or even rollout collection versus parameter updates. DualTune, for example, splits tool calling into tool selection and argument generation and trains dedicated LoRA adapters with separate loss masking (Kadekodi et al., 30 Sep 2025). Other work decouples rollback from task fitting in the optimizer (Ning et al., 21 Apr 2026), domain-specific heads from a frozen shared backbone (Wang et al., 2023), prompt stability from adapter plasticity (Fu et al., 2024), or offline trajectory generation from supervised optimization (Mu et al., 29 May 2026). This suggests that decoupled fine-tuning is best understood as a general design principle for isolating conflicting learning signals rather than as a single algorithm.

1. Conceptual basis and research lineage

A useful synthesis is that decoupled fine-tuning emerges whenever a pretrained model is asked to satisfy multiple adaptation requirements that are not well served by a single update rule. The literature identifies several such tensions explicitly: tool-name classification versus structured argument generation in local agentic systems (Kadekodi et al., 30 Sep 2025); preservation of upstream weights versus downstream adaptation in fine-tuning optimizers (Ning et al., 21 Apr 2026); general-domain retention versus domain specialization in LLM adaptation (Tang et al., 30 Sep 2025); and stability versus plasticity in few-shot detection and continual action recognition (Yin et al., 2024, Fu et al., 2024).

This broad usage has antecedents in earlier decoupling work on representation geometry and transfer architecture. “Decoupled Networks” reformulates convolution as a magnitude term multiplied by an angular term, with the norm associated with intra-class variation and the angle associated with semantic difference (Liu et al., 2018). “Rethinking embedding coupling in pre-trained LLMs” separates input and output embeddings and shows that larger output embeddings can improve downstream transfer even though the output embedding is discarded after pre-training (Chung et al., 2020). These papers are not fine-tuning procedures in the narrow post-training sense, but they establish the recurring premise that different functional roles should not necessarily share the same parameterization.

Across recent work, the central claim is not merely that decoupling adds modularity. Rather, the claim is that different adaptation targets produce different optimization signals. When these signals are entangled, fine-tuning can blur supervision, induce catastrophic forgetting, amplify domain domination, or increase inference cost. Decoupling is proposed as the corrective mechanism.

2. Recurrent design patterns

The literature uses “decoupled” in several technically distinct ways. The following patterns recur.

Pattern	Representative mechanism	Example papers
Subtask decoupling	Separate tool selection from argument generation with masked losses and dedicated adapters	(Kadekodi et al., 30 Sep 2025)
Phase decoupling	Train different components in different stages while freezing others	(Wang et al., 2023, Fu et al., 2024, Yin et al., 2024)
Optimizer decoupling	Anchor updates to pretrained weights or precomputed general gradients	(Ning et al., 21 Apr 2026, Tang et al., 30 Sep 2025)
Geometry decoupling	Separate update direction from update magnitude in low-rank adaptation	(Bini et al., 23 Mar 2025)
Modality decoupling	Split modality-specific and modality-independent signals, or preserve visual knowledge separately from task alignment	(Lin et al., 15 Apr 2025, Wu et al., 17 Feb 2025, Ma et al., 27 May 2026)
Trajectory decoupling	Collect rollouts offline, then optimize with weighted SFT	(Mu et al., 29 May 2026)

This taxonomy is interpretive, but it reflects the concrete mechanisms used in the cited papers. A plausible implication is that “decoupled fine-tuning” has become an umbrella term for methods that enforce separation where standard end-to-end fine-tuning would otherwise couple incompatible objectives.

3. Task- and trajectory-level decoupling

DualTune is the clearest recent instance of task-structured decoupling in LLM tool use. The method starts from the claim that “tool calling” is two qualitatively different problems: choosing the right tool and filling in its arguments. It trains one shared LoRA adapter for tool selection and a separate LoRA adapter for each tool’s argument generation, with loss computed only on tool-name tokens for the selector and only on argument tokens for the corresponding argument generator (Kadekodi et al., 30 Sep 2025). At inference time, the base model first routes the query to a toolset or MCP server, then vLLM loads the tool-selection adapter for that toolset, then swaps in the chosen tool’s argument-generation adapter, and the loop continues until the selector emits a special “summarize” token. The system is implemented in Rust with containerized tool execution.

The reported evidence is unusually direct. On MCP-Bench filesystem, DualTuneModel-7B, built from Qwen-2.5-7B, reaches 61.5 ToolFit versus 16.0 for the original Qwen-2.5-7B baseline; the abstract summarizes this as a 46% improvement (Kadekodi et al., 30 Sep 2025). The ablations isolate the source of the gain: replacing only the selector with GPT-5-mini raises ToolFit from 16.0 to 60.8, whereas replacing only argument generation raises it from 16.0 to 28.8. On the filesystem MCP-Bench ablation, hierarchical orchestration alone improves Qwen-2.5-7B from 10.4% to 16%, traditional fine-tuning raises performance to 39%, and decoupled fine-tuning reaches 61.5%. When the toolset is expanded from 12 tools to 30, tool descriptions and schemas consume about 9.8K tokens and ToolFit drops from 16% to 10.4%, which the paper presents as the failure mode addressed by hierarchical routing.

DRIFT applies decoupling at a different level: it separates rollout generation from optimization in multi-turn correction tasks. Instead of online RL with fresh trajectories at every update, it samples offline trajectories from a frozen reference policy, computes importance weights from a KL-regularized RL objective, and then performs weighted supervised fine-tuning (Mu et al., 29 May 2026). Its core relation is

$$p^*(\tau \mid x) \propto p_{\mathrm{ref}}(\tau \mid x)\exp\!\left(\frac{R(\tau)}{\beta}\right),$$

which yields a weighted SFT objective under the reference-policy trajectory distribution. The method retains only the terminal turn from each rollout and trains with weighted cross-entropy on that final response conditioned on the full interaction history. Empirically, DRIFT is reported to match or exceed multi-turn reinforcement-learning baselines while maintaining the training efficiency and simplicity of standard supervised fine-tuning (Mu et al., 29 May 2026).

A common misconception is that task decoupling is only a prompt-engineering device. The DualTune and DRIFT results suggest otherwise: in both cases, the decoupling alters the training objective itself, either through token-span masking or through importance-weighted supervision derived from a KL-regularized policy objective.

4. Update- and parameter-level decoupling

A second major line of work decouples the geometry of parameter updates. DeLoRA modifies LoRA by normalizing the low-rank update and learning a separate boundary parameter so that update direction and update strength are optimized independently (Bini et al., 23 Mar 2025). Its central update is

$\Delta W = \frac{\lambda\|\bar W\|}{r} B\Xi A,$

with the bound

$$\left\|\frac{\lambda}{r}B\Xi A\right\| \le \lambda.$$

The paper describes this as decoupling angular learning from adaptation strength. It reports stronger robustness than LoRA and DoRA under learning-rate changes and prolonged training, while retaining expressivity beyond bounded methods such as ETHER. On GLUE with RoBERTa-base, DeLoRA achieves an average score of 85.6, compared with 84.7 for LoRA and 85.0 for Adapter; on LLaMA-2-7B instruction tuning, it reports the best average performance, 41.42, versus 40.19 for LoRA and 40.73 for DoRA (Bini et al., 23 Mar 2025).

Dynamic Boosted Annealing decouples fine-tuning in a different sense: general-domain learning is isolated into a one-time gradient-extraction stage, and later domain training uses only domain data while being guided by the stored global gradient (Tang et al., 30 Sep 2025). The paper computes a global gradient under zero learning rate, $\eta_G = 0$, and then forms the boosted gradient

$$g_{B,t} = \gamma_t \hat{g}_G + (1-\gamma_t)g_{D,t}.$$

Dynamic correction uses gradient similarity, and annealed learning uses a very small learning rate with linear decay to zero. The paper reports an average 5.8% improvement in joint performance over vanilla fine-tuning and a 91.0% GPU-hour reduction. On News QA with Qwen3, direct fine-tuning collapses to $S_G=3.83$, whereas DBA preserves general knowledge with $S_G=68.52$ while maintaining $S_D=80.82$, giving $S=74.16$ (Tang et al., 30 Sep 2025).

These methods indicate that decoupled fine-tuning does not require freezing most of the model or introducing task-specific heads. It can instead act directly on the update geometry, constraining how far the solution moves, or on the provenance of the optimization signal, separating general guidance from domain adaptation.

5. Stability, plasticity, and forgetting mitigation

Several papers present decoupled fine-tuning as a remedy for stability–plasticity conflict. DualOpt explicitly distinguishes scratch training from fine-tuning and argues that fine-tuning must preserve useful upstream knowledge while adapting to downstream data (Ning et al., 21 Apr 2026). Its rollback update

$$\theta_{t+1} = (1-\lambda)(\theta_t - \eta_t g_t) + \lambda \theta_0$$

anchors each step toward the pretrained weights $\Delta W = \frac{\lambda\|\bar W\|}{r} B\Xi A,$0, and a layer-wise penalty sets stronger rollback in shallow layers and weaker rollback in deeper layers. On a PACS ablation with ViT-B + Adam, baseline full fine-tuning achieves 87.79, adding weight rollback yields 93.41, adding layer-wise penalty gives 93.70, and adding diversified decay rate reaches 94.38 (Ning et al., 21 Apr 2026). The paper interprets rollback as the main contributor and layer-wise decoupling as a further refinement.

MDGD addresses forgetting in multimodal instruction tuning by separating visual preservation from task alignment. The paper defines visual forgetting as a reduction in effective rank,

$\Delta W = \frac{\lambda\|\bar W\|}{r} B\Xi A,$1

and constructs task gradients orthogonal to the visual-drift direction while adding a visual alignment loss (Wu et al., 17 Feb 2025). It reports that, on LLaVA fine-tuned on OKVQA, MDGD reduces average pre-trained-task performance drop to 3.59%; on LLaVA fine-tuned on Flickr30K, MDGD yields a 1.45% improvement on pre-trained tasks. Its gradient-masking variant, MDGD-GM, updates only about 10% of the original parameters in the reported experiments (Wu et al., 17 Feb 2025).

The same stability–plasticity language appears in few-shot end-to-end object detection. For Sparse R-CNN, the stability–plasticity contradiction is stated as follows: the re-initialized classifiers need high plasticity, while inherited detector modules need high stability (Yin et al., 2024). The proposed Plasticity Classifier Fine-tuning stage updates only the cascade classifiers $\Delta W = \frac{\lambda\|\bar W\|}{r} B\Xi A,$2 on $\Delta W = \frac{\lambda\|\bar W\|}{r} B\Xi A,$3 while freezing the remaining modules $\Delta W = \frac{\lambda\|\bar W\|}{r} B\Xi A,$4, after which a final fine-tuning stage applies additional stabilization through partial backbone freezing and multi-source ensemble. In the ablation table, baseline Sparse R-CNN has 9.7 AP, adding PCF raises it to 13.5 AP, adding ME alone gives 11.9 AP, and combining PCF and ME yields 17.7 AP (Yin et al., 2024).

A plausible synthesis is that stability–plasticity decoupling is one of the most mature forms of decoupled fine-tuning: it appears in optimizers, gradient surgery, detector training schedules, and continual learning, but the shared aim is always to prevent adaptation-critical parameters from being regularized in the same way as knowledge-bearing parameters.

6. Domain-, modality-, and deployment-oriented decoupling

Another major branch decouples adaptation according to domain structure or modality structure. D-Train in multi-domain learning uses a tri-phase general-to-specific schedule: pre-train a root model on all domains, post-train by splitting into multi-heads, and finally fine-tune the heads with the backbone fixed (Wang et al., 2023). The third phase is the decisive one: each domain head is optimized independently on its own domain while the shared representation is frozen. The paper reports average accuracy: 86.0 on Office-Home, and on DomainNet average accuracy: 77.3 with worst accuracy: 72.2; it also states that D-Train (w/o Fine-tune) performs much worse, making the third phase critical (Wang et al., 2023).

DPAT in continual action recognition likewise uses a frozen pretrained ViT backbone but separates prompts and adapters across two stages per task (Fu et al., 2024). Stage 1 tunes prompts and classifier head while adapters are frozen; Stage 2 freezes prompts and tunes adapters, task key, and classifier head. The paper frames prompts as the source of stability and generalization, and adapters as the source of plasticity and task adaptation. On Kinetics-400 with CLIP pretraining, DPAT reports 61.3% Acc and 22.3% BWF; the ablation shows that removing the temporal adapter drops accuracy from 61.3% to 33.1%, removing all adapters lowers it to 25.0%, and removing all prefixes increases forgetting to 33.6% BWF (Fu et al., 2024).

DMPT and DAPT transpose the same principle to multimodal prompt tuning. DMPT freezes the CLIP/ViT backbone and decouples visual prompts into modality-specific prompts and modality-independent semantic prompts, then reconnects the semantic prompts through Prompt Inverse Bind (Lin et al., 15 Apr 2025). The paper states that only 17.9M parameters are updated, which is 6.5% of the total backbone parameters, and that the method saves up to 93.5% of training memory compared with full fine-tuning. On RGBNT201, using only semantic prompts gives 71.7 mAP / 74.4 Rank-1, adding bind prompts gives 73.5 / 76.9, adding modality prompts gives 76.7 / 80.6, and the full DMPT gives 78.5 / 81.3 (Lin et al., 15 Apr 2025). DAPT, by contrast, decouples the visual modality into foreground and background representations before alignment and reports an average improvement of DAPT-S around +1.94% across shot settings in few-shot recognition; it also reports that DAPT reaches comparable performance to MaPLe using only 50% of the training data (Zhang et al., 1 Aug 2025).

FedDTL extends decoupled fine-tuning to federated VLM adaptation. The image encoder is trained locally on clients, while the text encoder is trained globally on the server using class prompts and uploaded image embeddings (Ma et al., 27 May 2026). Local adaptation is itself two-stage: supervised fine-tuning first, then reinforcement learning to reduce over-specialization. The paper reports that decoupled encoder training significantly improves base accuracy, especially under Non-IID and full-data settings, and that two-stage local fine-tuning improves novel accuracy and harmonic mean (Ma et al., 27 May 2026). This suggests that decoupling can be used not only to separate objectives inside a model, but also to allocate optimization responsibility across a distributed system.

A final deployment-oriented example is DKDL-Net, where the pipeline itself is decoupled: teacher training with cross-entropy, student distillation with decoupled knowledge distillation, and then LoRA fine-tuning of the frozen student (Petrosian et al., 2024). The final model has 6,838 trainable parameters versus 69,626 for the teacher, achieves 99.48% accuracy on the test set, and reduces average inference time from 3816 μs/sample to 1757 μs/sample (Petrosian et al., 2024).

7. Benefits, limitations, and scope conditions

The empirical record across these papers points to several recurring benefits. Decoupled fine-tuning often improves accuracy relative to monolithic fine-tuning in settings where objectives are visibly heterogeneous: DualTune improves tool calling from 16.0 to 61.5 ToolFit on MCP-Bench filesystem (Kadekodi et al., 30 Sep 2025); DualOpt improves PACS accuracy from 87.79 to 94.38 with rollback and layer-wise control (Ning et al., 21 Apr 2026); D-Train outperforms its shared-bottom counterparts on Office-Home and DomainNet (Wang et al., 2023). It also frequently reduces forgetting or collapse: MDGD preserves effective rank and pre-trained performance (Wu et al., 17 Feb 2025), DBA preserves general capability while training on domain data only (Tang et al., 30 Sep 2025), and DPAT reduces forgetting in continual action recognition without replay memory (Fu et al., 2024). In deployment-oriented work, decoupling is also used to improve latency, memory, or operational feasibility, as in DualTune’s adapter swapping on consumer GPUs (Kadekodi et al., 30 Sep 2025) and DMPT’s 6.5% trainable-parameter budget (Lin et al., 15 Apr 2025).

The limitations are equally consistent. Decoupling can create maintenance overhead: in DualTune, adding a new tool requires updating the shared selector and training a new argument adapter (Kadekodi et al., 30 Sep 2025). It can depend on structured task boundaries: DPAT assumes task-specific prompt selection and is not directly suited to online learning (Fu et al., 2024). It can be limited by offline coverage: DRIFT can only upweight behaviors already present in trajectories sampled from the fixed reference policy (Mu et al., 29 May 2026). Some methods are explicitly approximations rather than exact objective optimizers, such as DRIFT’s terminal-only supervision in place of full-trajectory weighting (Mu et al., 29 May 2026). Others introduce new hyperparameters, such as the rollback schedule in DualOpt or the balance coefficients in DBA (Ning et al., 21 Apr 2026, Tang et al., 30 Sep 2025).

A useful corrective to overgeneralization is that decoupled fine-tuning is not synonymous with PEFT. Some methods are PEFT-based, including DualTune, DeLoRA, DMPT, and FedDTL (Kadekodi et al., 30 Sep 2025, Bini et al., 23 Mar 2025, Lin et al., 15 Apr 2025, Ma et al., 27 May 2026). Others modify the optimizer, the training schedule, or the dataflow while leaving the basic parameterization intact, as in DualOpt, D-Train, DRIFT, and stability–plasticity few-shot detection (Ning et al., 21 Apr 2026, Wang et al., 2023, Mu et al., 29 May 2026, Yin et al., 2024). Nor is decoupling always about freezing more parameters. In several cases, it is instead about reorganizing supervision so that different parameters, losses, or data sources are not forced into the same update channel.

The broader implication is that decoupled fine-tuning has become a general methodology for transferring pretrained models under structural mismatch. Where standard fine-tuning assumes one objective, one update rule, and one adaptation path, decoupled fine-tuning asks which parts of the problem should actually remain coupled. The strongest results in the current literature come precisely from answering that question narrowly and operationally rather than treating “fine-tuning” as a single indivisible procedure.