Downstream Fine-Tuning: Adaptation Strategies

• Downstream fine-tuning is a transfer-learning procedure that adapts a pretrained model to a target task using task-specific supervision.
• It includes full-model, layer-selective, and parameter-efficient methods that update only crucial parameters to balance specialization and retention.
• Practical challenges involve preserving pretrained capabilities, optimizing sample construction, and managing efficiency and deployability constraints.

Downstream fine-tuning is the transfer-learning procedure in which a pretrained model is adapted to a target task, domain, or deployment constraint using task-specific supervision after pretraining. In current practice it is the standard adaptation setting for language encoders, vision and vision-language backbones, 3D point-cloud models, multimodal LLMs, diffusion models, mixture-of-experts systems, and mobile-edge LLM deployments. Across these settings, the central technical question is not merely whether to fine-tune, but which parameters, objectives, sample constructions, and systems mechanisms should govern adaptation so that specialization is obtained without unnecessary loss of pretrained capability, robustness, or deployability (Lodha et al., 2023, Yu et al., 2023, Shachaf et al., 2021).

1. Problem setting and scope

In the language-encoder literature, downstream fine-tuning is the standard setting in which a pretrained encoder such as BERT-base-cased is adapted to supervised target tasks from GLUE and SuperGLUE, with full-model fine-tuning serving as the usual baseline and selective alternatives introduced as efficiency-oriented variants (Lodha et al., 2023). In work on LLM sample construction, the setting is distinguished from prompt engineering and instruction tuning: “downstream-tuning” is defined for specific tasks and domains in which output stability and machine-readability matter, including multi-aspect sentiment analysis, event extraction, and nested named entity recognition (Guo et al., 2024).

The same term covers more structurally varied models. In multimodal LLMs, the model is described as a composition of a vision encoder $f$, connector $\varphi$, and LLM $g$, with downstream fine-tuning usually updating the connector and some LLM layers while freezing the vision encoder (Huang et al., 2024). In partial-class classification, the setting consists of fine-tuning a pretrained classifier on a subset of classes $S \subset Y$ and then evaluating on the full label space $Y=S\cup U$, so that the effect of adaptation on classes absent during fine-tuning can be isolated (Mai et al., 2024). In wireless systems, the same adaptation step is recast as a distributed optimization problem constrained by device memory, bandwidth, and local private data (Zhang et al., 16 Jan 2025).

This breadth suggests that downstream fine-tuning is better regarded as a family of post-pretraining adaptation procedures than as a single algorithm. The common denominator is parameter update under a target objective; the differences lie in parameter scope, auxiliary constraints, and the degree to which the pretrained representation is preserved or structurally redirected.

2. Principal adaptation regimes

A first axis of variation is the amount of the pretrained model that is updated. The language-encoder literature contrasts full-model fine-tuning with selective or “surgical” fine-tuning, in which only a subset of layers is updated. One concrete criterion is the diagonal Fisher Information Matrix score, aggregated per layer and used to rank layers before training; empirically, fine-tuning the top 5 ranked layers is usually sufficient to approach full-model performance on many GLUE and SuperGLUE tasks, and the ranking remains stable across epochs $0,2,5,8,10$ (Lodha et al., 2023).

A second axis is parameter-efficient fine-tuning. The visual-tuning survey organizes this regime into prompt tuning, adapter tuning, parameter tuning, and remapping tuning. Prompt methods reparameterize the input; adapters insert small trainable modules; parameter methods update only biases, low-rank factors, or reparameterized weights; remapping methods distill or transfer into a downstream architecture. The survey explicitly treats these as downstream tuning because they reuse the source model while tuning only a small portion of parameters, often for edge deployment and cloud-to-edge reuse of large backbones (Yu et al., 2023).

Task-specific architectures refine this picture. TS-SAM keeps the SAM ViT image encoder frozen and trains only the Convolutional Side Adapter, Multi-scale Refinement Module, and Feature Fusion Decoder; it discards the original SAM mask decoder and does not rely on point or box prompts for decoding. TwinTURBO likewise freezes the RADIOv2.5-B backbone and updates only a lightweight transformer-projector stack, predictor $h_\psi$, and discriminator $D_\alpha$, using mutual-information lower bounds in both task space and latent space to exploit labelled and unlabelled data under low-label regimes (Yu et al., 2024, Quétant et al., 10 Mar 2025).

Other regimes modify not just parameter count but the adaptation target itself. RA-MoE augments standard supervised fine-tuning for multilingual MoE models with a routing-alignment term over middle-layer task experts on $ci$-type examples, where the model is correct in English and incorrect in the target language. Diffusion-Sharpening reformulates diffusion-model fine-tuning as trajectory-level reward optimization, selecting denoising paths during training rather than paying extra inference-time sampling costs (Deng et al., 27 May 2026, Tian et al., 17 Feb 2025).

These variants indicate that downstream fine-tuning increasingly operates at the level of learned structure—routing, trajectory, or task-relevant subspace—rather than merely at the level of “all weights” versus “few weights.”

3. Specialization, retention, and robustness

A recurring theme is the tension between downstream specialization and preservation of pretrained capability. In pretrained 3D point-cloud models, linear probing preserves robustness but often underfits the target task, whereas full fine-tuning improves in-distribution accuracy but can distort the original backbone features and reduce robustness under distribution shift. WiSE-FT-LP addresses this by interpolating pretrained and fine-tuned weights,

$$\theta_{WiSE}(\alpha)=(1-\alpha)\theta_{PT}+\alpha\theta_{FT},$$

then freezing the interpolated backbone and retraining only the head. On Point-M2AE for ScanObjectNN PB_T50_RS, full fine-tuning reaches 86.43% downstream accuracy with 89.59% backbone robustness, while WiSE-FT-LP reaches 86.99% accuracy and 90.68% robustness; on OBJ_ONLY it reaches 89.33% accuracy and 91.29% robustness versus 88.81% and 89.14% for full fine-tuning (Zhang et al., 2024).

In partial-class fine-tuning, a different diagnosis emerges. “Fine-Tuning is Fine, if Calibrated” argues that the main failure is not catastrophic destruction of absent-class features, but discrepant logit scales between fine-tuning classes and absent classes. The feature extractor often improves absent-class discrimination, while the classifier becomes biased toward seen classes. The proposed correction is a simple additive calibration,

$\varphi$0

which restores substantial absent-class accuracy without retraining on absent-class labels (Mai et al., 2024).

Comparable preservation-oriented designs appear in multimodal and visual settings. SPIDER for MLLM fine-tuning measures parameter importance from frozen pretrained weight magnitude and accumulated fine-tuning gradient values, then updates mainly those parameters whose downstream importance exceeds pretrained importance, restoring the rest toward frozen values. DR-Tune regularizes the downstream head on the pretrained feature distribution rather than directly constraining the backbone, and then applies semantic calibration through global rotation and class-level translation so that the pretrained and downstream feature spaces remain aligned despite semantic drift (Huang et al., 2024, Zhou et al., 2023).

Taken together, these results challenge the simple claim that downstream fine-tuning necessarily “forgets” pretraining. The more precise picture is that forgetting may arise from distorted feature geometry, misallocated parameter updates, or score-calibration mismatch, depending on the regime.

4. Data regime, task-head preparation, and sample construction

The quality of downstream fine-tuning depends not only on which parameters are updated, but also on the optimization signal presented to them at the start of adaptation. “How to prepare your task head for finetuning” decomposes feature adaptation into a kernel term, a direction term, and an energy term, and defines the Average Initial Energy

$\varphi$1

Its central finding is that as this initial energy increases, Euclidean and cosine distances between resulting and original features increase, while dot products and resulting feature norms first increase and then decrease. The practical recommendation is not to head-probe to full convergence by default: earlier stopping of head probing, label smoothing during head probing, or changes in head capacity can produce better downstream performance by preserving useful residual adaptation energy (Ren et al., 2023).

Sample construction itself also matters. Sample Design Engineering studies input design, output design, and reasoning design for LLM downstream-tuning and reports several stable patterns: explicit instructions help, Inst-first is usually better than Inst-last, No-MI is better than MI, Lines is the most reliable output format, TxtLabel tends to beat NumLabel, PU is generally better than OU, and CoT helps more in out-of-domain than in-domain settings. The integrated strategy Inst-first + No-MI + Lines + PU + TxtLabel is proposed as ES-SDE, and the paper reports that 500 ES-SDE samples can be worth roughly 2,000 EW-SDE or heuristic samples on GENIA and Review11 (Guo et al., 2024).

When labelled data are extremely scarce, fine-tuning can be regularized with unlabelled data. TwinTURBO couples a downstream-task mutual-information bound with a latent-space InfoNCE-style term and a discriminator-based marginal-matching critic. On SVHN with only 100 labelled pairs, adding the critic yields about a 5% accuracy gain over the baseline, and the binary cross-entropy variant is reported to improve performance by roughly 1–3% in several settings while reducing seed variance (Quétant et al., 10 Mar 2025).

These studies imply that downstream fine-tuning quality depends on the initial optimization landscape induced by the head, the formatting of training examples, and the availability of auxiliary unlabelled structure—not only on model scale.

5. Efficiency, sparsity, and deployment constraints

As foundation models scale, downstream fine-tuning is increasingly shaped by cost. The visual-tuning survey argues that partial-parameter tuning can match or exceed full fine-tuning while using far fewer trainable parameters, and associates this with lower storage overhead, lower training and inference memory, and better deployability on autonomous vehicles, drones, and robots. The survey cites, for example, DAN as achieving comparable or higher performance with only 13% of the parameters of standard fine-tuning, and DP-BiTFiT as saving over 60% training time (Yu et al., 2023).

GEM pushes this logic into static sparse fine-tuning. It ranks parameters by the gradient-to-weight ratio,

$\varphi$2

and allocates layer budgets using entropy of the score distribution. On GLUE/SuperGLUE-style tasks it updates only 0.1% of parameters and reports the best seven-task average for OPT-125M, OPT-1.3B, and Phi-2. For OPT-125M, the reported average is 68.85 for GEM versus 67.23 for full fine-tuning, a gain of +1.62; the abstract states that the method achieves up to a 1.6% improvement in fine-tuning accuracy over full fine-tuning while updating only 0.1% of model parameters (Kang et al., 22 Aug 2025).

Deployment constraints become explicit in wireless systems. Split Fine-Tuning partitions an LLM between a mobile device and an edge server, keeps LoRA adapters on both sides, compresses activations and gradients through sparsification, stochastic quantization, and lossless encoding, and jointly optimizes transformer-block allocation, compression rate, and spectrum resources. The paper reports up to 80.2% reduction in fine-tuning delay and 93.6% reduction in communication overhead compared to state-of-the-art benchmarks, while satisfying device-side memory and model accuracy constraints (Zhang et al., 16 Jan 2025).

Efficiency-oriented downstream fine-tuning is therefore not a single PEFT recipe. It includes selective layer updates, static sparse masks, frozen-backbone side networks, and systems-level model partitioning, each optimized against a different bottleneck.

6. Security, safety, and persistence under later updates

Downstream fine-tuning also exposes security and safety risks that are specific to post-pretraining adaptation. One line of work shows that the creator of an open-source LLM can backdoor the released checkpoint so that, after the downstream developer fine-tunes it on proprietary data, the creator can later extract the downstream training queries with only black-box access. The reported extraction performance reaches as much as 76.3% of 5,000 downstream queries in practical settings and 94.9% in more ideal settings, and the paper further shows that detection based on natural-language probing can be bypassed by semantically meaningless triggers (Zhang et al., 21 May 2025).

Another line of work studies whether downstream fine-tuning can undo prior safety interventions. ILU argues that existing LLM unlearning methods such as NPO and RMU are fragile because unrelated downstream fine-tuning can recover forgotten information. By adding an IRM-style invariance penalty during unlearning, ILU improves robustness under later fine-tuning: average robust accuracy rises from 0.42 to 0.65 for RMU and from 0.47 to 0.56 for NPO, while downstream fine-tuning accuracy remains strong (Wang et al., 2 Jun 2025).

The same fragility appears in diffusion models. ResAlign treats downstream fine-tuning as an implicit optimization problem via a Moreau Envelope-based reformulation and adds meta-learning over diverse simulated fine-tuning scenarios. The paper reports that state-of-the-art safety-driven unlearning methods can lose safety even after fine-tuning on benign datasets, whereas ResAlign retains substantially lower harmfulness; after fine-tuning on DreamBench++, “Ours” reports IP 0.0186 and US 0.0050, compared with IP 0.1038 and US 0.0317 for AdvUnlearn (Li et al., 22 Jul 2025).

Conversely, downstream fine-tuning is not always a defense against malicious preconditioning. INFUSE identifies fine-tune-insensitive VLA modules using parameter drift, Fisher-weighted drift, and activation shift, and injects the backdoor only there. The paper reports mean attack success rates of 91.0% on simulation environments and 79.8% on real-world robot tasks after user-side fine-tuning, far above BadVLA’s 38.8% and 36.6%, respectively (Zhou et al., 31 Jan 2026).

These results show that downstream fine-tuning can either recover suppressed behaviors or preserve implanted ones. A plausible implication is that adaptation procedures must increasingly be audited not only for utility but also for what they reactivate, reveal, or fail to erase.

7. Theoretical characterizations and checkpoint selection

Theoretical work explains why downstream fine-tuning succeeds only under certain forms of source-target relation. For linear teachers, the exact population risk after fine-tuning from pretrained initialization is

$\varphi$3

so success depends not simply on $\varphi$4, but on the component of the source-target difference that lies in directions not recovered by target samples. In deep linear regression, the relevant similarity becomes increasingly directional with depth; for aligned source and target differing only by a scalar, the paper shows that the risk goes to zero as depth tends to infinity, and it further shows that freezing early layers can collapse transferability in this theoretical model (Shachaf et al., 2021).

A complementary viewpoint treats checkpoint choice itself as a downstream-fine-tuning problem. The downstream free energy criterion ranks pretraining checkpoints by local Bayesian evidence for downstream adaptation, measuring how much nearby parameter mass is favorable for the eventual target task. Because downstream data are unavailable at checkpoint-selection time, the paper introduces a pretraining analogue estimated by WBIC and argues that lower pretraining free energy predicts better downstream fine-tuning. On CIFAR-FS with ResNet-18, lower pretraining WBIC correlates strongly with improved full and 5-shot fine-tuning accuracy across sweeps of learning rate, batch size, and momentum (Munn et al., 2024).

The combined theoretical picture is that downstream fine-tuning is controlled by local geometry: covariance structure, implicit bias induced by architecture and initialization, the head-induced energy available for feature movement, and the local volume of good solutions around a checkpoint. This suggests that successful adaptation is less a matter of maximizing parameter change than of steering change into the right subspace, at the right scale, and from the right starting point.