Parameter-Based Transfer Learning

• Parameter-based transfer learning is a family of techniques that reuse learned parameters from source tasks to efficiently adapt models for target tasks.
• It encompasses approaches such as adapter-based, prompt-based, and low-rank reparameterization methods that finely control parameter updates.
• These methods improve efficiency by reducing retraining costs, mitigating overfitting on limited data, and preserving pretrained knowledge.

Parameter-based transfer learning denotes a family of transfer-learning strategies in which knowledge acquired on a source task or domain is reused through parameters rather than only through shared data representations or output labels. In a general formulation, the target predictor has the form $h_{T,w,\theta}(x)=\langle w,\psi_\theta(x)\rangle$, so that a parameter $\theta$ learned or estimated in one setting determines the feature map used in another (Kumagai, 2016). In contemporary large-model practice, the dominant realization is parameter-efficient transfer learning (PETL, also PEFT), where a pretrained backbone remains mostly frozen and only a small task-specific parameter set is optimized on the downstream task (Ding et al., 2024). The literature accordingly spans direct parameter inheritance, learned parameter projection, gated source–target fusion, sparse “model patches,” adapters, prompts, low-rank updates, channel or token selection, and, in some application-specific settings, transfer of optimized procedure parameters such as architecture choices and stopping points rather than network weights alone (Chen et al., 2018, Meyer et al., 17 Jan 2026).

1. Conceptual scope and relation to adjacent paradigms

Parameter-based transfer learning is best understood against two contrasting baselines. At one extreme, linear probing freezes the pretrained backbone and learns only a task head; at the other, full fine-tuning updates all model weights. PETL occupies the intermediate regime: it preserves most pretrained parameters while allowing controlled target adaptation through a deliberately small or structured parameter subset (Mai et al., 2024). This framing is motivated by recurring drawbacks of full fine-tuning reported across vision, speech, and multimodal settings: large storage cost from maintaining a separate model per task, high compute and memory demand, overfitting under limited downstream data, and catastrophic forgetting of pretrained knowledge (Zhao et al., 2023, Li et al., 2023).

The literature also distinguishes parameter transfer from feature- or transform-based transfer. In the PTELM line, feature-based methods learn a shared representation or transformation that reduces source–target discrepancy, whereas parameter transfer assumes that the transferable object is encoded directly in model parameters and that the target model should be improved by reusing or constraining those parameters (Chen et al., 2018). This distinction remains useful even in modern PETL: some methods primarily alter attention or MLP weights, some modulate normalization parameters, and some act on latent activations through auxiliary modules, but all treat parameters as the carrier of reusable knowledge.

A further broadening appears in domain-specific work where the transferred object is not a trained predictor but a parameter configuration for an optimization procedure. AUTO-DIP, for example, transfers Deep Image Prior architecture settings and stopping iterations between similar fluorescence microscopy images, explicitly presenting this as parameter-based transfer learning because the transferred object is the optimal hyperparameter configuration rather than learned denoiser weights (Meyer et al., 17 Jan 2026). This suggests that the concept is broader than backbone adaptation alone.

2. Mechanisms of parameter reuse and adaptation

One mechanistic family transfers parameters directly and learns how strongly or how faithfully they should influence the target model. PTELM replaced the conventional penalty $\|\Theta_s-\Theta_t\|^2$ with a projective relation $\Theta_t=\mathcal{M}\Theta_s$, instantiated in extreme learning machines as $\beta_t=M\beta_s$; the target classifier is therefore a projected version of the source classifier rather than a merely proximate copy (Chen et al., 2018). PTU introduced a finer-grained alternative: instead of hand-crafted “freeze” or “fine-tune” states, it learns a fine-tune gate $r_l$ and an update gate $z_l$ that nonlinearly combine source and target activations at each layer, so that transferability is learned end-to-end rather than specified heuristically (Zhang et al., 2018). In model-based RL, fractional transfer learning made this interpolation explicit through $W_T=W_T+\omega W_S$, with $\omega\in[0,1]$ controlling the degree of inheritance between random initialization and full transfer (Sasso et al., 2021).

A second family adds small task-specific parameter overlays to a frozen backbone. The early “model patch” formulation showed that a small distributed patch can replace full retraining: scale-and-bias parameters across the network, often corresponding to normalization parameters, and lightweight depthwise-convolution patches were sufficient to specialize pretrained networks to new tasks while reusing most shared weights (Mudrakarta et al., 2018). In current PETL terminology, the main overlay families are adapter-based, prompt-based, and reparameterization-based methods. Adapter-based methods insert a bottleneck module, typically with residual form $h' = h + \mathrm{Adapter}(h)$; prompt-based methods prepend or inject learnable prompt vectors, often written as $\theta$0; and reparameterization-based methods encode a full update through a low-rank factorization such as $\theta$1 or by tuning only bias terms (Ding et al., 2024).

A third family focuses on where the adaptation resides inside the network. Visual-PETL treats prompt tuning, adapters, prefix tuning, LayerNorm tuning, and related methods as parameter-insertion strategies whose performance depends strongly on the position of trainable parameters within Transformer blocks (Yu et al., 2022). This view supports a more structural understanding of transfer: parameters inserted into attention, MLP, normalization, or classifier components do not play interchangeable roles, and the same parameter budget can yield materially different behavior depending on placement.

3. Parameter placement, selection, and staged adaptation

A central finding in visual PETL is that the position of trainable parameters matters as much as the parameter count itself. In Swin-style backbones, tuning only the FC layer is cheap but weak on larger datasets, LayerNorm tuning alone is insufficient, and attention QKV and MLP-related positions are substantially more effective; the proposed PATT module accordingly adapts the attention branch by deriving prefix-like parameters from the previous layer output and inserting them in parallel to the standard attention pathway (Yu et al., 2022). The resulting Swin-BAPAT hybrid combines attention-side and MLP-side adaptation, illustrating the broader principle that transfer quality depends on matching the trainable subspace to the architecture’s functional bottlenecks.

Recent work has also shifted from uniform adaptation toward explicit parameter selection. TTC-Tuning argues that PETL should first address task distribution shift and only then allocate limited adaptation capacity to task-relevant channels. Its first stage tunes only LayerNorm scale and shift parameters to align the pretrained model to the target distribution; its second stage computes a Taylor expansion-based importance score, approximated by a gradient–weight product, aggregates those scores at the channel level, selects the top-$\theta$2 channels, and tunes only that small subset through a residual linear module (Zhao et al., 2023). The method thereby departs from end-to-end PEFT that assumes all channels are equally available from the outset.

A related but broader notion appears in unsupervised domain adaptation. TransPar proposes that only a subset of network parameters is genuinely transferable across domains, while the remainder tends to fit domain-specific details. It estimates parameter importance iteratively through $\theta$3, partitions parameters into transferable and untransferable subsets according to the distribution discrepancy, and updates the two groups differently: transferable parameters receive task gradients, while untransferable ones are pushed toward zero by weight decay alone (Han et al., 2021). In this formulation, parameter selection is not merely an efficiency device; it is a mechanism for controlling domain specificity.

4. Efficiency objectives beyond tunable-parameter count

Although PETL is often introduced as a way to reduce the number of trainable parameters, several studies emphasize that trainable-parameter reduction does not automatically imply low memory usage or fast inference. SHERL and S2A explicitly target this gap. SHERL decouples adaptation into early anti-redundant aggregation of shallow intermediate outputs and late regulation through minimal pretrained layers, aiming to combine PETL-like adaptation quality with memory-efficient transfer learning; on GLUE with T5-base it used 0.85M trainable parameters, 2.9 GB training memory, and reached 84.3 average score, compared with LST’s 1.74M parameters, 5.5 GB, and 84.1 average, while reducing training memory by about 84% relative to full fine-tuning (Diao et al., 2024). S2A adopts an activation-aware perspective: it redesigns parametric tuning modules to avoid storing full backbone activations and quantizes non-parametric activations such as Softmax and GELU to 4 bits, yielding, for ViT-B/16, 0.90% trainable parameters and 640 MB memory versus 4099 MB for full fine-tuning, with roughly a fourfold reduction in GPU memory footprint on average compared with prior PETL methods (Jin et al., 11 Mar 2025).

Inference efficiency introduces a further axis. CoDA keeps a dense pretrained Transformer frozen but activates its expensive branch only for a routed subset of tokens, while a lightweight adapter branch processes all tokens. The router uses a differentiable soft top-$\theta$4 mechanism, and the reduction factor $\theta$5 controls the speed–accuracy trade-off. Across language, vision, and speech tasks, CoDA reports a 2x to 8x inference speed-up compared to state-of-the-art Adapter approaches with moderate to no accuracy loss and the same parameter efficiency, while routing iterations add less than 2% of total decoding latency (Lei et al., 2023). In this formulation, parameter-based transfer becomes inseparable from conditional computation.

Parameter sharing provides a complementary route to efficiency. MambAdapter inserts a Mamba state-space block inside low-rank bottleneck adapters and shares the same down- and up-projection matrices across all adapter layers, reducing projection parameters from $\theta$6 to $\theta$7 when there are $\theta$8 adapters (Ali et al., 14 Jun 2026). This is still PETL in the usual sense, but it treats redundancy across adapter layers as an optimization target in its own right.

5. Empirical regimes and application domains

Visual recognition remains the most intensively studied regime. On VTAB-1K with ViT-B/16, TTC-Tuning achieved 74.8% average accuracy across 19 downstream tasks using only 0.19M parameters, outperforming full fine-tuning on 18 of 19 tasks and surpassing SSF by 1.7% average accuracy; on Swin-B it achieved 74.5% average accuracy (Zhao et al., 2023). A broader unifying study of 14 PEFT methods on ViTs found that, once hyperparameters were tuned carefully, most PETL methods clustered between about 74.7 and 77.6 overall mean on VTAB-1K, whereas linear probing reached 61.9 and full fine-tuning 70.0; the same study further reported that PETL remains useful in many-shot regimes and preserves CLIP robustness to distribution shifts better than full fine-tuning alone (Mai et al., 2024).

Multimodal retrieval has produced several specialized PETL architectures. For remote sensing image–text retrieval, a frozen CLIP backbone with a multimodal remote sensing adapter and a hybrid multi-modal contrastive objective required only 0.16M training parameters, corresponding to a 98.9% reduction relative to full fine-tuning, and exceeded traditional methods by 7–13% while achieving comparable or better performance than full fine-tuning (Yuan et al., 2023). For text-based person retrieval, UP-Person combined Prefix, LoRA, and Adapter modules in a unified CLIP-based architecture, fine-tuning 4.7% of parameters and reaching R@1 = 74.17 and mAP = 65.91 on CUHK-PEDES, with further strong results on ICFG-PEDES and RSTPReid (Liu et al., 14 Apr 2025). These systems illustrate a recurrent theme: different PETL submodules are assigned distinct roles, such as local feature adaptation, task-specific prompting, and global distribution correction.

Speech, audio, and music show a similar diversification. On the SURE benchmark for speech understanding, ConvAdapter used only 0.94% of the trainable parameters of full fine-tuning and was comparable to prefix tuning and LoRA on many tasks while improving over standard adapters on some speaker-recognition settings (Li et al., 2023). MambAdapter, applied to AST and Whisper, achieved 89.85 average accuracy in the Houlsby configuration on four audio classification tasks and 49.9 average WER across five ASR languages while training only about 0.45% of Whisper’s parameters (Ali et al., 14 Jun 2026). In music foundation models, PETL methods outperformed both probing and full fine-tuning on music auto-tagging, and matched fine-tuning on key detection and tempo estimation with significantly less training cost, while simultaneously raising the caution that a small model trained from scratch can achieve similar results on some tasks (Ding et al., 2024).

Outside foundation-model adaptation, parameter-based transfer has also been applied to physical-system identification and inverse problems. A transfer-learning-based neural estimator for building RC thermal models pretrained on 450 source buildings improved performance by 18.6–24.0% with only 12 days of training data and by up to 49.4% with 72 days, while eliminating the need for an initial parameter guess (Raisch et al., 7 Apr 2026). In fluorescence microscopy denoising, AUTO-DIP transferred U-net depth, width, skip-connection choice, and stopping iteration from a calibration set to new images using similarity-based lookup; on the validation set, grouping by the same microscope type and specimen type yielded mean PSNR = 34.99 dB and mean LPIPS = 0.098, close to the image-specific optimum of 35.56 dB, and transferred configurations were about 3× faster than the original DIP parameter setting in practice (Meyer et al., 17 Jan 2026). These examples show that parameter-based transfer learning is not confined to supervised representation learning.

6. Theory, negative transfer, and unresolved questions

Theoretical work frames parameter transfer as a stability and estimation problem. The learning-bound analysis of parameter transfer learning introduces local stability for parametric feature mappings and parameter transfer learnability, then derives an excess-risk bound that depends on three terms: the target-sample estimation term, the source-parameter estimation error, and an interaction term capturing how source-parameter error perturbs target learning (Kumagai, 2016). Applied to sparse coding in self-taught learning, this yields a first explicit learning bound for a classical unsupervised-transfer setting.

More recent theory addresses partial inheritance directly. In a two-layer ReLU CNN analysis, transferred parameters are characterized as carriers of universal knowledge shared between tasks, and their value is shown to depend on the inheritance fraction $\theta$9, the amount of source-task data, source-task noise, and the strength of the shared signal (Yuan et al., 26 Sep 2025). The same analysis also formalizes negative transfer: when the shared signal is weak relative to task-specific structure or noise, inherited parameters can amplify nuisance directions and lead to lower target test accuracy than training from scratch. This challenges the common assumption that any nonzero amount of source reuse is preferable to reinitialization.

Empirically, several misconceptions have likewise been revised. One is that PEFT is only a low-shot regularizer; the visual recognition study shows that it can remain competitive or superior in many-shot settings while using far fewer trainable parameters (Mai et al., 2024). A second is that similar average accuracy implies similar learned behavior: PETL methods with near-identical VTAB performance were shown to disagree on about 20% of predictions on DTD and Retinopathy and about 35% on DMLab, implying distinct inductive biases and making simple ensemble averaging beneficial (Mai et al., 2024). A third is that foundation-model adaptation is uniformly advantageous; music results indicate that for key detection and tempo estimation, the current generation of music foundation models may offer limited benefit over a small model trained from scratch (Ding et al., 2024).

Taken together, the literature suggests that parameter-based transfer learning is no longer a single technique but a design space organized around several orthogonal questions: what should be transferred, where should transfer be inserted, how should source–target mismatch be handled, how should limited adaptation capacity be allocated, and which efficiency metric—trainable parameters, activation memory, storage, or inference latency—should dominate the design objective. The field’s current trajectory points toward increasingly selective and structured transfer mechanisms, stronger accounting for distribution shift, and a more explicit treatment of when partial reuse is beneficial and when it is not.