Frozen Encoder Strategies in PETL

• Frozen encoder strategies are PETL methods that freeze backbone model weights and update only light auxiliary modules (e.g., adapters, LoRA) for efficient transfer learning.
• They reduce trainable parameters to as low as 0.05%–5% of full fine-tuning, achieving up to 10× faster training and significantly lower GPU memory usage.
• Applied in domains like music auto-tagging and multimodal medical imaging, these methods deliver competitive accuracy while mitigating overfitting and calibration issues.

Frozen Encoder Strategies refer to a class of parameter-efficient transfer learning (PETL) and fine-tuning methodologies in which the large, pre-trained backbone of an encoder model is held fixed ("frozen") during downstream adaptation. Instead of updating the entire model (as in conventional fine-tuning), only a small, typically task-specific subset of parameters is trained. This approach is motivated by the pressing need to reduce computational, memory, and overfitting costs associated with full model updates—especially in domains with large foundation models or limited resource budgets. Frozen encoder strategies are now widely validated in music information retrieval, multimodal medical imaging, natural language, vision-language, and speech processing applications, and are the backbone for modern PETL variants that include adapters, low-rank updates (LoRA), prompt/prefix-based modules, and bias-only tuning.

1. Mathematical Formulation and Implementation

Let $W_\text{enc}$ denote the (frozen) pre-trained encoder parameters, and $\theta_\text{aux}$ the set of trainable parameters introduced for downstream adaptation. Frozen encoder strategies restrict training so that

$$W_\text{enc} \gets W_\text{enc}^{(0)}, \qquad \text{(no updates)}$$

and only $\theta_\text{aux}$ (which may include adapters, prompts, fusion module weights, or low-rank updates) is optimized.

Typical frozen encoder PETL variants:

• Adapter-based: For intermediate representation $h \in \mathbb{R}^d$, insert a bottleneck MLP adapter:

$$h_a = W_2 \, \text{ReLU}(W_1 h), \quad h' = h + \alpha h_a,$$

with bottleneck dimension $r \ll d$ (Ding et al., 2024).

• Prompt-based: Learn a set of continuous prompt tokens $P \in \mathbb{R}^{p \times d}$; prepend to each input $X$ (Ding et al., 2024).
• Low-rank (LoRA): Over weight matrices $W$, learn additive updates $\Delta W = UV$ with $$U \in \mathbb{R}^{d \times r}, V \in \mathbb{R}^{r \times d}$$, and

$W' = W + \Delta W$

(Ding et al., 2024).

• Bias-only (BitFit): Update only the bias vectors in select layers (Khan et al., 25 Dec 2025).

In multimodal scenarios, frozen vision and/or text encoders are combined via a trainable fusion module, e.g. cross-modal attention and linear projections (Khan et al., 25 Dec 2025). Fixed-budget adaptation constrains $\lVert \theta_\text{aux} \rVert$ to a given proportion of total parameters.

2. Parameter and Computational Efficiency

Frozen encoder strategies yield major efficiency gains by:

• Reducing trainable parameters to $0.05\%$–$5\%$ of full fine-tuning (e.g., 2.5% in multimodal chest X-ray classification vs. 100% for full FT) (Khan et al., 25 Dec 2025, Ding et al., 2024).
• Lowering GPU memory and training time, as only small auxiliary modules are updated. For instance, LoRA/adapters with $r=64$, $d=1024$, $L=12$: $1.3$–$1.6$M trainable params (∼1.5% of backbone) (Ding et al., 2024).
• Achieving >10× faster training and reduced memory: e.g., music auto-tagging adapters require 6h versus 48h for full fine-tuning (Ding et al., 2024).

A representative cost table:

Method	Trainable Params	% of Full FT	GPU-hours (AutoTag)
Full FT	110M	100%	48h
Adapters (r=64)	1.6M	1.5%	6h
Prompt-tuning	0.05M	0.05%	4h
LoRA (r=64)	1.3M	1.2%	5h

This efficiency is crucial for real-world deployment where training cost, hardware demands, or privacy requirements preclude large-scale fine-tuning (Ding et al., 2024, Khan et al., 25 Dec 2025).

3. Empirical Performance Across Downstream Tasks

Frozen encoder PETL variants consistently yield strong results on challenging discriminative tasks and maintain competitive accuracy on structured tasks.

• Music Auto-tagging (semantic): Adapters, LoRA, and prompts outperform both probing and full fine-tuning. For example, adapters (r=64) reach ROC-AUC $0.920 \pm 0.001$ vs $0.905$ for full FT (paired $t$-test $p<0.01$) (Ding et al., 2024).
• Key/Tempo Estimation (structured): PETL methods match full FT, but small models from scratch achieve comparable performance, questioning the marginal utility of foundation model encoders for these tasks (Ding et al., 2024).
• Multimodal medical imaging: Fixed-budget strategies with frozen ResNet-50 and DistilBERT encoders achieve AUROC $0.892$–$0.908$, outperforming full FT ($0.770$) with $40\times$ fewer trainable parameters. External validation confirms robust scaling on larger datasets (Khan et al., 25 Dec 2025).
• Calibration: PETL methods show higher expected calibration error (ECE $\sim 0.29$–$0.34$) compared to simple vision-only models (ECE $\sim 0.05$); post-hoc calibration methods are recommended before clinical deployment (Khan et al., 25 Dec 2025).

Typical findings are summarized below.

Method	Params	AUROC (Chest X-Ray)	ECE
Full FT	94.3M	0.770	0.327
Frozen Enc.	2.37M	0.9079	0.339
LoRA	2.37M	0.9027	0.304
Adapter	2.37M	0.9009	0.293
BitFit	2.37M	0.8916	0.303

4. Integration Workflow and Implementation Recipes

Frozen encoder strategies are implemented by freezing all backbone model weights and inserting auxiliary modules for adaptation:

• Adapter modules are injected after MHA and FFN sub-layers, with $r$ chosen to fit budget (Ding et al., 2024).
• LoRA modules are applied to all Q/K/V and FFN-in matrices; scaling factors ($\alpha$, $r$) control overhead.
• Prompt-based PETL prepends $p$ learnable tokens at each layer; prompt length and initialization impact stability.
• Fusion modules (multimodal) consist of vision/text projections, cross-modal attention, and classification heads. All encoder weights remain fixed; only fusion parameters are trained (Khan et al., 25 Dec 2025).

Training settings:

• Optimizer: AdamW, $lr=10^{-4}$ (PETL), batch size 32, $20$ epochs, early-stopping on validation (Ding et al., 2024).
• Input: For music/audio, Mel-spectrograms. For chest X-Ray, ResNet-50 image features and redacted DistilBERT text embeddings (Khan et al., 25 Dec 2025).
• Hardware: Single V100 or similar GPU for practical resource constraint assessment.

5. Strategic Considerations and Practical Guidelines

Selection of PETL strategy under frozen encoder regimes depends on task characteristics, resource constraints, and desired interpretability:

• Semantic, multi-label tasks (auto-tagging): Adapters or LoRA (with bottleneck) deliver optimal trade-off in accuracy and efficiency (Ding et al., 2024).
• Structured tasks (key/tempo, regression): PETL matches full FT, but small scratch models may be competitive. No significant benefit from foundation features, questioning utility for these tasks (Ding et al., 2024).
• Low-resource/multitask scenarios: Prompt-tuning offers the smallest disk/parameter footprint (Khan et al., 25 Dec 2025).
• Interpretability/layer-wise control: Adapters provide explicit per-layer hooks.
• Calibration needs: Adapter and LoRA offer slightly lower ECE, but post-hoc temperature scaling is mandatory for clinical reliability (Khan et al., 25 Dec 2025).

Further recommendations:

• For rapid prototyping: LoRA can be integrated with minimal code adjustments (e.g., PEFT library).
• For ultra-tight parameter budgets: allocate capacity preferentially to fusion modules, not to multimodal encoders (when applicable), maximizing return on parameter investment (Khan et al., 25 Dec 2025).
• When deploying in medical or safety-critical domains, calibration procedures must complement the high discrimination of PETL approaches.

6. Limitations and Open Directions

Frozen encoder strategies exhibit key limitations:

• On simple structured tasks, marginal foundation model utility vs scratch models suggests limited benefit (Ding et al., 2024).
• Adapters add runtime latency if not merged; LoRA requires additional inference computation unless fully merged (Ding et al., 2024).
• Prompt-based methods are sensitive to prompt length $p$ and initialization; stability is nontrivial (Ding et al., 2024).
• Calibration degradation is substantial; ECE correction is essential for reliable decision support (Khan et al., 25 Dec 2025).
• In multimodal settings, under tight budget constraints, the vision encoder can outperform multimodal fusion, indicating that cross-modal synergy is not free and is budget-dependent (Khan et al., 25 Dec 2025).

Future work may include:

• Advanced fusion architectures to unlock synergistic gains under severe bandwidth/parameter constraints.
• Dynamic PETL modules that adapt selection of frozen/trained blocks as more data or compute becomes available.
• Deeper theoretical analysis of why foundation encoders provide limited advantage on certain tasks and new guidelines for PETL allocation.

---

Frozen encoder strategies, as validated across diverse domains and task modalities, fundamentally enable rapid and resource-constrained adaptation of foundation models via selective training of small auxiliary modules. By rigorously budget-matching parameter allocation to task demands and systematically freezing the bulk of model weights, these approaches deliver state-of-the-art discriminative performance, marked efficiency, and practical deployment paths while imposing tractable limitations in calibration and fusion synergy (Ding et al., 2024, Khan et al., 25 Dec 2025).