Layer-wise Defrosting: Efficient Fine-Tuning

• Layer-wise defrosting is a fine-tuning technique that selectively unfreezes top layers of a pretrained model to adapt efficiently to new tasks.
• This method balances adaptation flexibility with statistical efficiency by using protocols like incremental, cascade, and semantic-aware defrosting.
• Empirical studies indicate that defrosting only the upper quarter of layers can recover over 90% of full fine-tuning performance while saving computational cost.

Layer-wise defrosting, also referred to as “freezing layers,” is a transfer learning and fine-tuning technique in deep neural networks whereby only a selected subset of layers is made trainable (defrosted) during adaptation to a new task or dataset, while the remaining layers remain fixed (frozen). This strategy is motivated by empirical evidence that adjusting only a fraction of the highest network layers during transfer or task adaptation recovers nearly the entirety of downstream task performance, resulting in significant reductions in memory, computation cost, and often, overfitting (Lee et al., 2019, Gerace et al., 2023, Gu et al., 2024, Zhang et al., 2021). Layer-wise defrosting also enables more systematic trade-offs between flexibility (via more defrosted layers) and statistical efficiency (via deeper freezing).

1. Formal Definition and Notation

Let a model consist of an embedding layer $\theta^{(0)}$, a stack of $L$ main layers $\{\theta^{(1)},\ldots,\theta^{(L)}\}$, and a task-specific output head $\theta^{(\text{out})}$. In the typical defrosting protocol, one chooses an integer $N$ and holds fixed (freezes) parameters $\theta^{(0)},\ldots,\theta^{(N)}$ (“freeze set”), while allowing $$\theta^{(N+1)},\ldots,\theta^{(L)},\theta^{(\text{out})}$$ (“defrost set”) to be updated during training or fine-tuning (Lee et al., 2019, Gerace et al., 2023). The fine-tuning process then minimizes the downstream loss $\mathcal{L}(\theta)$ over the defrost set $D_k$ (where $k=L-N$ is the number of trainable layers) while parameters in the freeze set $F_N$ remain unchanged.

This formalism generalizes across architectures (transformers, CNNs, LSTMs) and use cases (language modeling, vision, speech recognition), and can be accompanied by hard freezing or compliant regularization (e.g., strong $L_2$ penalty to keep weights near pre-trained values on frozen layers) (Gerace et al., 2023).

2. Empirical Motivation and Cost-Quality Trade-offs

Pretrained models, such as BERT and RoBERTa, contain hundreds of millions of parameters and many stacked layers. Full fine-tuning back-propagates through all layers, incurring both high memory/computational cost and a risk of overfitting, especially for simple or data-scarce target tasks (Lee et al., 2019, Gu et al., 2024). Empirical findings consistently show that:

• For a wide range of tasks, defrosting only the top $\lfloor L/4 \rfloor$ layers typically recovers at least 90% of full fine-tuning quality.
• Freezing all but the classifier drastically underperforms, recovering only 60–65% of full accuracy on average.
• For some tasks (e.g., sentiment analysis with large models), partial defrosting achieves even slightly higher accuracy than full tuning, suggesting an implicit regularization effect via the restriction of adaptation capacity.
• The CoLA linguistic acceptability task is an outlier, requiring more layers to be defrosted to attain comparable quality (Lee et al., 2019).

The cost-benefit rationale applies both to dense full-parameter adaptation and to parameter-efficient fine-tuning (PEFT) methods (such as adapters and LoRA). Defrosting fewer layers directly reduces the number of trainable parameters and the computation/memory required for backpropagation (Gu et al., 2024).

3. Algorithms and Selection Protocols for Defrosting

Several principled procedures for determining the number and identity of layers to defrost have been proposed:

• Incremental Layer Defrosting (ILD): For each possible freeze depth $k=0,1,\ldots,L$, freeze the first $k$ layers and retrain the remainder. The optimal defrosting point $d^*$ is selected as the $k$ maximizing downstream accuracy $\mathrm{Acc}(k;n_t,\rho)$ on a validation set, where $n_t$ is the amount of target data and $\rho$ expresses source-target task relatedness (Gerace et al., 2023).
• Top-down (Cascade) Defrosting: Train the classifier first with upper layers defrosted, then progressively “defrost” additional lower layers layer-wise, stopping when validation error ceases to improve (Zhang et al., 2021).
• Semantic-aware Layer-freezing (SEFT): Use semantic deviation scores, derived from comparing each layer’s latent representation to a straight-line interpolation between the embedding and ground-truth output semantic bases, to rank layers by benefit (Gu et al., 2024). Only layers with high deviation (largest mismatch from ideal “transition trace”) are defrosted. This approach allows for dynamic and budgeted cost-quality balancing, e.g., freezing a specified fraction of layers according to cost constraints.

Table: Representative Layer-wise Defrosting Protocols

Protocol/Method	Selection Criterion	Key Empirical Result
ILD (Gerace et al., 2023)	Validation accuracy maximization	$d^*$ increases with $n_t$ and $\rho$
Cascade (Zhang et al., 2021)	Stop when val. error ceases to decrease	Improved ASR WER and LM perplexity
SEFT (Gu et al., 2024)	Semantic deviation minimization	30–50% back-prop savings, no loss
Elsa (Lee et al., 2019)	Sweep $N$ for >=90% of baseline	25% of layers sufficient in BERT/RoBERTa

4. Experimental Findings Across Architectures and Tasks

Extensive benchmarks validate the ubiquity and impact of layer-wise defrosting:

• Transformer LMs (BERT, RoBERTa): On GLUE tasks, defrosting only $k = \lfloor L/4 \rfloor$ top layers (e.g., 3 of 12 in BERT_BASE) preserves $\geq 90\%$ of the full-fine-tuning score on sentiment, paraphrase, and inference tasks; larger fractions or full defrosting sometimes led to negligible gains or mild overfitting (Lee et al., 2019).
• CNNs (ResNet, Wide-ResNet): On object recognition, the optimal defrost depth depends nontrivially on dataset size ($n_t$) and source-target correlation ($\rho$). Incremental scans over $k$ (frozen depth) reveal a single accuracy peak, with $d^*$ increasing with available data and task similarity. Representation similarity measures (CKA, SVCCA, Information Imbalance) track this defrosting breakpoint (Gerace et al., 2023).
• Speech/Linguistics (BLSTM, Transformer for ASR): Top-down cascade defrosting yields lower error rates compared to classical curricula. Freezing a strong classifier and adapting lower feature-extractors results in better regularization and improved generalization, with empirical improvements on WSJ and Switchboard benchmarks (Zhang et al., 2021).
• Semantic-aware LMs (SEFT): Defrosting guided by semantic deviation identifies cost-efficient layers to unfreeze, outperforming naïve and structural policies, for example achieving higher test accuracy at the same cost-saving ratio on Pythia and Llama-3 models (Gu et al., 2024).

5. Quantitative Performance and Cost Analyses

Layer-wise defrosting delivers concrete, reproducible efficiency benefits while preserving or improving downstream task accuracy:

• For BERT_BASE on GLUE: defrosting top 3 of 12 layers ($k=3$) yields SST-2 accuracy of 90.8 versus 92.7 for full fine-tuning, i.e., $>\!97\%$ performance for $25\%$ of the cost (Lee et al., 2019).
• In vision models, the correct defrosting choice can result in 5$\times$ less target data required to reach a given performance as compared to the classical “freeze all but last” method (Gerace et al., 2023).
• SEFT achieves 30–50% reduction in total back-prop FLOPs, with performance meeting or exceeding that of budget-equivalent naïve or structural baselines (Gu et al., 2024).
• In ASR, top-down defrosting produces 16–38% relative reduction in word error rate over baseline (Zhang et al., 2021).

6. Theoretical Analysis and Representation Similarity

The optimal depth to defrost is controlled by a trade-off between bias and variance. Formally, expected error at freeze depth $k$ is:

$$E(k; n_t, \rho) = \mathrm{Bias}(k; \rho) + \mathrm{Var}(k; n_t)$$

where bias decreases and variance increases with less freezing. Empirically, $d^*$ grows with $\log n_t$ and inversely with $\rho$. Representation similarity metrics (CKA, II) computed between source-trained and target-trained feature maps provide effective proxies for deciding where to defrost: the optimal $k$ is close to the layer where similarity drops sharply (Gerace et al., 2023).

In semantic-aware approaches, per-layer deviation (e.g., cosine distance from semantic anchors) acts as an unsupervised indicator of adaptation need. Budgeted variants balance cost over layers using geometric or arithmetic allocation schedules (Gu et al., 2024).

7. Best-Practice Guidelines and Limitations

Best-practices synthesized from the literature include:

• For modern large LMs, freeze the bottom 75% of transformer layers ($k = \lfloor L/4 \rfloor$) to save resources while maintaining performance (Lee et al., 2019).
• For scarce data or highly dissimilar tasks, prefer freezing more layers. For closely related or high-data targets, incrementally unfreeze lower layers (Gerace et al., 2023).
• PEFT methods are orthogonal to defrosting: use SEFT or similar logic to select which layers to adapt, and instantiate parameter-efficient modules only therein (Gu et al., 2024).
• Monitor task- or representation-based metrics (validation accuracy, CKA) to inform defrost depth choice.
• Regularization is amplified by freezing: overfitting can be mitigated, and in simple tasks, partial defrosting may exceed joint training.

Limitations include the assumption that linear or monotonic semantic transitions align well with adaptation needs (challenged in highly non-isotropic architectures or tasks), the static nature of most defrosting schedules (dynamic layer selection remains an open area), and restrictions to architectures where layers can be modularly frozen/unfrozen (Gu et al., 2024, Gerace et al., 2023).

---

Layer-wise defrosting constitutes a principled, empirically validated protocol for selecting which neural network layers to adapt during transfer or fine-tuning, with predictable trade-offs in accuracy, data efficiency, memory/computation, and overfitting. Its mechanisms and best practices are now established across transformers, CNNs, RNNs, and multi-modal networks (Lee et al., 2019, Gerace et al., 2023, Gu et al., 2024, Zhang et al., 2021).