Gradual Unfreezing in Neural Networks

• Gradual unfreezing is a staged training strategy in deep learning where network layers are sequentially unfrozen to ensure model stability and enhanced generalization.
• It employs either top-down or bottom-up schedules, using metrics like Fisher Information and sharpness to determine optimal unfreeze intervals.
• Its practical application spans transfer learning, federated optimization, and even experimental physics, leading to measurable improvements in performance.

Gradual unfreezing is a staged training strategy wherein a neural network's layers or modules are sequentially transitioned from frozen (parameters fixed) to unfrozen (parameters trainable), rather than enabling all parameters to update simultaneously. This approach is motivated by both stability and generalization concerns in various deep learning settings, including transfer learning, adversarial training, federated learning, and even experimental physics. Gradual unfreezing typically proceeds in either a top-down (from output to input layers) or bottom-up (from input to output layers) order, with custom scheduling and metric-driven decision points. The technique is closely associated with transferability, robust adaptation, and improved optimization dynamics.

1. Foundational Schedules and Design Patterns

Two canonical variants of gradual unfreezing are established in the literature: (i) top-down unfreezing, historically associated with transfer/adaptation tasks, and (ii) bottom-up unfreezing, particularly suited for federated and distributed learning.

• Top-Down Unfreezing (e.g., GU, FUN, PUPGAN): Initially only the head (output layer) and possibly the highest-level feature layers are made trainable. Over subsequent training steps or epochs, lower (deeper) layers are unfrozen one by one. The unfreeze interval ($k$ steps per layer) is either fixed or selected based on learning dynamics, often guided by the trace of Fisher Information or sharpness (Liu et al., 2024, Liu et al., 2023).
• Bottom-Up Unfreezing (e.g., FedBug): Training proceeds by thawing layers from the input upwards, ensuring a persistent anchor in the downstream layers for cross-client consistency in federated settings (Kao et al., 2023).

A general schedule consists of partitioning parameters into logical blocks, initializing only the head as trainable, and iteratively adding (or probabilistically selecting, as in PUPGAN) new blocks to the set of trainable parameters.

2. Mathematical Formalization and Algorithmic Implementation

The core of gradual unfreezing is the management of the trainable parameter set $\mathcal{S}$ as a function of training step $i$ and schedule parameter $k$. For top-down unfreezing, at each unfreeze step,

$$\mathcal{S}_{t} = \{C\} \cup \{\theta_j | j \in \{L-1, L-2, ..., L-1-t\}\},$$

where $C$ is the classifier head and $\theta_j$ denotes block $j$ (Liu et al., 2023, Liu et al., 2024). The SELECT function in pseudocode determines which layer to unfreeze next (heuristic order or using a metric such as Fisher Information). For the bottom-up schedule, as in FedBug, the newly thawed layer projects inputs into a latent space, preserving the decision boundaries imposed by still-frozen upstream modules (Kao et al., 2023).

In PUPGAN, each pre-trained layer is unfrozen stochastically per epoch, with a sampled probability exceeding a set threshold $\varphi$ triggering the activation of an additional layer, inducing progressive adaptation in the GAN discriminator (Sun et al., 2020).

3. Theoretical Motivations and Metric-Based Scheduling

Empirical and theoretical analyses converge on early-phase training dynamics as decisive for generalization properties, particularly for out-of-distribution (OOD) performance (Liu et al., 2024). The following metrics play central roles:

• Fisher Information Trace: $$\text{tr}(F) = \mathbb{E}_{x} \mathbb{E}_{\hat{y}} \|\nabla_w \log p_w(\hat{y}|x)\|^2$$ represents model sensitivity to parameter perturbations. Schedules inducing a pronounced early "Fisher hill" (i.e., high Fisher trace before unfreezing) correlate with superior cross-lingual and OOD generalization (Liu et al., 2023, Liu et al., 2024).
• Sharpness: Quantifies the expected/worst-case loss increase under small perturbations. Schedules timing their transition from frozen to unfrozen regimes based on sharpness stabilization yield Pareto-optimal ID/OOD tradeoffs (Liu et al., 2024).

Optimization theory in federated settings (FedBug) shows that gradual (bottom-up) unfreezing provably contracts client drift faster than updating all layers at once, leading to improved convergence rates (Kao et al., 2023).

4. Applications and Empirical Outcomes

Natural Language Processing and Adapter Fine-Tuning

Gradual unfreezing is extensively studied in transformer-based adapter frameworks for cross-lingual transfer. In these domains, scheduled unfreezing of task adapters (rather than base model weights) bridges the performance gap between parameter-efficient fine-tuning and full model adaptation. Both heuristic (top-down) and Fisher-based layer selection achieve consistent +2–4 percentage point gains in OOD (cross-lingual) transfer tasks, with further improvements for languages with lower baseline transferability (Liu et al., 2023, Liu et al., 2024).

Federated and Distributed Optimization

FedBug exemplifies the use of bottom-up gradual unfreezing in federated learning. Here, the sequential thawing of local client layers aligns intermediate latent representations, suppressing client drift and accelerating global objective convergence. Experimental results show that even modestly slow unfreezing (e.g., 10–40% of local steps) consistently yields accuracy gains (e.g., +1.8% on CIFAR-10, +4.9% on CIFAR-100) over FedAvg and earlier approaches (Kao et al., 2023).

Generative Models and Transfer Learning

Progressive unfreezing in GAN discriminators (PUPGAN) ensures smooth transfer from classification pretraining to generation tasks. This stabilizes adversarial dynamics and enhances perceptual quality, reflected in improved PSNR/SSIM in SRGAN and Pix2Pix by 0.4–2 dB and 0.02–0.13 respectively, and substantial improvement in Perceptual Quality Index on unpaired translation tasks (Sun et al., 2020).

Physics and Experimental Systems

In phase transition studies (e.g., ice–water interface or magnetic shape-memory alloys), "gradual unfreezing" refers to physical control over the melting/de-arrest process via temperature/field schedules, governed by kinetic models such as the Stefan problem and CHUF protocol. These protocols produce stepwise or smooth transitions in order parameters (magnetization or phase fraction) directly analogous to staged unfreezing in neural updates (Chaddah et al., 2012, Chasnitsky et al., 2020).

5. Comparative Empirical Analysis

Domain/Task	Scheduling Direction	Metric/Trigger	Reported Gains
Adapter transfer (NLP, cross-lingual) (Liu et al., 2023)	Top-down	Fisher trace, fixed interval	+2–4 pp OOD F1/acc, robust ID
Federated learning (FedBug) (Kao et al., 2023)	Bottom-up	Fraction of local steps (P)	+0.3–5 pp test acc; faster conv.
GAN perceptual transfer (Sun et al., 2020)	Top-down	Probabilistic per-epoch	↑PSNR/SSIM, ↓PQI; less instability
OOD generalization (Liu et al., 2024)	Top-down	Fisher/sharpness stabilization	+1–30 pp OOD with minimal ID loss

Notably, in domain-specific applications such as factoid question answering (BioASQ9b), gradual unfreezing did not deliver statistically significant accuracy improvements, highlighting substantial context dependence (Khanna et al., 2021).

6. Best Practices, Limitations, and Open Questions

Effective deployment of gradual unfreezing requires:

• Careful partitioning into logical blocks and selection of unfreeze intervals, either fixed or dynamically determined via information-based metrics (Liu et al., 2024).
• Ensuring the total unfreezing phase remains a small fraction of total training (e.g., $\mathcal{S}$0) to allow sufficient time for joint adaptation after all layers are active (Liu et al., 2023).
• In federated settings, bottom-up unfreezing aligns local feature spaces, while top-down is critical for transfer learning and OOD generalization.

Limitations include diminished or negligible efficacy in certain low-data or robust-initialization regimes (e.g., minimal effect on BioASQ9b with DistilBERT (Khanna et al., 2021)) and incomplete theoretical characterization for large-scale, nonlinear, and highly heterogeneous cases. The field lacks universal criteria to predict ahead of time in which cases gradual unfreezing will substantially impact performance.

7. Physical and Non-ML Analogues: Phase Transition and Kinetic Unfreezing

Outside machine learning, gradual unfreezing describes physically regulated devitrification and de-arrest in systems exhibiting kinetic constraints, such as magnetic shape-memory alloys (CHUF protocol) and directional melting (Stefan problem) (Chaddah et al., 2012, Chasnitsky et al., 2020). In these cases, precise sequences of extrinsic parameter changes (e.g., warming field, block temperatures) produce a stepwise or continuous progression from frozen to equilibrium states, underpinned by kinetic theory and observable as a sequence of sharp or gradual macroscopic transitions, often with substantial implications for controllable material properties.

---

Gradual unfreezing thus constitutes a principled intervention in both machine learning and physical sciences to modulate adaptation dynamics for stability, robustness, and accurate tracking or transfer of underlying structures. The schedule, metrics, and theoretical basis are subject to active research to refine both empirical utility and foundational understanding.