Partial Layer Fine-Tuning

• Partial layer fine-tuning is a transfer-learning regime that updates only select layers or submodules while freezing the remainder of a pre-trained model.
• It leverages depth-wise representational differences, using metrics like cosine similarity and FIM scores to target critical layers for efficient adaptation.
• The approach encompasses various strategies, including layer grouping, sparse updates, and submodule tuning, to address task- and architecture-specific challenges.

Partial layer fine-tuning is a transfer-learning regime in which only a subset of a pre-trained model’s layers, layer groups, blocks, or sub-layer parameters are updated, while the remainder stay frozen. In the literature, this regime appears both as a direct alternative to full fine-tuning and as a design dimension inside parameter-efficient fine-tuning (PEFT). Across transformers, convolutional networks, U-Nets, Vision Transformers, and LLMs, the recurring premise is that not all depths are equally important for downstream adaptation, and that selective updating can improve efficiency while preserving or even improving task performance in some settings (Lee et al., 2019, Lodha et al., 2023, Ye et al., 2023).

1. Depth-wise basis of selective adaptation

A central empirical rationale for partial layer fine-tuning is that representational change during adaptation is not uniformly distributed across depth. In BERT, analysis based on activation similarity, Sparse AutoEncoders (SAEs), and token-level visualizations shows a steady progression across layers: early layers primarily retain general linguistic features such as syntax, part-of-speech, and structural words; middle layers act as a transition zone; and late layers become highly specialized for the downstream task, with sparse but strong activations on task-critical tokens or classes after fine-tuning. In the same study, cosine similarity between average pre-trained and fine-tuned activation vectors starts high at roughly $1.0$ in early layers and declines with depth, with Layer 12 dropping to $0.3$–$0.5$ for complex tasks such as News topic classification (Nadipalli, 23 Feb 2025).

The cosine similarity used in that analysis is

$$\text{Cosine Similarity}(a,b)=\frac{a\cdot b}{\|a\|\,\|b\|}$$

where $a$ and $b$ are average activation vectors from a selected layer for the pre-trained and fine-tuned models. The same work associates late-layer specialization with SAE features sharply tuned to polarity, domain-critical vocabulary, and class markers, while early layers remain comparatively task-agnostic (Nadipalli, 23 Feb 2025).

A broader depth-profile study across 240 fine-tuning runs, 15 models, and four architecture families reports that representational change concentrates in output-proximal layers in every standard-training run except one. It defines the “locality gradient” as the slope of layer-wise representational change over relative depth, and shows that this slope remains positive under standard training in 189 of 190 runs. However, when per-layer updates are rescaled to equalize $\|\Delta W\|/\|W\|$ after each optimizer step, the profile persists in some conditions and collapses in others. Sequential-block architectures such as BERT, OPT, and GPT-2 retain the slope across tested objectives at $125$M–$350$M, whereas parallel-block architectures such as Pythia and CodeGen retain it only for causal-language-modeling objectives; this distinction narrows at $1.3$B–$0.3$0B (Billa, 19 Apr 2026).

This body of evidence supports partial layer fine-tuning as a response to measured depth-wise locality rather than as a purely heuristic simplification. At the same time, the equal-step results indicate that output-localized adaptation is a composite phenomenon: part intrinsic to model architecture and objective, part attributable to ordinary gradient-flow asymmetries (Billa, 19 Apr 2026).

2. Selection mechanisms and search procedures

The literature formalizes layer selection in several distinct ways. One line of work freezes the first $0.3$1 layers and fine-tunes only the top $0.3$2 layers. Another ranks layers by an information-theoretic score before fine-tuning. A third treats layer choice and learning-rate assignment as a search problem. A fourth embeds layer grouping and parameter allocation into an explicit PEFT design space (Lee et al., 2019, Lodha et al., 2023, Shen et al., 2021, Chen et al., 2023).

A direct freezing study on BERT and RoBERTa asks how many final layers need to be updated. For BASE models with 12 layers, fine-tuning just the top 3–5 layers attains at least $0.3$3 of the task-specific effectiveness of full fine-tuning; for LARGE models with 24 layers, the last 6–7 layers suffice for similar performance. The same study also reports that fine-tuning all layers does not always help, with a consistent increase in SST-2 quality when freezing 12–16 layers of RoBERTaLARGE or BERTLARGE (Lee et al., 2019).

A more explicit selection criterion is the diagonal Fisher Information Matrix (FIM) score. For a parameter vector $0.3$4,

$0.3$5

Layer-level FIM scores are obtained by aggregating diagonal elements per layer, using the Frobenius norm over all parameters in that layer. In practice, the FIM diagonal is estimated using only about 100 target samples from the validation set, and the top 1 to 5 layers—usually 5—are selected for fine-tuning. The reported layer rankings remain nearly constant before and throughout fine-tuning, which means important layers can be identified at initialization (Lodha et al., 2023).

Few-shot work extends the problem from binary freeze/unfreeze decisions to layer-wise learning-rate assignment. If a network has $0.3$6 layers, the fine-tuning strategy is a vector $0.3$7, where each $0.3$8 is chosen from a discrete set and $0.3$9 means the layer is frozen. The search space has size $0.5$0, and an evolutionary algorithm initializes candidate strategies, evaluates them on validation episodes, retains top-$0.5$1 parents, and generates offspring through mutation and crossover. This procedure typically discovers partial strategies in which some layers—more likely final ones—are updated and others are frozen (Shen et al., 2021).

The PEFT design-space formulation generalizes selective tuning into four components: layer grouping, trainable parameter allocation, tunable groups, and strategy assignment. In that framework, the empirically preferred grouping is the spindle pattern,

$0.5$2

with $0.5$3 the number of layers in group $0.5$4. The same study reports four design patterns: group layers in a spindle pattern, allocate trainable parameters uniformly, tune all groups, and assign proper tuning strategies to different groups (Chen et al., 2023).

Vision Transformers add another metric-driven selection rule. The fine-tuned angle metric measures the angle between pre-trained and post-tuning weights of a layer $0.5$5:

$0.5$6

Layers with larger angles are treated as more influential, and layers with smaller angles as more redundant. The reported guideline is to fine-tune layers with the largest angles for challenging tasks and layers with the smallest angles for easy tasks (Ye et al., 2023).

3. Task, architecture, and modality dependence

Partial layer fine-tuning is not governed by a single depth rule across tasks or modalities. In language encoders, selective late-layer tuning often works well, but the effective subset is task-dependent. FIM-based surgical fine-tuning reports that top-5 layers achieve within $0.5$7 of full fine-tuning on most GLUE and SuperGLUE tasks, and can outperform full-model fine-tuning on tasks such as MRPC and BoolQ. The same study identifies limits for tasks requiring complex world knowledge or reasoning spanning many layers, specifically RTE, CB, and COPA, where partial fine-tuning underperforms full-model fine-tuning (Lodha et al., 2023).

The assumption that early layers are always generic is contradicted in medical imaging. For ultrasound image segmentation with U-Net pre-trained on natural images, tuning the contracting path and fixing the expanding path gives a Dice score of $0.5$8, pixel error of $0.5$9, and Rand score of $$\text{Cosine Similarity}(a,b)=\frac{a\cdot b}{\|a\|\,\|b\|}$$0, whereas tuning the expanding path and freezing the contracting path gives $$\text{Cosine Similarity}(a,b)=\frac{a\cdot b}{\|a\|\,\|b\|}$$1, $$\text{Cosine Similarity}(a,b)=\frac{a\cdot b}{\|a\|\,\|b\|}$$2, and $$\text{Cosine Similarity}(a,b)=\frac{a\cdot b}{\|a\|\,\|b\|}$$3, respectively. The same study shows that shallow-to-deep inclusion yields progressively better results than deep-to-shallow inclusion for ultrasound, while in X-ray segmentation both strategies yield similarly high performance, with Dice around $$\text{Cosine Similarity}(a,b)=\frac{a\cdot b}{\|a\|\,\|b\|}$$4 (Amiri et al., 2020).

Vision Transformers exhibit yet another pattern. Manual strategies such as attention-only or FFN-only tuning can match or exceed full fine-tuning while updating fewer parameters, and layer choice is reported as critical. On ViT-B/16 pre-trained on ImageNet-21K and fine-tuned on ImageNet-1K, full fine-tuning reaches $$\text{Cosine Similarity}(a,b)=\frac{a\cdot b}{\|a\|\,\|b\|}$$5 top-1 accuracy with $$\text{Cosine Similarity}(a,b)=\frac{a\cdot b}{\|a\|\,\|b\|}$$6M tunable parameters, FFN-only reaches $$\text{Cosine Similarity}(a,b)=\frac{a\cdot b}{\|a\|\,\|b\|}$$7 with $$\text{Cosine Similarity}(a,b)=\frac{a\cdot b}{\|a\|\,\|b\|}$$8M, and the angle-guided FAPFT method reaches $$\text{Cosine Similarity}(a,b)=\frac{a\cdot b}{\|a\|\,\|b\|}$$9 with $a$0M (Ye et al., 2023).

A complementary ViT method, ALaST, makes the selection dynamic rather than fixed. It estimates per-layer importance from the class-token delta at each mini-batch, updates layer budgets online, and allocates compute by either freezing low-budget layers or reducing their token counts. The reported gains relative to traditional full fine-tuning are up to $a$1 reduction in training time, up to $a$2 reduction in FLOPs, and up to $a$3 reduction in memory load; the method can also be combined with LoRA (Devoto et al., 2024).

These results suggest that partial layer fine-tuning should be treated as architecture- and modality-specific model selection. Later-layer adaptation is often effective in language transformers, but shallow adaptation can be decisive when low-level statistics differ sharply from pre-training, as in ultrasound (Amiri et al., 2020, Nadipalli, 23 Feb 2025).

4. Partiality beyond whole layers

The scope of partial layer fine-tuning has expanded from choosing entire layers to choosing parts of layers, partial connections, sparse update directions, or pruned layer sets. In these variants, “partial” no longer means only “some layers,” but also “some weights within a layer” or “some layers after structured removal” (Woo et al., 28 Feb 2025, Chekalina et al., 2024, Huang et al., 27 Jan 2026).

PaCA, or Partial Connection Adaptation, directly fine-tunes a randomly selected subset of pretrained weights rather than adding adapter modules. For a linear layer with $a$4, PaCA randomly selects $a$5 columns of $a$6; only those partial connections are trainable. The method is motivated by eliminating sequential adapter overhead during training and reducing activation memory because only partial activations need to be stored for gradient computation. Relative to LoRA, PaCA reports a $a$7 reduction in training time and a $a$8 reduction in total memory usage, while maintaining comparable accuracy across scenarios including MMLU and instruction tuning on Oasst1. It also enables training with $a$9 longer sequence and improves throughput by $b$0 on both NVIDIA A100 GPU and INTEL Gaudi2 HPU; in quantized form it can fine-tune LLaMA3.1-70B on a single A100 GPU (Woo et al., 28 Feb 2025).

SparseGrad targets MLP layers, which the paper notes comprise about half of transformer parameters. It learns an orthogonal basis in which MLP gradients become highly sparse, then keeps only the largest $b$1 of elements in the transformed gradient matrix during fine-tuning. Under identical memory requirements, the method reports better performance than LoRA and MeProp on BERT, RoBERTa, and LLaMA-2. On the LLaMA-2 question-answering experiment, SparseGrad reaches an MT-Bench score of $b$2, versus $b$3 for LoRA and $b$4 for MeProp, at the same $b$5 updated-parameter level (Chekalina et al., 2024).

Structured layer pruning provides a different extension. A sequence-to-sequence framework combining LoRA with structured layer pruning tunes $b$6 of parameters and prunes over $b$7 of Transformer layers, reporting up to $b$8 reduction in GPU memory usage, $b$9 speedup in training, and retention of over $\|\Delta W\|/\|W\|$0 of the original model’s generation quality on free-text sequence-to-sequence tasks (Zhu et al., 2023). GradPruner similarly uses early cumulative gradients to rank and prune layers during fine-tuning, then sparsifies and merges pruned layers with retained layers; it reports $\|\Delta W\|/\|W\|$1 parameter reduction with only $\|\Delta W\|/\|W\|$2 decrease in accuracy, together with about $\|\Delta W\|/\|W\|$3 training and $\|\Delta W\|/\|W\|$4 inference reductions in time and memory usage (Huang et al., 27 Jan 2026).

On low-cost edge devices, Skip2-LoRA inserts trainable LoRA adapters between the last layer and every other layer, which permits caching of frozen forward-pass intermediates across epochs. The reported result is a $\|\Delta W\|/\|W\|$5 average reduction in fine-tuning time relative to a counterpart with the same number of trainable parameters, while preserving accuracy (Matsutani et al., 2024).

5. Submodule tuning, hybrid compositions, and dual-system partitioning

Some methods preserve the idea of partial layer fine-tuning but shift the trainable locus to submodules within each layer. FL-Tuning adds trainable units to the feed-forward network of each Transformer layer while freezing the original weights. The modified FFN can be written as the sum of the frozen original FFN and a trainable addFFN, and only about $\|\Delta W\|/\|W\|$6 of the Transformer’s parameters are trained. On the CLUE benchmark, the method improves accuracy by $\|\Delta W\|/\|W\|$7 on WSC 1.0 in full-data settings and F1 by $\|\Delta W\|/\|W\|$8 on CLUENER in few-shot settings over P-tuning v2, converges about $\|\Delta W\|/\|W\|$9 times faster, and is comparable with full fine-tuning on most datasets. Its ablations report that performance is positively correlated with both the number and depth of tuned layers (Liu et al., 2022).

Other approaches combine a frozen pre-trained model with new trainable pathways rather than selecting a subset of the original layers to update. LEVI constructs a layer-wise ensemble between a usually frozen pre-trained model and a small task-specific model trained from scratch. Adapting layers concatenate intermediate pre-trained representations with the task-specific representation and produce predictions whose average forms the final output. The reported effect is improved out-of-distribution generalization, especially when a new task comes from different subdomains than pre-training (Roh et al., 2024).

Dual-system partitioning pushes the idea further by assigning different parameter subsets to different reasoning demands. LoRA-PAR divides data into System 1 and System 2 subsets via multi-model role-play and voting, computes parameter importance by a Taylor/Fisher-based score, and partitions LoRA parameters into System 1-only, System 2-only, and shared subsets. For $125$0, the paper reports that typically only about $125$1 of LoRA parameters are updated. On GSM8K, LoRA-PAR reaches $125$2 accuracy, compared with $125$3 for PiSSA and $125$4 for baseline LoRA, while using fewer active parameters (Huang et al., 28 Jul 2025).

These formulations indicate that partial layer fine-tuning has broadened into a family of selective adaptation mechanisms: selecting depths, selecting submodules inside a layer, or partitioning trainable capacity by task type (Liu et al., 2022, Huang et al., 28 Jul 2025).

6. Practice, limits, and recurrent misconceptions

Several papers translate their empirical analyses into operational guidance. For BERT fine-tuning, one study recommends freezing early layers $125$5–$125$6, aggressively fine-tuning late layers $125$7–$125$8, and selectively fine-tuning middle layers $125$9–$350$0 when the new task diverges semantically from pre-training; it also proposes SAE-based or cosine-similarity analysis to target fine-tuning more precisely (Nadipalli, 23 Feb 2025). In PEFT design spaces, the reported best pattern is spindle grouping with uniform parameter allocation, tuning all groups, and heterogeneous strategy assignment (Chen et al., 2023).

Convolutional transfer-learning studies offer analogous but not identical prescriptions. One ImageNet transfer study reports that the best practice is to freeze the initial low-level convolutional layers and fine-tune from the third convolutional layer upward; it further argues that the pre-trained classification layer should not be universally discarded, and that careful normalization, especially Batch Normalization, is essential for harmony between pre-trained and new layers (Shermin et al., 2019). A separate block-wise optimization study on Tf_flower across VGG16, MobileNet-v1, MobileNet-v2, MobileNet-v3, and ResNet50v2 reports that block-wise approaches achieve better performances than both baseline methods and the layer-wise method (Barakat et al., 2023).

The main misconception in this area is that one universal rule determines which layers should always be frozen or tuned. The evidence does not support that. Ultrasound segmentation requires shallow adaptation rather than the conventional strategy of preserving shallow layers (Amiri et al., 2020). Surgical fine-tuning of language encoders is effective on most GLUE and SuperGLUE tasks but underperforms on RTE, CB, and COPA (Lodha et al., 2023). The depth-profile study further shows that output-localized change can collapse under equal-step control in some architecture–objective combinations, especially parallel-block architectures under non-CausalLM objectives at smaller scale (Billa, 19 Apr 2026).

A plausible implication is that partial layer fine-tuning is best viewed not as a fixed recipe but as a controlled allocation problem over depth, submodules, and parameter budget. The empirical criteria already proposed in the literature—cosine similarity, SAE features, FIM scores, angle metrics, early gradient accumulation, block-wise validation search, and adaptive compute budgets—are all attempts to make that allocation explicit rather than heuristic (Nadipalli, 23 Feb 2025, Lodha et al., 2023, Ye et al., 2023, Huang et al., 27 Jan 2026).