Less is More: Selective Layer Finetuning with SubTuning

Abstract: Finetuning a pretrained model has become a standard approach for training neural networks on novel tasks, resulting in fast convergence and improved performance. In this work, we study an alternative finetuning method, where instead of finetuning all the weights of the network, we only train a carefully chosen subset of layers, keeping the rest of the weights frozen at their initial (pretrained) values. We demonstrate that \emph{subset finetuning} (or SubTuning) often achieves accuracy comparable to full finetuning of the model, and even surpasses the performance of full finetuning when training data is scarce. Therefore, SubTuning allows deploying new tasks at minimal computational cost, while enjoying the benefits of finetuning the entire model. This yields a simple and effective method for multi-task learning, where different tasks do not interfere with one another, and yet share most of the resources at inference time. We demonstrate the efficiency of SubTuning across multiple tasks, using different network architectures and pretraining methods.

• The paper presents SubTuning, a method that selectively finetunes network layers using a data-driven finetuning profile to maximize validation gains.
• It demonstrates improved performance over full finetuning in low-data, data-corruption, and multi-task settings, achieving up to 3% greater accuracy under corruption.
• The approach reduces generalization error and computational cost while enabling scalable multi-task learning with efficient parameter updates.

Selective Layer Finetuning via SubTuning: A Technical Exposition

Introduction

The "Less is More: Selective Layer Finetuning with SubTuning" (2302.06354) presents a principled approach to parameter-efficient transfer learning by introducing SubTuning. This algorithmic strategy selectively updates a subset of network layers (and the readout head) while keeping the remaining parameters frozen at their pretrained values. The work advances the understanding of layer importance in finetuning, constructs a methodological finetuning profile for layer selection, and empirically and theoretically demonstrates the efficacy of SubTuning in various regimes—including low-data, data-corrupted, and multi-task learning (MTL) settings.

Finetuning Profile: Dissecting Layer Importance

A core contribution is the empirical construction of the finetuning profile, which quantifies the individual impact of each layer or block within a pretrained network when finetuned for a downstream task. The authors systematically finetune each block of a ResNet-50 (pretrained on ImageNet), one at a time, on CIFAR-10 and observe strong non-monotonicity in performance as a function of layer depth. Contrary to conventional wisdom, deeper layers are not always more beneficial for transfer, and block importance varies with architecture, pretraining, and target task.

Figure 1: Finetuning profile of ResNet-50 on CIFAR-10; SubTuning outperforms full finetuning under data corruption and across all dataset sizes.

The finetuning profile shows that optimal performance does not strictly coincide with updating the final blocks (with the largest parameter footprints), nor is it trivially determined by architectural position. This observation motivates a learnable, data-driven selection of layers for transfer, eschewing simplistic heuristics.

SubTuning Algorithm

Building on the profiling, SubTuning employs a greedy layer selection heuristic. The process iteratively selects layers that yield the maximal marginal validation gain, halting when further improvement drops below a threshold. This mechanism efficiently approximates the combinatorial subset selection, leveraging the finetuning profile to retain only a fraction of trainable weights commensurate with task requirements. Theoretical analysis establishes that generalization error is reduced compared to full finetuning, as the sample complexity now scales with the number of selected parameters, not the total network size. Specifically, for $k$ selected layers with $r' \ll r$ parameters, the error bound exhibits only a logarithmic dependence on overall layer count.

Low-Data and Data Corruption Settings

Empirical results demonstrate that SubTuning achieves superior performance to both linear probing and full finetuning in sample-starved and corrupted data scenarios. On benchmarks such as VTAB-1k (CIFAR-100, Flowers102, Caltech101, DMLab), SubTuning consistently surpasses alternatives like Head2Toe and LoRA, sometimes by large margins. Notably, with limited annotated samples and under distribution shifts (e.g., CIFAR-10-C with 14 distinct corruptions), SubTuning delivers up to 3% greater accuracy than full finetuning and outperforms layer-contiguous "surgical" finetuning baselines.

In the data scarcity regime, Figure 2 illustrates that when the labeled set is small, finetuning layers closer to the output is more effective, while with increasing data, finetuning earlier layers yields greater benefits.

Figure 2: Data size versus block selection in single-block SubTuning; optimal block moves earlier as dataset grows.

Additionally, SubTuning is shown to be robust and beneficial under active learning protocols, outperforming other transfer strategies in both random and margin-based sample acquisition schemes.

SubTuning Under Distribution Shift

SubTuning's ability to mitigate performance degradation under distribution shift is established in controlled CIFAR-10 to CIFAR-10-C experiments. By tuning a minimal set of strategically chosen blocks, SubTuning consistently exceeds full finetuning and "surgical finetuning" across a spectrum of corruptions, challenging prevailing wisdom about which layers should be adapted for robustness.

Multi-Task Learning and Computational Efficiency

SubTuning addresses a key bottleneck in MTL and continual learning: the inability to independently deploy finetuned networks without prohibitive compute/memory costs or catastrophic forgetting. By restricting updates to a designated layer subset for new tasks, SubTuning enables efficient deployment of multiple task-specialized models, each sharing most computation with the original backbone and forking only at selected blocks.

Figure 3: SubTuning for MTL in which tasks share backbone layers but diverge at selected blocks and readout heads.

The implementation reduces redundant computation and IO, as only the diff in weights between original and SubTuned branches are incurred. Latency–accuracy tradeoffs are empirically favorable; Figure 4 confirms that SubTuning achieves substantial gains in accuracy for minimal additional latency, cementing its value in resource-constrained deployments.

Figure 4: Accuracy versus A100 latency for SubTuning on CIFAR-10; significant gains with minimal inference cost.

Extensions: Pruning, Siamese SubTuning, and Initialization

Auxiliary analyses (Figures 9–13, 14, 15) validate further flexibility: (i) integrating channel pruning with SubTuning reduces the parameter overhead with negligible accuracy loss, (ii) Siamese SubTuning—concatenating features from both frozen and SubTuned paths—yields further accuracy boosts in low-data regimes, and (iii) the necessity of initialization from pretrained weights, as random reinitialization leads to degraded performance even with extended training schedules.

Theoretical and Practical Implications

This work challenges assumptions regarding blockwise transferability and demonstrates that adaptive, data-driven selection of trainable layers is critical for efficient and robust transfer. Practically, SubTuning provides a readily implementable mechanism for scalable adaptation in settings with multiple tasks, variable compute/memory budgets, or annotation bottlenecks. Theoretically, the results motivate deeper investigation into the mechanistic underpinnings of transfer and parameter reusability, and suggest opportunities for further synthesis with PETL techniques (e.g., adapters, LoRA, masking).

Conclusion

The investigation establishes SubTuning as a compelling method for parameter-efficient transfer learning. By leveraging the finetuning profile and greedy adaptive selection, SubTuning delivers enhanced sample efficiency, increased robustness to data corruption and distribution shifts, and enables scalable multi-task inference under compute constraints. Future developments may explore hybridization with orthogonal PETL approaches and automated layer selection strategies, potentially informing both the theoretical analysis of transfer learning phenomena and real-world neural network deployment.