Statistical mechanics of transfer learning in fully-connected networks in the proportional limit (2407.07168v1)

Abstract: Transfer learning (TL) is a well-established machine learning technique to boost the generalization performance on a specific (target) task using information gained from a related (source) task, and it crucially depends on the ability of a network to learn useful features. Leveraging recent analytical progress in the proportional regime of deep learning theory (i.e. the limit where the size of the training set $P$ and the size of the hidden layers $N$ are taken to infinity keeping their ratio $\alpha = P/N$ finite), in this work we develop a novel single-instance Franz-Parisi formalism that yields an effective theory for TL in fully-connected neural networks. Unlike the (lazy-training) infinite-width limit, where TL is ineffective, we demonstrate that in the proportional limit TL occurs due to a renormalized source-target kernel that quantifies their relatedness and determines whether TL is beneficial for generalization.

• The paper introduces a novel statistical mechanics framework using the single-instance Franz-Parisi formalism to examine transfer dynamics in neural networks.
• The paper demonstrates that a renormalized kernel with coupling parameter γ captures source-target relationships, enabling effective transfer learning in finite-width networks.
• The paper validates its theoretical predictions with empirical results across diverse tasks, revealing the critical role of network size in enabling adaptive feature learning.

Overview of Transfer Learning in Fully-Connected Networks

The paper "Statistical mechanics of transfer learning in fully-connected networks in the proportional limit" by Ingrosso et al. presents a detailed theoretical framework to understand Transfer Learning (TL) within the context of fully-connected neural networks, specifically when both the training set size $P$ and hidden layer size $N$ approach infinity while maintaining a finite ratio $\alpha = P/N$. This scenario, termed the proportional regime, underscores how neural networks can effectively utilize knowledge from pre-trained models on related tasks, despite traditional infinite-width (lazy-training) setups proving inadequate for TL.

Key Contributions

1. Single-Instance Franz-Parisi Formalism: The authors introduce an effective theory leveraging a single-instance Franz-Parisi framework. This novel approach, rooted in statistical mechanics, facilitates the examination of TL dynamics within Bayesian neural networks, particularly under the proportional limit where typical TL mechanisms such as feature-learning remain operational.
2. Transfer Dynamics: The paper elucidates how TL becomes viable through a renormalized kernel capturing source-target task relationships, quantified by a parameter $\gamma$ that governs the coupling strength. This kernel aids in determining TL efficacy, enabling feature transfer when similarities exist between the tasks.
3. Quantitative Agreement: Through rigorous analytical and experimental work, the paper demonstrates substantial alignment between theoretical predictions and empirical results across different tasks, specifically in one-hidden layer (1HL) network configurations. Such validation encompasses diverse source-target setups, e.g., C-EMNIST and C-CIFAR tasks, presenting how TL performance varies with data structure and network size.
4. Theoretical Insights on Model Size: Analysis includes insights into the network size's influence on TL. The findings stress that finite-width networks significantly benefit from TL over their infinite-width counterparts, which show negligible performance gains due to the absence of adaptive feature learning intrinsic to the infinite-width lazy-training regime.
5. Extended Applicability: Theoretical constructs are expected to hold more broadly for complex architectures, such as deep linear networks and convolutional layers. These extensions open new vistas for applying these frameworks to increasingly sophisticated TL problems.

Implications and Future Directions

Practically, the findings highlight the significance of network architecture choices and the intertwining of task-related features when deploying TL in deep learning applications, particularly in data-constrained scenarios. The results suggest optimizing coupling parameters and network size for maximal TL benefits, tailored to specific task characteristics, could prove advantageous. Additionally, the approach paves the way for understanding more intricate non-linear networks, where deep learning's lack of a comprehensive theoretical understanding has historically been a limitation.

Theoretically, the incorporation of statistical mechanics into machine learning, as exhibited in this paper, indicates a fruitful cross-disciplinary approach, potentially bridging gaps between AI and physical sciences. The exploration of TL in broader settings, such as beyond the proportional limit, remains an area ripe for further inquiry to enhance generalization capabilities in artificial neural systems.

In conclusion, this work lays foundational insights into the statistical underpinnings of TL, enriching both our theoretical knowledge and practical approaches in harnessing fully-connected networks for related task domains. The broader implications suggest directions for future research, including kernel-driven network designs and efficient, scalable TL methods across various domains of AI applications.