Deep Reference Priors: What is the best way to pretrain a model? (2202.00187v2)

Abstract: What is the best way to exploit extra data -- be it unlabeled data from the same task, or labeled data from a related task -- to learn a given task? This paper formalizes the question using the theory of reference priors. Reference priors are objective, uninformative Bayesian priors that maximize the mutual information between the task and the weights of the model. Such priors enable the task to maximally affect the Bayesian posterior, e.g., reference priors depend upon the number of samples available for learning the task and for very small sample sizes, the prior puts more probability mass on low-complexity models in the hypothesis space. This paper presents the first demonstration of reference priors for medium-scale deep networks and image-based data. We develop generalizations of reference priors and demonstrate applications to two problems. First, by using unlabeled data to compute the reference prior, we develop new Bayesian semi-supervised learning methods that remain effective even with very few samples per class. Second, by using labeled data from the source task to compute the reference prior, we develop a new pretraining method for transfer learning that allows data from the target task to maximally affect the Bayesian posterior. Empirical validation of these methods is conducted on image classification datasets. Code is available at https://github.com/grasp-lyrl/deep_reference_priors.

• The paper formalizes model pretraining by leveraging deep reference priors to maximize data influence on posterior distributions, improving performance with limited samples.
• It reformulates semi-supervised learning by deriving reference priors from unlabeled data, offering a theoretical basis for techniques like consistency regularization.
• Empirical results on CIFAR datasets show the approach achieves 85.45% accuracy on CIFAR-10 with just five labels per class, highlighting its effectiveness in transfer learning.

An Exploration of Deep Reference Priors for Model Pretraining

In this paper, the authors address the challenge of optimal data exploitation during the model pretraining phase, focusing particularly on how additional data, whether labeled from a related task or unlabeled from the same task, can be effectively utilized to enhance learning in Bayesian frameworks. They accomplish this by reconsidering pretraining through the lens of reference priors—a class of uninformative Bayesian priors designed to maximize the mutual information between the data and the model weights.

The principal motivation behind reference priors is to allow the available data to have the maximal influence on the posterior distributions rather than being overly prescriptive through prior assumptions. This is particularly salient in scenarios involving small sample sizes where the reference prior guides learning towards lower-complexity models—a form of regularization that naturally occurs due to limited data.

Contributions and Methodological Framework

The authors make several key contributions, providing both theoretical and empirical advancements for semi-supervised and transfer learning:

1. Formalizing Pretraining with Reference Priors: By leveraging the concept of mutual information maximization between tasks and model parameters, the paper formalizes how reference priors can intrinsically balance the likelihood of observed data in the posterior distribution, a technique that is particularly novel for deep networks.
2. Semi-supervised Learning and Reference Priors: Herein, the authors reformulate semi-supervised learning as a process of deriving reference priors using unlabeled data. This approach clarifies why existing methods such as FixMatch, which utilize consistency regularization and entropy minimization, demonstrate efficacy—they align well with the properties of reference priors.
3. Transfer Learning as a Two-stage Experiment: A comprehensive formulation is devised wherein knowledge is transferred from a source to a target task by constructing multi-stage reference priors. This adaptive approach optimizes the retention of task-relevant information and discards the irrelevant, drawing a parallel with the information bottleneck principle.

Empirical Evaluation and Results

The efficacy of these theoretical formulations is empirically validated using well-known image classification datasets like CIFAR-10 and CIFAR-100. Noteworthy findings include achieving 85.45% accuracy on CIFAR-10 with only five labeled samples per class—a result evidencing its competitiveness against state-of-the-art semi-supervised learning methods. Additionally, the empirical work showcases the superiority of reference priors over conventional fine-tuning techniques in transfer learning tasks, especially with small sample sizes.

Implications and Future Directions

From a theoretical standpoint, the development of reference priors for deep learning enriches the Bayesian inference toolkit, presenting a robust technique for incorporating diverse data sources effectively. These insights could dramatically impact the landscape of transfer learning, hinting at potential enhancements in domains such as continual learning, where the goal is to leverage historical data optimally as the task distribution evolves.

Future research could explore the scalability of these methods to larger, more complex architectures and tasks. There might also be a promising direction in integrating these ideas with neural architecture search or automated feature engineering to further harness the potential of these data-driven priors.

In conclusion, the paper provides substantial insight into how deep learning processes can be theoretically grounded and empirically optimized by incorporating reference priors, potentially transforming methodologies within the nuanced field of semi-supervised and transfer learning.