Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes (1810.05148v4)

Abstract: There is a previously identified equivalence between wide fully connected neural networks (FCNs) and Gaussian processes (GPs). This equivalence enables, for instance, test set predictions that would have resulted from a fully Bayesian, infinitely wide trained FCN to be computed without ever instantiating the FCN, but by instead evaluating the corresponding GP. In this work, we derive an analogous equivalence for multi-layer convolutional neural networks (CNNs) both with and without pooling layers, and achieve state of the art results on CIFAR10 for GPs without trainable kernels. We also introduce a Monte Carlo method to estimate the GP corresponding to a given neural network architecture, even in cases where the analytic form has too many terms to be computationally feasible. Surprisingly, in the absence of pooling layers, the GPs corresponding to CNNs with and without weight sharing are identical. As a consequence, translation equivariance, beneficial in finite channel CNNs trained with stochastic gradient descent (SGD), is guaranteed to play no role in the Bayesian treatment of the infinite channel limit - a qualitative difference between the two regimes that is not present in the FCN case. We confirm experimentally, that while in some scenarios the performance of SGD-trained finite CNNs approaches that of the corresponding GPs as the channel count increases, with careful tuning SGD-trained CNNs can significantly outperform their corresponding GPs, suggesting advantages from SGD training compared to fully Bayesian parameter estimation.

• The paper establishes a theoretical link proving that deep convolutional networks with many channels converge to Gaussian processes, extending prior findings for fully connected networks.
• It introduces Monte Carlo estimation methods to practically compute the complex kernels of these CNN-GP equivalences, especially when pooling layers are present.
• Empirical findings show CNN-GPs achieve state-of-the-art for non-trainable kernel GPs, but finite CNNs trained with SGD can outperform them, highlighting practical differences in optimization benefits.

Bayesian Deep Convolutional Networks with Many Channels as Gaussian Processes

The paper under discussion establishes a significant theoretical connection between deep convolutional neural networks (CNNs) with a large number of channels and Gaussian processes (GPs). This work extends the previous understanding of fully connected networks (FCNs) acting as GPs by deriving a similar relationship for CNNs, both with and without pooling layers. The research highlights several implications for model selection and training methodologies in deep learning from a Bayesian perspective.

Analytical Contribution:

1. GP Equivalence Extension: The authors successfully extend the GP representation, previously established for infinitely wide fully connected layers, to CNNs with many channels. This is rigorously derived for various architectural settings, including those with pooling, strided convolutions, and different padding techniques. The derivation relies on proving that the network's output converges to a Gaussian distribution as the number of channels tends to infinity while holding the number of output channels fixed.
2. Monte Carlo Estimation: Given the impracticality of evaluating certain GP kernels analytically, especially with pooling layers, the authors introduce a Monte Carlo method. This computational approach allows for the estimation of NN-GP kernels, enabling practical use of these theoretical insights outside simplified contexts.
3. Convergence and Assumptions: The convergence proof is robust and requires that the neural network's activation function must be absolutely continuous with an exponentially bounded derivative. The paper presents detailed mathematical rigor for ensuring that as channels increase, the activations indeed follow a Gaussian process.

Empirical Findings:

• Performance Insights: The paper achieves state-of-the-art results for Gaussian processes with non-trainable kernels on the CIFAR10 dataset. This is notable given that typically, GPs are less competitive on complex datasets without introducing trainable kernel components.
• SGD vs. Bayesian Approaches: A pivotal empirical finding is that finite CNNs trained with stochastic gradient descent (SGD) can sometimes outperform their GP counterparts, especially with correct hyperparameter tuning, such as larger learning rates for ReLU networks. This suggests that the benefits of SGD, particularly in the presence of equivariance without pooling, can be leveraged effectively, possibly due to overparameterization advantages.

Theoretical and Practical Implications:

From a theoretical standpoint, the equivalence between CNNs and GPs in the Bayesian framework offers a new perspective for analyzing the bias-variance tradeoffs in neural network architectures. Understanding these models as GPs allows researchers to interpret weight priors explicitly, rather than implicitly through hyperparameters and initialization protocols.

Practically, the insights gained regarding the performance of CNN-GPs without pooling stress the importance of pooling layers for achieving translation invariance, which is crucial in tasks like image recognition. This can influence both the design and tuning of deep learning models, advocating for a more nuanced approach than purely increasing network width or channel count.

Future Directions:

The authors suggest several avenues for future work. These include exploring the extension of these results to networks using newer architectures like attention mechanisms or setups with batch normalization, where similar theoretical convergence guarantees could be investigated. Additionally, understanding the discrete differences where empirical SGD outpaces Bayesian models could yield new innovations in training regimens and neural architecture design.

Overall, this paper presents a rigorous theoretical development with substantial implications in both the understanding and practical deployment of convolutional neural networks in a Bayesian context, enriching the toolset of methods to approach deep learning from a probabilistic standpoint.