Deep Neural Networks as Gaussian Processes (1711.00165v3)

Abstract: It has long been known that a single-layer fully-connected neural network with an i.i.d. prior over its parameters is equivalent to a Gaussian process (GP), in the limit of infinite network width. This correspondence enables exact Bayesian inference for infinite width neural networks on regression tasks by means of evaluating the corresponding GP. Recently, kernel functions which mimic multi-layer random neural networks have been developed, but only outside of a Bayesian framework. As such, previous work has not identified that these kernels can be used as covariance functions for GPs and allow fully Bayesian prediction with a deep neural network. In this work, we derive the exact equivalence between infinitely wide deep networks and GPs. We further develop a computationally efficient pipeline to compute the covariance function for these GPs. We then use the resulting GPs to perform Bayesian inference for wide deep neural networks on MNIST and CIFAR-10. We observe that trained neural network accuracy approaches that of the corresponding GP with increasing layer width, and that the GP uncertainty is strongly correlated with trained network prediction error. We further find that test performance increases as finite-width trained networks are made wider and more similar to a GP, and thus that GP predictions typically outperform those of finite-width networks. Finally we connect the performance of these GPs to the recent theory of signal propagation in random neural networks.

• The paper presents a theoretical derivation extending GP equivalence from single-layer neural networks to deep, infinitely wide architectures.
• It introduces an efficient computational pipeline for calculating covariance functions, facilitating precise Bayesian inference.
• Experimental validation on MNIST and CIFAR-10 demonstrates that widening finite networks improves performance to mirror GP predictions.

Deep Neural Networks as Gaussian Processes

The paper "Deep Neural Networks as Gaussian Processes" by Jaehoon Lee et al., explores an equivalence previously noted for single-layer neural networks and extends it to deeper architectures. This paper is grounded in the established notion that an infinitely wide, single-layer fully-connected neural network with independent identically distributed (i.i.d.) parameters aligns with a Gaussian Process (GP). The authors elaborate on the exact equivalence between infinitely wide, deep neural networks and GPs, introducing a computationally efficient method to calculate the covariance function for such GPs. They further validate this theoretical framework by performing Bayesian inference on classic benchmarks such as MNIST and CIFAR-10 datasets.

Core Contributions and Findings

The paper's contributions can be segmented as follows:

1. Theoretical Derivation: The authors meticulously derive the exact equivalence between deep, infinitely wide neural networks and GPs. They compute the covariance function for these GPs based on recursive and deterministic kernel function calculations. This is an important theoretical step that substantiates the neural network-GP correspondence beyond the single-layer case.
2. Efficient Computational Pipeline: To make the theory practically applicable, an efficient computational method is developed to construct the covariance matrices requisite for Bayesian inference in the GP framework. This pipeline ensures that even for networks with numerous layers, calculation remains feasible.
3. Experimental Validation: The derived GPs were used to perform Bayesian inference on the MNIST and CIFAR-10 datasets. Results showed that neural network accuracy, as the layer width increases, approaches that of the corresponding GP. Moreover, the performance of the GP was found to correlate strongly with the performance of trained neural networks. Intriguingly, as finite-width networks become wider, their test performance improves and starts to mirror the GP predictions, which typically outperform finite-width networks.
4. Practical Implications: The paper makes a bold claim that GP uncertainty is significantly correlated with network prediction errors, underscoring the practical utility of GP-based Bayesian inference in predictively reliable scenarios. It demonstrates that GP models provide uncertainty estimates which are crucial for robust machine learning applications where prediction confidence matters, such as in autonomous systems and medical diagnosis.
5. Connection to Existing Theory: The GP formulations relate to the recent advancements in the understanding of signal propagation in deep, random neural networks. This connection is emphasized by showing how the structure of the GP kernel aligns with fixed points discovered in random signal propagation theories.

Implications and Future Directions

The implications of this research are manifold:

• Bayesian Deep Learning: The presented framework offers a clear pathway for adopting Bayesian methods in deep learning, allowing for exact Bayesian inference as opposed to approximate methods traditionally used in neural networks.
• Improved Generalization: The ability of neural networks to generalize better with larger layer widths, making them more GP-like, signals that leveraging GP methodologies could enhance model robustness and performance.
• Inference Efficiency: The computational efficiency achieved contributes towards making Bayesian methods more accessible for deep learning practitioners, bridging the gap between theoretical advancement and practical applicability.

Future work could explore larger and more complex datasets to determine scalability limits, the integration of these GP methods with convolutional neural networks (CNNs) and recurrent neural networks (RNNs), and the use of these methods for hyperparameter tuning and architecture search. Additionally, there is ample scope for developing new kernel functions inspired by various neural network architectures to further potentiate the synergy between GPs and deep learning.

Conclusion

The paper firmly establishes that deep neural networks, when infinitely wide, can be equivalently expressed as GPs, providing a robust theoretical underpinning. This equivalence not only enriches our theoretical understanding but also opens practical doors to apply Bayesian inference to deep learning, ensuring more reliable and interpretable AI systems. The work by Lee et al., thus offers valuable insights and tools to the field, promoting a more holistic integration of neural networks and probabilistic methods.