Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functions (1905.06076v2)

Abstract: A simple, flexible approach to creating expressive priors in Gaussian process (GP) models makes new kernels from a combination of basic kernels, e.g. summing a periodic and linear kernel can capture seasonal variation with a long term trend. Despite a well-studied link between GPs and Bayesian neural networks (BNNs), the BNN analogue of this has not yet been explored. This paper derives BNN architectures mirroring such kernel combinations. Furthermore, it shows how BNNs can produce periodic kernels, which are often useful in this context. These ideas provide a principled approach to designing BNNs that incorporate prior knowledge about a function. We showcase the practical value of these ideas with illustrative experiments in supervised and reinforcement learning settings.

• The paper introduces a framework for designing Bayesian Neural Network architectures that integrate expressive priors by leveraging Gaussian Process kernel combinations.
• Novel derivations show how BNNs can produce periodic kernels through input warping and activation functions, mimicking structures useful for time series data.
• Experiments demonstrate that these BNNs achieve superior predictive capability in time series forecasting and enhanced learning performance in reinforcement learning tasks.

Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functions

The integration of expressive priors into Bayesian neural networks (BNNs) represents a methodologically robust approach enhancing the flexibility and adaptability of neural networks. The paper discussed herein introduces a framework for designing BNN architectures that integrate expressive priors, as informed by the established relationship between Gaussian Processes (GPs) and BNNs.

Primarily, this research leverages the ability to construct expressive priors in GP models by combining basic kernels, such as summing a periodic and linear kernel to model variability with trends. While this approach has been well-explored within the domain of GPs, its application to BNNs has remained largely uncharted. The paper proposes architectures for BNNs that resemble these kernel combinations, thereby extending the analogous principles of GPs to Bayesian deep learning models.

A significant focus is placed on the generation of periodic functions within BNNs, which are frequently beneficial due to their structural resemblance to certain data patterns encountered in real-world scenarios. Novel derivations are presented for BNN architectures that produce periodic kernels through input warping followed by activation functions, achieving equivalence with popular GP kernels such as the exponential sine squared kernel.

The authors implement illustrative experiments in supervised and reinforcement learning settings to underscore the practical utility of these theoretical advancements. For time series prediction, architectures combining BNNs with GP-like kernels demonstrated superior predictive capability relative to traditional BNN forms, capturing trends and seasonality effectively. Similarly, in reinforcement learning, scenarios involving the pendulum swing-up task show enhanced learning performance and policy quality when leveraging BNN architectures with appropriately specified priors.

The implications of this research are twofold:

1. Theoretical Implications: It solidifies the connection between GPs and BNNs, offering a pathway to leverage the rich theoretical foundation of GPs within BNN frameworks. As BNNs approach the behavior of GPs at infinite width, the research outlines methods to exploit this convergence for practical model design.
2. Practical Implications: By providing a structured approach to incorporate prior knowledge into BNN model design, the findings improve model performance in tasks where specific structural properties of the data can inform the choice of model architecture. This can enhance data efficiency and performance in learning tasks.

Looking to the future, the developments outlined in this paper suggest exciting prospects for automating BNN design processes. This could involve creating mechanisms for dynamic adjustment of priors based on evolving data structures or abstract level specifications. Furthermore, as AI systems continue to evolve, the integration of structured priors informed by GP frameworks into BNNs promises enhanced model scalability and applicability across diverse domains, potentially bridging foundational machine learning concepts with cutting-edge applications.