Scalable Generalized Bayesian Online Neural Network Training for Sequential Decision Making (2506.11898v1)

Abstract: We introduce scalable algorithms for online learning and generalized Bayesian inference of neural network parameters, designed for sequential decision making tasks. Our methods combine the strengths of frequentist and Bayesian filtering, which include fast low-rank updates via a block-diagonal approximation of the parameter error covariance, and a well-defined posterior predictive distribution that we use for decision making. More precisely, our main method updates a low-rank error covariance for the hidden layers parameters, and a full-rank error covariance for the final layer parameters. Although this characterizes an improper posterior, we show that the resulting posterior predictive distribution is well-defined. Our methods update all network parameters online, with no need for replay buffers or offline retraining. We show, empirically, that our methods achieve a competitive tradeoff between speed and accuracy on (non-stationary) contextual bandit problems and Bayesian optimization problems.

• The paper introduces a hybrid methodology that combines frequentist low-rank updates with Bayesian posterior predictive distributions for online training in sequential decision-making tasks.
• It presents a novel parameter update strategy that efficiently approximates error variance while maintaining valid predictive distributions despite improper posteriors.
• Empirical results demonstrate superior performance in non-stationary contextual bandits and competitive Bayesian optimization compared to existing methods.

Scalable Generalized Bayesian Online Neural Network Training for Sequential Decision Making

This paper presents novel computational strategies for online training of neural networks leveraging scalable algorithms that integrate efficient frequentist inference with Bayesian posterior predictive distributions, specifically tailored for sequential decision-making tasks like contextual bandits and Bayesian optimization. The primary challenge in deploying Bayesian Neural Networks (BNNs) for these tasks lies in computationally expensive inference, which limits their application, especially in time-sensitive environments.

Key Contributions

1. Hybrid Methodology: The authors propose a methodological fusion that utilizes both frequentist and Bayesian filtering techniques. The frequentist approach facilitates efficient low-rank updates by focusing on parameter estimation and error variance quantification without demanding proper density characterization. This is complemented by deriving a well-defined posterior predictive distribution employing the Martingale posterior framework, supporting principled sequential decision-making even in the absence of traditional prior-likelihood-posterior updates.
2. Parameter Update Strategy: The paper introduces a novel update procedure for neural network parameters. Hidden layer parameters use a low-rank error covariance approximation, whereas the final layer parameters rely on a full-rank covariance. This is handled by maintaining an approximate error variance-covariance matrix represented partially as block-diagonal. Despite the use of improper posteriors, the resulting posterior predictive distribution remains valid, thus empowering sample-based decision-making.
3. Empirical Performance: Empirical evaluations illustrate that these methods provide robust trade-offs between computational speed and accuracy, outperforming existing methods in sequential decision-making scenarios characterized by non-stationarity. Specifically, the proposed methodology demonstrated superior performance in non-stationary neural contextual bandits and matched the efficacy of replay-buffer-based approaches in stationary settings such as Bayesian optimization.

Implications and Future Directions

The authors have successfully showcased the scalability of their approach, making it viable for real-world sequential decision-making applications like adaptive control and financial forecasting. The ability to train neural networks online without replay buffers or offline retraining presents a practical advancement for environments where data streams continuously and dynamically evolve.

Theoretically, the proposed hybrid filtering framework challenges traditional paradigms by sidestepping stringent requirements for proper density updates and focusing on error variance encapsulation. This perspective could stimulate further exploration into generalized Bayesian frameworks suitable for high-dimensional and non-stationary data environments.

As artificial intelligence progresses, methodologies like those presented herein could foster advancements in fully-online reinforcement learning and expand into broader domains requiring adaptive, real-time learning capabilities. One promising avenue for future research might be extending these algorithms to address fixed-lag smoothing, incorporating sensitivity adjustments for outliers, and reducing dependence on hyperparameter selection.

In conclusion, this work presents a robust and theoretically sound framework for scaling online inference in neural networks, positioning itself as a significant tool for sequential decision-making applications across diverse fields. The practical and theoretical insights offered here may pave the way for further innovations in efficient and scalable Bayesian learning.