Kalman Bayesian Neural Networks Overview

• Kalman Bayesian Neural Networks are Bayesian models that integrate Kalman filtering techniques to compute predictive distributions and update weight uncertainties.
• They employ closed-form Bayesian learning for sequential/online training without relying on gradient descent, enabling efficient adaptation under changing data distributions.
• Recent implementations combine KBNNs with Kalman correction steps for transformer fine-tuning and uncertainty-aware tracking in real-time applications.

Kalman Bayesian Neural Networks (KBNNs) are Bayesian neural-network formulations in which the calculation of the predictive distribution of the output and the update of the weight distribution are treated as Bayesian filtering and smoothing problems, where the weights are modeled as Gaussian random variables (Wagner et al., 2021). In this formulation, closed-form expressions are used for training the network’s parameters in a sequential/online fashion without gradient descent (Wagner et al., 2021). In later work, Kalman Bayesian Neural Networks are used as a core component for sequential fine-tuning of transformers under data distribution shift, and closely related hybrid systems couple a trained BNN with a Kalman correction step or transform the KF into a stochastic machine learning architecture for uncertainty-aware tracking (Jing et al., 12 Sep 2025, Gupta et al., 30 Apr 2026, Dahan et al., 2023).

1. Origins in neural implementations of Kalman inference

An early neural-network precursor showed that optimal Kalman prediction and control (KPC), as well as system identification, can be learned and executed by a recurrent neural network composed of linear-response nodes, using as input only a stream of noisy measurement data. To work with noisy data, the Kalman equations are reformulated in measurement space, since the true plant state is unobservable and the transformation $H$ is unknown. The resulting learning rules are local/Hebbian, and the task imposes strong, non-arbitrary constraints on the network’s architecture, signal flows, and processing modes (0805.4247).

A complementary line of work showed that a gradient-descent approximation to the Kalman filter requires only local computations with variance weighted prediction errors, and that it is possible under the same scheme to adaptively learn the dynamics model with a learning rule that corresponds directly to Hebbian plasticity (Millidge et al., 2021). In parallel, KalmanNet retained the recursive, two-step KF structure and replaced the model-based computation of the Kalman gain with a small neural network trained end-to-end from data, thereby learning to carry out Kalman filtering under non-linear dynamics with partial information (Revach et al., 2021).

This suggests that KBNNs emerged within a broader program: preserving recursive Bayesian filtering structure while replacing analytically unavailable components with learned neural computations.

2. Closed-form Bayesian learning over neural-network weights

In the closed-form KBNN formulation, the model is a feed-forward multilayer perceptron with $L$ layers, ReLU or sigmoid activations, and all weights modeled as random variables with Gaussian distributions. The key structural assumption is that neurons are pairwise independent, but weights within a neuron can be correlated. For each neuron $i$ in layer $l$,

$$p(\mathbf{W}^l) = \prod_{i=1}^{M_l} \mathcal{N}(\mathbf{w}_i^l \mid \mu_{w}^i, \Sigma_{w}^i).$$

The forward pass propagates predictive means and covariances layer by layer: $\mu_{a}^{n} = (\mu_{w}^n)^{\top} \mu_z^l,$

$$(\sigma_{a}^{n})^2 = (\mu_{w}^n)^{\top} \Sigma_z^l \mu_{w}^n + (\mu_z^l)^{\top} \Sigma_{w}^n \mu_z^l + \operatorname{tr}(\Sigma_{w}^n \Sigma_z^l),$$

$$\mu_{z}^{l+1,n} = \mathbb{E}[f(a_n^l)], \qquad (\sigma_{z}^{l+1,n})^2 = \mathbb{E}[f(a_n^l)^2] - [\mu_{z}^{l+1,n}]^2.$$

This forward propagation is described as analogous to the prediction step of a Kalman filter (Wagner et al., 2021).

The backward pass is cast as Bayesian smoothing. If

$$p\left(\begin{bmatrix}\mathbf{s} \ \mathbf{t}\end{bmatrix}\right) = \mathcal{N}\left( \begin{bmatrix}\mu_s \ \mu_t\end{bmatrix}, \begin{bmatrix}\Sigma_s & \Sigma_{st} \ \Sigma_{st}^\top & \Sigma_t\end{bmatrix} \right),$$

then conditioning on the updated mean and covariance of $\mathbf{t}$ yields [ \mu_s^+ = \mu_s + \Sigma_{st} (\Sigma_t)^{-1} (\mu_t^+ - \mu_t), \