---
title: 'BONG: Bayesian Online Natural Gradient'
url: https://www.emergentmind.com/topics/bayesian-online-natural-gradient-bong
type: topic
---

# BONG: Bayesian Online Natural Gradient

The Bayesian Online Natural Gradient (BONG) algorithm is a principled approach to sequential Bayesian inference that leverages the natural gradient—i.e., preconditioning by the Fisher Information Matrix (FIM)—to produce efficient, curvature-aware online updates for probabilistic models. BONG’s core insight is the theoretical equivalence between online natural gradient descent and extended Kalman filtering, enabling the integration of second-order information, rigorous uncertainty quantification, and scalable, deterministic approximations for high-dimensional problems such as Bayesian neural networks and fine-tuned vision-language models.

## 1. Theoretical Foundations and Equivalence to Kalman Filtering

BONG is grounded in the observation that natural gradient descent—a learning algorithm that scales parameter updates by the inverse Fisher information—admits an algebraic equivalence to the extended Kalman filter (EKF) in both the i.i.d. and recurrent settings. Specifically, estimating a parameter $\theta \in \mathbb{R}^n$ from a stream of observations via online natural gradient can be recast as applying an EKF to track a hidden state (the parameter vector) in light of new “pseudo-observations.” In this framework, the Fisher information accumulated from the data directly corresponds to the updated precision (inverse covariance) in the Kalman formalism [1703.00209, 2511.01694].

In the exponential family, the EKF’s information-filter step is:
\[
P_t^{-1} = P_{t-1}^{-1} + H_t^\top R_t^{-1} H_t = P_{t-1}^{-1} + F_t,
\]
with $H_t$ the Jacobian of the output w.r.t. $\theta$ and $R_t$ the observation noise covariance. The parameter update takes the form:
\[
\theta_t = \theta_{t-1} - P_t \nabla_\theta \ell_t,
\]
which matches the natural gradient step $\theta \leftarrow \theta - \eta F^{-1} \nabla \ell$ when $\eta$ and other hyperparameters are aligned.

## 2. Variational Bayesian Formulation and Single-Step Update

In online variational Bayes, a mean-field or structured Gaussian variational posterior $q_\lambda(\theta)$ is recursively updated as new data arrive. Standard VB minimizes:
\[
-\mathbb{E}_{q_\lambda}[\log p(y_t|x_t,\theta)] + \mathrm{KL}(q_\lambda(\theta)\|\pi_{t|t-1}(\theta)),
\]
where $\pi_{t|t-1}$ is the prior predictive. BONG simplifies this by performing a single natural gradient ascent step on the expected log-likelihood term, initialized at the prior parameter, and dropping the explicit KL regularizer. For an exponential family,
\[
\lambda_t = \lambda_{t|t-1} + F(\lambda_{t|t-1})^{-1} \nabla_\lambda \mathbb{E}_{q_{t|t-1}}[\log p(y_t|x_t,\theta)],
\]
and in the mean parameterization,
\[
\mu_t = \mu_{t|t-1} + \nabla_\mu \mathbb{E}_{q_{t|t-1}}[\log p(y_t|x_t,\theta)].
\]
BONG is thus a single-pass, one-step natural gradient mirror descent applied sequentially [2405.19681].

## 3. Exactness for Conjugate Exponential-Family Models

In the special case where the variational family and the likelihood form a conjugate exponential family, the BONG update precisely recovers the exact Bayesian posterior in one step. If
\[
p(y_t|\theta) \propto \exp(s(y_t)^\top T(\theta) - A(T(\theta))),
\]
with $q_{t|t-1}(\theta)$ also exponential family, then the update is
\[
\eta_t = \eta_{t|t-1} + \nabla_\eta \mathbb{E}_{q_{t|t-1}} [s(y_t)^\top T(\theta) - A(T(\theta))] = \eta_{t|t-1} + [s(y_t); 1],
\]
matching the canonical Bayesian filter.

## 4. Practical Approximations: Gaussian, Diagonal, and Low-Rank BONG

For non-conjugate problems, notably neural network parameterizations, full-covariance computations are infeasible. BONG employs approximations:
- **Diagonal approximation**: Store and update only the diagonal elements of the covariance or Fisher, leading to $\mathcal{O}(n)$ cost [2511.01694, 2511.13224].
- **Diagonal plus low-rank (DLR)**: Represent the precision as $\Lambda + W W^\top$, where $\Lambda$ is diagonal and $W$ is $n \times r$ with $r \ll n$; the Fisher/precision updates and associated SVD projections allow memory and computation savings at a small cost in fidelity [2405.19681].
- **Monte Carlo and linearized EKF**: Estimate the required moments $\mathbb{E}[\nabla \log p], \mathbb{E}[\nabla^2 \log p]$ via sampling or first-order Taylor approximations, respectively.

The BONG step for a Gaussian family is
\[
m_t = m_{t|t-1} + \Sigma_{t|t-1} \mathbb{E}_{q_{t|t-1}} [\nabla_\theta \log p(y_t|x_t,\theta)],
\]
\[
\Sigma_t^{-1} = \Sigma_{t|t-1}^{-1} - \mathbb{E}_{q_{t|t-1}} [\nabla_\theta^2 \log p(y_t|x_t,\theta)].
\]
For minibatch or Bayesian neural network training, these are computed efficiently via MC or diagonal/dlr approximations [2511.01694, 2511.13224, 2405.19681].

## 5. Iterative Natural Gradient Filtering in Nonlinear Systems

For highly nonlinear systems, the BONG approach can be extended to iterative, locally optimal natural gradient flows on the manifold of Gaussians. At each step, the update seeks stationary points of a variational objective combining expected loss and KL divergence to the prediction, leveraging the Fisher metrics of both mean and precision blocks. The resulting algorithm—termed NANO—proceeds by repeated natural-gradient steps on:
\[
J(\hat x, P) = \mathbb{E}_{\mathcal{N}(\hat x, P)}[-\log p(y|x)] + D_{\mathrm{KL}}(\mathcal{N}(\hat x, P) \| \mathcal{N}(\hat x_{t|t-1}, P_{t|t-1})),
\]
with Fisher-based preconditioning on both mean and covariance [2410.15832]. Specializing to the linear-Gaussian case, a single BONG step recovers EKF.

## 6. Computational Complexity and Scalability

BONG is explicitly designed for computational tractability in high-dimensional inference:
- Full-covariance Kalman/natural gradient is $\mathcal{O}(n^2)$ per step, impractical for $n \sim 10^6$.
- Diagonalization reduces cost and storage to $\mathcal{O}(n)$; DLR maintains scalability for modest-rank updates.
- Per-step complexity can match or marginally exceed first-order optimizers (SGD/Adam), making BONG suitable for online training of neural adapters or large language-vision models [2511.01694, 2511.13224].

## 7. Uncertainty Quantification, Trust-Region Mechanisms, and Empirical Results

BONG’s maintenance of an updated approximate posterior $q(\theta) = \mathcal{N}(m, P)$ endows each parameter with a calibrated uncertainty. Applications exploit this for Bayesian trust-region regularization, notably scaling updates by a Mahalanobis distance factor when the incoming data are out-of-distribution:
\[
\lambda = e^{-\alpha d_M},
\]
resulting in enhanced OOD robustness. Experimental results on few-shot vision-language adaptation demonstrate consistent outperformance or parity with first-order baselines in both in-distribution and OOD tasks, with marked improvements under severe domain shift (e.g., +9pp OOD robustness on corrupted ImageNet-C injection) [2511.01694]. Analogous benefits for BNNs include improved calibration and accelerated convergence compared to variational inference with first-order optimizers [2511.13224].

---

**References**:  
- "Bayesian Natural Gradient Fine-Tuning of CLIP Models via Kalman Filtering" [2511.01694]  
- "Nonlinear Bayesian Filtering with Natural Gradient Gaussian Approximation" [2410.15832]  
- "Natural gradient descent for improving variational inference based classification of radio galaxies" [2511.13224]  
- "Bayesian Online Natural Gradient (BONG)" [2405.19681]  
- "Online Natural Gradient as a Kalman Filter" [1703.00209]

Source: https://www.emergentmind.com/topics/bayesian-online-natural-gradient-bong