Bella: Bayesian Low-Rank Neural Learning

• Bayesian Low-Rank LeArning (Bella) is a framework that models uncertainty in neural networks using low-rank perturbations of a fixed pre-trained model.
• It drastically reduces trainable parameters and memory usage while achieving competitive performance on tasks such as ImageNet, CAMELYON17, and DomainNet.
• Bella supports both deep ensemble training and SVGD-based updates, enabling scalable uncertainty quantification with efficient low-rank kernel evaluations.

Bayesian Low-Rank LeArning (Bella) is a framework for practical Bayesian neural networks that represents each Bayesian particle as a low-rank perturbation of a pre-trained deterministic network rather than as a full copy of all parameters. In its canonical form, Bella freezes a shared base model, introduces multiple low-rank parameter perturbations, and then applies either deep-ensemble-style training or Stein Variational Gradient Descent (SVGD) in that reduced parameter space. The stated objective is to mitigate the computational burden of Bayesian learning while preserving, and in some cases improving, robustness, out-of-distribution behavior, and uncertainty quality on large-scale tasks including ImageNet, CAMELYON17, DomainNet, VQA with CLIP, and LLaVA (Doan et al., 2024).

1. Conceptual lineage and scope

Bella belongs to a broader Bayesian low-rank learning tradition in which low-dimensional structure is not fixed by hand as a hard rank constraint alone, but is induced through hierarchical priors, latent-factor constructions, or adaptive posterior approximations. Earlier work on sparse Bayesian methods for low-rank matrix estimation framed matrix completion and robust PCA through sparse Bayesian learning principles, starting from a matrix factorization formulation and enforcing the low-rank constraint in the estimates as a sparsity constraint; that work emphasized effective rank determination and high recovery performance (Babacan et al., 2011). Subsequent Bayesian matrix reconstruction methods used Gaussian priors with Kronecker-structured covariance matrices, evidence approximation, and EM to recover low-rank matrices when neither the rank nor the noise power is known a priori (Sundin et al., 2015). Online variational Bayes subspace learning likewise used sparse Bayesian learning principles to impose low-rankness explicitly and to adapt the rank recursively under incomplete observations (Giampouras et al., 2016).

Within that lineage, Bella transfers the low-rank idea from latent matrix estimation to neural parameter perturbations. Instead of placing the full posterior approximation directly over all network weights, Bella assumes that most posterior variability relevant for prediction can be expressed through low-rank updates around a strong pre-trained model. The paper states this implicitly through two assumptions: the pre-trained base $\theta_0$ is a good starting point close to high-density regions of the posterior, and the relevant posterior variability can be captured by low-rank perturbations around $\theta_0$ rather than arbitrary full-rank shifts (Doan et al., 2024). This places Bella alongside a class of Bayesian low-rank methods that trade unrestricted parameterization for tractable geometry, reduced memory, and scalable uncertainty quantification.

2. Low-rank parameterization and posterior approximation

For a dense layer with pre-trained weight $W_0 \in \mathbb{R}^{d_1 \times d_2}$, Bella defines particle $i$ by

$$W_i = W_0 + \Delta_i = W_0 + B_i A_i,\quad i=1,\ldots,n,$$

where $B_i \in \mathbb{R}^{d_1 \times r}$, $A_i \in \mathbb{R}^{r \times d_2}$, and $r \ll \min(d_1,d_2)$ (Doan et al., 2024). Across layers, a particle is therefore the collection of low-rank factor pairs $\{(A_i^{(\ell)}, B_i^{(\ell)})\}_\ell$, while the base weights $\{W_0^{(\ell)}\}_\ell$ are shared and frozen.

This parameterization gives a per-layer trainable parameter count of

$\theta_0$0

instead of $\theta_0$1, with relative fraction

$\theta_0$2

which is small for large layers and small $\theta_0$3 (Doan et al., 2024). Inference is then performed by averaging over particles: $\theta_0$4

Bella does not place its prior on the full parameter vector $\theta_0$5. Instead, the random variables are the low-rank factors $\theta_0$6, while the base model $\theta_0$7 is deterministic. In the SVGD formulation used in the main paper, the prior over $\theta_0$8 is an improper uniform prior, so the target posterior over $\theta_0$9 is driven entirely by the data likelihood (Doan et al., 2024). The approximate posterior is the empirical distribution over low-rank-adapted models,

$W_0 \in \mathbb{R}^{d_1 \times d_2}$0

which is a particle approximation rather than a factorized Gaussian variational family. This is one of Bella’s defining distinctions from conventional VI-based BNNs.

3. Training mechanisms: ensemble Bella and SVGD Bella

Bella supports two training regimes. The first is an ensemble-style construction in which each particle’s low-rank factors are trained independently with standard optimization, and predictive probabilities are averaged at inference time. The second is Bella-SVGD, which applies Stein Variational Gradient Descent directly to the low-rank factors (Doan et al., 2024).

For Bella-SVGD, the updates for $W_0 \in \mathbb{R}^{d_1 \times d_2}$1 and $W_0 \in \mathbb{R}^{d_1 \times d_2}$2 are

$W_0 \in \mathbb{R}^{d_1 \times d_2}$3

with

$W_0 \in \mathbb{R}^{d_1 \times d_2}$4

and an analogous expression for $W_0 \in \mathbb{R}^{d_1 \times d_2}$5 (Doan et al., 2024). The kernel is defined in original weight space, but for shift-invariant kernels such as the RBF kernel it can be evaluated using only the low-rank products $W_0 \in \mathbb{R}^{d_1 \times d_2}$6.

A notable technical feature is the low-rank evaluation of the RBF kernel

$W_0 \in \mathbb{R}^{d_1 \times d_2}$7

where the Frobenius norm term is expanded through trace identities so that kernel evaluation costs $W_0 \in \mathbb{R}^{d_1 \times d_2}$8 rather than $W_0 \in \mathbb{R}^{d_1 \times d_2}$9 per pair (Doan et al., 2024). This is the key step that makes repulsive particle methods feasible at CLIP and LLaVA scale.

The resulting training protocol is structurally simple: load a pre-trained base model, freeze it, initialize low-rank factors for each particle, and update only those factors. For ensemble Bella, the particles are decoupled; for Bella-SVGD, the updates additionally include the repulsive kernel-gradient term that discourages particle collapse (Doan et al., 2024).

4. Computational profile and empirical behavior

The central empirical claim of Bella is that low-rank posterior parameterization changes the scaling regime of Bayesian deep learning. For CLIP ViT-B/32, the paper reports the following parameter and memory profile (Doan et al., 2024):

Model	Particles $i$0	Trainable params
SVGD base	5	567M
SVGD Bella	5	1.84M
SVGD base	40	3.51B
SVGD Bella	100	36.86M

The same experiments report RAM and storage figures. With $i$1, full SVGD uses $i$2 GB RAM and $i$3 MB storage, whereas SVGD Bella uses $i$4 GB RAM and $i$5 MB storage; the single deterministic model uses $i$6 GB RAM and $i$7 MB storage (Doan et al., 2024). The practical implication is direct: Bella’s GPU memory footprint stays close to that of a single model even when the number of particles grows.

Performance-wise, Bella is positioned as a competitive approximation rather than a purely efficiency-oriented surrogate. On CLIP-based vision tasks, Bella-SVGD attains $i$8 on CIFAR-10, $i$9 on CIFAR-100, $$W_i = W_0 + \Delta_i = W_0 + B_i A_i,\quad i=1,\ldots,n,$$0 on CAMELYON17, $$W_i = W_0 + \Delta_i = W_0 + B_i A_i,\quad i=1,\ldots,n,$$1 on DomainNet, and $$W_i = W_0 + \Delta_i = W_0 + B_i A_i,\quad i=1,\ldots,n,$$2 on ImageNet, compared with $$W_i = W_0 + \Delta_i = W_0 + B_i A_i,\quad i=1,\ldots,n,$$3, $$W_i = W_0 + \Delta_i = W_0 + B_i A_i,\quad i=1,\ldots,n,$$4, $$W_i = W_0 + \Delta_i = W_0 + B_i A_i,\quad i=1,\ldots,n,$$5, $$W_i = W_0 + \Delta_i = W_0 + B_i A_i,\quad i=1,\ldots,n,$$6, and $$W_i = W_0 + \Delta_i = W_0 + B_i A_i,\quad i=1,\ldots,n,$$7 for full SVGD, and substantially lower results for the single-model baseline on the same tasks (Doan et al., 2024). On CIFAR-100, Bella-SVGD exceeds the reported full-SVGD number. On CAMELYON17 and DomainNet, the gap to full SVGD is small while the parameter reduction is dramatic.

The rank and particle-count ablations are equally central. The paper reports that performance rises sharply from $$W_i = W_0 + \Delta_i = W_0 + B_i A_i,\quad i=1,\ldots,n,$$8 to $$W_i = W_0 + \Delta_i = W_0 + B_i A_i,\quad i=1,\ldots,n,$$9, then shows diminishing returns as $B_i \in \mathbb{R}^{d_1 \times r}$0 increases further; with $B_i \in \mathbb{R}^{d_1 \times r}$1, Bella reaches approximately $B_i \in \mathbb{R}^{d_1 \times r}$2 of single-model parameters and nearly matches full SVGD on CAMELYON17 and ImageNet (Doan et al., 2024). Increasing the number of particles improves Bella monotonically on CIFAR-10, which is consistent with the particle approximation view of the posterior.

Bella is also evaluated on multimodal VQA with LLaVA-1.5-7B. Because of engineering constraints, the LLaVA study uses several fine-tuned variants as Bella-like particles rather than a full low-rank SVGD implementation, but the reported Yes/No and Number results still show improved accuracy and exact match for the SVGD-style Bella construction, together with lower entropy on correct answers and higher entropy on incorrect ones (Doan et al., 2024).

5. Relation to other Bayesian low-rank neural models

Bella is not the only neural instantiation of Bayesian low-rank learning. A closely related construction is LR-BPFL, which learns a global deterministic model together with personalized low-rank Bayesian corrections in federated learning. There, the effective weight matrix is

$B_i \in \mathbb{R}^{d_1 \times r}$3

with Gaussian variational posteriors on the low-rank factors and an adaptive rank selection mechanism based on continuous gates and hard thresholding (Zhang et al., 2024). Both Bella and LR-BPFL therefore share the basic design pattern of a deterministic backbone plus stochastic low-rank corrections, but they target different inference regimes: Bella emphasizes deep ensembles and SVGD for large centralized models, whereas LR-BPFL addresses client-specific uncertainty and personalization under communication constraints (Doan et al., 2024).

Outside neural networks, Bayesian low-rank learning has developed along several complementary lines. Bayesian low-rank matrix reconstruction with Kronecker-structured Gaussian priors uses evidence maximization and EM to infer both noise precision and effective rank (Sundin et al., 2015). Online Bayesian subspace learning from incomplete data uses hierarchical sparse Bayesian priors to induce low rank and optional sparsity in the learned dictionary (Giampouras et al., 2016). Tensor methods extend the same logic to tensor ring and tensor train decompositions, using Student-$B_i \in \mathbb{R}^{d_1 \times r}$4-inducing Gaussian-Gamma hierarchies or Bayesian ALS updates to infer low-rank structure and posterior uncertainty (Long et al., 2020, Menzen et al., 2020). A later probabilistic analysis of the nuclear norm introduced the nuclear norm distribution

$B_i \in \mathbb{R}^{d_1 \times r}$5

showed that it is a proper prior, that it is rotationally invariant, and that $B_i \in \mathbb{R}^{d_1 \times r}$6 is Gamma distributed, thereby supplying a direct Bayesian interpretation of nuclear-norm-regularized low-rank inference (Segert et al., 6 Oct 2025).

This broader literature suggests that Bella is best understood not as an isolated trick, but as the neural-network specialization of a larger program: replace full Bayesian parameter spaces with structured low-rank latent spaces, then recover posterior diversity with hierarchical priors, particle methods, or both.

6. Misconceptions, limitations, and open directions

A recurring misconception is to treat Bella as a full Bayesian posterior over all network weights. The paper does not do that. The base parameters are deterministic and frozen, the prior over low-rank factors in the SVGD formulation is improper uniform, and the approximate posterior is the empirical distribution over particles $B_i \in \mathbb{R}^{d_1 \times r}$7 rather than a density over the full parameter tensor (Doan et al., 2024). Bella is therefore a low-rank Bayesian approximation around a fixed backbone, not a full-weight Bayesian model.

The stated limitations follow directly from that approximation. Bella depends on a strong pre-trained base model; if $B_i \in \mathbb{R}^{d_1 \times r}$8 is poor, low-rank perturbations may not suffice. It assumes that relevant posterior variability lies in a low-dimensional subspace around that base model. Its SVGD variant still incurs kernel-evaluation overhead, even though the low-rank kernel formulas make that overhead tractable. For LLaVA, the paper explicitly notes that engineering constraints prevented a true low-rank SVGD implementation, so the multimodal experiments are not yet a complete end-to-end realization of the framework (Doan et al., 2024).

The open directions identified in the paper are correspondingly structural: more expressive priors and posteriors on low-rank parameters, hierarchical or multi-rank perturbations, improved integration with large language and multimodal models, alternative kernels and repulsion schemes, and broader modality coverage (Doan et al., 2024). A plausible extension is adaptive rank selection in the style of LR-BPFL, where rank components are continuously gated and then thresholded client- or task-specifically (Zhang et al., 2024). Another plausible implication is that the probabilistic and geometric analyses developed for matrix priors such as the nuclear norm distribution may eventually inform more explicit prior design for Bella-style neural posterior approximations (Segert et al., 6 Oct 2025).

In that sense, Bella occupies a specific and technically consequential position in Bayesian deep learning: it operationalizes the low-rank principle not at the level of observed matrices or latent tensors, but at the level of neural parameter perturbations, and uses that choice to make particle-based Bayesian methods practical on model scales where full-parameter BNNs had largely become computationally prohibitive (Doan et al., 2024).