LoRA-VBLL: Bayesian & Vector Bank Adaptation

• LoRA-VBLL is an umbrella term for parameter-efficient fine-tuning methods that integrate low-rank adaptation with Bayesian or vector-bank techniques for enhanced uncertainty quantification and storage efficiency.
• IVON-LoRA employs full variational treatment to guide uncertainty-based pruning, improving calibration and performance compared to standard AdamW in commonsense reasoning tasks.
• VBLL-based constructions combine a deterministic backbone with a stochastic last layer, enabling scalable Bayesian inference with analytic updates for high-dimensional surrogate modeling.

LoRA-VBLL denotes a cluster of parameter-efficient fine-tuning constructions built around low-rank adaptation and variational Bayesian inference. The available literature suggests that the label is not yet standardized: one line of work uses it for variational Bayesian optimization of all LoRA parameters with IVON, another for a deterministic LoRA-adapted backbone paired with a Variational Bayesian Last Layer (VBLL), and another source uses “LoRA-VBLL” for the vector-bank-based reparameterization VB-LoRA (Cong et al., 17 Jun 2025, Xiang et al., 1 Oct 2025, Xiang et al., 3 Apr 2026, Li et al., 2024). Across these usages, the recurring objective is to preserve LoRA’s parameter efficiency while improving uncertainty quantification, calibration, recursive updating, or storage efficiency.

1. Terminology and scope

In recent arXiv literature, “LoRA-VBLL” is best understood as an umbrella label rather than a single canonical algorithm. The main usages are summarized below.

Usage in literature	Core construction	Representative source
Variational Bayesian low-rank LoRA	All LoRA parameters are treated as Gaussian random variables and optimized variationally with IVON	(Cong et al., 17 Jun 2025)
LoRA with Variational Bayesian Last Layer	A deterministic LoRA-adapted feature extractor is combined with a Bayesian linear or classification head	(Xiang et al., 1 Oct 2025, Xiang et al., 3 Apr 2026)
Vector-bank-based LoRA	LoRA factors are composed from a shared vector bank with a differentiable top-$k$ admixture module	(Li et al., 2024)

The first two usages are explicitly Bayesian in the probabilistic sense, but the locus of uncertainty differs. In IVON-LoRA, the posterior is over the LoRA parameters themselves. In VBLL-based constructions, the LoRA-adapted backbone is deterministic and only the last layer is stochastic. The third usage is terminologically distinct: VB-LoRA is primarily a storage and parameter-sharing scheme rather than a Bayesian posterior model (Li et al., 2024).

This divergence matters because the empirical and computational trade-offs differ sharply across formulations. Full variational treatment of LoRA parameters exposes posterior variance for pruning and test-time sampling, whereas last-layer Bayesianism supports analytic or near-analytic uncertainty estimates with a single backbone pass. Vector-bank formulations instead target extreme compression and cross-layer parameter sharing.

2. IVON-LoRA as variational Bayesian low-rank adaptation

A direct “LoRA-VBLL” construction is given by IVON-LoRA, which replaces AdamW with Implicit Variational Online Newton and treats all LoRA weights as Gaussian latent variables (Cong et al., 17 Jun 2025). For a pretrained weight matrix $W_0 \in \mathbb{R}^{d \times k}$, LoRA adds

$$\Delta W_0 = BA,\qquad A \in \mathbb{R}^{r \times k},\; B \in \mathbb{R}^{d \times r},\; r \ll \min(d,k),$$

so that

$W = W_0 + BA.$

Only $A$ and $B$ are trained, while $W_0$ is frozen. In the reported transformer setup, LoRA is applied to query and value matrices only, not the output projection, because of numerical issues observed by the authors.

The variational objective replaces point-estimate minimization by

$$\min_{q(\theta)} \; \mathbb{E}_{q(\theta)}[L(\theta)] + \lambda^{-1} D_{\mathrm{KL}}(q(\theta)\,\|\,p(\theta)),$$

with diagonal Gaussian posterior

$q(\theta)=\mathcal{N}(\mu,\mathrm{diag}(v))$

and isotropic Gaussian prior

$p(\theta)=\mathcal{N}(0,\sigma_p^2 I).$

IVON maintains a mean vector $W_0 \in \mathbb{R}^{d \times k}$0 and a scale vector from which the posterior variance is derived. The key relation is

$W_0 \in \mathbb{R}^{d \times k}$1

where $W_0 \in \mathbb{R}^{d \times k}$2 is an online estimate of the diagonal Hessian and $W_0 \in \mathbb{R}^{d \times k}$3 is weight decay. Training uses one Monte Carlo sample per step, with reparameterization

$W_0 \in \mathbb{R}^{d \times k}$4

and adds less than $W_0 \in \mathbb{R}^{d \times k}$5 wall-clock overhead.

A distinctive consequence of this formulation is post-training uncertainty-guided pruning. Parameters with the largest posterior variance are interpreted as weakly constrained by the data and are pruned by setting the corresponding posterior means to zero. The reported default is to prune $W_0 \in \mathbb{R}^{d \times k}$6 of LoRA parameters per matrix. On commonsense reasoning with Llama-3.2-3B, pruning $W_0 \in \mathbb{R}^{d \times k}$7–$W_0 \in \mathbb{R}^{d \times k}$8 substantially improves calibration while retaining or slightly improving accuracy, and uncertainty-guided pruning dominates random pruning at the same sparsity.

The empirical results are unusually strong for a Bayesian LoRA method at this scale. On a set of commonsense reasoning tasks, IVON-LoRA improves accuracy over AdamW by $W_0 \in \mathbb{R}^{d \times k}$9 and reduces ECE by $$\Delta W_0 = BA,\qquad A \in \mathbb{R}^{r \times k},\; B \in \mathbb{R}^{d \times r},\; r \ll \min(d,k),$$0. In Table 1, the posterior mean model attains average accuracy $$\Delta W_0 = BA,\qquad A \in \mathbb{R}^{r \times k},\; B \in \mathbb{R}^{d \times r},\; r \ll \min(d,k),$$1 versus $$\Delta W_0 = BA,\qquad A \in \mathbb{R}^{r \times k},\; B \in \mathbb{R}^{d \times r},\; r \ll \min(d,k),$$2 for AdamW, with ECE $$\Delta W_0 = BA,\qquad A \in \mathbb{R}^{r \times k},\; B \in \mathbb{R}^{d \times r},\; r \ll \min(d,k),$$3 versus $$\Delta W_0 = BA,\qquad A \in \mathbb{R}^{r \times k},\; B \in \mathbb{R}^{d \times r},\; r \ll \min(d,k),$$4 and NLL $$\Delta W_0 = BA,\qquad A \in \mathbb{R}^{r \times k},\; B \in \mathbb{R}^{d \times r},\; r \ll \min(d,k),$$5 versus $$\Delta W_0 = BA,\qquad A \in \mathbb{R}^{r \times k},\; B \in \mathbb{R}^{d \times r},\; r \ll \min(d,k),$$6. Under Bayesian prediction with $$\Delta W_0 = BA,\qquad A \in \mathbb{R}^{r \times k},\; B \in \mathbb{R}^{d \times r},\; r \ll \min(d,k),$$7 posterior samples, IVON-LoRA reaches accuracy $$\Delta W_0 = BA,\qquad A \in \mathbb{R}^{r \times k},\; B \in \mathbb{R}^{d \times r},\; r \ll \min(d,k),$$8, ECE $$\Delta W_0 = BA,\qquad A \in \mathbb{R}^{r \times k},\; B \in \mathbb{R}^{d \times r},\; r \ll \min(d,k),$$9, and NLL $W = W_0 + BA.$0, and is the only method reported there that improves accuracy over AdamW while using Bayesian test-time prediction. The same study reports average GLUE score $W = W_0 + BA.$1 for IVON-LoRA@mean on DeBERTa-v3-base, compared with $W = W_0 + BA.$2 for vanilla LoRA and $W = W_0 + BA.$3 for full fine-tuning. Test-time temperature scaling through $W = W_0 + BA.$4 provides an additional control knob: $W = W_0 + BA.$5 or $W = W_0 + BA.$6 yields modest accuracy gains of about $W = W_0 + BA.$7 on MMLU OOD evaluation with similar ECE and NLL.

3. Variational Bayesian last layers on top of LoRA-adapted features

A second major meaning of LoRA-VBLL places Bayesian uncertainty only in the last layer while keeping the pretrained backbone and the LoRA adapters deterministic. In high-dimensional Bayesian optimization, the surrogate model is

$W = W_0 + BA.$8

with Gaussian prior

$W = W_0 + BA.$9

and variational posterior

$A$0

The LoRA parameters $A$1, the variational parameters $A$2, and $A$3 are jointly optimized by maximizing the ELBO. The posterior predictive mean and variance are

$A$4

with predictive noise added through $A$5 if desired (Xiang et al., 1 Oct 2025).

For classification, PoLAR-VBLL uses the same deterministic-backbone, stochastic-last-layer template but with a multiclass softmax head. The logits are

$A$6

with prior

$A$7

and factorized Gaussian posterior across classes,

$A$8

where each $A$9 is a full covariance matrix. Because the expectation of the softmax log-normalizer is intractable, the method applies a Jensen-tightened ELBO. The resulting objective is analytic and avoids Monte Carlo through the backbone (Xiang et al., 3 Apr 2026).

This architectural split has several consequences. First, inference requires one backbone pass and only cheap last-layer sampling. Second, uncertainty is explicitly Bayesian only in the last layer, not in the LoRA-adapted features. Third, the head can support exact or near-exact recursive Bayes updates when the likelihood is linear-Gaussian, which is especially important in sequential decision problems such as Bayesian optimization. A plausible implication is that VBLL-based LoRA-VBLL methods trade some posterior expressiveness for a substantial gain in scalability relative to approaches that randomize all adapter parameters.

4. Recursive Bayes, ensembles, and Bayesian optimization

Within Bayesian optimization, LoRA-VBLL is developed as a surrogate for expensive high-dimensional black-box functions over continuous, categorical, ordinal, and mixed spaces (Xiang et al., 1 Oct 2025). Inputs are converted to text through LIFT-style prompting, processed by a pretrained LLM such as GPT-2 or LLaMA-3.1-8B, and mapped to a feature vector $B$0. The BO loop alternates between fitting or updating the surrogate and selecting the next point with an acquisition function, primarily Thompson sampling.

A central contribution is recursive updating between full fine-tuning events. When LoRA parameters are frozen, the model becomes a linear-Gaussian regression in $B$1, and the posterior update for $B$2 is exact. The reported update is Kalman filter-like: $B$3

$B$4

Re-fine-tuning of LoRA is triggered only when the predictive likelihood of a new observation falls below a threshold $B$5. Feature caching is used between fine-tuning steps for discrete candidate sets, so that acquisition optimization and recursive updates need only cheap linear operations.

The ensemble extension, ENS-LoRA-VBLL, addresses rank and hyperparameter sensitivity by maintaining multiple LoRA-VBLL surrogates with different ranks $B$6 and distinct priors or noise scales. Model weights are updated recursively by Bayes: $B$7 Thompson sampling is then performed under the model-averaged posterior by sampling both a model index and the corresponding last-layer parameters.

The reported empirical behavior is favorable in high-dimensional discrete and molecular settings. On Pest Control, ENS-LoRA-VBLL reaches approximately $B$8 by $B$9 evaluations. On MAXSAT60 and other discrete or continuous benchmarks, it is described as consistently competitive or superior, especially where GP-based methods struggle. In molecular tasks, different fixed ranks are preferred by different datasets, with rank $W_0$0 best on Redoxmer and rank $W_0$1 on Kinase, while ENS-LoRA-VBLL matches or exceeds the best single rank across tasks. Runtime and memory are also emphasized: BLoB uses about $W_0$2 GB GPU memory, LLLA about $W_0$3 GB, ENS-LoRA-VBLL about $W_0$4 GB, and small MLP-VBLL about $W_0$5 GB, with the paper attributing the gains to analytical ELBOs, rank-1 recursive updates, feature caching, and avoidance of Monte Carlo over LLM parameters.

5. PoLAR-VBLL and the geometry of uncertainty-aware adapters

PoLAR-VBLL is a refinement of LoRA-VBLL motivated by a geometric criticism of standard LoRA: rank collapse and “directional diversity collapse” (Xiang et al., 3 Apr 2026). The paper defines the stable rank of an update matrix $W_0$6 as

$W_0$7

and argues that standard LoRA often collapses toward effective rank $W_0$8, which in turn compresses features into a narrow subspace and harms Bayesian last-layer uncertainty estimation.

To address this, PoLAR replaces the standard LoRA update by

$W_0$9

where $$\min_{q(\theta)} \; \mathbb{E}_{q(\theta)}[L(\theta)] + \lambda^{-1} D_{\mathrm{KL}}(q(\theta)\,\|\,p(\theta)),$$0 and $$\min_{q(\theta)} \; \mathbb{E}_{q(\theta)}[L(\theta)] + \lambda^{-1} D_{\mathrm{KL}}(q(\theta)\,\|\,p(\theta)),$$1 have orthonormal columns and $$\min_{q(\theta)} \; \mathbb{E}_{q(\theta)}[L(\theta)] + \lambda^{-1} D_{\mathrm{KL}}(q(\theta)\,\|\,p(\theta)),$$2 is unconstrained. The Stiefel constraints are maintained approximately through landing fields rather than explicit retractions. The method is reported to be $$\min_{q(\theta)} \; \mathbb{E}_{q(\theta)}[L(\theta)] + \lambda^{-1} D_{\mathrm{KL}}(q(\theta)\,\|\,p(\theta)),$$3–$$\min_{q(\theta)} \; \mathbb{E}_{q(\theta)}[L(\theta)] + \lambda^{-1} D_{\mathrm{KL}}(q(\theta)\,\|\,p(\theta)),$$4 faster than explicit retractions on GPUs.

The Bayesian part remains a VBLL. The last-layer posterior is Gaussian with full covariance per class, the ELBO is Jensen-tightened and analytic, and training alternates between VB last-layer updates and PoLAR-parameter updates. The overall complexity is dominated by a single backbone pass per batch, with the VB contribution only in the last layer at $$\min_{q(\theta)} \; \mathbb{E}_{q(\theta)}[L(\theta)] + \lambda^{-1} D_{\mathrm{KL}}(q(\theta)\,\|\,p(\theta)),$$5. At inference time, features are computed once, then class-wise last-layer weights are sampled repeatedly: $$\min_{q(\theta)} \; \mathbb{E}_{q(\theta)}[L(\theta)] + \lambda^{-1} D_{\mathrm{KL}}(q(\theta)\,\|\,p(\theta)),$$6 and predictions are averaged over $$\min_{q(\theta)} \; \mathbb{E}_{q(\theta)}[L(\theta)] + \lambda^{-1} D_{\mathrm{KL}}(q(\theta)\,\|\,p(\theta)),$$7 samples.

The reported evidence supports the geometric argument. PoLAR maintains much higher stable rank, approximately $$\min_{q(\theta)} \; \mathbb{E}_{q(\theta)}[L(\theta)] + \lambda^{-1} D_{\mathrm{KL}}(q(\theta)\,\|\,p(\theta)),$$8 versus approximately $$\min_{q(\theta)} \; \mathbb{E}_{q(\theta)}[L(\theta)] + \lambda^{-1} D_{\mathrm{KL}}(q(\theta)\,\|\,p(\theta)),$$9 for LoRA. In Table 7, the Jensen-tightened ELBO differs from a $q(\theta)=\mathcal{N}(\mu,\mathrm{diag}(v))$0-sample Monte Carlo estimate by an initial gap of about $q(\theta)=\mathcal{N}(\mu,\mathrm{diag}(v))$1, but the gap drops below $q(\theta)=\mathcal{N}(\mu,\mathrm{diag}(v))$2 after $q(\theta)=\mathcal{N}(\mu,\mathrm{diag}(v))$3 steps and remains stable. On ARC-E inference, BLoB variants need about $q(\theta)=\mathcal{N}(\mu,\mathrm{diag}(v))$4–$q(\theta)=\mathcal{N}(\mu,\mathrm{diag}(v))$5 s, whereas PoLAR-VBLL needs about $q(\theta)=\mathcal{N}(\mu,\mathrm{diag}(v))$6 s, described as approximately $q(\theta)=\mathcal{N}(\mu,\mathrm{diag}(v))$7 faster; memory is about $q(\theta)=\mathcal{N}(\mu,\mathrm{diag}(v))$8 GB for PoLAR-VBLL versus $q(\theta)=\mathcal{N}(\mu,\mathrm{diag}(v))$9 GB for full Laplace approximation. The same study reports that PoLAR-VBLL is best or second-best in accuracy on most in-distribution datasets while also attaining low ECE and NLL, and that PoLAR-VBLL $p(\theta)=\mathcal{N}(0,\sigma_p^2 I).$0 LA slightly improves ECE and NLL over PoLAR-VBLL without LA without hurting accuracy. This suggests that, within the VBLL family, adapter geometry can be as consequential as the posterior family itself.

6. Vector-bank-based LoRA and the nomenclature conflict

A distinct usage of “LoRA-VBLL” appears in the VB-LoRA paper, where the term refers to a vector-bank-based LoRA rather than to variational Bayesian last layers or posteriors over LoRA parameters (Li et al., 2024). VB-LoRA begins from the standard LoRA decomposition

$p(\theta)=\mathcal{N}(0,\sigma_p^2 I).$1

then divides each vector $p(\theta)=\mathcal{N}(0,\sigma_p^2 I).$2 into fixed-length sub-vectors,

$p(\theta)=\mathcal{N}(0,\sigma_p^2 I).$3

and reconstructs each sub-vector from a global vector bank

$p(\theta)=\mathcal{N}(0,\sigma_p^2 I).$4

through a differentiable top-$p(\theta)=\mathcal{N}(0,\sigma_p^2 I).$5 admixture module: $p(\theta)=\mathcal{N}(0,\sigma_p^2 I).$6

The method’s “divide-and-share” paradigm breaks standard LoRA’s isolation across matrix dimensions, modules, and layers. During training, the vector bank and the per-subvector logits are optimized jointly. After training, the logits are discarded and only the bank, the top-$p(\theta)=\mathcal{N}(0,\sigma_p^2 I).$7 indices, and the admixture weights are stored. Under the paper’s storage accounting with $p(\theta)=\mathcal{N}(0,\sigma_p^2 I).$8 and $p(\theta)=\mathcal{N}(0,\sigma_p^2 I).$9-bit indices, the stored-parameter estimate is

$W_0 \in \mathbb{R}^{d \times k}$00

The reported compression is extreme. For Llama2-13B instruction tuning, standard LoRA stores $W_0 \in \mathbb{R}^{d \times k}$01M parameters, whereas VB-LoRA stores $W_0 \in \mathbb{R}^{d \times k}$02M, a ratio of

$W_0 \in \mathbb{R}^{d \times k}$03

In the same setup, the paper reports MT-Bench score $W_0 \in \mathbb{R}^{d \times k}$04 for VB-LoRA versus $W_0 \in \mathbb{R}^{d \times k}$05 for the authors’ LoRA run. On RoBERTa-large GLUE, VB-LoRA(qv) uses $W_0 \in \mathbb{R}^{d \times k}$06M parameters with average $W_0 \in \mathbb{R}^{d \times k}$07, and VB-LoRA(all) uses $W_0 \in \mathbb{R}^{d \times k}$08M with average $W_0 \in \mathbb{R}^{d \times k}$09. On GPT-2 Large E2E, VB-LoRA attains $W_0 \in \mathbb{R}^{d \times k}$10 for BLEU/NIST/METEOR/ROUGE-L/CIDEr with $W_0 \in \mathbb{R}^{d \times k}$11M parameters.

Because this line of work uses “LoRA-VBLL” for a non-Bayesian vector-bank formulation, terminological ambiguity is unavoidable. A practical reading is that “VB” may denote either “variational Bayesian” or “vector bank,” depending on the paper. This suggests that citations are essential whenever the term is used without expansion.

7. Serving and systems context

Separate systems work addresses the deployment problem that arises once many LoRA-style adapters must be served concurrently. S-LoRA is a serving system for thousands of concurrent LoRA adapters that stores all adapters in host memory, fetches the adapters used by the currently running queries to GPU memory, and uses Unified Paging to manage both dynamic adapter weights and KV cache tensors in a single memory pool (Sheng et al., 2023). It introduces heterogeneous batching with custom CUDA or Triton kernels and a LoRA-aware tensor parallelism strategy. The paper reports throughput improvements of up to $W_0 \in \mathbb{R}^{d \times k}$12 versus vLLM with naive support of LoRA serving and up to $W_0 \in \mathbb{R}^{d \times k}$13 versus HuggingFace PEFT, while scaling to $W_0 \in \mathbb{R}^{d \times k}$14 adapters on a single A100-80GB GPU in the reported configurations.

In multimodal vision serving, VaLoRA presents an end-to-end system that combines accuracy-aware LoRA adapter generation, an adaptive-tiling LoRA adapters batching operator, and a flexible orchestration mechanism for merged, unmerged, and mixture execution modes (Mi et al., 2024). It is evaluated on five vision tasks and three LMMs, and the paper reports $W_0 \in \mathbb{R}^{d \times k}$15–$W_0 \in \mathbb{R}^{d \times k}$16 accuracy improvements over the original LMMs and $W_0 \in \mathbb{R}^{d \times k}$17–$W_0 \in \mathbb{R}^{d \times k}$18 latency reductions relative to state-of-the-art LoRA model serving systems. Its adaptive-tiling matrix multiplication operator is reported to deliver average speedups of $W_0 \in \mathbb{R}^{d \times k}$19 over S-LoRA, $W_0 \in \mathbb{R}^{d \times k}$20 over Punica, and $W_0 \in \mathbb{R}^{d \times k}$21 over dLoRA for unmerged matmul operations.

These systems papers are not themselves definitions of LoRA-VBLL, but they clarify an important boundary. Bayesian fine-tuning, vector-bank compression, and last-layer variational inference address model construction; S-LoRA and VaLoRA address the orthogonal problem of runtime multiplexing, paging, batching, and scheduling. A plausible implication is that, as uncertainty-aware or highly compressed LoRA variants proliferate, practical deployment will increasingly depend on co-design between posterior structure and serving infrastructure.