Bi-LoRA: Dual Low-Rank Adaptation

• Bi-LoRA is defined as a dual-adapter system where one low-rank module uses gradient descent for task adaptation and another uses gradient ascent to probe sharpness.
• It reformulates SAM for efficient PEFT by decoupling the adaptation and perturbation modules, thereby avoiding the collapse of the perturbation space seen in naive LoRA-SAM models.
• Empirical results show Bi-LoRA achieves flatter minima and improved accuracy across tasks such as language understanding and diffusion-model personalization with minimal extra computational overhead.

Searching arXiv for the named method and closely related papers to ground the article in recent literature. Bi-directional Low-Rank Adaptation (Bi-LoRA) most commonly denotes a parameter-efficient fine-tuning scheme that reformulates sharpness-aware training for large pre-trained models by pairing two disjoint low-rank adapters: a primary adapter optimized by gradient descent for task adaptation and an auxiliary adapter optimized by gradient ascent to simulate adversarial weight perturbations. In this usage, introduced in "Bi-LoRA: Efficient Sharpness-Aware Minimization for Fine-Tuning Large-Scale Models" (Liu et al., 27 Aug 2025), the method is designed to retain the generalization benefits associated with Sharpness-Aware Minimization (SAM) while avoiding SAM’s doubled training cost and substantial extra memory overhead. The term is, however, overloaded: "BiLoRA" has also been used for a bi-level optimization framework for overfitting-resilient LoRA (Qiang et al., 2024), and "Bi-LORA" has been used for a vision-language synthetic-image detection method that inserts LoRA adapters into two parts of a BLIP-2 stack (Keita et al., 2024). A related later development, D²-LoRA, is explicitly described as an instance of a more general bi-directional low-rank adaptation family (Fujisawa et al., 16 Feb 2026).

1. Core definition and problem setting

In the 2025 Bi-LoRA formulation, the starting point is the tension between generalization and efficiency in low-data fine-tuning of large models. SAM seeks solutions that remain low-loss under small worst-case perturbations, with objective

$$\min_{\theta}\max_{\|\epsilon\|\le \rho} L(\theta+\epsilon),$$

and in practice approximates the inner maximization by

$$\epsilon^{*}\simeq \rho \cdot \frac{\nabla_{\theta}L(\theta)}{\|\nabla_{\theta}L(\theta)\|}.$$

LoRA, by contrast, freezes a pre-trained matrix $W_0\in\mathbb{R}^{m\times n}$ and learns a low-rank update

$W=W_0+\Delta W,\qquad \Delta W = B A,$

with $B\in\mathbb{R}^{m\times r}$, $A\in\mathbb{R}^{r\times n}$, and $r\ll \min(m,n)$ (Liu et al., 27 Aug 2025).

Bi-LoRA is motivated by the observation that a direct SAM-on-LoRA construction restricts the perturbation to a narrow low-rank subspace. The method therefore introduces a second low-rank adapter and separates two roles that are entangled in naïve LoRA-SAM: task fitting and sharpness probing. The resulting construction is still parameter-efficient, since only low-rank factors are trainable, but it changes the optimization geometry by allowing the perturbation module to evolve independently of the task-adaptation module.

A common source of confusion is nomenclature. In the 2024 bi-level "BiLoRA" paper, the central idea is not SAM but splitting pseudo-singular vectors and pseudo-singular values across two disjoint data subsets inside a nested optimization procedure (Qiang et al., 2024). In the 2024 vision-language "Bi-LORA" paper, "bi-directional" refers to placing LoRA adapters in both the Q-Former and the decoder to reframe synthetic-image detection as caption generation (Keita et al., 2024). D²-LoRA, in turn, uses additive and subtractive low-rank branches plus a train-time column-wise projection, and presents itself as a particular instance of a broader bi-directional low-rank adaptation design space (Fujisawa et al., 16 Feb 2026).

2. From LoRA-SAM to the dual-adapter min-max formulation

The motivating baseline in the 2025 paper is a naïve "LoRA-SAM" objective,

$$\min_{B,A}\max_{\|\epsilon_{B},\epsilon_{A}\|\le \rho} L\bigl(W_0+(B+\epsilon_B)(A+\epsilon_A)\bigr).$$

Expanding the adapted weight gives an effective perturbation

$$\epsilon_W = B\epsilon_A + \epsilon_B A + \epsilon_B\epsilon_A,$$

with $\epsilon_B\epsilon_A$ shown to be negligible. The crucial proposition is that the effective perturbation space of LoRA-SAM is spanned by $$\epsilon^{*}\simeq \rho \cdot \frac{\nabla_{\theta}L(\theta)}{\|\nabla_{\theta}L(\theta)\|}.$$0 through $$\epsilon^{*}\simeq \rho \cdot \frac{\nabla_{\theta}L(\theta)}{\|\nabla_{\theta}L(\theta)\|}.$$1 and $$\epsilon^{*}\simeq \rho \cdot \frac{\nabla_{\theta}L(\theta)}{\|\nabla_{\theta}L(\theta)\|}.$$2 through $$\epsilon^{*}\simeq \rho \cdot \frac{\nabla_{\theta}L(\theta)}{\|\nabla_{\theta}L(\theta)\|}.$$3. As $$\epsilon^{*}\simeq \rho \cdot \frac{\nabla_{\theta}L(\theta)}{\|\nabla_{\theta}L(\theta)\|}.$$4 and $$\epsilon^{*}\simeq \rho \cdot \frac{\nabla_{\theta}L(\theta)}{\|\nabla_{\theta}L(\theta)\|}.$$5 converge, this subspace shrinks, which limits SAM’s ability to explore full-space sharpness (Liu et al., 27 Aug 2025).

Bi-LoRA replaces that construction with two disjoint adapters per frozen weight:

$$\epsilon^{*}\simeq \rho \cdot \frac{\nabla_{\theta}L(\theta)}{\|\nabla_{\theta}L(\theta)\|}.$$6

Here $$\epsilon^{*}\simeq \rho \cdot \frac{\nabla_{\theta}L(\theta)}{\|\nabla_{\theta}L(\theta)\|}.$$7 is the primary adapter used for task adaptation, while $$\epsilon^{*}\simeq \rho \cdot \frac{\nabla_{\theta}L(\theta)}{\|\nabla_{\theta}L(\theta)\|}.$$8 is the auxiliary adapter used to model adversarial perturbation. The joint objective becomes

$$\epsilon^{*}\simeq \rho \cdot \frac{\nabla_{\theta}L(\theta)}{\|\nabla_{\theta}L(\theta)\|}.$$9

This decouples adaptation from perturbation. The auxiliary branch is constrained by a Frobenius-norm ball, and the corresponding proposition states that the perturbation subspace is $W_0\in\mathbb{R}^{m\times n}$0, independent of $W_0\in\mathbb{R}^{m\times n}$1. By choosing the auxiliary rank $W_0\in\mathbb{R}^{m\times n}$2 sufficiently large and exploiting the slower convergence of $W_0\in\mathbb{R}^{m\times n}$3, the method captures broader directions of sharpness (Liu et al., 27 Aug 2025).

This formulation is the sense in which the method is "bi-directional": one low-rank branch follows the descent direction of the task loss, while the other follows the ascent direction associated with sharpness. The term does not denote bidirectional sequence modeling, nor does it denote the row/column symmetry used in BoRA.

3. Optimization mechanics and systems profile

Let $W_0\in\mathbb{R}^{m\times n}$4 be evaluated at

$W_0\in\mathbb{R}^{m\times n}$5

Bi-LoRA updates the two adapters in opposite directions using the same backward pass:

$W_0\in\mathbb{R}^{m\times n}$6

$W_0\in\mathbb{R}^{m\times n}$7

The primary branch therefore performs gradient descent, and the auxiliary branch performs gradient ascent. Because both use the same $W_0\in\mathbb{R}^{m\times n}$8, the method requires no extra forward/backward pass relative to standard LoRA training, aside from the small overhead of the auxiliary factors (Liu et al., 27 Aug 2025).

After the ascent step, the auxiliary perturbation is projected back into the admissible norm ball. With multiple adapted layers indexed by $W_0\in\mathbb{R}^{m\times n}$9, the method computes

$W=W_0+\Delta W,\qquad \Delta W = B A,$0

If $W=W_0+\Delta W,\qquad \Delta W = B A,$1, it rescales both factors as

$W=W_0+\Delta W,\qquad \Delta W = B A,$2

The algorithmic loop is correspondingly simple: initialize $W=W_0+\Delta W,\qquad \Delta W = B A,$3 to zero; sample a batch; compute loss and merged weight; backpropagate $W=W_0+\Delta W,\qquad \Delta W = B A,$4; update the primary adapter by descent and the auxiliary adapter by ascent; project the auxiliary adapter; repeat until convergence; and discard the auxiliary adapter at inference, leaving final weight $W=W_0+\Delta W,\qquad \Delta W = B A,$5 (Liu et al., 27 Aug 2025).

The efficiency claims are explicit. LoRA-SAM requires two gradient computations per step, doubling time and roughly doubling memory. Bi-LoRA instead adds only the storage for $W=W_0+\Delta W,\qquad \Delta W = B A,$6, approximately the same size as $W=W_0+\Delta W,\qquad \Delta W = B A,$7, leading to $W=W_0+\Delta W,\qquad \Delta W = B A,$8 extra memory and $W=W_0+\Delta W,\qquad \Delta W = B A,$9 extra time. On T5-CoLA, reported single-step times are $B\in\mathbb{R}^{m\times r}$0 for LoRA, $B\in\mathbb{R}^{m\times r}$1 for LoRA-SAM, and $B\in\mathbb{R}^{m\times r}$2 for Bi-LoRA; GPU memory overhead is reported as less than $B\in\mathbb{R}^{m\times r}$3 extra versus LoRA (Liu et al., 27 Aug 2025).

4. Geometric and theoretical interpretation

The principal theoretical claim is that Bi-LoRA avoids the perturbation-space collapse that arises when SAM is applied directly to the LoRA factors. Because the auxiliary perturbation branch is decoupled from the adaptation branch, the adversarial directions do not collapse with the convergence of $B\in\mathbb{R}^{m\times r}$4 and $B\in\mathbb{R}^{m\times r}$5. The paper further states that the auxiliary adapter’s slower convergence ensures continued exploration of new sharp directions throughout training (Liu et al., 27 Aug 2025).

Within this interpretation, the primary adapter can be viewed as approximating the task-specific low-rank displacement, while the auxiliary adapter approximates a low-rank adversarial perturbation constrained in Frobenius norm. This suggests a separation between representation fitting and curvature probing that is absent in standard LoRA. The paper’s own language is sharper: the dual-module design enables Bi-LoRA to capture broader sharpness for achieving flatter minima while remaining memory-efficient.

The empirical geometric evidence reported for this claim is based on line-search plots in the full-parameter space. Those plots show that Bi-LoRA finds significantly flatter minima than either LoRA or LoRA-SAM (Liu et al., 27 Aug 2025). Since the perturbation adapter is discarded after training, the flatter solution is attributed to the training trajectory rather than to a more expensive inference-time model.

A further implication, stated in the paper’s conclusion, is that the method aims at "full-space flatness" while preserving LoRA-style efficiency. That phrasing distinguishes Bi-LoRA from constructions whose perturbation search is intrinsically tied to the column and row spaces of the task adapter itself.

5. Empirical behavior across domains

The 2025 Bi-LoRA paper evaluates the method on natural language understanding, LLMs, and diffusion-model personalization, and reports consistent gains in average accuracy or similarity metrics relative to LoRA and LoRA-SAM (Liu et al., 27 Aug 2025).

On T5-base for GLUE and SuperGLUE, the datasets are MNLI, SST-2, CoLA, QNLI, and MRPC for GLUE, and BoolQ, CB, COPA, RTE, and WiC for SuperGLUE. Metrics are accuracy, with Matthews correlation for CoLA. Average results are reported as 84.34% on GLUE and 76.11% on SuperGLUE for LoRA, 84.49% and 76.20% for LoRA-SAM at twice the cost, and 84.81% and 76.80% for Bi-LoRA at single-pass cost. The gains are especially strong on small-data tasks, with CoLA improving by +1.36% and MRPC by +0.82%.

For LLMs, the paper reports experiments on Llama 2-7B and 3.1-8B over mathematical reasoning, code, dialogue, and instruction-following tasks, including GSM8K, HumanEval, MT-Bench, MMLU, DROP, and BBH. The qualitative summary is that Bi-LoRA consistently beats LoRA and LoRA-SAM, often matching or exceeding full fine-tuning. A concrete example on GSM8K reports 60.32% for Bi-LoRA, versus 58.21% for LoRA and 59.16% for LoRA-SAM.

For diffusion models, under SDXL DreamBooth on 3D Icons, Bi-LoRA improves average CLIP image-text similarity by +0.70% and text-text similarity by +3.85% over vanilla LoRA. Against efficient SAM variants on GSM8K and HumanEval, Bi-LoRA achieves the highest average, 43.76%, compared with 43.44% for Flat-LoRA and 42.98% for LoRA-nBAR. The method is also reported to be composable with other LoRA variants: it can be "plugged into" LoRA-GA, PiSSA, and DoRA, further improving their performance by at least 1.4% on MRPC and CoLA.

These results support the paper’s central claim that sharper-aware optimization can be integrated into PEFT without inheriting the standard computational penalty of SAM. A plausible implication is that the method is particularly attractive in regimes where limited data makes flat-minimum bias valuable but full-parameter SAM is prohibitively expensive.

6. Hyperparameters, deployment, and relation to similarly named methods

The practical recipe in the 2025 paper is intentionally close to standard LoRA. The primary rank $B\in\mathbb{R}^{m\times r}$6 is typically 4–16, the auxiliary rank $B\in\mathbb{R}^{m\times r}$7 is 2–16 with default $B\in\mathbb{R}^{m\times r}$8, and using the same learning rate for both branches works well. Reported learning-rate settings are $B\in\mathbb{R}^{m\times r}$9 in LLMs and $A\in\mathbb{R}^{r\times n}$0 to $A\in\mathbb{R}^{r\times n}$1 in T5. The perturbation radius $A\in\mathbb{R}^{r\times n}$2 is taken in $A\in\mathbb{R}^{r\times n}$3, with default $A\in\mathbb{R}^{r\times n}$4, and performance is described as robust across that range. Batch size, epochs, scheduler, and warmup follow the base LoRA protocol in each domain (Liu et al., 27 Aug 2025).

Integration is correspondingly lightweight. Bi-LoRA replaces a single LoRA adapter with two equal-sized adapters, one trained in ascent, and requires no model-code change beyond doubling the adapter modules and adding one line of norm projection after each step. At inference, the auxiliary adapter is discarded, so runtime cost is exactly the same as standard LoRA (Liu et al., 27 Aug 2025).

Because the label "Bi-LoRA" is overloaded, terminological disambiguation is often necessary:

Paper	Use of the term	Core mechanism
(Liu et al., 27 Aug 2025)	Efficient SAM for PEFT	Primary LoRA descent plus auxiliary LoRA ascent
(Qiang et al., 2024)	Overfitting-resilient LoRA	Bi-level optimization over $A\in\mathbb{R}^{r\times n}$5 and $A\in\mathbb{R}^{r\times n}$6
(Keita et al., 2024)	Synthetic image detection	LoRA in both Q-Former and OPT decoder
(Fujisawa et al., 16 Feb 2026)	Generalized bi-directional PEFT instance	Signed residual branches plus column-wise projection

The 2024 bi-level BiLoRA framework parameterizes each adapter as $A\in\mathbb{R}^{r\times n}$7, trains pseudo-vectors on one subset and pseudo-values on a held-out subset, and reports improvements over LoRA and AdaLoRA on GLUE and NLG benchmarks, at the cost of an inner-loop backward pass and an outer-loop hypergradient pass (Qiang et al., 2024). The 2024 vision-language Bi-LORA framework uses BLIP-2, a frozen ViT, a Q-Former, and OPT-2.7B, recasting binary real/fake detection as a captioning problem and reporting 93.41% average accuracy on unseen generators (Keita et al., 2024). D²-LoRA, finally, combines additive and subtractive low-rank branches with a train-time column-wise projection and preserves algebraic mergeability with near-exact numerical equivalence at inference (Fujisawa et al., 16 Feb 2026).

For that reason, "Bi-directional Low-Rank Adaptation" is best treated not as a single universally standardized method name, but as a family label that has been applied to multiple distinct PEFT designs. In current usage, however, the sharpness-aware dual-adapter construction of (Liu et al., 27 Aug 2025) is the most direct and literal instance of Bi-LoRA as a bi-directional descent/ascent low-rank adaptation scheme.