BID-LoRA: Bi-Directional Low-Rank Adaptation

• The paper introduces BID-LoRA, a framework that employs three separate low-rank adapter pathways—retain, new, and forget—to balance continual learning unlearning tasks.
• It uses an escape unlearning objective that geometrically drives forget-class embeddings away from retain-class centroids, effectively mitigating knowledge leakage.
• The approach achieves competitive performance on benchmarks like CIFAR-100 and CASIA-Face100 with approximately 5% trainable parameters through isolated, staged adapter updates.

Searching arXiv for the cited BID-LoRA and related LoRA variants to ground the article in the relevant papers. arXiv search query: BID-LoRA (Rachapudi et al., 14 Apr 2026) BoRA (Wang et al., 2024) Bi-LoRA (Liu et al., 27 Aug 2025) Bi-Directional Low-Rank Adaptation (BID-LoRA) is a parameter-efficient framework for Continual Learning Unlearning (CLU), a setting in which a model must repeatedly forget designated data or classes, learn new data or classes, and retain previously useful knowledge over multiple adaptation cycles. In its defining formulation, BID-LoRA uses three dedicated low-rank adapter pathways—retain, new, and unlearn—applied to attention layers, together with an escape unlearning objective that pushes forget-class embeddings to positions maximally distant from retained knowledge, while updating only about $5\%$ of parameters (Rachapudi et al., 14 Apr 2026). The framework is motivated by the claim that naively combining existing continual learning (CL) and machine unlearning (MU) methods causes knowledge leakage, understood as a gradual degradation of foundational retained knowledge across repeated learn–forget cycles (Rachapudi et al., 14 Apr 2026).

1. CLU as the problem setting

BID-LoRA is defined for a unified regime called Continual Learning Unlearning. In this regime, a pretrained model $\mathcal{M}$ must support three simultaneous requirements at each step of adaptation: precise deletion of unwanted knowledge, efficient integration of new knowledge while preserving prior information, and minimizing knowledge leakage across cycles (Rachapudi et al., 14 Apr 2026). The motivating scenarios explicitly include identity management systems, privacy compliance under GDPR/CCPA, dynamic face recognition, and models that must adapt to changing users, policies, or classes (Rachapudi et al., 14 Apr 2026).

The paper formalizes a pretrained model as a mapping $f_{\mathcal{M}} : \mathcal{X}_D \to \mathcal{Y}_D$, with $D_f$ denoting data to forget, $D_r^{\text{full}}$ the full retain set, $D_{\text{new}}$ new data to learn, and $D_r \subset D_r^{\text{full}}$ a replay buffer. The assumptions are

$$D = D_r^{\text{full}} \cup D_f,\qquad D_r^{\text{full}} \cap D_f = \emptyset,\qquad D_{\text{new}} \cap D = \emptyset.$$

Before adaptation, the intended behavior is that the model maps $\mathcal{X}_{D_f}$ to $\mathcal{Y}_{D_f}$ and $\mathcal{M}$0 to $\mathcal{M}$1, but not $\mathcal{M}$2 to $\mathcal{M}$3. After adaptation, the desired behavior reverses this status for the forget and new sets: the model should no longer map $\mathcal{M}$4 to $\mathcal{M}$5, should preserve mapping on $\mathcal{M}$6, and should now map $\mathcal{M}$7 to $\mathcal{M}$8 (Rachapudi et al., 14 Apr 2026).

For the continual setting over tasks $\mathcal{M}$9, the update rule is written as

$f_{\mathcal{M}} : \mathcal{X}_D \to \mathcal{Y}_D$0

Successful CLU is then defined through three operational criteria:

• forgetting: $f_{\mathcal{M}} : \mathcal{X}_D \to \mathcal{Y}_D$1,
• retention: $f_{\mathcal{M}} : \mathcal{X}_D \to \mathcal{Y}_D$2,
• learning: $f_{\mathcal{M}} : \mathcal{X}_D \to \mathcal{Y}_D$3, where $f_{\mathcal{M}} : \mathcal{X}_D \to \mathcal{Y}_D$4 is the number of classes (Rachapudi et al., 14 Apr 2026).

This formulation places BID-LoRA in a distinct category from one-shot unlearning and standard class-incremental learning. The emphasis is sustained adaptation without cumulative drift. A plausible implication is that the method is intended less as an isolated PEFT variant and more as a systems-level mechanism for repeated state correction under retention constraints.

2. Tri-path low-rank architecture

The architectural point of departure is standard LoRA. In the paper’s notation, standard LoRA modifies a frozen weight matrix $f_{\mathcal{M}} : \mathcal{X}_D \to \mathcal{Y}_D$5 by

$f_{\mathcal{M}} : \mathcal{X}_D \to \mathcal{Y}_D$6

with $f_{\mathcal{M}} : \mathcal{X}_D \to \mathcal{Y}_D$7, $f_{\mathcal{M}} : \mathcal{X}_D \to \mathcal{Y}_D$8, $f_{\mathcal{M}} : \mathcal{X}_D \to \mathcal{Y}_D$9 frozen, $D_f$0, $D_f$1, and $D_f$2 a scaling factor (Rachapudi et al., 14 Apr 2026). BID-LoRA argues that, in CLU, a single adapter is overloaded because it must simultaneously support retention, learning, and forgetting; the consequence is gradient interference and eventually knowledge leakage (Rachapudi et al., 14 Apr 2026).

BID-LoRA replaces that single update space with three dedicated low-rank pathways: $D_f$3 These are applied jointly as

$D_f$4

while the backbone $D_f$5 remains frozen (Rachapudi et al., 14 Apr 2026).

The separation is the central design principle. The retain adapter preserves prior knowledge, the new adapter learns new classes, and the forget adapter removes unwanted knowledge. Because they are trained with separate objectives and gradient masking, they do not directly interfere (Rachapudi et al., 14 Apr 2026). The paper applies these adapters to attention layers and classifier heads, preserving the PEFT premise that only small low-rank modules are updated.

This decomposition is called “bi-directional” in the paper’s title, but the operative mechanism is actually tri-path. The term therefore refers less to two opposing optimization directions than to coordinated directional control over retention and change within the CLU pipeline. This suggests that the distinctive contribution lies in pathway specialization rather than in a single algebraic modification of the LoRA parameterization.

3. Escape unlearning and the geometry of forgetting

The forget mechanism in BID-LoRA is escape unlearning. Its goal is to delete knowledge without damaging retained classes by moving forget-class embeddings toward a target that is maximally distant from retained knowledge (Rachapudi et al., 14 Apr 2026).

The construction proceeds geometrically. For each class $D_f$6, the class centroid is

$D_f$7

where $D_f$8 is the learned embedding of sample $D_f$9 (Rachapudi et al., 14 Apr 2026). Using retain-class centroids $D_r^{\text{full}}$0, the paper defines the escape direction as

$D_r^{\text{full}}$1

Here, $D_r^{\text{full}}$2 measures alignment with retain centroid $D_r^{\text{full}}$3; the inner $D_r^{\text{full}}$4 selects the retain centroid most aligned with $D_r^{\text{full}}$5; and the outer minimization chooses the direction least aligned with all retain classes (Rachapudi et al., 14 Apr 2026). The interpretation given is that $D_r^{\text{full}}$6 is the direction maximally distant from retained knowledge.

Because placing the escape point on the unit sphere can be unstable, the paper scales it as

$D_r^{\text{full}}$7

with $D_r^{\text{full}}$8 a scaling factor (Rachapudi et al., 14 Apr 2026). Forget samples are then driven toward this target through

$D_r^{\text{full}}$9

The stated effect is that the model maps forget samples toward the same escape target, producing a many-to-one collapse that destroys class-discriminative structure (Rachapudi et al., 14 Apr 2026).

The retain and new pathways use distinct objectives. Retention combines classification and embedding anchoring: $D_{\text{new}}$0 where $D_{\text{new}}$1 are logits on retain samples, $D_{\text{new}}$2 retain labels, $D_{\text{new}}$3 student embeddings, and $D_{\text{new}}$4 frozen teacher embeddings from the initial model (Rachapudi et al., 14 Apr 2026). New knowledge uses the standard classification loss

$D_{\text{new}}$5

Taken together, these losses assign distinct geometric roles to the three pathways: anchoring for retention, discrimination for acquisition, and collapse toward a distant point for deletion. A plausible implication is that BID-LoRA treats unlearning not as simple error induction but as controlled relocation in embedding space.

4. Optimization protocol, isolation, and efficiency

Training is organized as isolated updates. The algorithm performs three stages per cycle:

1. retention update: freeze forget and new adapters, and update the retain adapter and retain classifier head using $D_{\text{new}}$6;
2. forget update: freeze retain and new adapters, and update the forget adapter and forget head using $D_{\text{new}}$7;
3. new knowledge update: freeze retain and forget adapters, and update the new adapter and new head using $D_{\text{new}}$8 (Rachapudi et al., 14 Apr 2026).

At inference, the adapters are merged into the frozen backbone as

$D_{\text{new}}$9

The subtractive forget term appears in the merge expression exactly as written in the algorithm description (Rachapudi et al., 14 Apr 2026).

The framework is explicitly parameter-efficient. BID-LoRA trains small low-rank adapters on attention layers and classifier heads while keeping the backbone frozen. The reported tunable-parameter ratio is about $D_r \subset D_r^{\text{full}}$0 on CIFAR-100 and $D_r \subset D_r^{\text{full}}$1 on CASIA-Face100 (Rachapudi et al., 14 Apr 2026). The paper presents this as practical for large pretrained ViTs and transformers, repeated adaptation cycles, settings where full retraining is too costly, and privacy-sensitive deployment where “surgical” updates are preferred (Rachapudi et al., 14 Apr 2026).

The adapter configuration used in the main protocol assigns rank $D_r \subset D_r^{\text{full}}$2 to the retain adapter, rank $D_r \subset D_r^{\text{full}}$3 to the new adapter, and rank $D_r \subset D_r^{\text{full}}$4 to the forget adapter. The replay buffer $D_r \subset D_r^{\text{full}}$5 is $D_r \subset D_r^{\text{full}}$6 of the full retain set (Rachapudi et al., 14 Apr 2026). These design choices matter because the framework depends on both pathway separation and limited replay. The ablations report that performance saturates around rank $D_r \subset D_r^{\text{full}}$7, that more buffer helps retention, and that removing any one of the three pathways harms its corresponding objective (Rachapudi et al., 14 Apr 2026).

5. Experimental protocol and empirical behavior

The empirical evaluation covers CIFAR-100 and CASIA-Face100, the latter described as a curated subset of $D_r \subset D_r^{\text{full}}$8 identities from CASIA-WebFace. The backbones are Data-efficient image transformer (DeiT) for classification and Face Transformer for face recognition (Rachapudi et al., 14 Apr 2026). The evaluation uses a six-task sliding window protocol:

• start from classes $D_r \subset D_r^{\text{full}}$9–$$D = D_r^{\text{full}} \cup D_f,\qquad D_r^{\text{full}} \cap D_f = \emptyset,\qquad D_{\text{new}} \cap D = \emptyset.$$0,
• task 1: $$D = D_r^{\text{full}} \cup D_f,\qquad D_r^{\text{full}} \cap D_f = \emptyset,\qquad D_{\text{new}} \cap D = \emptyset.$$1–$$D = D_r^{\text{full}} \cup D_f,\qquad D_r^{\text{full}} \cap D_f = \emptyset,\qquad D_{\text{new}} \cap D = \emptyset.$$2,
• task 2: $$D = D_r^{\text{full}} \cup D_f,\qquad D_r^{\text{full}} \cap D_f = \emptyset,\qquad D_{\text{new}} \cap D = \emptyset.$$3–$$D = D_r^{\text{full}} \cup D_f,\qquad D_r^{\text{full}} \cap D_f = \emptyset,\qquad D_{\text{new}} \cap D = \emptyset.$$4,
• task 3: $$D = D_r^{\text{full}} \cup D_f,\qquad D_r^{\text{full}} \cap D_f = \emptyset,\qquad D_{\text{new}} \cap D = \emptyset.$$5–$$D = D_r^{\text{full}} \cup D_f,\qquad D_r^{\text{full}} \cap D_f = \emptyset,\qquad D_{\text{new}} \cap D = \emptyset.$$6,
• task 4: $$D = D_r^{\text{full}} \cup D_f,\qquad D_r^{\text{full}} \cap D_f = \emptyset,\qquad D_{\text{new}} \cap D = \emptyset.$$7–$$D = D_r^{\text{full}} \cup D_f,\qquad D_r^{\text{full}} \cap D_f = \emptyset,\qquad D_{\text{new}} \cap D = \emptyset.$$8,
• task 5: $$D = D_r^{\text{full}} \cup D_f,\qquad D_r^{\text{full}} \cap D_f = \emptyset,\qquad D_{\text{new}} \cap D = \emptyset.$$9–$\mathcal{X}_{D_f}$0,
• task 6: $\mathcal{X}_{D_f}$1–$\mathcal{X}_{D_f}$2.

Each task retains $\mathcal{X}_{D_f}$3 classes, forgets $\mathcal{X}_{D_f}$4 classes, and learns $\mathcal{X}_{D_f}$5 new classes (Rachapudi et al., 14 Apr 2026). The reported metrics are forget accuracy $\mathcal{X}_{D_f}$6, retain accuracy $\mathcal{X}_{D_f}$7, new accuracy $\mathcal{X}_{D_f}$8, overall accuracy $\mathcal{X}_{D_f}$9, MIA success rate, KL divergence to oracle, and tunable parameter ratio (Rachapudi et al., 14 Apr 2026).

On CIFAR-100, BID-LoRA uses only $\mathcal{Y}_{D_f}$0 tunable parameters and achieves very low forget accuracy, roughly $\mathcal{Y}_{D_f}$1–$\mathcal{Y}_{D_f}$2 across tasks; retain accuracy around $\mathcal{Y}_{D_f}$3–$\mathcal{Y}_{D_f}$4; new accuracy up to $\mathcal{Y}_{D_f}$5; overall accuracy close to oracle; MIA near $\mathcal{Y}_{D_f}$6–$\mathcal{Y}_{D_f}$7; and low KL divergence (Rachapudi et al., 14 Apr 2026). The examples given are task 1 overall accuracy $\mathcal{Y}_{D_f}$8 and task 6 overall accuracy $\mathcal{Y}_{D_f}$9, corresponding to only about a $\mathcal{M}$00 drop across the six-task sequence (Rachapudi et al., 14 Apr 2026).

On CASIA-Face100, BID-LoRA again uses about $\mathcal{M}$01 tunable parameters and maintains forget accuracy near $\mathcal{M}$02–$\mathcal{M}$03, high retain and new accuracy, overall accuracy around $\mathcal{M}$04–$\mathcal{M}$05, MIA close to $\mathcal{M}$06, and low KL divergence (Rachapudi et al., 14 Apr 2026). The paper reports task 1 overall accuracy $\mathcal{M}$07 and task 6 overall accuracy $\mathcal{M}$08, an overall drop of about $\mathcal{M}$09 (Rachapudi et al., 14 Apr 2026).

The baselines are LSF, CLPU-DER++, UniCLUN, UG-CLU, and UnCLe. The paper characterizes these as representing replay, distillation, hypernetworks, and gradient-based unified strategies (Rachapudi et al., 14 Apr 2026). The comparative interpretation is that BID-LoRA gives the best tradeoff between forgetting, retaining, learning new classes, and parameter efficiency, while several baselines either forget well at the cost of overall accuracy, show unstable retention, or exhibit inconsistent MIA or lower stability (Rachapudi et al., 14 Apr 2026).

The ablation results sharpen this interpretation. Standard LoRA is reported to be inferior to BID-LoRA in CLU, supporting the claim that pathway separation matters. A larger escape scaling factor improves forgetting, with $\mathcal{M}$10 giving the best reported forgetting performance. The paper also presents t-SNE and 3D sphere plots showing forget embeddings migrating toward the escape point (Rachapudi et al., 14 Apr 2026). This suggests that the claimed deletion mechanism is not only metric-based but also geometrically observable in the learned representation space.

6. Relation to adjacent “bi-directional” LoRA variants, misconceptions, and limitations

The expression “Bi-Directional Low-Rank Adaptation” is potentially ambiguous because several LoRA variants use related nomenclature while solving different problems. The following comparison helps disambiguate the literature.

Method	Defining mechanism	Primary setting
BID-LoRA	retain, new, and unlearn adapters plus escape unlearning	continual learning and unlearning
Bi-LoRA	primary LoRA and auxiliary LoRA for SAM-style perturbations	sharpness-aware fine-tuning
BoRA	row-wise and column-wise magnitude adaptation around a low-rank direction update	bi-dimensional weight decomposition

BID-LoRA in the strict sense refers to the CLU framework with three dedicated pathways and escape unlearning (Rachapudi et al., 14 Apr 2026). By contrast, “Bi-LoRA: Efficient Sharpness-Aware Minimization for Fine-Tuning Large-Scale Models” introduces a dual-module design in which a primary LoRA module performs task adaptation via gradient descent and an auxiliary LoRA module models SAM-style perturbations via gradient ascent; only the primary branch is retained for inference (Liu et al., 27 Aug 2025). That method addresses the mismatch that arises when SAM perturbations are forced into the same low-rank subspace used for adaptation, and it is framed around generalization and efficient sharpness-aware training rather than CLU (Liu et al., 27 Aug 2025).

BoRA, in turn, is “Bi-dimensional Weight-Decomposed Low-Rank Adaptation,” a symmetric extension of DoRA that learns both row-wise and column-wise magnitude information around a low-rank directional update. Its claim is symmetry across horizontal and vertical dimensions of the weight matrix, with trainable magnitude vectors $\mathcal{M}$11 and $\mathcal{M}$12 inserted into a two-stage normalization and scaling pipeline (Wang et al., 2024). The paper explicitly states that BoRA can be interpreted as a bi-directional or bi-dimensional LoRA variant, but it does not use “BID-LoRA” as the method name (Wang et al., 2024).

A common misconception is therefore to treat BID-LoRA, Bi-LoRA, and BoRA as interchangeable. They are not. BID-LoRA is a unified continual learning plus machine unlearning framework; Bi-LoRA is a SAM-inspired fine-tuning method; and BoRA is a symmetric magnitude-direction decomposition for PEFT (Rachapudi et al., 14 Apr 2026).

The limitations of BID-LoRA are also explicit. The framework relies on a replay buffer $\mathcal{M}$13, assumed to be at least $\mathcal{M}$14 of the retain set; it is validated mainly on vision classification and face recognition; it assumes a structured tri-partite setting with known retain, forget, and new partitions; and future work is needed to remove the buffer entirely and extend the method to other biometric modalities (Rachapudi et al., 14 Apr 2026). These constraints indicate that the framework is presently strongest in controlled, transformer-based identity or classification pipelines where repeated enrollment, revocation, and retention are all first-class requirements.