RegCL: Recursive Continual Learning for SAM

• RegCL is a non-replay continual learning framework that recursively merges task-specific LoRA adapters to prevent catastrophic forgetting.
• It adapts the Segment Anything Model to diverse domains like medical, shadow, and camouflaged-object segmentation without storing historical data.
• Experimental results show RegCL outperforms sequential LoRA fine-tuning while maintaining constant model size and achieving a strong stability–plasticity balance.

Searching arXiv for “RegCL” and closely related terms to ground the article in current papers. RegCL is a non-replay continual-learning framework for adapting the Segment Anything Model (SAM) to multiple specialized domains by merging parameter-efficient adaptation modules—primarily LoRA adapters—rather than sequentially overwriting a single adapter. It is formulated for dynamic, multi-domain settings such as medical, shadow, and camouflaged-object segmentation, with the explicit objectives of reducing catastrophic forgetting, avoiding historical data storage, and keeping model size constant regardless of the number of tasks (Shu et al., 16 Jul 2025).

1. Problem formulation and motivation

RegCL is motivated by a limitation of existing SAM adaptation practice. SAM is strong in zero-shot segmentation on natural images, but it performs poorly on several specialized domains, including medical images, camouflaged object segmentation, and shadow segmentation. Existing remedies typically use adapter-based fine-tuning, but these methods are largely one-step and domain-specific: an adapter trained for one domain often becomes specialized to that domain, and subsequent adaptation to another domain can degrade earlier-domain performance. In the continual setting, this manifests as catastrophic forgetting.

The paper studies this problem in a domain-incremental learning (DIL) setup. Tasks share the same label space, namely segmentation masks, while input distributions differ across domains, and task identity is not provided at test time. Within this setting, “non-replay continual learning” means that, when learning task $t$, the method cannot access data from previous tasks. There is no rehearsal buffer, no exemplar memory, no replay of old images, and no historical labels during future stages. This constraint is presented as practically relevant for privacy-sensitive domains, especially medical imaging, and for storage-efficient deployment.

The paper also argues that standard continual-learning families are not an ideal fit for SAM adapter tuning. Architecture-based methods tend to add parameters and cause model growth, while regularization-based methods constrain updates in ways that can suppress meaningful adapter specialization. RegCL therefore reframes continual adaptation as post-training consolidation: each arriving domain produces a task-specific adapter, and those adapters are subsequently integrated by model merging rather than by continued sequential optimization on a single parameter state (Shu et al., 16 Jul 2025).

2. Core method: continual adaptation by recursive model merging

RegCL stands for Regression Continual Learning. Its central mechanism is a RegMean-inspired merging rule applied to SAM adaptation modules. For each task $t$, a task-specific LoRA adapter $W_t$ is trained from the same initialization $W_0$. The method then merges the current adapter with the previously accumulated merged adapter $\overline{W}_{t-1}$.

The construction builds on the RegMean formulation, in which merging minimizes prediction discrepancy between a merged model and the task-specific models. Using task feature inner-product matrices

$C_t = X_t^\top X_t,$

RegCL defines the recursive merge for $t>1$ as

$$\overline{W}_t = (P_t + C_t)^{-1}\big(P_t \overline{W}_{t-1} + C_t W_t\big),$$

with the historical accumulator updated by

$P_{t+1} = P_t + C_t.$

For the first task, the merged adapter is simply

$\overline{W}_1 = W_1.$

This formulation decomposes the merge into a historical term $t$0 and a new-knowledge term $t$1, with $t$2 providing adaptive weighting. The effect is to preserve prior merged behavior while incorporating current-domain behavior, without revisiting old training data. For nonlinear-layer weights, the method falls back to simple averaging: $t$3

A notable property of this design is that old raw data are not needed at merge time. The continual state is compressed into the previous merged adapter $t$4 and the accumulated statistic $t$5. The paper also claims order independence for the merging process because it depends on sums of $t$6, whose addition is commutative. This suggests that RegCL is not a replay method disguised as merging; it is explicitly a recursive closed-form consolidation procedure over parameter-efficient domain adapters (Shu et al., 16 Jul 2025).

3. Architecture, trainable components, and optimization workflow

The implementation uses SAM with a ViT-B/16 backbone. LoRA modules are inserted into the image encoder, while the image encoder backbone weights, prompt encoder, and mask decoder remain frozen. The prompt encoder uses point-type prompts. Thus, the only trainable parameters are the LoRA parameters, and only those parameters are merged across tasks. The paper further notes that it “consolidate[s] the low-rank $t$7 of each layer into a single entity” to reduce computational cost and facilitate extraction of $t$8.

Training proceeds in two nested stages. First, for each incoming domain $t$9, a fresh task-specific LoRA adapter $W_t$0 is trained from the common initialization $W_t$1. The fine-tuning loss is

$W_t$2

Second, after that domain-specific training, the method computes $W_t$3, merges $W_t$4 into the current continual adapter using the recursive rule above, and updates the historical accumulator.

Operationally, the full pipeline is:

1. Freeze SAM.
2. Insert LoRA modules into the image encoder.
3. Initialize LoRA parameters $W_t$5 with Kaiming initialization.
4. Set $W_t$6.
5. For each task $W_t$7, train $W_t$8 on $W_t$9 only.
6. Compute $W_0$0.
7. Merge $W_0$1 with $W_0$2.
8. Update $W_0$3.
9. After the final task, deploy the single merged adapter $W_0$4.

The reported training configuration uses 20 epochs per dataset, batch size 8, learning rate 0.005, and a cosine annealing schedule. The framework is therefore not full-model continual tuning; it is SAM adaptation through LoRA-specific recursive merging (Shu et al., 16 Jul 2025).

4. Continual-learning properties and computational profile

RegCL is explicitly designed to maintain parameter efficiency under sequential domain arrival. At inference time, it carries a single merged adapter rather than one adapter per task, so model size does not grow with the number of tasks. This distinguishes it from both per-domain adapter banks and architecture-expansion continual-learning methods. Because test-time task identity is not required, deployment is also aligned with the DIL assumption used in evaluation.

The computational argument of RegCL centers on the statistic $W_0$5. For linear layers, $W_0$6 can be computed in one forward pass over training data after the individual models are trained. The paper presents this as cheaper than Fisher-based merging, which requires backward passes. The storage overhead is

$W_0$7

where $W_0$8 is the number of linear layers and $W_0$9 is each layer’s input dimension; the paper states that this is comparable to the number of LoRA parameters.

These properties give RegCL a particular deployment profile. It requires current task data, current task LoRA parameters, the current $\overline{W}_{t-1}$0, the previous merged adapter, and the accumulator $\overline{W}_{t-1}$1. It does not require old raw data, old labels, replay buffers, or exemplar storage. The authors frame this as both storage-efficient and privacy-friendly, since only low-dimensional matrices $\overline{W}_{t-1}$2 rather than raw data need to be retained or shared. A plausible implication is that RegCL occupies an intermediate point between classical regularization-based continual learning and generic model merging: it inherits the no-replay regime of the former and the closed-form consolidation logic of the latter (Shu et al., 16 Jul 2025).

5. Experimental evaluation

The empirical study uses five datasets across three broad domains: Kvasir-SEG, CAMO, ISTD, ISIC, and COD10K-v2, arranged in the continual sequence

$\overline{W}_{t-1}$3

Evaluation uses segmentation metrics—mIoU, mF1, and mMAE—and continual-learning metrics from GEM: $\overline{W}_{t-1}$4

$\overline{W}_{t-1}$5

$\overline{W}_{t-1}$6

On the 5-task DIL benchmark, RegCL reports ACC values of 0.751 mIoU, 0.840 mF1, and 0.048 mMAE; BWT values of -0.028 mIoU, -0.021 mF1, and 0.006 mMAE; and FWT values of 0.651 mIoU, 0.763 mF1, and 0.084 mMAE. Compared with naïve sequential LoRA fine-tuning (LoRA-Seq), which obtains ACC mIoU 0.696, ACC mF1 0.802, ACC mMAE 0.063, and BWT mIoU -0.107, RegCL improves average performance and substantially reduces forgetting. The paper summarizes the gains over LoRA-Seq as +0.055 mIoU ACC, +0.038 mF1 ACC, and 0.015 lower mMAE ACC.

Against non-replay continual-learning baselines such as EWC, SPPA, LAG, and O-LoRA, RegCL is reported as the strongest overall method. Relative to EWC, which yields ACC mIoU 0.716, ACC mF1 0.816, and ACC mMAE 0.058, RegCL improves by roughly +0.035 mIoU, +0.024 mF1, and 0.010 lower mMAE. The paper interprets this as better stability–plasticity balance.

The study also compares RegCL with simple mean merging. In the ablation table, Mean obtains ACC values of 0.739 mIoU, 0.831 mF1, and 0.055 mMAE, whereas RegCL reaches 0.751, 0.840, and 0.048. Relative to the Upper Bound, reported as 0.820 mIoU, 0.890 mF1, and 0.030 mMAE, RegCL attains 91.6% of Upper Bound on mIoU ACC and 94.4% on mF1 ACC. The paper also notes a qualification: RegCL is not uniformly best on every individual dataset, and on CAMO and COD the Mean baseline has slightly higher mIoU/mF1 than RegCL.

Although RegCL is designed for non-replay continual learning, the paper evaluates a replay-augmented variant using 300 replay samples per dataset. RegCL+Replay reaches ACC values of 0.809 mIoU, 0.882 mF1, and 0.036 mMAE, with BWT of -0.018 mIoU and -0.013 mF1, outperforming RegCL alone and slightly surpassing replay baselines such as ER and DER. This suggests that the recursive merge can function as a strong core mechanism even when replay becomes available (Shu et al., 16 Jul 2025).

6. Terminological scope and nearby but distinct usages

In the cited literature, RegCL in the strict sense refers to the SAM continual-adaptation method above. Several nearby names can create confusion. RC$\overline{W}_{t-1}$7L, from “Region-level Contrastive and Consistency Learning,” is a semi-supervised semantic-segmentation framework built on MaskFormer and teacher–student region-level regularization, but it is not named RegCL (Zhang et al., 2022). CLReg, from “From Logits to Latents: Contrastive Representation Shaping for LLM Unlearning,” is a contrastive representation regularizer for LLM unlearning and is sometimes a transposed source of “RegCL” references, but the paper itself uses CLReg rather than RegCL (Tang et al., 29 Jan 2026). RegCCRF denotes a regular-constrained CRF for regular-language-constrained sequence prediction, again not RegCL (Papay et al., 2021).

There is also a broader compliance-oriented usage in which “RegCL” might be interpreted as a regulatory or compliance control layer for language systems. In that broader sense, “RAGulating Compliance: A Multi-Agent Knowledge Graph for Regulatory QA” presents a triplet-centric, provenance-linked, multi-agent QA architecture for regulatory compliance, but it explicitly does not define a formal control language or use the term RegCL (Agarwal et al., 13 Aug 2025). “Citation-Closure Retrieval and Per-Rule Attribution for Real-World Regulatory Compliance Question Answering” formalizes regulatory compliance QA as citation-closure retrieval plus per-rule attribution, and is highly relevant to a regulation-grounded LLM agenda, but again it is not a paper about RegCL as a named method (Ju et al., 28 May 2026). By contrast, “GREx: Generalized Referring Expression Segmentation, Comprehension, and Generation” is unrelated to RegCL and introduces GREx and ReLA instead (Ding et al., 8 Jan 2026).

7. Limitations and future directions

The paper’s scope is deliberately narrow. RegCL is demonstrated in domain-incremental segmentation rather than class-incremental or task-incremental settings. Its implementation mainly targets LoRA-based SAM adaptation, and its merging rule is grounded in a linear-layer approximation; nonlinear layers are handled by simple averaging. It also requires training a fresh adapter for each incoming task before merging, so it is not eliminating per-task training cost.

Empirically, the evaluation does not include extensive ablations on LoRA rank or placement, alternative optimization forms, explicit task-order sensitivity experiments, or scaling studies over longer task sequences. Although the paper claims order independence theoretically through the additive form of $\overline{W}_{t-1}$8, it does not report a dedicated empirical task-order table. The per-dataset results likewise show that the weighted merge is not universally better than simple mean merging on every domain.

The conclusion positions RegCL as a method for adapting foundation models under dynamic environments. The stated future directions include broader integration with replay when available, extension beyond SAM and beyond segmentation, application to other parameter-efficient tuning modules, and investigation of broader continual-learning scenarios. This suggests that RegCL is best understood not as a universal continual-learning solution, but as a specific and technically constrained proposal for multi-domain SAM adaptation via recursive model merging (Shu et al., 16 Jul 2025).