On-Device Online Continual Merging

• On-device online continual merging is the process of sequentially integrating task-specific adapters on devices under strict storage limits without retaining full historical data.
• The paper introduces similarity-based selection and incremental running-average merging (e.g., K-Merge++) to effectively preserve individual task performance despite memory constraints.
• Experiments demonstrate that strategic merging techniques can recover 80–93% of single-task performance, highlighting a critical storage–accuracy trade-off in practical deployments.

On-device online continual merging denotes a deployment regime in which a resource-constrained device must integrate task-adapted model components over time without retaining the full historical set of components or retraining from scratch. In the most direct formulation for LLMs, task-specific LoRA adapters arrive sequentially, while the device can store only a small fixed number of adapters; each new adapter must therefore be stored in a free slot or merged into an existing slot while preserving performance on previously supported tasks (Shenaj et al., 15 Oct 2025). Closely related lines of work include offline storage-constrained clustering and merging of many adapters before deployment (Bohdal et al., 24 Jan 2026), sequential continual model merging of full checkpoints via barrier-aware trajectories in parameter space (Lin et al., 19 May 2026), and wake/sleep continual consolidation that merges stream experience into a persistent model rather than merging separately trained models (Harun et al., 2023).

1. Scope and conceptual boundaries

The literature distinguishes several regimes that are adjacent but not identical. The narrowest and most literal use of the topic refers to online continual merging of adapters on a device under a hard storage budget. A broader use includes continual model merging of full task checkpoints as they arrive sequentially. A still broader adjacent interpretation treats continual learning itself as a staged form of knowledge merging across time, even when no separately trained models are exchanged.

Work	Core regime	Relation to the topic
"K-Merge: Online Continual Merging of Adapters for On-device LLMs" (Shenaj et al., 15 Oct 2025)	Sequential arrival of LoRAs under budget $K$	Direct formulation
"Data-driven Clustering and Merging of Adapters for On-device LLMs" (Bohdal et al., 24 Jan 2026)	Offline clustering and merging of many adapters into $K$ clusters	Storage-constrained precursor
"Unlocking the Potential of Continual Model Merging: An ODE Perspective" (Lin et al., 19 May 2026)	Sequential merging of full task models	Direct continual merging, not evaluated as on-device
"SIESTA: Efficient Online Continual Learning with Sleep" (Harun et al., 2023)	Online continual learning with deferred consolidation	Related temporal knowledge merging

The distinction is important because the algorithmic constraints differ. K-Merge assumes that the device does not get all LoRAs upfront, cannot revisit the original single-task training data, and must make every merging decision online using only the current stored adapters and the newly arrived adapter (Shenaj et al., 15 Oct 2025). By contrast, D²C is explicitly better understood as an offline clustering-and-merging method rather than a fully online continual merging method, even though it is motivated by the same storage-constrained on-device setting (Bohdal et al., 24 Jan 2026). ODE-M is explicitly sequential and continual, but the paper states that it is not an on-device paper in the systems sense (Lin et al., 19 May 2026). SIESTA is likewise on-device-oriented, but it is not merging multiple separately trained models in the federated or adapter-merging sense (Harun et al., 2023).

2. Formal problem setting for adapter streams on device

In the adapter-stream formulation, each arriving adapter $L^{(t)}$ corresponds to a new user-requested task $\tau_t$. At step $t$, the device already stores a collection $\mathcal{L}^{(t-1)}$ of at most $K$ adapters and must update it after receiving the new LoRA. The update has only two admissible forms: either the device merges the new LoRA into one existing stored adapter and replaces that slot,

$$\mathcal{L}^{(t)} = \{\mathcal{L}^{(t-1)} \setminus L_c\} \cup \{\mathrm{merge}(L_c, L^{(t)})\},$$

or it allocates a new slot,

$$\mathcal{L}^{(t)} = \{L^{(t)}\} \cup \mathcal{L}^{(t-1)},$$

with the second option available only if $|\mathcal{L}^{(t-1)}|<K$ (Shenaj et al., 15 Oct 2025).

This formulation introduces a specific stability–plasticity dilemma. If early adapters are stored too eagerly, the device may waste slots on redundant or highly similar tasks and later have no space for genuinely different tasks. If merging is too aggressive, task-specific performance may be damaged. The device also does not know future tasks, so it cannot compute an optimal clustering in advance. The paper further assumes that merging must happen on-device rather than on the server, since keeping copies of all deployed adapters on both sides would be inefficient and impractical (Shenaj et al., 15 Oct 2025).

The benchmark used in K-Merge makes the storage pressure explicit. It consists of $K$0 problem types and $K$1 languages, giving $K$2 distinct tasks or adapters. Tasks arrive in random order, one per timestep, and the system is evaluated on all tasks seen so far. Because the underlying tasks use heterogeneous metrics, the paper defines a normalized aggregate score $K$3 that divides each task’s achieved score by the score of its corresponding single-task LoRA. This normalizes each task to its own single-task upper bound, so $K$4 measures how well the stored collection preserves single-task performance (Shenaj et al., 15 Oct 2025).

3. Adapter-level online continual merging methods

K-Merge addresses the online decision problem through similarity-based target selection and history-aware incremental merging. For each transformer layer $K$5 and projection type $K$6, it computes the LoRA update matrix

$K$7

flattens it with $K$8, and uses cosine similarity on the flattened updates. The similarity between the incoming adapter $K$9 and a stored adapter $L^{(t)}$0 is averaged across layers and projection types, and the merge target is chosen by

$L^{(t)}$1

This makes the online clustering rule explicitly structural: the closest stored adapter in LoRA-update space becomes the absorber if merging is required (Shenaj et al., 15 Oct 2025).

The second component is a lightweight history map $L^{(t)}$2 that records which task indices have been merged into each current slot. If a new adapter is assigned to cluster $L^{(t)}$3, the merge is not a simple two-model average; instead, the paper uses a running average,

$L^{(t)}$4

and then updates the history as

$L^{(t)}$5

The stated consequences are that the rule is incremental, requires no retention of all prior adapters in memory, and is order-invariant in the sense that all adapters in a cluster contribute equally to the final merged representation. The same history map is also used as a routing table at inference time: for task $L^{(t)}$6, the system finds the cluster key $L^{(t)}$7 such that $L^{(t)}$8 and loads adapter $L^{(t)}$9 (Shenaj et al., 15 Oct 2025).

Two variants are proposed. K-Merge stores the incoming LoRA separately while free storage remains and otherwise merges it into the nearest stored adapter. K-Merge++ adds a similarity threshold $\tau_t$0: if the new adapter is already sufficiently similar to the nearest stored adapter, it is merged immediately even when capacity remains; otherwise it is stored separately until the budget is exhausted. The threshold is estimated empirically from pairwise similarities on held-out LoRA pairs from unseen tasks and languages, using the median as the threshold. The resulting deployment-time decision is data-free, although it depends on an offline calibration stage on held-out adapters (Shenaj et al., 15 Oct 2025).

The empirical behavior of these rules is central to the topic. K-Merge and especially K-Merge++ outperform all baselines across storage budgets; with 8 stored adapters, they recover roughly $\tau_t$1–$\tau_t$2 of single-task performance. On Llama-3.2-1B, K-Merge reaches about $\tau_t$3 at $\tau_t$4, with K-Merge++ around $\tau_t$5–$\tau_t$6 depending on ordering, while on Qwen-2.5-1.5B, K-Merge++ reaches up to about $\tau_t$7 at $\tau_t$8. The paper further reports that K-Merge++ is the most robust under random, problem-type-grouped, and deliberately worst-case orderings, and attributes this to the threshold mechanism preserving slots early instead of filling them with redundant adapters. Runtime for integrating a new LoRA is roughly $\tau_t$9s to $t$0s for 2 to 8 stored adapters, and memory overhead is described as negligible relative to text generation itself. The storage pressure motivating the problem is also quantified: each LoRA is 27 MB for Llama-3.2-1B and 34 MB for Qwen-2.5-1.5B, so storing all 40 would require about 1 GB (Shenaj et al., 15 Oct 2025).

4. Offline storage-constrained clustering and merging

D²C addresses the same deployment bottleneck from an offline perspective. The paper considers $t$1 single-task LoRA adapters, one per task, but assumes that the device can store only $t$2 adapters due to memory and storage limits. The proposed solution is Data-driven Clustering and merging of adapters: first cluster the $t$3 task-specific LoRAs into $t$4 groups, then merge the adapters within each cluster to obtain $t$5 multi-task adapters for deployment. The key distinguishing feature is that cluster quality is evaluated directly by downstream task loss on a few task examples rather than by an explicit adapter embedding or similarity metric (Bohdal et al., 24 Jan 2026).

Algorithmically, D²C begins with a random partition map $t$6 assigning the $t$7 adapters to $t$8 clusters. For each of $t$9 iterations, the method samples two clusters $\mathcal{L}^{(t-1)}$0 and $\mathcal{L}^{(t-1)}$1, samples one adapter $\mathcal{L}^{(t-1)}$2 from cluster $\mathcal{L}^{(t-1)}$3, merges the adapters currently in cluster $\mathcal{L}^{(t-1)}$4, and evaluates the merged adapter on task $\mathcal{L}^{(t-1)}$5’s example set $\mathcal{L}^{(t-1)}$6 of size $\mathcal{L}^{(t-1)}$7. It then temporarily moves $\mathcal{L}^{(t-1)}$8 to cluster $\mathcal{L}^{(t-1)}$9, merges the adapters in $K$0, and evaluates again on $K$1. The loss is cross-entropy,

$K$2

with an analogous $K$3 for the tentative destination cluster. If $K$4, the move is undone. The resulting update rule is therefore an accept-if-lower-loss greedy local search over cluster assignments. After $K$5 iterations, all adapters in each final cluster are merged into a single multi-task adapter, using an existing merging method such as TIES; the main experimental setup uses TIES merging with unary weights and density factor $K$6, while the paper also reports compatibility with linear merging (Bohdal et al., 24 Jan 2026).

The experimental setup is explicitly aligned with on-device LLM deployment. The evaluated backbones are Llama 3.2 3B, Qwen 2.5 1.5B, and StableLM 2 1.6B. The task suite contains 40 text-generation tasks spanning Grammar Error Correction, Smart Reply, Summarization, Tone Adjustment, and Question Answering across English, Spanish, French, German, Italian, Chinese (simplified), Korean, and Japanese. D²C uses 10 examples per task, usually stores 5 clusters, and runs for 200 iterations. In the motivating example, 40 task-specific adapters are compressed to 5 stored adapters, or $K$7 of the storage required to keep all adapters separately (Bohdal et al., 24 Jan 2026).

The paper’s main result is that D²C performs best among clustering methods under this storage constraint. For Llama 3.2 3B, the reported aggregate scores are 14.7 for zero-shot, 32.9 for separate single-task LoRAs, 21.5 for random clustering, 20.4 for K-Means, 22.3 for K-Means + SVD, and 26.0 for D²C. Merging all LoRAs into one is worse than D²C, with reported score 18.7% for Llama 3.2 3B. Additional analyses show that performance generally improves with more clusters, that the discovered clusters are mostly task-based rather than language-based, that performance is not very sensitive to the number of examples per task, and that the clustering procedure for Llama 3.2 3B takes about 1.3 hours on one GPU. The paper explicitly states that this is an offline preprocessing step before deployment rather than a true online continual merging protocol (Bohdal et al., 24 Jan 2026).

5. Continual merging beyond adapter storage

ODE-M generalizes the continual-merging problem from LoRA slots to full task-specific model checkpoints. It formalizes a stream of task-specific models $K$8, all sharing the same architecture and parameter space, and defines the sequential update

$K$9

where $$\mathcal{L}^{(t)} = \{\mathcal{L}^{(t-1)} \setminus L_c\} \cup \{\mathrm{merge}(L_c, L^{(t)})\},$$0 is the shared pretrained backbone. The paper’s central claim is that prior continual merging rules are too endpoint-only: they combine the current merged checkpoint and the new model algebraically, without controlling the transition path in parameter space. ODE-M instead treats the merge as a continuous trajectory between the current model $$\mathcal{L}^{(t)} = \{\mathcal{L}^{(t-1)} \setminus L_c\} \cup \{\mathrm{merge}(L_c, L^{(t)})\},$$1 and the incoming model $$\mathcal{L}^{(t)} = \{\mathcal{L}^{(t-1)} \setminus L_c\} \cup \{\mathrm{merge}(L_c, L^{(t)})\},$$2, motivated by mode connectivity and the possibility of low-loss connecting paths even when straight-line interpolation crosses a barrier (Lin et al., 19 May 2026).

The method defines a base transport field $$\mathcal{L}^{(t)} = \{\mathcal{L}^{(t-1)} \setminus L_c\} \cup \{\mathrm{merge}(L_c, L^{(t)})\},$$3 with $$\mathcal{L}^{(t)} = \{\mathcal{L}^{(t-1)} \setminus L_c\} \cup \{\mathrm{merge}(L_c, L^{(t)})\},$$4, computes a calibration-loss gradient $$\mathcal{L}^{(t)} = \{\mathcal{L}^{(t-1)} \setminus L_c\} \cup \{\mathrm{merge}(L_c, L^{(t)})\},$$5, decomposes the motion into a gradient-aligned component $$\mathcal{L}^{(t)} = \{\mathcal{L}^{(t-1)} \setminus L_c\} \cup \{\mathrm{merge}(L_c, L^{(t)})\},$$6 and an orthogonal component $$\mathcal{L}^{(t)} = \{\mathcal{L}^{(t-1)} \setminus L_c\} \cup \{\mathrm{merge}(L_c, L^{(t)})\},$$7, and then uses a rectified velocity

$$\mathcal{L}^{(t)} = \{\mathcal{L}^{(t-1)} \setminus L_c\} \cup \{\mathrm{merge}(L_c, L^{(t)})\},$$8

with $$\mathcal{L}^{(t)} = \{\mathcal{L}^{(t-1)} \setminus L_c\} \cup \{\mathrm{merge}(L_c, L^{(t)})\},$$9 damping only the loss-increasing motion. If $$\mathcal{L}^{(t)} = \{L^{(t)}\} \cup \mathcal{L}^{(t-1)},$$0, then $$\mathcal{L}^{(t)} = \{L^{(t)}\} \cup \mathcal{L}^{(t-1)},$$1; if $$\mathcal{L}^{(t)} = \{L^{(t)}\} \cup \mathcal{L}^{(t-1)},$$2, a clipped expression based on $$\mathcal{L}^{(t)} = \{L^{(t)}\} \cup \mathcal{L}^{(t-1)},$$3 and $$\mathcal{L}^{(t)} = \{L^{(t)}\} \cup \mathcal{L}^{(t-1)},$$4 is used. The trajectory is integrated numerically with Euler integration,

$$\mathcal{L}^{(t)} = \{L^{(t)}\} \cup \mathcal{L}^{(t-1)},$$5

using step size $$\mathcal{L}^{(t)} = \{L^{(t)}\} \cup \mathcal{L}^{(t-1)},$$6 in the main setup. The paper also defines utility-weighted metrics,

$$\mathcal{L}^{(t)} = \{L^{(t)}\} \cup \mathcal{L}^{(t-1)},$$7

and the operating time schedule

$$\mathcal{L}^{(t)} = \{L^{(t)}\} \cup \mathcal{L}^{(t-1)},$$8

with equal-utility special case $$\mathcal{L}^{(t)} = \{L^{(t)}\} \cup \mathcal{L}^{(t-1)},$$9 (Lin et al., 19 May 2026).

Empirically, ODE-M is reported as best on macro-average ACC across CLIP backbones ViT-B/32, ViT-B/16, and ViT-L/14 and stream lengths of 8, 14, and 20 tasks. Examples include ViT-B/32 with 8 tasks, where ODE-M achieves $|\mathcal{L}^{(t-1)}|<K$0 versus OPCM at $|\mathcal{L}^{(t-1)}|<K$1, and ViT-L/14 with 20 tasks, where it reaches $|\mathcal{L}^{(t-1)}|<K$2 versus $|\mathcal{L}^{(t-1)}|<K$3. For heterogeneous task utility, it also achieves the best weighted accuracy in all reported settings. At the same time, the paper states that backward transfer is not always best, because the method prioritizes favorable accuracy–stability trade-offs rather than always maximizing BWT. Its relevance to on-device deployment is indirect: it uses a small calibration set of 1024 examples, Euler integration, and per-task overheads of 123.9 s for ViT-B/32, 200.8 s for ViT-B/16, and 246.8 s for ViT-L/14, but does not provide explicit memory-complexity analysis, device benchmarks, or energy measurements (Lin et al., 19 May 2026).

SIESTA occupies a different but related position. It is a supervised continual learning method that alternates between a wake or online phase and a sleep or offline phase. During wake, the network $|\mathcal{L}^{(t-1)}|<K$4 receives labeled samples one by one, stores compressed latent features $|\mathcal{L}^{(t-1)}|<K$5 using product quantization, and updates only the classifier weight for the observed class using a running mean,

$|\mathcal{L}^{(t-1)}|<K$6

During sleep, it updates $|\mathcal{L}^{(t-1)}|<K$7 and $|\mathcal{L}^{(t-1)}|<K$8 with supervised backpropagation on reconstructed latent features, under a fixed compute budget $|\mathcal{L}^{(t-1)}|<K$9. The wake phase is rehearsal-free and backprop-free, while replay is reserved for sleep. The paper explicitly presents this as on-device learning for phones, robots, AR headsets, and appliances, and reports continual learning on ImageNet-1K in under 2 hours on a single NVIDIA A5000 GPU, with 1.9 hours for 900 classes in one setting, compared with 8.1 hours for REMIND under the same conditions. In the augmentation-free ImageNet-1K setting, it reports final top-5 accuracy 83.59 for SIESTA versus 83.31 for the offline learner, 74.31 for REMIND, 70.15 for DER, and 63.92 for ER, with about 2.02 GB auxiliary memory in the main setup. The paper explicitly notes, however, that this is not model merging across multiple models, but a hybrid of cheap online prototype updates and deferred consolidation (Harun et al., 2023).

6. Empirical regularities, misconceptions, and limitations

A recurrent empirical regularity across the adapter-focused papers is that task structure dominates language structure. D²C reports that its discovered clusters are mostly task-based and that clustering by language homogeneity does not help much, while K-Merge reports that LoRAs tend to cluster more strongly by problem type than by language and that some language groups, especially European languages, are more similar to each other than to Asian languages (Bohdal et al., 24 Jan 2026, Shenaj et al., 15 Oct 2025). This suggests that, in the evaluated multilingual task suites, low-rank update geometry is driven more by problem type than by language alone.

Another regularity is the storage–accuracy trade-off. D²C explicitly reports that storing fewer adapters lowers memory and storage cost but that too aggressive merging reduces performance, and its results show a better storage–accuracy trade-off than random or K-Means clustering under the same budget (Bohdal et al., 24 Jan 2026). K-Merge makes the same trade-off dynamic rather than static: K-Merge++ improves robustness by preserving capacity for later diverse tasks, and the appendix threshold ablation on Llama-3.2-1B with $K$00 reports $K$01 at $K$02, $K$03 at $K$04, $K$05 at $K$06, and $K$07 at $K$08, making the median-based choice $K$09 the best among the reported values. When $K$10, K-Merge++ also achieves the best clustering consistency at 88.3%, compared with 82.5% for Linear, 83.3% for K-Merge, and 86.7% for TIES, even though TIES has worse performance overall (Shenaj et al., 15 Oct 2025).

The literature also requires careful terminological discipline. D²C is iterative and task-aware, but the paper explicitly states that it is not an online continual merging algorithm in the strict sense because it does not describe continuous arrival of new tasks after deployment, persistent incremental updates to clusters in production, revisiting old assignments under stream processing, or an explicit lifelong learning schedule (Bohdal et al., 24 Jan 2026). ODE-M is directly about continual model merging under sequential task arrival, but the paper equally explicitly states that it is not designed around strict on-device constraints and does not demonstrate edge-device feasibility (Lin et al., 19 May 2026). SIESTA, despite fitting the broader idea of incrementally integrating new knowledge into a persistent model, is not merging multiple separately trained models at all (Harun et al., 2023).

The main limitations recorded in the cited works follow from these boundaries. K-Merge notes that it only studies LLMs, only considers LoRA adapters, focuses on on-device-sized models, and warns that merging adapters may weaken safeguarding mechanisms, so safety evaluation should accompany deployment (Shenaj et al., 15 Oct 2025). ODE-M reports no explicit memory-complexity analysis, quantized or mobile implementation, latency or energy measurements, or strict on-device evaluation (Lin et al., 19 May 2026). D²C lacks a streaming update mechanism for newly arriving tasks in deployment (Bohdal et al., 24 Jan 2026). SIESTA assumes that the frozen lower network $K$11 produces sufficiently universal features after base initialization, depends on memory buffer size and sleep schedule, and is demonstrated primarily on CNNs and vision tasks (Harun et al., 2023).

Taken together, these works define a layered research landscape rather than a single settled paradigm. The most literal instance of on-device online continual merging is the budgeted streaming-LoRA setting exemplified by K-Merge. D²C provides the offline storage-constrained counterpart, ODE-M provides a trajectory-controlled continual merging formalism for sequential task checkpoints, and SIESTA provides an on-device continual consolidation paradigm in which stream experience is merged into persistent weights over time.