---
title: On-Device Online Continual Merging
url: https://www.emergentmind.com/topics/on-device-online-continual-merging
type: topic
---

# On-Device Online Continual Merging

On-device online continual merging denotes a deployment regime in which a resource-constrained device must integrate task-adapted model components over time without retaining the full historical set of components or retraining from scratch. In the most direct formulation for large language models, task-specific LoRA adapters arrive sequentially, while the device can store only a small fixed number of adapters; each new adapter must therefore be stored in a free slot or merged into an existing slot while preserving performance on previously supported tasks [2510.13537]. Closely related lines of work include offline storage-constrained clustering and merging of many adapters before deployment [2601.17441], sequential continual model merging of full checkpoints via barrier-aware trajectories in parameter space [2605.19409], and wake/sleep continual consolidation that merges stream experience into a persistent model rather than merging separately trained models [2303.10725].

## 1. Scope and conceptual boundaries

The literature distinguishes several regimes that are adjacent but not identical. The narrowest and most literal use of the topic refers to online continual merging of adapters on a device under a hard storage budget. A broader use includes continual model merging of full task checkpoints as they arrive sequentially. A still broader adjacent interpretation treats continual learning itself as a staged form of knowledge merging across time, even when no separately trained models are exchanged.

| Work | Core regime | Relation to the topic |
|---|---|---|
| "K-Merge: Online Continual Merging of Adapters for On-device Large Language Models" [2510.13537] | Sequential arrival of LoRAs under budget \(K\) | Direct formulation |
| "Data-driven Clustering and Merging of Adapters for On-device Large Language Models" [2601.17441] | Offline clustering and merging of many adapters into \(K\) clusters | Storage-constrained precursor |
| "Unlocking the Potential of Continual Model Merging: An ODE Perspective" [2605.19409] | Sequential merging of full task models | Direct continual merging, not evaluated as on-device |
| "SIESTA: Efficient Online Continual Learning with Sleep" [2303.10725] | Online continual learning with deferred consolidation | Related temporal knowledge merging |

The distinction is important because the algorithmic constraints differ. K-Merge assumes that the device does not get all LoRAs upfront, cannot revisit the original single-task training data, and must make every merging decision online using only the current stored adapters and the newly arrived adapter [2510.13537]. By contrast, D²C is explicitly better understood as an offline clustering-and-merging method rather than a fully online continual merging method, even though it is motivated by the same storage-constrained on-device setting [2601.17441]. ODE-M is explicitly sequential and continual, but the paper states that it is not an on-device paper in the systems sense [2605.19409]. SIESTA is likewise on-device-oriented, but it is not merging multiple separately trained models in the federated or adapter-merging sense [2303.10725].

## 2. Formal problem setting for adapter streams on device

In the adapter-stream formulation, each arriving adapter \(L^{(t)}\) corresponds to a new user-requested task \(\tau_t\). At step \(t\), the device already stores a collection \(\mathcal{L}^{(t-1)}\) of at most \(K\) adapters and must update it after receiving the new LoRA. The update has only two admissible forms: either the device merges the new LoRA into one existing stored adapter and replaces that slot,
\[
\mathcal{L}^{(t)} = \{\mathcal{L}^{(t-1)} \setminus L_c\} \cup \{\mathrm{merge}(L_c, L^{(t)})\},
\]
or it allocates a new slot,
\[
\mathcal{L}^{(t)} = \{L^{(t)}\} \cup \mathcal{L}^{(t-1)},
\]
with the second option available only if \(|\mathcal{L}^{(t-1)}|<K\) [2510.13537].

This formulation introduces a specific stability–plasticity dilemma. If early adapters are stored too eagerly, the device may waste slots on redundant or highly similar tasks and later have no space for genuinely different tasks. If merging is too aggressive, task-specific performance may be damaged. The device also does not know future tasks, so it cannot compute an optimal clustering in advance. The paper further assumes that merging must happen on-device rather than on the server, since keeping copies of all deployed adapters on both sides would be inefficient and impractical [2510.13537].

The benchmark used in K-Merge makes the storage pressure explicit. It consists of \(\alpha=5\) problem types and \(\beta=8\) languages, giving \(\gamma=\alpha\beta=40\) distinct tasks or adapters. Tasks arrive in random order, one per timestep, and the system is evaluated on all tasks seen so far. Because the underlying tasks use heterogeneous metrics, the paper defines a normalized aggregate score \(S^{(t)}\) that divides each task’s achieved score by the score of its corresponding single-task LoRA. This normalizes each task to its own single-task upper bound, so \(S^{(t)}\) measures how well the stored collection preserves single-task performance [2510.13537].

## 3. Adapter-level online continual merging methods

K-Merge addresses the online decision problem through similarity-based target selection and history-aware incremental merging. For each transformer layer \(n\) and projection type \(p \in \{key, query, value, output\}\), it computes the LoRA update matrix
\[
\Delta W^{n,p} = B^{n,p}A^{n,p},
\]
flattens it with \(f(\cdot)\), and uses cosine similarity on the flattened updates. The similarity between the incoming adapter \(L^{(t)}\) and a stored adapter \(L_i\) is averaged across layers and projection types, and the merge target is chosen by
\[
c = \arg\max_{i=1,\ldots, |\mathcal{L}^{(t-1)}|} \text{sim}(L^{(t)}, L_i).
\]
This makes the online clustering rule explicitly structural: the closest stored adapter in LoRA-update space becomes the absorber if merging is required [2510.13537].

The second component is a lightweight history map \(\mathcal{H}^{(t)}\) that records which task indices have been merged into each current slot. If a new adapter is assigned to cluster \(c\), the merge is not a simple two-model average; instead, the paper uses a running average,
\[
\mathrm{merge}(L_c,L^{(t)}) = \frac{L^{(t)} + |\mathcal{H}^{(t-1)}[c]| \cdot L_c}{|\mathcal{H}^{(t-1)}[c]| + 1},
\]
and then updates the history as
\[
\mathcal{H}^{(t)}[c] = \mathcal{H}^{(t-1)}[c] \cup \{t\}.
\]
The stated consequences are that the rule is incremental, requires no retention of all prior adapters in memory, and is order-invariant in the sense that all adapters in a cluster contribute equally to the final merged representation. The same history map is also used as a routing table at inference time: for task \(\tau_i\), the system finds the cluster key \(\hat{c}\) such that \(i \in \mathcal{H}^{(t)}[\hat{c}]\) and loads adapter \(L_{\hat{c}}\) [2510.13537].

Two variants are proposed. K-Merge stores the incoming LoRA separately while free storage remains and otherwise merges it into the nearest stored adapter. K-Merge++ adds a similarity threshold \(s\): if the new adapter is already sufficiently similar to the nearest stored adapter, it is merged immediately even when capacity remains; otherwise it is stored separately until the budget is exhausted. The threshold is estimated empirically from pairwise similarities on held-out LoRA pairs from unseen tasks and languages, using the median as the threshold. The resulting deployment-time decision is data-free, although it depends on an offline calibration stage on held-out adapters [2510.13537].

The empirical behavior of these rules is central to the topic. K-Merge and especially K-Merge++ outperform all baselines across storage budgets; with 8 stored adapters, they recover roughly \(80\)–\(90\%\) of single-task performance. On Llama-3.2-1B, K-Merge reaches about \(0.83\) at \(K=8\), with K-Merge++ around \(0.82\)–\(0.83\) depending on ordering, while on Qwen-2.5-1.5B, K-Merge++ reaches up to about \(0.93\) at \(K=8\). The paper further reports that K-Merge++ is the most robust under random, problem-type-grouped, and deliberately worst-case orderings, and attributes this to the threshold mechanism preserving slots early instead of filling them with redundant adapters. Runtime for integrating a new LoRA is roughly \(0.04\)s to \(0.18\)s for 2 to 8 stored adapters, and memory overhead is described as negligible relative to text generation itself. The storage pressure motivating the problem is also quantified: each LoRA is 27 MB for Llama-3.2-1B and 34 MB for Qwen-2.5-1.5B, so storing all 40 would require about 1 GB [2510.13537].

## 4. Offline storage-constrained clustering and merging

D²C addresses the same deployment bottleneck from an offline perspective. The paper considers \(N\) single-task LoRA adapters, one per task, but assumes that the device can store only \(K < N\) adapters due to memory and storage limits. The proposed solution is Data-driven Clustering and merging of adapters: first cluster the \(N\) task-specific LoRAs into \(K\) groups, then merge the adapters within each cluster to obtain \(K\) multi-task adapters for deployment. The key distinguishing feature is that cluster quality is evaluated directly by downstream task loss on a few task examples rather than by an explicit adapter embedding or similarity metric [2601.17441].

Algorithmically, D²C begins with a random partition map \(\theta\) assigning the \(N\) adapters to \(K\) clusters. For each of \(T\) iterations, the method samples two clusters \(c\) and \(c'\), samples one adapter \((A_t, B_t)\) from cluster \(c\), merges the adapters currently in cluster \(c\), and evaluates the merged adapter on task \(t\)’s example set \(\mathcal{D}_t\) of size \(n\). It then temporarily moves \((A_t, B_t)\) to cluster \(c'\), merges the adapters in \(c'\), and evaluates again on \(\mathcal{D}_t\). The loss is cross-entropy,
\[
\ell = \mathrm{CE}\big(\text{merge}(\{(A_{\theta^c_i}, B_{\theta^c_i})\}), \mathcal{D}_t\big),
\]
with an analogous \(\ell'\) for the tentative destination cluster. If \(\ell \le \ell'\), the move is undone. The resulting update rule is therefore an accept-if-lower-loss greedy local search over cluster assignments. After \(T\) iterations, all adapters in each final cluster are merged into a single multi-task adapter, using an existing merging method such as TIES; the main experimental setup uses TIES merging with unary weights and density factor \(0.5\), while the paper also reports compatibility with linear merging [2601.17441].

The experimental setup is explicitly aligned with on-device LLM deployment. The evaluated backbones are Llama 3.2 3B, Qwen 2.5 1.5B, and StableLM 2 1.6B. The task suite contains 40 text-generation tasks spanning Grammar Error Correction, Smart Reply, Summarization, Tone Adjustment, and Question Answering across English, Spanish, French, German, Italian, Chinese (simplified), Korean, and Japanese. D²C uses 10 examples per task, usually stores 5 clusters, and runs for 200 iterations. In the motivating example, 40 task-specific adapters are compressed to 5 stored adapters, or \(12.5\%\) of the storage required to keep all adapters separately [2601.17441].

The paper’s main result is that D²C performs best among clustering methods under this storage constraint. For Llama 3.2 3B, the reported aggregate scores are 14.7 for zero-shot, 32.9 for separate single-task LoRAs, 21.5 for random clustering, 20.4 for K-Means, 22.3 for K-Means + SVD, and 26.0 for D²C. Merging all LoRAs into one is worse than D²C, with reported score 18.7% for Llama 3.2 3B. Additional analyses show that performance generally improves with more clusters, that the discovered clusters are mostly task-based rather than language-based, that performance is not very sensitive to the number of examples per task, and that the clustering procedure for Llama 3.2 3B takes about 1.3 hours on one GPU. The paper explicitly states that this is an offline preprocessing step before deployment rather than a true online continual merging protocol [2601.17441].

## 5. Continual merging beyond adapter storage

ODE-M generalizes the continual-merging problem from LoRA slots to full task-specific model checkpoints. It formalizes a stream of task-specific models \(\{\psi_k\}_{k=1}^K\), all sharing the same architecture and parameter space, and defines the sequential update
\[
\Psi_{k+1}=\mathcal{M}(\Psi_k,\psi_{k+1},\psi_0), \qquad \Psi_1=\psi_1,
\]
where \(\psi_0\) is the shared pretrained backbone. The paper’s central claim is that prior continual merging rules are too endpoint-only: they combine the current merged checkpoint and the new model algebraically, without controlling the transition path in parameter space. ODE-M instead treats the merge as a continuous trajectory between the current model \(\theta_0\) and the incoming model \(\theta_1\), motivated by mode connectivity and the possibility of low-loss connecting paths even when straight-line interpolation crosses a barrier [2605.19409].

The method defines a base transport field \(u_t(\theta)=\alpha(t)(\theta_1-\theta)\) with \(\alpha(t)=\frac{1}{1-t}\), computes a calibration-loss gradient \(g_t=\nabla_{\theta_t}\mathcal{L}\), decomposes the motion into a gradient-aligned component \(u_{\parallel}\) and an orthogonal component \(u_{\perp}\), and then uses a rectified velocity
\[
v_t(\theta)=u_{\perp}+\gamma(\theta,t)\,u_{\parallel},
\]
with \(\gamma(\theta,t)\in[0,1]\) damping only the loss-increasing motion. If \(\langle g_t,u_t\rangle\le 0\), then \(\gamma(\theta,t)=1\); if \(\langle g_t,u_t\rangle>0\), a clipped expression based on \(\Delta\mathcal{L}=\mathcal{L}(\theta_1)-\mathcal{L}(\theta_0)\) and \(\langle g_t,u_t\rangle\) is used. The trajectory is integrated numerically with Euler integration,
\[
\theta \leftarrow \theta + h\,v,
\]
using step size \(h=0.05\) in the main setup. The paper also defines utility-weighted metrics,
\[
\mathrm{ACC}_w(\Psi_k)=\frac{1}{\sum_{i=1}^k w_i}\sum_{i=1}^k w_i\, a_{k,i},
\qquad
\mathrm{BWT}_w(\Psi_k)=\frac{1}{\sum_{i=1}^{k-1} w_i}\sum_{i=1}^{k-1} w_i\,(a_{k,i}-a_{i,i}),
\]
and the operating time schedule
\[
t_k = \frac{w_k}{W_{k-1}+w_k} = \frac{w_k}{\sum_{i=1}^k w_i},
\]
with equal-utility special case \(t_k=1/k\) [2605.19409].

Empirically, ODE-M is reported as best on macro-average ACC across CLIP backbones ViT-B/32, ViT-B/16, and ViT-L/14 and stream lengths of 8, 14, and 20 tasks. Examples include ViT-B/32 with 8 tasks, where ODE-M achieves \(79.6\) versus OPCM at \(75.5\), and ViT-L/14 with 20 tasks, where it reaches \(81.1\) versus \(76.0\). For heterogeneous task utility, it also achieves the best weighted accuracy in all reported settings. At the same time, the paper states that backward transfer is not always best, because the method prioritizes favorable accuracy–stability trade-offs rather than always maximizing BWT. Its relevance to on-device deployment is indirect: it uses a small calibration set of 1024 examples, Euler integration, and per-task overheads of 123.9 s for ViT-B/32, 200.8 s for ViT-B/16, and 246.8 s for ViT-L/14, but does not provide explicit memory-complexity analysis, device benchmarks, or energy measurements [2605.19409].

SIESTA occupies a different but related position. It is a supervised continual learning method that alternates between a wake or online phase and a sleep or offline phase. During wake, the network \(\mathcal{F}(\mathcal{G}(\mathcal{H}(\cdot)))\) receives labeled samples one by one, stores compressed latent features \(\mathbf{Z}_t=\mathcal{H}(\mathbf{X}_t)\) using product quantization, and updates only the classifier weight for the observed class using a running mean,
\[
\mathbf{f}_k \leftarrow \frac{c_k}{c_k + 1}\mathbf{f}_k + \frac{1}{c_k + 1}\mathbf{z}_t,
\qquad
c_k \leftarrow c_k + 1.
\]
During sleep, it updates \(\mathcal{F}\) and \(\mathcal{G}\) with supervised backpropagation on reconstructed latent features, under a fixed compute budget \(m=q \times n\). The wake phase is rehearsal-free and backprop-free, while replay is reserved for sleep. The paper explicitly presents this as on-device learning for phones, robots, AR headsets, and appliances, and reports continual learning on ImageNet-1K in under 2 hours on a single NVIDIA A5000 GPU, with 1.9 hours for 900 classes in one setting, compared with 8.1 hours for REMIND under the same conditions. In the augmentation-free ImageNet-1K setting, it reports final top-5 accuracy 83.59 for SIESTA versus 83.31 for the offline learner, 74.31 for REMIND, 70.15 for DER, and 63.92 for ER, with about 2.02 GB auxiliary memory in the main setup. The paper explicitly notes, however, that this is not model merging across multiple models, but a hybrid of cheap online prototype updates and deferred consolidation [2303.10725].

## 6. Empirical regularities, misconceptions, and limitations

A recurrent empirical regularity across the adapter-focused papers is that task structure dominates language structure. D²C reports that its discovered clusters are mostly task-based and that clustering by language homogeneity does not help much, while K-Merge reports that LoRAs tend to cluster more strongly by problem type than by language and that some language groups, especially European languages, are more similar to each other than to Asian languages [2601.17441][2510.13537]. This suggests that, in the evaluated multilingual task suites, low-rank update geometry is driven more by problem type than by language alone.

Another regularity is the storage–accuracy trade-off. D²C explicitly reports that storing fewer adapters lowers memory and storage cost but that too aggressive merging reduces performance, and its results show a better storage–accuracy trade-off than random or K-Means clustering under the same budget [2601.17441]. K-Merge makes the same trade-off dynamic rather than static: K-Merge++ improves robustness by preserving capacity for later diverse tasks, and the appendix threshold ablation on Llama-3.2-1B with \(K=5\) reports \(0.68\) at \(s=0.010\), \(0.74\) at \(s=0.015\), \(0.81\) at \(s=0.020\), and \(0.80\) at \(s=0.025\), making the median-based choice \(s=0.020\) the best among the reported values. When \(K=\alpha=5\), K-Merge++ also achieves the best clustering consistency at 88.3%, compared with 82.5% for Linear, 83.3% for K-Merge, and 86.7% for TIES, even though TIES has worse performance overall [2510.13537].

The literature also requires careful terminological discipline. D²C is iterative and task-aware, but the paper explicitly states that it is not an online continual merging algorithm in the strict sense because it does not describe continuous arrival of new tasks after deployment, persistent incremental updates to clusters in production, revisiting old assignments under stream processing, or an explicit lifelong learning schedule [2601.17441]. ODE-M is directly about continual model merging under sequential task arrival, but the paper equally explicitly states that it is not designed around strict on-device constraints and does not demonstrate edge-device feasibility [2605.19409]. SIESTA, despite fitting the broader idea of incrementally integrating new knowledge into a persistent model, is not merging multiple separately trained models at all [2303.10725].

The main limitations recorded in the cited works follow from these boundaries. K-Merge notes that it only studies LLMs, only considers LoRA adapters, focuses on on-device-sized models, and warns that merging adapters may weaken safeguarding mechanisms, so safety evaluation should accompany deployment [2510.13537]. ODE-M reports no explicit memory-complexity analysis, quantized or mobile implementation, latency or energy measurements, or strict on-device evaluation [2605.19409]. D²C lacks a streaming update mechanism for newly arriving tasks in deployment [2601.17441]. SIESTA assumes that the frozen lower network \(\mathcal{H}\) produces sufficiently universal features after base initialization, depends on memory buffer size and sleep schedule, and is demonstrated primarily on CNNs and vision tasks [2303.10725].

Taken together, these works define a layered research landscape rather than a single settled paradigm. The most literal instance of on-device online continual merging is the budgeted streaming-LoRA setting exemplified by K-Merge. D²C provides the offline storage-constrained counterpart, ODE-M provides a trajectory-controlled continual merging formalism for sequential task checkpoints, and SIESTA provides an on-device continual consolidation paradigm in which stream experience is merged into persistent weights over time.

Source: https://www.emergentmind.com/topics/on-device-online-continual-merging