Adaptive Knowledge Merging

• Adaptive Knowledge Merging is a process that dynamically consolidates multiple task-specific models using context-aware scaling to accommodate varying inputs, layers, and domains.
• It employs methods such as task arithmetic, Bayesian updating, and layer-wise adaptations to balance stability and plasticity without re-accessing original training data.
• Practical implementations demonstrate improved accuracy and reduced interference, with some methods reporting accuracy gains up to 80% and significant performance boosts across diverse tasks.

Adaptive knowledge merging denotes a family of methods that consolidate multiple specialized models, task vectors, or knowledge sources into a single system while adapting the merger to input, layer, task, domain, or training state rather than relying on a fixed global combination rule. In contemporary model merging, this usually means combining task-specific parameter updates around a shared pretrained model without re-accessing original training data; in continual learning, it means balancing stability and plasticity under sequential updates; in symbolic settings, it means restoring consistency under integrity constraints while retaining as much source information as possible (Lu et al., 2024, Yang et al., 2023, Li et al., 3 Apr 2025, Xu et al., 2013).

1. Formal problem setting and recurring objectives

A common neural formulation starts from a pretrained parameter vector $\theta_0$ and task-specific experts $\{\theta_t\}_{t=1}^T$, and seeks a merged model $\theta^*$ satisfying the generic relation

$$\theta_{\mathrm{opt}} \approx \theta^* = F(\theta_0,\theta_1,\ldots,\theta_T),$$

with task vectors defined as differences from the base model, such as $T_k=\theta_k-\theta_{\mathrm{pre}}$ or $\Delta_i=\theta_i-\theta_0$ (Lu et al., 2024, Yang et al., 2023, Li et al., 3 Jun 2025). Classical task arithmetic instantiates this as

$$\theta^*=\theta_0+\sum_{t=1}^T \gamma_t(\theta_t-\theta_0),$$

but a central empirical finding across the literature is that fixed coefficients are often too rigid when tasks conflict, when layers play different roles, or when test distributions are heterogeneous (Lu et al., 2024, Yang et al., 2023).

Several distinct objective families recur. One family minimizes the gap between a merged model and the task-specific experts on their respective tasks, sometimes under an explicit shared-knowledge constraint, as in DOGE’s constrained optimization view (Wei et al., 2 Jan 2025). A second family casts merging as continual Bayesian updating, where a new-task solution is merged with a stability-preserving solution using curvature-aware coefficients derived from Fisher information (Li et al., 3 Apr 2025). A third family preserves feature representations rather than raw parameters, exemplified by LOT Merging’s layer-wise minimization of feature drift (Sun et al., 29 May 2025). In symbolic knowledge merging, the objective is instead to reconcile multiple knowledge bases or ontologies under an integrity constraint $\mu$, or to restore consistency by forgetting variables or relaxing qualitative constraints (Xu et al., 2013, Bouraoui et al., 2022).

A plausible implication is that “adaptive” does not refer to a single mechanism. It denotes a shift from static parameter averaging toward procedures that condition the merge on evidence unavailable to static rules: unlabeled target samples, input embeddings, subspace structure, rehearsal losses, normalization statistics, or logical conflict patterns.

2. Sample-conditional and uncertainty-aware composition

A major branch of adaptive knowledge merging performs composition at inference time or at the level of output distributions. Twin-Merging first extracts a shared expert $\theta_s$, isolates exclusive components $v_t=\theta_t-\theta_s$, compresses them with SVD, and then uses an input-aware router to produce simplex-constrained weights

$\{\theta_t\}_{t=1}^T$0

yielding the adaptive merge

$\{\theta_t\}_{t=1}^T$1

The method is explicitly designed to avoid directly and statically merging exclusive task-specific knowledge that would otherwise interfere across heterogeneous inputs (Lu et al., 2024).

A related but distribution-level formulation appears in Fusion-$\{\theta_t\}_{t=1}^T$2. There, each source LLM produces aligned token-level distributions $\{\theta_t\}_{t=1}^T$3, an adaptive selection network computes source probabilities $\{\theta_t\}_{t=1}^T$4, threshold gating chooses a subset $\{\theta_t\}_{t=1}^T$5, and a fused distribution is formed as

$\{\theta_t\}_{t=1}^T$6

The target model is trained with a language-modeling loss, a fusion loss against $\{\theta_t\}_{t=1}^T$7, and a coefficient-of-variation feedback term

$\{\theta_t\}_{t=1}^T$8

where $\{\theta_t\}_{t=1}^T$9 and $\theta^*$0 in the reported setting. This framework reduces knowledge interference by up to $\theta^*$1 compared to existing approaches on BBH in the reported experiments (Kong et al., 28 May 2025).

Uncertainty-aware knowledge integration provides a third sample-conditional mechanism. MUKI estimates teacher uncertainty with Monte-Carlo Dropout,

$\theta^*$2

then either selects the most certain teacher or forms a soft mixture using confidence scores

$\theta^*$3

with $\theta^*$4. The resulting “virtual golden supervision” is further reweighted instance-wise by the margin between the top two confidence scores. This allows adaptive merging across heterogeneous PLMs and disjoint label spaces without human annotations (Li et al., 2022).

These methods share a common premise: the optimal combination is input-dependent. Static globally optimal coefficients may perform acceptably on average, but they are systematically mismatched to distribution shifts, ambiguous examples, or mixed-domain prompts.

3. Layer-wise, modular, and subspace-aware parameter merging

A second branch adapts the merger inside parameter space, often at layer or subspace granularity.

Method	Adaptive signal	Representative composition
AdaMerging	Unlabeled-sample entropy; task-wise or layer-wise coefficients	$\theta^*$5
DOGE	Shared-subspace projection and adaptive learning rates	$\theta^*$6
FroM	Per-layer Frobenius norm $\theta^*$7	$\theta^*$8
AdaLTM	Learned layer-wise ASR/SER coefficients	$\theta^*$9
LOT Merging	Feature covariances $$\theta_{\mathrm{opt}} \approx \theta^* = F(\theta_0,\theta_1,\ldots,\theta_T),$$0	$$\theta_{\mathrm{opt}} \approx \theta^* = F(\theta_0,\theta_1,\ldots,\theta_T),$$1
AXIS	Singular-value saliency and top-$$\theta_{\mathrm{opt}} \approx \theta^* = F(\theta_0,\theta_1,\ldots,\theta_T),$$2 rank-one components	$$\theta_{\mathrm{opt}} \approx \theta^* = F(\theta_0,\theta_1,\ldots,\theta_T),$$3

AdaMerging learns coefficients from unlabeled target-distribution samples by minimizing predictive entropy,

$$\theta_{\mathrm{opt}} \approx \theta^* = F(\theta_0,\theta_1,\ldots,\theta_T),$$4

with either task-wise $$\theta_{\mathrm{opt}} \approx \theta^* = F(\theta_0,\theta_1,\ldots,\theta_T),$$5 or layer-wise $$\theta_{\mathrm{opt}} \approx \theta^* = F(\theta_0,\theta_1,\ldots,\theta_T),$$6. The paper reports that entropy correlates strongly and positively with prediction loss, with average Spearman correlation $$\theta_{\mathrm{opt}} \approx \theta^* = F(\theta_0,\theta_1,\ldots,\theta_T),$$7, and that layer-wise AdaMerging reaches $$\theta_{\mathrm{opt}} \approx \theta^* = F(\theta_0,\theta_1,\ldots,\theta_T),$$8 average accuracy on ViT-B/32, compared with $$\theta_{\mathrm{opt}} \approx \theta^* = F(\theta_0,\theta_1,\ldots,\theta_T),$$9 for Task Arithmetic and $T_k=\theta_k-\theta_{\mathrm{pre}}$0 for Ties-Merging (Yang et al., 2023).

DOGE instead models merging as adaptive projective gradient descent. It reconstructs a data-free first-order objective from task vectors, identifies a shared subspace $T_k=\theta_k-\theta_{\mathrm{pre}}$1 from top singular directions, and updates a modification vector $T_k=\theta_k-\theta_{\mathrm{pre}}$2 only in directions orthogonal to that shared subspace:

$T_k=\theta_k-\theta_{\mathrm{pre}}$3

Merging coefficients are treated as adaptive learning rates, with a layer-wise rule $T_k=\theta_k-\theta_{\mathrm{pre}}$4. This formulation is intended to preserve shared knowledge rather than eliminate it by over-aggressive orthogonalization (Wei et al., 2 Jan 2025).

FroM uses a different data-free signal: per-layer Frobenius norm. In full fine-tuning, it minimizes

$T_k=\theta_k-\theta_{\mathrm{pre}}$5

which yields the closed-form weighted average

$T_k=\theta_k-\theta_{\mathrm{pre}}$6

Applied layer-wise, the weights become

$T_k=\theta_k-\theta_{\mathrm{pre}}$7

The reported ablations place the best $T_k=\theta_k-\theta_{\mathrm{pre}}$8 around $T_k=\theta_k-\theta_{\mathrm{pre}}$9 or $\Delta_i=\theta_i-\theta_0$0 for full fine-tuning and around $\Delta_i=\theta_i-\theta_0$1 to $\Delta_i=\theta_i-\theta_0$2 for LoRA merging (Li et al., 3 Jun 2025).

LOT Merging moves the adaptive signal from parameter magnitude to representation geometry. For linear layers it solves a convex quadratic problem whose closed-form solution depends on the per-task Gram matrices $\Delta_i=\theta_i-\theta_0$3. The merged layer minimizes

$\Delta_i=\theta_i-\theta_0$4

thereby explicitly reducing feature drift between expert and merged representations. The paper reports $\Delta_i=\theta_i-\theta_0$5 average accuracy on ViT-B/32 and $\Delta_i=\theta_i-\theta_0$6 on ViT-L/14, with improvements up to $\Delta_i=\theta_i-\theta_0$7 over prior training-free methods (Sun et al., 29 May 2025).

Speech-specific adaptive merging follows the same layer-wise logic. AdaLTM merges in-domain ASR and SER task vectors into frozen WavLM-Large layers with learned scalars $\Delta_i=\theta_i-\theta_0$8 and $\Delta_i=\theta_i-\theta_0$9, plus learned layer-aggregation weights $$\theta^*=\theta_0+\sum_{t=1}^T \gamma_t(\theta_t-\theta_0),$$0. On MSP-Podcast, adaptive layer-wise merging reaches $$\theta^*=\theta_0+\sum_{t=1}^T \gamma_t(\theta_t-\theta_0),$$1 UAR in the dual-vector setting, while a fully trainable joint MTL baseline reports $$\theta^*=\theta_0+\sum_{t=1}^T \gamma_t(\theta_t-\theta_0),$$2 UAR (Lee et al., 26 Mar 2026).

4. Continual, sequential, and editing-oriented merging

In continual learning, adaptive knowledge merging must decide not only how to combine models, but also when to merge and how much historical knowledge to preserve. AIMMerging monitors a learning signal

$$\theta^*=\theta_0+\sum_{t=1}^T \gamma_t(\theta_t-\theta_0),$$3

and a forgetting signal derived from rehearsal-loss exceedance, then adapts the next merge interval $$\theta^*=\theta_0+\sum_{t=1}^T \gamma_t(\theta_t-\theta_0),$$4, allows early merges when forgetting spikes, and defers merges when learning is stable. At merge time it combines a new-task vector and a rehearsal-derived past-task vector through

$$\theta^*=\theta_0+\sum_{t=1}^T \gamma_t(\theta_t-\theta_0),$$5

with $$\theta^*=\theta_0+\sum_{t=1}^T \gamma_t(\theta_t-\theta_0),$$6 computed from the proportions of learning and forgetting signals. The reported gains average $$\theta^*=\theta_0+\sum_{t=1}^T \gamma_t(\theta_t-\theta_0),$$7 on FWT and $$\theta^*=\theta_0+\sum_{t=1}^T \gamma_t(\theta_t-\theta_0),$$8 on BWT over prior state of the art (Feng et al., 22 Sep 2025).

OPCM addresses a different sequential setting in which expert checkpoints arrive one by one. For each new task vector, it computes an orthogonal projection onto the complement of the low-rank subspace spanned by previously merged updates, and then rescales the accumulated merge to maintain a stable parameter distance from the base model. The update has the form

$$\theta^*=\theta_0+\sum_{t=1}^T \gamma_t(\theta_t-\theta_0),$$9

with empirical scaling close to $\mu$0. The method reports average accuracy improvements of $\mu$1–$\mu$2 over continual Task Arithmetic and continual TIES-Merging, while maintaining constant memory complexity in the number of models (Tang et al., 16 Jan 2025).

BECAME introduces a Bayesian formulation of continual merging. After a gradient-projection stage yields a stability-preserving solution $\mu$3 and a second unconstrained stage yields a plastic solution $\mu$4, the final model is

$\mu$5

with a closed-form coefficient

$\mu$6

This coefficient increases when new-task curvature dominates and decreases when accumulated prior precision dominates. On $\mu$7-Split CIFAR-100, the reported ACC improves from $\mu$8 for GPM to $\mu$9 for BECAME(GPM) (Li et al., 3 Apr 2025).

Sequential knowledge editing uses the same general principle. In “Model Merging for Knowledge Editing,” robust supervised fine-tuning first internalizes edits into $\theta_s$0, then the edited model is obtained by merging the sparse delta back into the base:

$\theta_s$1

The reported practical settings use $\theta_s$2 and $\theta_s$3, and the sequential protocol always merges back into $\theta_s$4 rather than the previous merged model in order to avoid compounding drift (Fu et al., 14 Jun 2025).

5. Cross-domain, multi-source, and multimodal transfer

When source models are highly heterogeneous, adaptive knowledge merging often departs from direct weight arithmetic and introduces staging, decomposition, or synthetic proxy data. DMM is designed for models that occupy disconnected modes. It first merges only sufficiently similar models using standard techniques, aggregates BatchNorm buffers, synthesizes pseudo-data by matching merged normalization statistics, and then distills high-confidence predictions from divergent teachers into the merged model. Under severe heterogeneity ($\theta_s$5 Dirichlet splits), the paper reports FedAvg $\theta_s$6 on CIFAR-10, $\theta_s$7 on CIFAR-100, and $\theta_s$8 on CrisisMMD when DMM refinement is added (Liu et al., 6 Mar 2026).

AXIS targets many-source transfer by decomposing each task matrix into rank-one SVD components, selecting only the most salient components globally, re-orthogonalizing them with a final SVD, and then adapting only the principal singular values on target data. This makes adaptation cost constant in the number of source models after the initial aggregation stage. In the reported ViT-B/32 experiment with $\theta_s$9 source task vectors, AXIS averages $v_t=\theta_t-\theta_s$0 versus $v_t=\theta_t-\theta_s$1 for aTLAS (Osial et al., 26 Aug 2025).

MergeNet addresses the more general case of heterogeneous architectures, tasks, and modalities. Its parameter adapter, coupled with an LPKA attention module and a knowledge-transfer cycle $v_t=\theta_t-\theta_s$2, repeatedly queries source parameters and synthesizes target updates during training, then is removed at test time for zero overhead. The reported configurations include $v_t=\theta_t-\theta_s$3 or $v_t=\theta_t-\theta_s$4 softmax attention heads per layer depending on the transfer scenario, with $v_t=\theta_t-\theta_s$5 ranging from $v_t=\theta_t-\theta_s$6 to $v_t=\theta_t-\theta_s$7 (Li et al., 2024).

A plausible implication is that heterogeneity changes the locus of adaptation. Homogeneous expert merging can often adapt in weight space alone; disconnected domains and mismatched architectures more often require routing over distributions, staged distillation, or intermediate low-rank bases.

6. Symbolic knowledge bases and ontology merging

Adaptive knowledge merging is not confined to neural parameters. In propositional belief merging, one can adaptively restore consistency by forgetting variables. If $v_t=\theta_t-\theta_s$8 is a profile of knowledge bases and $v_t=\theta_t-\theta_s$9 is an integrity constraint, the forgetting operators $\{\theta_t\}_{t=1}^T$00 and $\{\theta_t\}_{t=1}^T$01 choose common variable sets $\{\theta_t\}_{t=1}^T$02 to forget in all sources so that

$\{\theta_t\}_{t=1}^T$03

becomes consistent, with either minimal cardinality or inclusion-minimality. This variable-selection view satisfies desiderata such as preserving literals supported without opposition and blocking literals that are explicitly contradicted by different sources (Xu et al., 2013).

Region-based ontology merging introduces a qualitatively different adaptive mechanism. Multiple aligned terminological sources are translated into RCC5 constraint networks, aggregated using conceptual-neighborhood distances, and iteratively relaxed until consistency is restored. The relaxation operator

$\{\theta_t\}_{t=1}^T$04

and the controversy measure

$\{\theta_t\}_{t=1}^T$05

prioritize which pairwise concept relations should be weakened first. The final consistent scenario is then translated back into description-logic axioms (Bouraoui et al., 2022).

CoMerger scales symbolic merging to many ontologies by partitioning the multi-ontology merge into structurally coherent blocks. It ranks pivot classes by

$\{\theta_t\}_{t=1}^T$06

where connectivity combines taxonomic and non-taxonomic relations, then merges first within blocks and later across blocks according to inter-block relatedness. On the runtime side, the reported average speedup is $\{\theta_t\}_{t=1}^T$07 over balanced binary merging and $\{\theta_t\}_{t=1}^T$08 over ladder merging, reaching $\{\theta_t\}_{t=1}^T$09 over ladder at $\{\theta_t\}_{t=1}^T$10 ontologies (Babalou et al., 2020).

These symbolic frameworks show that adaptivity can mean variable forgetting, region-constraint relaxation, or structure-aware partition scheduling rather than gradient-based coefficient learning. The shared theme is selective weakening of conflicts instead of unconditional source aggregation.

7. Empirical regularities, misconceptions, and limitations

A recurring misconception is that removing parameter overlap is sufficient to remove interference. Twin-Merging reports that even without parameter interference, using non-overlapping LoRA modules, merging still drops to $\{\theta_t\}_{t=1}^T$11 normalized score, and even merging similar tasks such as XSUM and CNN-DM yields $\{\theta_t\}_{t=1}^T$12 normalized score, still $\{\theta_t\}_{t=1}^T$13 versus fine-tuned experts (Lu et al., 2024). This directly motivates modularization of shared versus exclusive knowledge, rather than simple overlap pruning.

A second misconception is that a single global coefficient is adequate. AdaMerging shows that multi-task performance is highly sensitive to $\{\theta_t\}_{t=1}^T$14, and that layer-wise adaptation on ViT-B/32 reaches $\{\theta_t\}_{t=1}^T$15 average accuracy whereas task-wise AdaMerging reaches $\{\theta_t\}_{t=1}^T$16 (Yang et al., 2023). FroM’s ablations similarly show that $\{\theta_t\}_{t=1}^T$17 controls a nontrivial trade-off between uniform averaging and near winner-take-all selection (Li et al., 3 Jun 2025).

A third misconception is that enforcing orthogonality is always beneficial. DOGE explicitly argues that methods which over-promote orthogonality discard task-specific information that is crucial for performance, and instead preserves shared knowledge by projecting updates around a shared subspace rather than eliminating overlap wholesale (Wei et al., 2 Jan 2025). LOT Merging reaches a similar conclusion from a different angle: performance degradation correlates strongly with feature drift, so preserving the geometry of intermediate representations can be more important than sparsifying or decorrelating raw deltas (Sun et al., 29 May 2025).

The limitations are equally consistent. Some methods require unlabeled target samples, as in AdaMerging; some require identical architectures, as in Twin-Merging’s basic setting; some depend on BatchNorm statistics, as in DMM; some are sensitive to sliding-window and threshold hyperparameters, as in AIMMerging; some require domain-consistent auxiliary vectors, as in AdaLTM; and several incur SVD or pseudoinverse costs, as in AXIS, LOT Merging, and FroM (Yang et al., 2023, Liu et al., 6 Mar 2026, Feng et al., 22 Sep 2025, Lee et al., 26 Mar 2026, Osial et al., 26 Aug 2025). This suggests that adaptive knowledge merging is best viewed not as a single universally dominant recipe, but as a design space in which the adaptive signal must match the source of interference.

Taken together, the literature indicates that effective adaptive knowledge merging depends on identifying what should remain invariant and what should be allowed to change. The invariant may be a shared subspace, a base model’s general capability, a set of important channels, or an integrity constraint; the adaptive component may be routing weights, layer-wise scalars, rehearsal-triggered intervals, singular-value reweighting, or selectively forgotten variables. The field’s unifying result is that merging improves most when it becomes conditional rather than static.