HAM: Hierarchical Adapter Merging

• The paper introduces HAM as a continual-learning framework that merges task-specific LoRA adapters into a single inference module, addressing catastrophic forgetting.
• HAM employs similarity-based grouping, selective magnitude pruning, and concatenation-based consolidation to preserve essential task information while controlling adapter growth.
• Empirical results show HAM improves average accuracy and reduces memory footprint on vision benchmarks compared to established PEFT methods.

Searching arXiv for the cited HAM paper and closely related adapter-merging papers to ground the article. Hierarchical Adapters Merging (HAM) is a continual-learning framework for LoRA-based parameter-efficient fine-tuning in which task-specific adapters are not kept permanently separate, but are dynamically consolidated into a bounded set of group adapters and then globally merged into a single inference-time adapter. In its original formulation, HAM targets class-incremental learning with disjoint class sets and no task identifiers at inference time; its defining mechanism is a hierarchy of task adapters, group adapters, and a final merged adapter, coupled with similarity-based grouping, selective pruning, concatenation-based intra-group consolidation, and importance-scaled final merging (Coleman et al., 16 Sep 2025).

1. Problem setting and conceptual scope

HAM was introduced for class-incremental continual learning, where a model receives a sequence of tasks

$\mathcal{T}=\{T_1,T_2,\dots,T_N\}$

with disjoint class sets and must retain performance on all encountered tasks without access to task identity at test time. The motivating difficulty is catastrophic forgetting: adaptation to new data interferes with previously learned knowledge. The framework is explicitly built on the observation that PEFT methods such as LoRA reduce adaptation cost but do not by themselves solve long-horizon scaling, because maintaining one adapter per task causes adapter count to grow linearly, complicates inference in task-agnostic settings, and limits transfer across related tasks (Coleman et al., 16 Sep 2025).

Within that setting, HAM is neither a pure LoRA-per-task method nor a purely post-hoc merge method. Its central claim is that consolidation should occur during continual learning itself. Each new task first receives its own low-rank adapter and an importance scalar; that adapter is then compared to previously consolidated groups, pruned, concatenated into a compatible group, and only later participates in a final global merge. The hierarchy is therefore operational rather than purely taxonomic: knowledge moves from task-local modules to intermediate group modules and finally to one deployed model.

A common misunderstanding is to treat HAM as a general recursive tree-merging framework. The formulation in the original paper is more specific. It is described as a two-stage consolidation pipeline with task-specific adapters at the lower level, group adapters at the intermediate level, and a final global merged adapter for inference, not as an arbitrary-depth merge tree (Coleman et al., 16 Sep 2025).

2. Mathematical formulation

HAM inherits the standard LoRA parameterization. For a pretrained weight matrix $W_0\in\mathbb{R}^{d\times k}$, adaptation is written as

$W = W_0 + \Delta W = W_0 + BA,$

where

$$B\in\mathbb{R}^{d\times r}, \qquad A\in\mathbb{R}^{r\times k}, \qquad r \ll \min(d,k).$$

This low-rank structure is the substrate on which the hierarchical consolidation mechanism operates (Coleman et al., 16 Sep 2025).

For a current task $T_i$, HAM trains a task-specific adapter $\Delta W_i=B_iA_i$ together with a scalar $\alpha_i$. Previously formed group adapters $\{\Delta W_{G_j}\}_{j=1}^M$ remain frozen, while their group-level importance scalars $\{\alpha_{G_j}\}_{j=1}^M$ are updated. The task-time model output is

$$h = \underbrace{W_0 x}_{\text{pre-trained model}} + \underbrace{\sum_{j=1}^{M} \alpha_{G_j} \Delta W_{G_j} x}_{\text{previous group adapters}} + \underbrace{\alpha_i \Delta W_i x}_{\text{current adapter}}.$$

This equation is central: the new adapter is optimized in the presence of already consolidated group knowledge rather than in isolation. The paper does not provide an explicit standalone task loss in LaTeX form, nor a separate optimization objective for $W_0\in\mathbb{R}^{d\times k}$0 (Coleman et al., 16 Sep 2025).

Group assignment is driven by cosine similarity between vectorized adapter matrices, computed using the last LoRA layer: $W_0\in\mathbb{R}^{d\times k}$1 A new adapter joins a group when

$W_0\in\mathbb{R}^{d\times k}$2

otherwise a new group is created if the current number of groups is below the budget $W_0\in\mathbb{R}^{d\times k}$3, and if the budget has already been reached the adapter is forced into the most similar group. The exact value of $W_0\in\mathbb{R}^{d\times k}$4 is not reported (Coleman et al., 16 Sep 2025).

Group importance is updated online. When a new adapter $W_0\in\mathbb{R}^{d\times k}$5 is added to group $W_0\in\mathbb{R}^{d\times k}$6, the scalar is updated as

$W_0\in\mathbb{R}^{d\times k}$7

and otherwise

$W_0\in\mathbb{R}^{d\times k}$8

The paper describes this as a running average of member-level importances (Coleman et al., 16 Sep 2025).

3. Hierarchical consolidation procedure

After group assignment, HAM applies selective magnitude pruning to the new adapter. For the LoRA factors $W_0\in\mathbb{R}^{d\times k}$9 and $W = W_0 + \Delta W = W_0 + BA,$0, the pruned matrices are

$W = W_0 + \Delta W = W_0 + BA,$1

where $W = W_0 + \Delta W = W_0 + BA,$2 and $W = W_0 + \Delta W = W_0 + BA,$3 retain only the top-$W = W_0 + \Delta W = W_0 + BA,$4 entries by absolute value. In the reported experiments, HAM keeps the top $W = W_0 + \Delta W = W_0 + BA,$5 of weights. This pruning stage is intended to suppress less important coefficients before consolidation (Coleman et al., 16 Sep 2025).

Intra-group consolidation is performed by concatenation rather than averaging or low-rank refitting: $W = W_0 + \Delta W = W_0 + BA,$6 This choice is methodologically important. Concatenation increases the effective rank of a group adapter, and the paper states that if a group contains $W = W_0 + \Delta W = W_0 + BA,$7 adapters then its rank becomes

$W = W_0 + \Delta W = W_0 + BA,$8

The authors report that they tested alternatives such as TIES for intra-group combination but found plain concatenation to work best, attributing that outcome to its preservation of more task-specific information than fixed-rank merge operators (Coleman et al., 16 Sep 2025).

Final inference is task-agnostic. Because class-incremental evaluation does not provide task identity, HAM does not deploy routing over task adapters or group adapters. Instead, all group adapters are merged into a single final adapter: $W = W_0 + \Delta W = W_0 + BA,$9 and the deployed model is

$$B\in\mathbb{R}^{d\times r}, \qquad A\in\mathbb{R}^{r\times k}, \qquad r \ll \min(d,k).$$0

Equal group weights are used at this last stage because the learned $$B\in\mathbb{R}^{d\times r}, \qquad A\in\mathbb{R}^{r\times k}, \qquad r \ll \min(d,k).$$1 values are taken to provide the needed scaling. The paper also notes that this final merging step can be executed at any point during training, although its practical discussion treats the final adapted model as effectively available only at the end (Coleman et al., 16 Sep 2025).

4. Empirical behavior and ablation findings

HAM is evaluated on three vision benchmarks—CIFAR-100, CUB-200, and Tiny-ImageNet—using a ViT-B/16 pretrained on ImageNet under class-incremental learning without task identity at inference time. The paper reports Average Accuracy, Forgetting Measure, and efficiency metrics, and emphasizes long task sequences, including a main 50-task setting and experiments up to 100 tasks on CUB-200. The fixed default configuration is LoRA rank $$B\in\mathbb{R}^{d\times r}, \qquad A\in\mathbb{R}^{r\times k}, \qquad r \ll \min(d,k).$$2, pruning retention $$B\in\mathbb{R}^{d\times r}, \qquad A\in\mathbb{R}^{r\times k}, \qquad r \ll \min(d,k).$$3, maximum number of groups $$B\in\mathbb{R}^{d\times r}, \qquad A\in\mathbb{R}^{r\times k}, \qquad r \ll \min(d,k).$$4, AdamW, learning rate $$B\in\mathbb{R}^{d\times r}, \qquad A\in\mathbb{R}^{r\times k}, \qquad r \ll \min(d,k).$$5, and batch size $$B\in\mathbb{R}^{d\times r}, \qquad A\in\mathbb{R}^{r\times k}, \qquad r \ll \min(d,k).$$6 (Coleman et al., 16 Sep 2025).

The reported headline result is that HAM is the top-performing method across all tested sequence lengths and shows the smallest accuracy degradation as task count increases. On CUB-200 with 100 tasks, HAM is reported to remain effective while SD-LoRA fails after 69 experiences due to GPU memory limitations. The paper further states that DualPrompt achieves the lowest forgetting numerically, but with much worse overall accuracy, and that HAM is about $$B\in\mathbb{R}^{d\times r}, \qquad A\in\mathbb{R}^{r\times k}, \qquad r \ll \min(d,k).$$7 better in accuracy on average than DualPrompt while also surpassing strong PEFT baselines such as SEMA, InfLoRA, and SD-LoRA in both accuracy and forgetting (Coleman et al., 16 Sep 2025).

Several ablations clarify which parts of the pipeline matter. Similarity-based grouping consistently outperforms orthogonality-based grouping, and the advantage grows as the number of tasks increases. Group adapters are not merely compressed surrogates: at pruning level $$B\in\mathbb{R}^{d\times r}, \qquad A\in\mathbb{R}^{r\times k}, \qquad r \ll \min(d,k).$$8, the paper reports task-level improvements of $$B\in\mathbb{R}^{d\times r}, \qquad A\in\mathbb{R}^{r\times k}, \qquad r \ll \min(d,k).$$9 on Task 4 and $T_i$0 on Task 8, while parameters drop from $T_i$1M to $T_i$2M, a $T_i$3 reduction, alongside a $T_i$4 accuracy improvement. This suggests that pruning plus concatenation can act as constructive consolidation rather than simple storage compression (Coleman et al., 16 Sep 2025).

The number of groups also matters. The paper reports that fewer groups generally improve performance: reducing from $T_i$5 to $T_i$6 with 10 tasks improves accuracy by $T_i$7, and the analogous reduction with 20 tasks yields a $T_i$8 improvement. For the final merge, linear merging performs best, slightly better than TIES and clearly better than DARE in the reported 10-task scenario. Pruning exhibits a non-monotonic behavior: Average Accuracy improves as retained weight percentage increases from $T_i$9 to $\Delta W_i=B_iA_i$0, then plateaus and declines slightly at $\Delta W_i=B_iA_i$1. Efficiency analysis on CIFAR-100 with 10 tasks reports that HAM is substantially faster than SD-LoRA and InfLoRA and is roughly twice as fast as SD-LoRA in both training and inference (Coleman et al., 16 Sep 2025).

5. Relation to neighboring adapter-merging research

HAM sits within a broader literature on adapter composition, but several nearby methods solve different problems. A useful baseline is flat additive LoRA merging in vision. The paper "Multi LoRA Meets Vision: Merging multiple adapters to create a multi task model" studies concatenation-based merging of separately trained vision LoRAs without retraining and shows that simple additive merging can work for some task combinations, while also exposing strong pairwise compatibility effects. That work is directly relevant as a flat baseline, but it does not introduce a hierarchy, clustering objective, routing mechanism, or staged consolidation procedure (Kesim et al., 2024).

HydraOpt is also related but structurally different. It is a data-free low-rank adapter compression method that shares a single global $\Delta W_i=B_iA_i$2 and retains a smaller bank of $\Delta W_i=B_iA_i$3 matrices, giving a one-level global-shared-plus-task/group-specific decomposition rather than a hierarchy. Its main contribution is a controllable storage/performance trade-off, not recursive or multi-level merging. The paper explicitly positions it as conceptually related and potentially complementary to HAM rather than as HAM itself (Ceritli et al., 23 Jul 2025).

Other neighboring works illuminate the distinction between hierarchical adapters and hierarchical adapter merging. $\Delta W_i=B_iA_i$4 performs flat data-driven clustering of adapters before within-cluster merging under a storage budget, but it does not build a tree or recursive schedule (Bohdal et al., 24 Jan 2026). HierAdaptMR is a hierarchical feature adaptation framework for cross-center cardiac MRI reconstruction, yet its hierarchy is realized by metadata-driven selection and sequential composition of protocol-level, center-level, and universal adapters rather than parameter-space merging (Xu et al., 18 Aug 2025). LatHAdapter organizes category, attribute, and image representations in a latent semantic hierarchy for VLM few-shot adaptation, but it operates through prompts, hyperbolic structure, and feature-level refinement rather than adapter merging (Zhao et al., 15 Aug 2025). OrthoFuse, by contrast, is a training-free pairwise fusion method for orthogonal multiplicative adapters in diffusion models and is relevant as a structured merge operator, though not as a hierarchical system (Aliev et al., 6 Apr 2026).

A further source of terminological confusion is that "hierarchical merging" is used outside the adapter literature. "Context-Aware Hierarchical Merging for Long Document Summarization" studies recursive merging of summaries for documents exceeding $\Delta W_i=B_iA_i$5K tokens, not adapter composition. Its relevance to HAM is conceptual at most: it concerns evidence-aware recursive summarization, not PEFT modules (Ou et al., 3 Feb 2025).

6. Limitations, misconceptions, and open directions

Several limitations are explicit in the original HAM formulation. The method depends on cosine similarity of vectorized adapter weights as its grouping signal, so grouping quality is only as good as that similarity proxy. When the number of groups has reached the budget $\Delta W_i=B_iA_i$6, a new adapter must be assigned to the most similar existing group even if similarity is low, which can increase interference. The paper also does not provide an explicit classification loss, a formal hierarchical consolidation objective, or asymptotic time and memory complexity expressions. These omissions matter for analytical comparison with alternative continual-learning schemes (Coleman et al., 16 Sep 2025).

Another important limitation is that the final adapted model is effectively available only at the end of training. Although the paper notes that final merging can be executed at any point, it does not develop a fully online deployment protocol for a continually updated merged model. The authors accordingly identify online adapters merging as a future direction and also suggest extension beyond LoRA to other PEFT methods such as prompts (Coleman et al., 16 Sep 2025).

Two misconceptions recur in discussion of HAM. First, HAM is not equivalent to one-shot post-hoc averaging of all task adapters. Its defining claim is incremental consolidation during learning through grouping, pruning, and concatenation before the final merge. Second, HAM should not be conflated with any work that contains the words “hierarchical” and “adapter.” In the current literature, “hierarchical” can refer to semantic organization, metadata-routed modular adaptation, recursive summary merging, or structured pairwise geometric fusion; only some of these settings involve parameter-space adapter merging in the specific sense used by HAM (Coleman et al., 16 Sep 2025).

Taken together, these points suggest a precise interpretation. HAM is best viewed as a bounded-group continual-learning framework in which hierarchy is a mechanism for scalable consolidation under task-agnostic inference constraints. Its significance lies less in arbitrary-depth recursive merging than in the combination of task-aware adapter training, similarity-based group formation, pruning-based noise suppression, concatenation-based capacity preservation, and importance-scaled final unification into a single deployed model (Coleman et al., 16 Sep 2025).