---
title: Hierarchical Adapters
url: https://www.emergentmind.com/topics/hierarchical-adapters
type: topic
---

# Hierarchical Adapters

Hierarchical adapters are parameter-efficient, modular neural modules that enable fine-grained adaptation of large pre-trained models to multiple domains, tasks, or modalities. The key property of a hierarchical adapter system is the existence of an explicit or implicit structure—often a tree, tiered layer, or multi-level grouping—through which parameter sharing and specialization are mediated. This structure allows adapters to capture both generic and specific knowledge in a scalable manner, reducing negative interference between unrelated tasks or domains and promoting positive transfer among related ones. Hierarchical adapters have been developed across language, vision, vision-language, speech, and medical imaging models for scenarios ranging from multi-domain and multi-task learning to continual and cross-center adaptation.

## 1. Theoretical Basis and Structural Principles

The foundational principle of hierarchical adapter systems is that target domains, tasks, or modalities exhibit varying degrees of relatedness and share common substructures. By modeling these relationships hierarchically, adapters can be organized such that:

- **Higher-level (coarse) adapters** encode broadly shared patterns, updated frequently as they are active for many domains or tasks.
- **Lower-level (fine) adapters** capture domain- or task-specific patterns, activated only for specialized subsets.

For example, in domain adaptation for language models, domains (e.g., individual websites) are mapped to leaves in a tree, with internal nodes aggregating related sites (e.g., all e-commerce) [2112.08786]. Each node in the tree has its own adapter parameters, and for any domain, the active adapter path is the set of nodes from root to leaf. This tree-based structural assignment encourages parameter sharing along shared subpaths and mitigates negative transfer between unrelated leaves, as unrelated domains share only root-level parameters.

Beyond explicit trees, hierarchical structures are instantiated in various forms: multi-level grouping by domain or modality [2508.13026], layered allocation for distinct loss objectives or representation roles [2509.16903], or latent semantic hierarchies in hyperbolic spaces [2508.11176].

## 2. Algorithms and Architectures

Numerous adapter constructions implement hierarchical organization across diverse model types:

### a. Tree-Structured Adapters for Domain Adaptation

For frozen transformer-based language models (e.g., GPT-2), each node of a domain tree is associated with a bottleneck adapter; only adapters on the path from root to the active domain are updated at train time, and their outputs are averaged at the corresponding transformer layers during inference [2112.08786]. This enables $O(\log k)$ parameter scaling for $k$ domains, significantly lower than $O(k)$ for flat adapter baselines.

### b. Hierarchical Regularization in VLMs

Latent Hierarchical Adapters for vision-language models employ a three-level semantic hierarchy—category, attribute (learnable prompts), and image instance—embedded in a hyperbolic (Poincaré ball) space [2508.11176]. Specialized attribute-aware refiners and hierarchical losses enforce structured alignment, supporting one-to-many mappings and improved adaptation to unseen classes.

### c. Hierarchical Grouping and Merging for Continual Learning

HAM (Hierarchical Adapter Merging) introduces group-based LoRA adapters: each task starts with its own low-rank adapter, but after training, adapters are grouped by similarity, pruned for compactness, and merged hierarchically [2509.13211]. This enables lifelong continual learning with bounded memory—at most $M \ll N$ group adapters are retained for $N$ tasks, circumventing linear growth and reducing catastrophic forgetting.

### d. Layerwise Hierarchies and Expert Mixtures

In multi-lingual discourse relation classification HiDAC, a dual adapter scheme is employed: lower layers use standard LoRA adapters under a contrastive loss, and upper layers use mixture-of-experts LoRA (MoE-LoRA) adapters, trained with cross-entropy [2509.16903]. This split captures local representation shaping and high-level classification specialization. Similarly, Hierarchical Recurrent Adapters in speech models separate global shared controller parameters from per-task heads, amortizing most trainable parameters [2403.19709].

### e. Multi-level Modality Adapters

In long video-to-text summarization, Hierarchical3D adapters perform global interaction among utterance-aligned multimodal tokens (text, audio, vision) alongside standard per-token adapters, enabling more effective fusion and global context propagation [2210.04829]. In medical imaging, protocol-level and center-level adapters are stacked to address both sequence-specific and scanner-specific distribution shifts, assisted by a universal adapter for unseen domains [2508.13026].

## 3. Training and Inference Procedures

The specifics of hierarchical adapter training depend on the structural embedding and modality:

- **Selective Parameter Updates:** For tree-based domain adaptation, only the adapters along the active path are updated for each target, with all other adapter weights and the backbone frozen. In contrast, dual-layer schemes (e.g., HiDAC) optimize distinct losses and adapters in different regions of the model.
- **Regularizer Design:** Hierarchical regularizers in hyperbolic space (as in LatHAdapter) enforce structured semantic proximity, while group-based merging (HAM) employs cosine similarity and pruning to consolidate adapters.
- **Inference-Time Aggregation:** For adaptation to unseen domains, strategies such as path-averaging over the hierarchy (e.g., two best domain paths in [2112.08786]) or on-the-fly group-merging (HAM) are applied, typically with minor computational overhead.

## 4. Empirical Results and Efficiency Trade-offs

Hierarchical adapters consistently outperform flat or per-task/domain adapter baselines across multiple modalities and tasks by achieving better trade-offs between performance, parameter count, and memory usage.

- In multi-domain language modeling, hierarchical adapters with a tree structure yield lower in-domain and out-of-domain perplexity compared to multi-domain or single-domain adapter baselines, with active parameters per forward pass scaling as $O(\log k)$ [2112.08786].
- In few-shot vision-language adaptation, Latent Hierarchical Adapters integrated with prompt-based baselines yield 0.7–4.7% accuracy gains on both base and novel classes with only minor increases in parameter count [2508.11176].
- HAM achieves robust continual learning on up to 100 tasks by containing memory to a small, fixed number of merged adapters and displaying 15–20% less forgetting than non-hierarchical PEFT baselines [2509.13211].
- HiDAC's dual-level adapter design achieves state-of-the-art multi-lingual discourse classification (67.5% vs. 66.8% for BERT 75% progressive unfreeze) while tuning only 3% of backbone parameters [2509.16903].
- Multimodal and cross-center hierarchical adapters (as in HierAdaptMR) yield pronounced domain generalization, e.g., 0.769→0.868 (+12.9%) improvement in SSIM for cardiac MRI reconstruction [2508.13026].
- Recurrent hierarchical adapters in ASR match full fine-tuning quality with just a fraction of the parameter footprint (mean WER 9.9 vs. 9.3; 0.2B vs. 232B parameters in the multi-task Euphonia benchmark) [2403.19709].

## 5. Limitations, Open Problems, and Generalization

Several practical and theoretical limitations remain:

- **Dependency on structural priors:** The quality of the hierarchy (e.g., clustering in tree-based methods) can limit effectiveness—poorly defined structures may underperform [2112.08786]. In many cases, domain or task groupings must be known or estimable.
- **Applicability across tasks:** Most work to date has been in modeling, classification, and representation adaptation; explicit downstream utilization in generative or open-ended tasks remains less explored.
- **Regularization and transfer:** Hierarchical sharing naturally regulates overfitting but can propagate noise if upper layers are not properly calibrated or if negative transfer across loosely related groups persists.
- **Model and adapter complexity:** Mixture-of-experts and group merging methods require additional gating, grouping, and pruning steps to avoid scaling bottlenecks—especially in large continual learning settings [2509.13211].

## 6. Extensions and Future Directions

Several clear research directions follow from the current literature:

- **Learned and adaptive hierarchies:** End-to-end training of both the hierarchy and adapter parameters, enabling data-driven, dynamic construction of sharing structures [2112.08786].
- **Cross-modal and multi-granular hierarchies:** Integrating multiple structural axes—semantic, modality, temporal—within a unified adapter framework (as in LatHAdapter and Hierarchical3D) [2508.11176, 2210.04829].
- **Synergy with other PEFT methods:** Combining hierarchical adapters with techniques such as LoRA, prompt tuning, or sparse updates for enhanced flexibility [2112.08786, 2509.16903].
- **Adaptive capacity control:** Dynamic growing, pruning, or reallocation of adapter banks at training or inference time for efficiency and transfer [2508.11176, 2509.13211].
- **Applications to novel tasks:** Testing hierarchical adapter frameworks beyond modeling/classification, such as joint generation, open-vocabulary retrieval, or multi-modal hierarchical generation [2508.11176].

## 7. Summary Table of Representative Hierarchical Adapter Architectures

| Approach / Paper             | Structure & Scope         | Core Mechanism                  |
|------------------------------|--------------------------|---------------------------------|
| "Efficient Hierarchical..." [2112.08786]         | Tree-structured domains (LMs)        | Node adapter per tree node; path-averaging; $O(\log k)$ params |
| "Latent Hierarchical Adapter" [2508.11176]   | Category–attribute–image (VLM)        | Attribute prompts; hyperbolic reg.; triplet miners              |
| "HAM: Hierarchical Adapter Merging" [2509.13211] | Continual tasks (vision)              | Adapter grouping & merging; LoRA; group-wise pruning            |
| "HiDAC" [2509.16903]                 | Layerwise dual adapters (DRC)           | LoRA in low layers (contrastive); MoE-LoRA upper (CE loss)      |
| "Hierarchical3D Adapters" [2210.04829]   | Multimodal utterance/global (summar.)  | Utterance-level fusion & interaction in enc. adapters           |
| "HierAdaptMR" [2508.13026]               | Protocol/center-level (MRI recon)     | Stacked adapters per scanner, protocol; universal adapter       |
| "Hierarchical Recurrent Adapter" [2403.19709]    | Task-level, recurrent-shared (speech)  | Global controller recur. adapter + task head per downstream     |

This comprehensive body of research demonstrates that hierarchical adapters can yield parameter-efficient, scalable, and robust adaptation mechanisms, adapting seamlessly to various architectures, domains, modalities, and task regimes.

Source: https://www.emergentmind.com/topics/hierarchical-adapters