---
title: MDAPT with Prototype-Based HyperAdapters
url: https://www.emergentmind.com/topics/adapter-based-parameter-efficient-mdapt
type: topic
---

# MDAPT with Prototype-Based HyperAdapters

Adapter-Based Parameter-Efficient MDAPT—Prototype-Based HyperAdapters

Adapter-based parameter-efficient Multi-Domain AdapTation (MDAPT) using Prototype-Based HyperAdapters (PHA) is a method for efficiently adapting pre-trained language models (PLMs) to a growing set of distinct domains or tasks. It achieves this by replacing exhaustive model fine-tuning with a small, dynamically generated set of trainable parameters via a hypernetwork and prototype-driven retrieval, resulting in strong generalization and sample efficiency, especially in low-data and multi-domain settings [2310.11670].

## 1. Architectural Foundations of Prototype-Based HyperAdapters

PHA extends classical adapter-tuning by inserting lightweight bottleneck adapters into every transformer layer, but avoids per-task storage and retraining by generating these adapters on-the-fly through a shared prototypical hypernetwork. For PLMs like T5-Base ($\theta \in \mathbb{R}^{220M}$), the backbone (encoder, decoder) remains frozen, while each layer $m$ contains a two-layer adapter $A^m$ controlled by:

- Down-projection $U^m \in \mathbb{R}^{b \times d}$
- Nonlinearity (ReLU)
- Up-projection $D^m \in \mathbb{R}^{d \times b}$

Here, $d$ is the hidden size (e.g., 768), $b$ is the adapter bottleneck ($b \ll d$). Adapter weights per layer and task/domain are not stored explicitly; instead, they are generated by a single hypernetwork $H_w$ which takes as input a learned task/domain prototype $k_i \in \mathbb{R}^{d'}$ and layer embedding $e_m$.

This design drastically reduces the parameter count required as the number of tasks/domains grows: total storage scales as $O(\tau d' + L d' + |H|)$ (prototypes, layer embeddings, hypernetwork), compared to $O(\tau d L)$ for separate per-task/domain adapters, where $\tau$ is the number of domains/tasks and $L$ the number of transformer layers.

## 2. Instance-Dense Retrieval and Prototype Learning

Task/domain prototypes are discovered and refined via an instance-dense retriever:

- Each instance $x$ is mapped to a latent embedding $z = G(h)$, with $h = \mathrm{Encoder}_\theta(x)$ and $G$ a dense MLP.
- Retrieval vectors are supervised by the InfoNCE contrastive loss $L_\mathrm{IR}$ to enforce that instance vectors from the same task/domain cluster in latent space, repelling vectors from different tasks/domains:
  $$
  L_\mathrm{IR}^i = -\frac{1}{N_i-1} \sum_{z_j \in \tilde{D}_i} \log \frac{\exp(f(z_i, z_j))}{\sum_{z_m \in S(i)} \exp(f(z_i, z_m))}
  $$
  with $f(a, b) = \cos(a,b)$.
- Prototype vectors $k_i$ are further optimized via a separate prototypical contrastive loss $L_\mathrm{Pro}$ to enhance their representativity and discrimination.

At inference, a new instance is mapped into the same latent space, and the nearest (in cosine similarity) or top-$K$ prototypes are selected to condition the hypernetwork, allowing rapid domain selection—even in the absence of explicit domain labels.

## 3. Adapter Parameter Generation via Prototypical Hypernetwork

- Each stabilized prototype $k_i$ is concatenated with a layer embedding $e_m$ and projected before being passed to the hypernetwork $H_w$ to yield domain- and layer-specific adapter parameters $(D_i^m, U_i^m)$.
- The layer output is modified as:
  $$
  y = \mathrm{FFN}(\mathrm{LN}(x)) + A_i^m(\mathrm{LN}(x)), \quad
  A_i^m(x) = D_i^m (\mathrm{ReLU}(U_i^m x)) + x.
  $$
- Only $H_w$, the set of prototypes $\{k_i\}$, and layer embeddings $\{e_m\}$ are stored; the full adapter parameter tensors are generated at run-time.

## 4. Parameter and Storage Efficiency

The crucial efficiency properties are as follows:

| Approach                        | Trainable Params | % of FF Tune (T5-Base) | Avg. GLUE+SG Score |
|----------------------------------|------------------|-----------------------|--------------------|
| Full fine-tune                   | 220M             | 100%                  | 84.9%              |
| Standard adapters                | 1.9M             | 0.86%                 | 84.5%              |
| Hyperformer++                    | 638K             | 0.29%                 | 84.7%              |
| HyperDecoder                     | 1.8M             | 0.82%                 | 83.7%              |
| **PHA (MDAPT, 12 tasks)**        | **616K**         | **0.28%**             | **85.5%**          |

At scale (e.g., $\tau = 12$ tasks/domains, $L = 12$ layers, $d' = 128$), PHA requires only $\sim$616K trainable parameters, a 0.28% fraction of full fine-tuning.

## 5. Multi-Domain Adaptation Mechanism

Transitioning from multi-task to multi-domain settings, each domain is treated as a "task," and domain prototypes are learned from in-domain (either labeled or unlabeled) data with the same InfoNCE and prototypical contrastive losses:

- **Prototype formation:** Domain prototypes $p_d$ capture the key characteristics of each domain.
- **Domain retrieval:** For unlabeled or mixed-domain data, domain assignment at inference is achieved by mapping each instance into the prototype space for retrieval and selection.
- **Top-K prototype mixture:** For instances spanning multiple domains, a weighted sum of the top-K prototypes can be fed to the hypernetwork.
- **Continuous/online domain shift:** Prototypes can be updated online to reflect emerging domains via a maintained buffer and small learning rate.

Benefits include: (1) maximal reuse of cross-domain knowledge in the hypernetwork, (2) almost negligible per-domain storage (one vector per domain), and (3) immediate adaptation with no backbone fine-tuning.

## 6. Training Objectives and Sample Efficiency

The end-to-end training objective for PHA-based MDAPT is:
$$
L_\mathrm{Total} = L_\mathrm{PLM} + \lambda \left( L_\mathrm{IR} + L_\mathrm{Pro} \right)
$$
where $L_\mathrm{PLM}$ is the supervised cross-entropy loss, $L_\mathrm{IR}$ is the retriever contrastive loss, $L_\mathrm{Pro}$ is the prototypical embedding loss, and $\lambda$ (default 0.1) balances sample efficiency and representation quality.

Empirically, PHA yields state-of-the-art robustness and efficiency in low-data settings:
- On 100 samples/task regimes, PHA attains 80% accuracy vs. 72% for classic adapters and 70% for Hyperformer.
- For few-shot transfer ($k=4,16,32$), PHA achieves 68–88% accuracy (+3–20% absolute improvement versus baselines).
- As the available data declines from full to 1%, PHA maintains a 5–10% absolute improvement in downstream metrics [2310.11670].

## 7. Practical Considerations and Extensions

Key design decisions and observations for adapter-based parameter-efficient MDAPT via PHA include:

- **Storage and deployment**: Models require only the frozen backbone, the hypernetwork, and a prototype vector per domain; domain adaptation is a matter of swapping in the appropriate prototype.
- **Mixed-domain and online settings**: Top-K prototype mixtures enable adaptation to samples with ambiguous or hybrid domain membership; buffer-based prototype updating supports streaming or evolving domains.
- **Training and convergence**: PHA achieves 10–15% faster convergence and 5–7% absolute accuracy improvements relative to adapters or domain-specific full fine-tuning in sentiment classification domains, while maintaining a per-domain parameter update of $<1\%$.

PHA-based MDAPT delivers a system in which (a) adapters are not explicitly stored per domain—a fundamental scalability improvement, (b) hypernetwork-based parameterization allows fully on-the-fly instantiation of domain adapters, and (c) sample efficiency is enhanced by prototype-driven retrieval and loss design.

## References

- Prototype-based HyperAdapter for Sample-Efficient Multi-task Tuning [2310.11670]

Source: https://www.emergentmind.com/topics/adapter-based-parameter-efficient-mdapt