---
title: LlamaMLP Adapter Layer
url: https://www.emergentmind.com/topics/llamamlp-adapter-layer
type: topic
---

# LlamaMLP Adapter Layer

LlamaMLP Adapter Layer refers to the insertion of lightweight, parameter-efficient adapter modules within the MLP (feed-forward) sub-layers of LLaMA, a variant of the Transformer architecture. As introduced and empirically evaluated in "LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models" [2304.01933], these adapters—implemented as either "Series adapters" or "Parallel adapters" (terminology from the original paper)—enable efficient fine-tuning by introducing a small bottleneck network per layer, allowing task adaptation by updating only a fraction of the model parameters while retaining most of the frozen LLM weights.

## 1. Architecture of MLP Adapter Layers

The LlamaMLP Adapter Layer modifies the standard MLP block found in the Transformer architecture. In each layer, the conventional feed-forward block operates as:

- $x \rightarrow W_1 x + b_1 \rightarrow \mathrm{GELU}(\cdot) \rightarrow W_2 \cdot + b_2 = y$

Adapters are added as follows:

- **Series Adapter**: Inserted after the MLP block's output, $y$. The process is:
  - Down-project $y$ to a lower-dimensional "bottleneck" of size $r$: $U = W_{\mathrm{down}}\,y + b_{\mathrm{down}}$, where $W_{\mathrm{down}} \in \mathbb{R}^{d \times r}$.
  - Pointwise nonlinearity: $V = \mathrm{ReLU}(U)$.
  - Up-project to the original dimension: $\Delta = W_{\mathrm{up}}\,V + b_{\mathrm{up}}$, $W_{\mathrm{up}} \in \mathbb{R}^{r \times d}$.
  - Residual addition: $y^{\prime} = y + \Delta$.

- **Parallel Adapter**: Operates in parallel with the MLP block, taking the MLP input $x$:
  - Down-project $x$: $U = W_{\mathrm{down}}\,x + b_{\mathrm{down}}$.
  - Nonlinearity and up-projection as above: $V = \mathrm{ReLU}(U)$, $\Delta = W_{\mathrm{up}}\,V + b_{\mathrm{up}}$.
  - Summed with the output of the MLP block: $y^{\prime} = y + \Delta$.

No auxiliary gating or sigmoid mechanisms are deployed in these adapters.

## 2. Placement Strategies in Transformer Blocks

Adapters are positioned within the Transformer encoder as follows:

- The **Series Adapter** is appended immediately after the MLP sub-layer's output, i.e., after the $W_2 \cdot + b_2$ operation.
- The **Parallel Adapter** computes its output in parallel to the MLP, with the result added to the MLP output.
  
Ablation results demonstrated optimal performance for Series Adapters when placed post-MLP and for Parallel Adapters when run parallel to the MLP sub-layer, rather than after attention or elsewhere.

## 3. Mathematical Formulation

The transformations governed by each adapter configuration are as follows:

- **Series Adapter**:
  $$
  U = W_{\mathrm{down}}\,y + b_{\mathrm{down}} \\
  V = \mathrm{ReLU}(U) \\
  \Delta = W_{\mathrm{up}}\,V + b_{\mathrm{up}} \\
  y' = y + \Delta
  $$
- **Parallel Adapter**:
  $$
  U = W_{\mathrm{down}}\,x + b_{\mathrm{down}} \\
  V = \mathrm{ReLU}(U) \\
  \Delta = W_{\mathrm{up}}\,V + b_{\mathrm{up}} \\
  y' = y + \Delta
  $$

Typically, $b_{\mathrm{down}}$ and $b_{\mathrm{up}}$ are either set to zero or learned.

## 4. Hyperparameters and Initialization

The adapter bottleneck size $r$ is a critical configuration parameter. Grid search over $\{64, 128, 256, 512\}$ identified $r=256$ as optimal for most tasks.

Hyperparameter choices:

- **Nonlinearity**: ReLU
- **Weight Initialization**:
  - $W_{\mathrm{down}} \sim \mathcal{N}(0, \sigma^2)$ with small $\sigma$ (e.g., $0.02$)
  - $W_{\mathrm{up}}$ initialized to zero, ensuring the adapter initially behaves as an identity function
  - $b_{\mathrm{down}}, b_{\mathrm{up}} = 0$
- No additional learnable scaling or gating is utilized beyond the two projection matrices.

*This suggests that the adapters are initialized to prevent divergence from original model behavior at the start of fine-tuning.*

## 5. Parameter Efficiency and Model Scale

Each adapter insertion results in $2dr$ extra parameters per MLP sub-layer.

A representative calculation for LLaMA-7B:

- $d \approx 4096$
- $r = 256$
- $2 \times 4096 \times 256 \approx 2.1$ million parameters per layer.

Given 32 layers, the total adapter parameter count is approximately $67$ million, which is roughly $1\%$ of the full LLaMA-7B model's size. 

| Model         | Dimension $d$ | Layers | $r$ | Params/Adapter | Total Adapter Params | Adapter % of Base |
|:--------------|:-------------:|:------:|:---:|:--------------:|:--------------------:|:-----------------:|
| LLaMA-7B      | 4096          | 32     | 256 | 2.1M           | 67M                  | 1%                |

This configuration enables most of the large model's capacity to be retained, while only a small fraction of the parameters are fine-tuned per task.

## 6. Empirical Performance and Ablation Studies

Adapter performance was validated on arithmetic and commonsense reasoning tasks using LLaMA models of different scales and various bottleneck sizes. Key findings:

- **Optimal Placement**:
  - Series after MLP: $59.5\%$
  - Series after attention: $58.3\%$
  - Parallel in MLP: $61.7\%$
- **Bottleneck Sweep (LLaMA-7B)**:

| $r$ | Series Adapter (%) | Parallel Adapter (%) |
|:---:|:-----------------:|:-------------------:|
| 64  | 57.3              | 59.1                |
| 128 | 59.2              | 60.8                |
| 256 | 59.5              | 61.7                |
| 512 | 56.6              | 58.0                |

- **Downstream Accuracy**:
  - Arithmetic (LLaMA-7B + series@MLP, $r=256$): $59.5\%$ (benchmark: GPT-3.5 zero-shot CoT $70.4\%$)
  - Arithmetic (LLaMA-13B + series): $63.0\%$
  - Commonsense (LLaMA-7B + series@MLP): $70.8\%$ (benchmark: ChatGPT zero-shot CoT $77.0\%$)
  - Commonsense (LLaMA-13B + series@MLP): $79.5\%$, exceeding ChatGPT ($77.0\%$)

These results demonstrate that with an adapter bottleneck $r=256$, approximately $1\%$ of parameters can be fine-tuned to recover $85-90\%$ of the performance of a $175$B-parameter model in math reasoning, whilst slightly exceeding strong baselines on commonsense reasoning [2304.01933].

## 7. Context, Significance, and Implications

LlamaMLP Adapter Layers exemplify parameter-efficient fine-tuning (PEFT) for large language models, reducing compute costs and storage requirements for model specialization. Their simple bottleneck architecture—with no gating and zero-initialized up-projection—facilitates rapid convergence and avoids interference with pre-trained model capabilities.

Empirical evidence suggests that adapter-based PEFT, as instantiated in LLaMA-MLP adapters, is a scalable alternative to full fine-tuning for LLMs, particularly when downstream model adaptation must be performed efficiently for multiple tasks or client settings [2304.01933].

Source: https://www.emergentmind.com/topics/llamamlp-adapter-layer