---
title: 'PaLM2-VAdapter: Efficient Vision-Language Adapter'
url: https://www.emergentmind.com/topics/palm2-vadapter
type: topic
---

# PaLM2-VAdapter: Efficient Vision-Language Adapter

PaLM2-VAdapter is a vision-language adapter framework that leverages a progressively aligned pretrained language model to bridge frozen vision encoders and large language models (LLMs), achieving accelerated convergence, improved parameter efficiency, and state-of-the-art zero-shot performance on multimodal tasks. The approach replaces the standard cross-attention-based perceiver resampler adapter with a two-stage paradigm in which a compact language model is first trained to interpret visual features and then used as an alignment module, providing a direct, supervised mapping into a frozen LLM. PaLM2-VAdapter yields significant improvements over prior architectures in visual understanding, multi-modal reasoning, and training efficiency while requiring substantially fewer parameters than leading large vision-language models [2402.10896].

## 1. System Architecture and Dataflow

PaLM2-VAdapter operates in the canonical frozen vision encoder → adapter → frozen LLM configuration, comparing two adapter schemes:

- **Baseline: Perceiver-Resampler Adapter**
  - **Vision Encoder:** A frozen, CoCa-pretrained Vision Transformer (ViT-B, ViT-L, ViT-g variants).
  - **Adapter:** Stacked cross-attention “perceiver resampler” layers (commonly depth 6).
  - **LLM Decoder:** Frozen PaLM 2 (1B or 8B parameters).
  - **Data Flow:**
    ```
    [Image]
      ↓
    ViT → {〈v₁,…,v_N〉}
      ↓
    Perceiver-Resampler
      ↓
    {q₁,…,q_L} (aligned visual tokens)
      ↓
    PaLM2 Decoder → generate text
    ```
- **PaLM2-VAdapter: Progressively Aligned Language Model as Adapter**
  - **Stage 1:** Fine-tuning a tiny PaLM2 (≈108M parameters) as a decoder on image-text pairs (no cross-attention resampling).
  - **Stage 2:** Freezing the tiny PaLM2, prepending a single-layer perceiver resampler, and training this composite adapter to map vision encoder features into a larger frozen PaLM2 decoder.
  - **Data Flow:**
    - *Stage 1 (Tiny LM as Decoder):*
      ```
      [Image] → ViT → Resampler₁₋layer → Tiny-PaLM2 → caption
      ```
    - *Stage 2 (LM as Adapter):*
      ```
      [Image] → ViT → Resampler₁₋layer → Tiny-PaLM2 → prefix for Large-PaLM2
      ```

The key technical distinction is that the tiny PaLM2, once trained to decode from visual tokens, aligns vision-language representations more efficiently and requires fewer adapter layers when serving as an intermediary to the larger LLM.

## 2. Mathematical Objectives and Layer Operations

Let $D = \{ (x_i, y_i) \}$ be the multimodal dataset of images or videos and corresponding text.

- $f_v(x) \in \mathbb{R}^{N \times d}$: Output of frozen vision encoder (patch embeddings).
- $g_{\mathrm{res}}(\cdot; \,\phi)$: Perceiver resampler mapping $\mathbb{R}^{N\times d} \rightarrow \mathbb{R}^{L \times d}$.
- $h_{\mathrm{tiny}}(\cdot; \theta)$: Tiny PaLM2 transformer.
- $h_{\mathrm{large}}(\cdot; \Theta)$: Large PaLM2 transformer, held frozen.

**Stage 1 (Tiny LM as Decoder; optimize $\theta$):**
\[
\mathcal{L}^{(1)}(\theta) = - \mathbb{E}_{(x,y)\sim D} \sum_{t=1}^{T} \log P\left(y_t \mid y_{<t},\,g_{\mathrm{res}}(f_v(x); \phi_0)\right)
\]
Only $\theta$ and optionally $\phi_0$ are updated.

**Stage 2 (Tiny LM + Resampler as Adapter; optimize $\phi$, optionally $\theta$):**
\[
\mathcal{L}^{(2)}(\phi,\,\theta) = - \mathbb{E}_{(x,y)\sim D} \sum_{t=1}^{T} \log P\left(y_t \mid y_{<t},\,h_{\mathrm{large}}\left(h_{\mathrm{tiny}}\left(g_{\mathrm{res}}(f_v(x);\,\phi),\,\theta\right)\right)\right)
\]

**Cross-attention in Perceiver Resampler:**
\[
Q = Q_{\mathrm{init}} W_q, \quad K = V W_k, \quad V' = V W_v
\]
\[
\mathrm{attn}(Q, K, V') = \mathrm{softmax}\left(\frac{Q K^\top}{\sqrt{d}}\right) V'
\]
Each layer adds a feedforward network and LayerNorm. This composite mapping, especially via the pretrained tiny LM, speeds up vision-language alignment.

## 3. Training Regimen and Datasets

- **Datasets**
  - *Images:* WebLI (≈30 million image-text pairs)
  - *Videos:* VTP and SMIT (sampled to 8 frames)

- **Optimization Hyperparameters**
  - Batch size: 2,048
  - Learning rate: $5 \times 10^{-4}$ (1,000 step warmup, linear decay)
  - Weight decay: $1 \times 10^{-4}$
  - Resolution: 288px, Patch size: 18px
  - Prompt: "Describe the following: <visual tokens> :"

**Stage 1** (Tiny LM as Decoder):
  - Initialize $\theta$ from pretrained tiny PaLM2 (108M).
  - Train ≈55,000 steps on combined image/video corpora.
  - Minimize $\mathcal{L}^{(1)}(\theta)$ by next-token teacher-forcing.

**Stage 2** (Adapter Training):
  - Freeze tiny LM, add/initialize 1-layer perceiver resampler, train ≈51,000 steps.
  - Minimize $\mathcal{L}^{(2)}(\phi, \theta)$ with the large PaLM2 frozen.

*Direct supervision using autoregressive cross-entropy loss enables faster convergence than contrastive or latent matching losses.*

## 4. Experimental Benchmarks and Comparisons

### Image Captioning (COCO, CIDEr)

| Adapter/Model   | ViT-B→1B CIDEr | ViT-L→1B CIDEr | ViT-L→VAdapter CIDEr |
|-----------------|----------------|---------------|---------------------|
| Perceiver-Res.  | 81.4           | 82.4          | —                   |
| PaLM2-VAdapter  | **83.0**       | —             | **89.6**            |

*PaLM2-VAdapter achieves higher CIDEr with fewer training steps (–76%).*

### Video Captioning (MSRVTT/VATEX, CIDEr)

| Variant                 | MSRVTT | VATEX |
|-------------------------|--------|-------|
| Baseline ViT-B          | 38.2   | —     |
| VAdapter ViT-B          | **42.1** | **38.3** |
| VAdapter ViT-g          | **45.6** | **51.2** |
| VAdapter 8B LM          | **47.7** | **53.0** |

### Visual QA (VQAv2, TextVQA, VizWiz, OKVQA; % Accuracy)

| VAdapter      | VQAv2 | TextVQA | VizWiz | OKVQA |
|---------------|-------|---------|--------|-------|
| 1B (ViT-g)    | 57.9  | 23.7    | 44.1   | 33.6  |
| 8B (ViT-g)    | **60.6**  | 24.8    | 43.7   | 40.9  |

### Parameter Efficiency

| Model                | Total Params | Trainable Params |
|----------------------|-------------|------------------|
| Flamingo-80B         | 80B         | 10.2B            |
| AnyMAL 15B           | 15B         | 0.1B             |
| PaLM2-VAdapter 8B    | 10.8B       | 0.13B            |

PaLM2-VAdapter variants operate with 120–130M adapter parameters, representing a 30–70% reduction vs. other state-of-the-art LVLMs.

### Qualitative and Behavioral Results

- Demonstrates accurate object counting and fine-grained reasoning about image and video content.
- Correctly responds to detailed queries such as “How many slices?” and “What is the man building?”

## 5. Ablation Studies and Insights

- **Perceiver Resampler Design:** Separate LayerNorm for queries and key/values yields superior CIDEr (46.8 vs. 38.4). Augmenting with FFN and time embedding increases CIDEr by 5–10 points. Depth six and hidden size/query dimension ≈768 optimal in baseline; with VAdapter, fewer layers suffice.
- **Progressive Pre-training:** Omitting adapter pretraining results in significantly lower downstream metrics (COCO: 79.2, VQAv2: 50.8). Including Stage 1 vision-language tuning increases scores (COCO: 83.0, VQAv2: 53.8). Language-only pretraining is less effective than multimodal.
- **Adapter Necessity:** A 1-layer perceiver outperforms attentional pooling plus tiny LM (CIDEr: 85.6 vs. 81.1); increasing perceiver layers beyond one can cause overfitting in the VAdapter context.
- **Convergence and Scaling:** VAdapter achieves 3× faster convergence (60k vs. 250k steps) and shows monotonic improvement as either vision backbone (ViT) or language model size increases, exceeding gains in the cross-attention baseline.

## 6. Significance and Comparative Position

PaLM2-VAdapter establishes that a progressively aligned, pretrained language model can function as an efficient, high-capacity vision-language adapter. This approach enables direct, supervised cross-modal alignment, leading to state-of-the-art zero-shot performance in both image and video captioning as well as question answering, with substantial gains in training efficiency. Empirically, it delivers higher accuracy and semantic fidelity than cross-attention resampler adapters, scales more effectively with increasing backbone size, and reduces required trainable parameters by 30–70% compared to prior large vision-language models. The findings support a paradigm shift favoring language-model-based adapters with progressive multimodal training for scalable and efficient vision-language modeling [2402.10896].

Source: https://www.emergentmind.com/topics/palm2-vadapter