PaLM2-VAdapter: Efficient Vision-Language Adapter

• PaLM2-VAdapter is a vision-language adapter that leverages a two-stage training approach to align frozen vision encoders with large language models for improved efficiency and zero-shot performance.
• It replaces conventional cross-attention resampler adapters with a compact, progressively aligned language model, resulting in faster convergence and fewer trainable parameters.
• Empirical benchmarks demonstrate significant gains in image/video captioning and visual QA, underscoring its practical impact on multimodal reasoning tasks.

PaLM2-VAdapter is a vision-language adapter framework that leverages a progressively aligned pretrained LLM to bridge frozen vision encoders and LLMs, achieving accelerated convergence, improved parameter efficiency, and state-of-the-art zero-shot performance on multimodal tasks. The approach replaces the standard cross-attention-based perceiver resampler adapter with a two-stage paradigm in which a compact LLM is first trained to interpret visual features and then used as an alignment module, providing a direct, supervised mapping into a frozen LLM. PaLM2-VAdapter yields significant improvements over prior architectures in visual understanding, multi-modal reasoning, and training efficiency while requiring substantially fewer parameters than leading large vision-LLMs (Xiao et al., 2024).

1. System Architecture and Dataflow

PaLM2-VAdapter operates in the canonical frozen vision encoder → adapter → frozen LLM configuration, comparing two adapter schemes:

• Baseline: Perceiver-Resampler Adapter
  • Vision Encoder: A frozen, CoCa-pretrained Vision Transformer (ViT-B, ViT-L, ViT-g variants).
  • Adapter: Stacked cross-attention “perceiver resampler” layers (commonly depth 6).
  • LLM Decoder: Frozen PaLM 2 (1B or 8B parameters).
  • Data Flow:
  • $g_{\mathrm{res}}(\cdot; \,\phi)$0
• PaLM2-VAdapter: Progressively Aligned LLM as Adapter
  • Stage 1: Fine-tuning a tiny PaLM2 (≈108M parameters) as a decoder on image-text pairs (no cross-attention resampling).
  • Stage 2: Freezing the tiny PaLM2, prepending a single-layer perceiver resampler, and training this composite adapter to map vision encoder features into a larger frozen PaLM2 decoder.
  • Data Flow:
  • Stage 1 (Tiny LM as Decoder):
  • $g_{\mathrm{res}}(\cdot; \,\phi)$1
  • Stage 2 (LM as Adapter):
  • $g_{\mathrm{res}}(\cdot; \,\phi)$2

The key technical distinction is that the tiny PaLM2, once trained to decode from visual tokens, aligns vision-language representations more efficiently and requires fewer adapter layers when serving as an intermediary to the larger LLM.

2. Mathematical Objectives and Layer Operations

Let $D = \{ (x_i, y_i) \}$ be the multimodal dataset of images or videos and corresponding text.

• $f_v(x) \in \mathbb{R}^{N \times d}$: Output of frozen vision encoder (patch embeddings).
• $g_{\mathrm{res}}(\cdot; \,\phi)$: Perceiver resampler mapping $$\mathbb{R}^{N\times d} \rightarrow \mathbb{R}^{L \times d}$$.
• $h_{\mathrm{tiny}}(\cdot; \theta)$: Tiny PaLM2 transformer.
• $h_{\mathrm{large}}(\cdot; \Theta)$: Large PaLM2 transformer, held frozen.

Stage 1 (Tiny LM as Decoder; optimize $\theta$):

$$\mathcal{L}^{(1)}(\theta) = - \mathbb{E}_{(x,y)\sim D} \sum_{t=1}^{T} \log P\left(y_t \mid y_{<t},\,g_{\mathrm{res}}(f_v(x); \phi_0)\right)$$

Only $\theta$ and optionally $\phi_0$ are updated.

Stage 2 (Tiny LM + Resampler as Adapter; optimize $f_v(x) \in \mathbb{R}^{N \times d}$0, optionally $f_v(x) \in \mathbb{R}^{N \times d}$1):

$f_v(x) \in \mathbb{R}^{N \times d}$2

Cross-attention in Perceiver Resampler:

$f_v(x) \in \mathbb{R}^{N \times d}$3

$f_v(x) \in \mathbb{R}^{N \times d}$4

Each layer adds a feedforward network and LayerNorm. This composite mapping, especially via the pretrained tiny LM, speeds up vision-language alignment.

3. Training Regimen and Datasets

• Datasets
  • Images: WebLI (≈30 million image-text pairs)
  • Videos: VTP and SMIT (sampled to 8 frames)
• Optimization Hyperparameters
  • Batch size: 2,048
  • Learning rate: $f_v(x) \in \mathbb{R}^{N \times d}$5 (1,000 step warmup, linear decay)
  • Weight decay: $f_v(x) \in \mathbb{R}^{N \times d}$6
  • Resolution: 288px, Patch size: 18px
  • Prompt: "Describe the following: <visual tokens> :"

Stage 1 (Tiny LM as Decoder): - Initialize $f_v(x) \in \mathbb{R}^{N \times d}$7 from pretrained tiny PaLM2 (108M). - Train ≈55,000 steps on combined image/video corpora. - Minimize $f_v(x) \in \mathbb{R}^{N \times d}$8 by next-token teacher-forcing.

Stage 2 (Adapter Training): - Freeze tiny LM, add/initialize 1-layer perceiver resampler, train ≈51,000 steps. - Minimize $f_v(x) \in \mathbb{R}^{N \times d}$9 with the large PaLM2 frozen.

Direct supervision using autoregressive cross-entropy loss enables faster convergence than contrastive or latent matching losses.

4. Experimental Benchmarks and Comparisons

Image Captioning (COCO, CIDEr)

Adapter/Model	ViT-B→1B CIDEr	ViT-L→1B CIDEr	ViT-L→VAdapter CIDEr
Perceiver-Res.	81.4	82.4	—
PaLM2-VAdapter	83.0	—	89.6

PaLM2-VAdapter achieves higher CIDEr with fewer training steps (–76%).

Video Captioning (MSRVTT/VATEX, CIDEr)

Variant	MSRVTT	VATEX
Baseline ViT-B	38.2	—
VAdapter ViT-B	42.1	38.3
VAdapter ViT-g	45.6	51.2
VAdapter 8B LM	47.7	53.0

Visual QA (VQAv2, TextVQA, VizWiz, OKVQA; % Accuracy)

VAdapter	VQAv2	TextVQA	VizWiz	OKVQA
1B (ViT-g)	57.9	23.7	44.1	33.6
8B (ViT-g)	60.6	24.8	43.7	40.9

Parameter Efficiency

Model	Total Params	Trainable Params
Flamingo-80B	80B	10.2B
AnyMAL 15B	15B	0.1B
PaLM2-VAdapter 8B	10.8B	0.13B

PaLM2-VAdapter variants operate with 120–130M adapter parameters, representing a 30–70% reduction vs. other state-of-the-art LVLMs.

Qualitative and Behavioral Results

• Demonstrates accurate object counting and fine-grained reasoning about image and video content.
• Correctly responds to detailed queries such as “How many slices?” and “What is the man building?”

5. Ablation Studies and Insights

• Perceiver Resampler Design: Separate LayerNorm for queries and key/values yields superior CIDEr (46.8 vs. 38.4). Augmenting with FFN and time embedding increases CIDEr by 5–10 points. Depth six and hidden size/query dimension ≈768 optimal in baseline; with VAdapter, fewer layers suffice.
• Progressive Pre-training: Omitting adapter pretraining results in significantly lower downstream metrics (COCO: 79.2, VQAv2: 50.8). Including Stage 1 vision-language tuning increases scores (COCO: 83.0, VQAv2: 53.8). Language-only pretraining is less effective than multimodal.
• Adapter Necessity: A 1-layer perceiver outperforms attentional pooling plus tiny LM (CIDEr: 85.6 vs. 81.1); increasing perceiver layers beyond one can cause overfitting in the VAdapter context.
• Convergence and Scaling: VAdapter achieves 3× faster convergence (60k vs. 250k steps) and shows monotonic improvement as either vision backbone (ViT) or LLM size increases, exceeding gains in the cross-attention baseline.

6. Significance and Comparative Position

PaLM2-VAdapter establishes that a progressively aligned, pretrained LLM can function as an efficient, high-capacity vision-language adapter. This approach enables direct, supervised cross-modal alignment, leading to state-of-the-art zero-shot performance in both image and video captioning as well as question answering, with substantial gains in training efficiency. Empirically, it delivers higher accuracy and semantic fidelity than cross-attention resampler adapters, scales more effectively with increasing backbone size, and reduces required trainable parameters by 30–70% compared to prior large vision-LLMs. The findings support a paradigm shift favoring language-model-based adapters with progressive multimodal training for scalable and efficient vision-language modeling (Xiao et al., 2024).