---
title: PaliGemma VLM Backbone
url: https://www.emergentmind.com/topics/paligemma-vision-language-backbone
type: topic
---

# PaliGemma VLM Backbone

PaliGemma is a family of open-weight Vision-Language Backbones (VLMs) that integrate a compute-optimal vision encoder with a high-capacity, decoder-only language model, designed for transfer across a wide variety of downstream vision-language tasks. Originating with a tightly-coupled architecture combining a SigLIP-So400m Vision Transformer and the Gemma-2 family of language models, the PaliGemma suite is characterized by modular design, multi-stage pretraining at various image resolutions, and demonstrated state-of-the-art results on diverse datasets [2412.03555][2407.07726].

## 1. Architectural Composition

PaliGemma relies on a modular backbone architecture consisting of three principal components:

- **SigLIP-So400m Vision Encoder**: A “shape-optimized” Vision Transformer, pre-trained on 400M image-text pairs with SigLIP’s sigmoid contrastive loss. Input images are partitioned into non-overlapping $14\times 14$ patches (at standard VLM resolutions of $224^2$, $448^2$, $896^2$), each patch yielding a $D_v$-dimensional token ($D_v\approx 768$ for PaliGemma 2; $D_v=1536$ in the original PaliGemma).
    - At $224^2$: 256 tokens; $448^2$: 1024; $896^2$: 4096.
    - SoViT-based: $L=24$ blocks, $H=12$ heads, MLP dimension $4\times D_v$ in PaliGemma 2; $L=12$, $H=16$, $D=1536$ in PaliGemma 1.
- **Gemma 2 Language Model Family**: Decoder-only, causal autoregressive Transformers ranging from 2B to 27B parameters, parameterized by increasing embedding and hidden dimension: $D_L \in \{768, 1024, 1280, 1536\}$, $L\in\{24,32,40,48\}$, $H\in\{12,16,20,24\}$.
- **Cross-modal Projection (Connector)**: A single linear transformation $W_{\text{proj}} \in \mathbb{R}^{D_v \times D_L}$ aligning vision token embeddings to the language model’s input space; optionally followed by LayerNorm. This component prepends visual tokens to the language prompt in “prefill” mode, without explicit cross-attention or auxiliary fusion modules.

This pipeline enables unified autoregressive modeling over a mixed visual-textual token sequence, simplifying architecture and transfer [2412.03555][2407.07726].

## 2. Multi-stage Training Paradigm

The development of PaliGemma employs a three-stage training procedure:

1. **Stage 0 – Unimodal Pretraining**: SigLIP-So400m and Gemma-2 are pretrained separately. SigLIP uses the contrastive loss,
   $$
   L_{\operatorname{SigLIP}} = - \sum_{i=1}^B \log\frac{\exp(f_v(x_i)\cdot f_t(t_i)/\tau)}{\sum_{j=1}^B \exp(f_v(x_i)\cdot f_t(t_j)/\tau)}.
   $$
   Gemma 2 uses standard causal next-token cross-entropy.

2. **Stage 1 – Joint Multimodal Pretraining ($224^2$)**: 1B multimodal examples (image captioning, VQA, OCR, detection recast as seq2seq, etc.), all parameters trainable. The loss is autoregressive, applied over concatenated vision and text tokens:
   $$
   L_{\text{CE}} = -\sum_{t=1}^T \log p(y_t|y_{<t}, \text{image}).
   $$
   Optimizer: Adam ($\beta_1=0.9, \beta_2=0.999$); learning rate for PaliGemma 2 models: $2\times 10^{-5}$ (3B), $1\times 10^{-5}$ (10B), $0.5\times 10^{-5}$ (28B); batch size $\sim 128{-}256$.

3. **Stage 2 – High-Resolution Multimodal Pretraining ($448^2$, $896^2$)**: 50M samples at $448^2$, 10M at $896^2$, up-weighting tasks sensitive to high resolution (e.g., fine-grained OCR).

4. **Stage 3 – Task-specific Fine-tuning**: No SigLIP contrastive term; all tasks reframed as conditional language modeling with an autoregressive decoding head. Fine-tuning draws from $224^2$ or $448^2$/$896^2$ checkpoints as appropriate.

This curriculum integrates transfer-oriented multimodal objectives and staged resolution scaling [2412.03555][2407.07726].

## 3. Unified Transfer and Fine-tuning Framework

All downstream tasks leverage the same backbone and input interface, eschewing additional task-specific heads. The vision tokens (after linear projection) are always prepended to the tokenized task prompt. Notable tasks covered in PaliGemma 2 include:

- **OCR (ICDAR’15, Total-Text):** Joint detection and recognition of word boxes and text.
- **Table Structure Recognition (PubTabNet, FinTabNet):** Output HTML with positional tokens for localization.
- **Molecular Structure Recognition:** Image to SMILES string mapping (MolScribe protocol).
- **Optical Music Score Recognition:** Image to **kern strings (measured by CER/SER/LER).
- **Long Fine-grained Captioning (DOCCI):** Generation of multi-sentence descriptive captions.
- **Radiology Report Generation (MIMIC-CXR):** “INDICATIONS:” prefix to “FINDINGS:” plus “IMPRESSION:” generation.
- **Spatial Reasoning (VSR) and Classic VLM Tasks:** Binary QA, VQA, referring expressions, video QA, segmentation, etc.

Fine-tuning schedules are hyperparameter-swept per task: for instance, OCR (15k steps, batch 256, LR grid $1\text{e-}5$ to $1\text{e-}4$), tables (15k steps, LR $1\text{e-}4$, padded/square-resize), molecules (30k steps, label smoothing 0.1), radiology (8 epochs, LR $5\times10^{-6}$, no regularizer), and so on. No adapters or frozen components by default; all parameters are usually updated [2412.03555].

## 4. Benchmarking and Empirical Findings

Performance analyses span standard VLM, OCR, structured document, molecular, spatial reasoning, and specialized domain tasks, revealing the following quantitative highlights (all from [2412.03555]):

| Task                       | Dataset             | PaliGemma 2 SOTA (size@res) | Prior Best (method) |
|----------------------------|---------------------|-----------------------------|---------------------|
| OCR (F1)                   | ICDAR’15/Total-Text | 75.9 (3B@$896^2$)           | 74.5/72.4 (HTS)     |
| Table Structure (TEDS)     | FinTabNet           | 0.83 (SOTA)                 | 0.80 (prior)        |
| Molecule Recognition (EM)  | --                  | 94.8 (10B@$448^2$)          | 93.8 (MolScribe)    |
| Music Scores (CER)         | GrandStaff          | 1.6% (3B@$896^2$)           | 3.9% (prior)        |
| DOCCI Non-Entailment (%)   | --                  | 20.3 (10B@$448^2$)          | lower (better)      |
| Radiology (RadGraph F1)    | MIMIC-CXR           | 29.5 (10B@$896^2$)          | 20.5 (Flamingo-CXR) |
| Detection (COCO mAP)       | COCO                | 43.6 (10B@$896^2$)          | Comparable (pix2seq)|
  
Ablations illuminate three task clusters: resolution-sensitive (OCR, document understanding), size-sensitive (multilingual, complex reasoning), and hybrid. Diminishing returns set in for model size beyond $10$B parameters on many tasks. Fine-tuning optimal learning rates decrease for larger models (e.g., $1\text{e-}5$ at 3B falls to $3\text{e-}6$ at 10B, $1\text{e-}6$ at 28B).

PaliGemma 1-to-2 transition yields average absolute improvements of $+0.65$ points @ $224^2$ and $+0.85$ @ $448^2$ across 30+ benchmarks [2412.03555].

## 5. Design Choices and Ablation Insights

Key architectural and procedural findings include:

- **Linear Connector**: A single linear projection from vision to language space is as effective as an MLP, both in pretraining perplexity and downstream accuracy.
- **No Cross-attention**: All fusion is effected via concatenation and a single projection layer; explicit cross-modal attention layers proved unnecessary for strong transfer.
- **Prefix-LM Masking**: Ablations confirm that prefix-LM masking (bidirectional vision/prompt, causal suffix prediction) gives superior results relative to other attention strategies.
- **Native High-Resolution Pretraining**: Stage 2 upcycling to $448/896$ px is critical; window-based “fallback” inference lags by $3{-}5\%$.
- **End-to-end Training**: Full fine-tuning outperforms freezing either vision or language backbone.
- **New Token Initialization**: Standard Gaussian (σ=0.02) init surpasses mean-embedding init post-stabilization.

These choices yield models with maximal transfer performance per parameter and per floating-point operation, underlining the impact of both architectural modularity and careful curriculum design [2412.03555][2407.07726].

## 6. Context, Scope, and Evolution

PaliGemma reflects a trend toward decomposable, “connector”-based VLM backbones capable of broad transfer by virtue of heavy data-driven pretraining and architectural simplicity. The transition from PaliGemma 1 [2407.07726] to PaliGemma 2 [2412.03555] expands task coverage (e.g., structured tables, molecules, music, radiology), model scale (Gemma 2 2B–27B), and extends systematic exploration of resolution/model size tradeoffs. Both iterations leverage pure autoregressive training (next-token cross-entropy over mixed vision+text), with all cross-modal alignment arising from generative supervision.

This design sets new state-of-the-art results across open benchmark landscapes for models of comparable (and often much larger) size, emphasizing that resolution scaling and backbone tuning synergistically determine transfer performance. The backbone paradigm introduced by PaliGemma provides a template for further research in efficient, unified VLMs for varied, low-shot downstream adaptation [2412.03555][2407.07726].

Source: https://www.emergentmind.com/topics/paligemma-vision-language-backbone