PaliGemma VLM Backbone

• PaliGemma is a family of modular vision-language backbones that integrates a compute-optimal SigLIP-So400m encoder with Gemma-2 language models across diverse tasks.
• It employs a multi-stage training paradigm with progressive resolution scaling, enabling robust performance on OCR, table recognition, molecular, and radiology tasks.
• The system’s unified fine-tuning framework uses a simple linear cross-modal connector and autoregressive decoding to streamline transfer learning.

PaliGemma is a family of open-weight Vision-Language Backbones (VLMs) that integrate a compute-optimal vision encoder with a high-capacity, decoder-only LLM, designed for transfer across a wide variety of downstream vision-language tasks. Originating with a tightly-coupled architecture combining a SigLIP-So400m Vision Transformer and the Gemma-2 family of LLMs, the PaliGemma suite is characterized by modular design, multi-stage pretraining at various image resolutions, and demonstrated state-of-the-art results on diverse datasets (Steiner et al., 2024, Beyer et al., 2024).

1. Architectural Composition

PaliGemma relies on a modular backbone architecture consisting of three principal components:

• SigLIP-So400m Vision Encoder: A “shape-optimized” Vision Transformer, pre-trained on 400M image-text pairs with SigLIP’s sigmoid contrastive loss. Input images are partitioned into non-overlapping $14\times 14$ patches (at standard VLM resolutions of $224^2$, $448^2$, $896^2$), each patch yielding a $D_v$-dimensional token ($D_v\approx 768$ for PaliGemma 2; $D_v=1536$ in the original PaliGemma).
  • At $224^2$: 256 tokens; $448^2$: 1024; $896^2$: 4096.
  • SoViT-based: $224^2$0 blocks, $224^2$1 heads, MLP dimension $224^2$2 in PaliGemma 2; $224^2$3, $224^2$4, $224^2$5 in PaliGemma 1.
• Gemma 2 LLM Family: Decoder-only, causal autoregressive Transformers ranging from 2B to 27B parameters, parameterized by increasing embedding and hidden dimension: $224^2$6, $224^2$7, $224^2$8.
• Cross-modal Projection (Connector): A single linear transformation $224^2$9 aligning vision token embeddings to the LLM’s input space; optionally followed by LayerNorm. This component prepends visual tokens to the language prompt in “prefill” mode, without explicit cross-attention or auxiliary fusion modules.

This pipeline enables unified autoregressive modeling over a mixed visual-textual token sequence, simplifying architecture and transfer (Steiner et al., 2024, Beyer et al., 2024).

2. Multi-stage Training Paradigm

The development of PaliGemma employs a three-stage training procedure:

1. Stage 0 – Unimodal Pretraining: SigLIP-So400m and Gemma-2 are pretrained separately. SigLIP uses the contrastive loss,

$448^2$0

Gemma 2 uses standard causal next-token cross-entropy.

2. Stage 1 – Joint Multimodal Pretraining ($448^2$1): 1B multimodal examples (image captioning, VQA, OCR, detection recast as seq2seq, etc.), all parameters trainable. The loss is autoregressive, applied over concatenated vision and text tokens:

$448^2$2

Optimizer: Adam ($448^2$3); learning rate for PaliGemma 2 models: $448^2$4 (3B), $448^2$5 (10B), $448^2$6 (28B); batch size $448^2$7.

1. Stage 2 – High-Resolution Multimodal Pretraining ($448^2$8, $448^2$9): 50M samples at $896^2$0, 10M at $896^2$1, up-weighting tasks sensitive to high resolution (e.g., fine-grained OCR).
2. Stage 3 – Task-specific Fine-tuning: No SigLIP contrastive term; all tasks reframed as conditional language modeling with an autoregressive decoding head. Fine-tuning draws from $896^2$2 or $896^2$3/$896^2$4 checkpoints as appropriate.

This curriculum integrates transfer-oriented multimodal objectives and staged resolution scaling (Steiner et al., 2024, Beyer et al., 2024).

3. Unified Transfer and Fine-tuning Framework

All downstream tasks leverage the same backbone and input interface, eschewing additional task-specific heads. The vision tokens (after linear projection) are always prepended to the tokenized task prompt. Notable tasks covered in PaliGemma 2 include:

• OCR (ICDAR’15, Total-Text): Joint detection and recognition of word boxes and text.
• Table Structure Recognition (PubTabNet, FinTabNet): Output HTML with positional tokens for localization.
• Molecular Structure Recognition: Image to SMILES string mapping (MolScribe protocol).
• Optical Music Score Recognition: Image to **kern strings (measured by CER/SER/LER).
• Long Fine-grained Captioning (DOCCI): Generation of multi-sentence descriptive captions.
• Radiology Report Generation (MIMIC-CXR): “INDICATIONS:” prefix to “FINDINGS:” plus “IMPRESSION:” generation.
• Spatial Reasoning (VSR) and Classic VLM Tasks: Binary QA, VQA, referring expressions, video QA, segmentation, etc.

Fine-tuning schedules are hyperparameter-swept per task: for instance, OCR (15k steps, batch 256, LR grid $896^2$5 to $896^2$6), tables (15k steps, LR $896^2$7, padded/square-resize), molecules (30k steps, label smoothing 0.1), radiology (8 epochs, LR $896^2$8, no regularizer), and so on. No adapters or frozen components by default; all parameters are usually updated (Steiner et al., 2024).

4. Benchmarking and Empirical Findings

Performance analyses span standard VLM, OCR, structured document, molecular, spatial reasoning, and specialized domain tasks, revealing the following quantitative highlights (all from (Steiner et al., 2024)):

Task	Dataset	PaliGemma 2 SOTA (size@res)	Prior Best (method)
OCR (F1)	ICDAR’15/Total-Text	75.9 (3B@$896^2$9)	74.5/72.4 (HTS)
Table Structure (TEDS)	FinTabNet	0.83 (SOTA)	0.80 (prior)
Molecule Recognition (EM)	--	94.8 (10B@$D_v$0)	93.8 (MolScribe)
Music Scores (CER)	GrandStaff	1.6% (3B@$D_v$1)	3.9% (prior)
DOCCI Non-Entailment (%)	--	20.3 (10B@$D_v$2)	lower (better)
Radiology (RadGraph F1)	MIMIC-CXR	29.5 (10B@$D_v$3)	20.5 (Flamingo-CXR)
Detection (COCO mAP)	COCO	43.6 (10B@$D_v$4)	Comparable (pix2seq)

Ablations illuminate three task clusters: resolution-sensitive (OCR, document understanding), size-sensitive (multilingual, complex reasoning), and hybrid. Diminishing returns set in for model size beyond $D_v$5B parameters on many tasks. Fine-tuning optimal learning rates decrease for larger models (e.g., $D_v$6 at 3B falls to $D_v$7 at 10B, $D_v$8 at 28B).

PaliGemma 1-to-2 transition yields average absolute improvements of $D_v$9 points @ $D_v\approx 768$0 and $D_v\approx 768$1 @ $D_v\approx 768$2 across 30+ benchmarks (Steiner et al., 2024).

5. Design Choices and Ablation Insights

Key architectural and procedural findings include:

• Linear Connector: A single linear projection from vision to language space is as effective as an MLP, both in pretraining perplexity and downstream accuracy.
• No Cross-attention: All fusion is effected via concatenation and a single projection layer; explicit cross-modal attention layers proved unnecessary for strong transfer.
• Prefix-LM Masking: Ablations confirm that prefix-LM masking (bidirectional vision/prompt, causal suffix prediction) gives superior results relative to other attention strategies.
• Native High-Resolution Pretraining: Stage 2 upcycling to $D_v\approx 768$3 px is critical; window-based “fallback” inference lags by $D_v\approx 768$4.
• End-to-end Training: Full fine-tuning outperforms freezing either vision or language backbone.
• New Token Initialization: Standard Gaussian (σ=0.02) init surpasses mean-embedding init post-stabilization.

These choices yield models with maximal transfer performance per parameter and per floating-point operation, underlining the impact of both architectural modularity and careful curriculum design (Steiner et al., 2024, Beyer et al., 2024).

6. Context, Scope, and Evolution

PaliGemma reflects a trend toward decomposable, “connector”-based VLM backbones capable of broad transfer by virtue of heavy data-driven pretraining and architectural simplicity. The transition from PaliGemma 1 (Beyer et al., 2024) to PaliGemma 2 (Steiner et al., 2024) expands task coverage (e.g., structured tables, molecules, music, radiology), model scale (Gemma 2 2B–27B), and extends systematic exploration of resolution/model size tradeoffs. Both iterations leverage pure autoregressive training (next-token cross-entropy over mixed vision+text), with all cross-modal alignment arising from generative supervision.

This design sets new state-of-the-art results across open benchmark landscapes for models of comparable (and often much larger) size, emphasizing that resolution scaling and backbone tuning synergistically determine transfer performance. The backbone paradigm introduced by PaliGemma provides a template for further research in efficient, unified VLMs for varied, low-shot downstream adaptation (Steiner et al., 2024, Beyer et al., 2024).