---
title: 'SigLIP-400M: Multilingual Vision-Language Model'
url: https://www.emergentmind.com/topics/siglip-400m-architecture
type: topic
---

# SigLIP-400M: Multilingual Vision-Language Model

SigLIP-400M (“So400m”) is an advanced multilingual vision–language encoder developed as part of the SigLIP 2 model family, designed to combine the representational capacity of a large Vision Transformer backbone with language-agnostic text processing and a unified multimodal training recipe. The architecture systematically integrates prior methodologies—captioning-based pretraining, self-supervision, attention-based pooling, and online data curation—to optimize semantic alignment, localization performance, and scalability across multiple resolutions and languages [2502.14786]. The nomenclature “400M” denotes the approximate parameter count of the vision encoder; the total checkpoint contains ~450M parameters, with a design emphasis on robust performance in zero-shot and transfer tasks as well as fairness and resource efficiency.

## 1. Model Backbone and Parameterization

The architecture employs a standard Vision Transformer (ViT) with significant scaling in both depth and width. The vision encoder consists of 24 sequential transformer blocks (each with hidden size $d=1024$, feed-forward dimension $d_{ff}=4096$, 16-headed self-attention with per-head width of 64) and patch size of 14 (“So/14”). The linear patch embedding projects each $14\times14\times3$ image patch into $\mathbb{R}^{1024}$. Learned 2D positional encodings are adopted, and a Multi-head Attention Pooling (MAP) head—an additional transformer block—pools the patch sequence into a single embedding.

The text encoder mirrors the vision architecture with 24 transformer layers, $d=1024$, and $d_{ff}=4096$. Tokenization is performed using the multilingual Gemma v2 with 256k wordpieces, supporting up to 64 input tokens with separate learned 1D positional embeddings. MAP pooling similarly reduces the token sequence to a single 1024-dimensional vector.

| Component          | Configuration/Value                | Notes                        |
|--------------------|------------------------------------|------------------------------|
| Vision backbone    | 24-layer ViT, $d=1024$, patch 14   | So/14, MAP head, 390M params |
| Text backbone      | Gemma v2 + 24-layer transformer    | $d=1024$, 60M params         |
| Pooling            | MAP head (multi-head attn)         | $h=16$, output $\in \mathbb{R}^{1024}$ |
| Total parameters   | $\approx$450M                      | 400M (vision), 60M (text)    |

MAP pooling and preservation of unpooled patch/token embeddings enable dense feature extraction for segmentation and localization [2502.14786].

## 2. Cross-Modal Objectives and Loss Functions

Training interleaves four complementary objectives, optimizing both coarse- and fine-grained multimodal correspondence:

1. **Sigmoid image–text loss** (SigLIP): A batchwise cross-pairwise logistic regression loss for $B^2$ image–text pairs; positive labels for matching pairs, negative otherwise:
   $$
   L_{sig} = - \sum_{i=1}^B \sum_{j=1}^B \left[ y_{ij}\log\sigma\left(\frac{v_i \cdot t_j}{\tau}\right) + (1-y_{ij})\log(1-\sigma\left(\frac{v_i \cdot t_j}{\tau}\right)) \right]
   $$
   where $v_i, t_j$ are the MAP embeddings, $\sigma(\cdot)$ is the sigmoid, and $\tau$ is a learned temperature.

2. **Captioning-based pretraining loss** (LocCa): A decoder predicts full-image captions, referring-expression box coordinates, and grounded captions using cross-entropy:
   $$
   L_{locca} = - \sum_{t=1}^L \log p(w_t|X, w_{<t})
   $$
   Decoder attaches to the last-layer unpooled patch sequence $X$.

3. **Self-distillation** (local-to-global, SILC): Teacher (EMA of student) produces full-image embedding $\varphi_{global}$; student produces $\varphi_{local}^{(k)}$ on 8 random crops; $L_2$ regression aligns student and teacher:
   $$
   L_{sd} = \sum_{k=1}^8 \|f(\varphi_{local}^{(k)}) - f(\varphi_{global})\|_2^2
   $$

4. **Masked-patch prediction** (TIPS): 50% of student patch embeddings masked and regressed to teacher features:
   $$
   L_{mp} = \sum_{p \in M} \| h_{masked}(p) - h_{full}(p) \|_2^2
   $$

For the last 20% of training, the aggregate loss is $L = L_{sig} + L_{locca} + \lambda_{sd}L_{sd} + \lambda_{mp}L_{mp}$ with $\lambda_{sd}=0.5$ and $\lambda_{mp}=0.125$ [2502.14786].

## 3. Architectural Innovations: Resolution, Pooling, and Dense Features

Native-aspect and variable-resolution (NaFlex) capability is implemented by resizing images so that both axes are multiples of 14, preserving aspect ratio within a distortion bound. Patch sequences with $N' < max\_seq$ are padded with learned [PAD] tokens, tracked by a masking mechanism. Positional embeddings are bilinearly interpolated for the new resolution grid.

Dense feature preservation is achieved by retaining unpooled last-layer patch embeddings, facilitating per-patch outputs for segmentation, depth estimation, and phrase grounding tasks. The LocCa decoder, when attached to these tokens, yields cross-attention maps usable for spatial localization.

| Feature             | Realization                    | Context                         |
|---------------------|-------------------------------|----------------------------------|
| NaFlex              | Input resize, patchwise pad    | Supports variable image size     |
| Dense features      | Pre-MAP token extraction       | Enables dense prediction/loc     |
| Pooling             | MAP with learned query         | Improves global alignment        |

## 4. Multilingual Data Processing and Fairness Objectives

SigLIP-400M trains on a WebLI-derived dataset with 10B image–alt-text pairs spanning 109 languages. The corpus sampling weights English at 90% and all other languages at 10%, a configuration chosen to balance accurate English performance with multilingual coverage, consistent with the approach of Pouget-Bosc et al. (2024). 

Fairness is enforced by pre-filtering alt-texts using the clip-de-bias technique (Alabdulmohsin et al., 2024), eliminating both first-order (gender) and second-order (occupation) bias artifacts without adding model parameters; all debiasing is in data preprocessing.

## 5. Computational Complexity and Resolution–Performance Tradeoffs

Inference compute depends on input resolution due to the quadratic scaling with patch sequence length. Performance improves sublinearly with increased resolution, with diminishing returns beyond 384px:

- 256px (196 tokens): $~$25 GFLOPs, zero-shot ImageNet $\sim$83.6%
- 384px (756 tokens): $~$40 GFLOPs, $\sim$84.1%
- 1024px (5,376 tokens): $~$100 GFLOPs, $\sim$84.3%

NaFlex enables efficiency–quality trade-off by supporting arbitrary resolutions with minimal overhead (bilinear interpolation + attention mask) [2502.14786].

## 6. Summary Table

| Component             | Value                                 | Notes                       |
|-----------------------|---------------------------------------|-----------------------------|
| Vision backbone       | ViT-So/14 (24 layers, $d=1024$)       | Patch size 14x14, MAP head  |
| Attention heads       | 16                                    | Head size 64                |
| MLP dim              | $d_{ff}=4096$                          |                             |
| Vision params         | $\approx 390$M                        | Excludes LocCa decoder      |
| Text encoder          | Gemma (multilingual) + 24L transf     | $d=1024$, $d_{ff}=4096$     |
| Text params           | $\approx 60$M                         |                             |
| Losses                | Sigmoid, LocCa, self-distil, TIPS     | Eqs (1)–(4)                 |
| Data                  | 10B img–alttext, 109 languages        | 90% EN, 10% non-EN          |
| Debiasing             | clip-de-bias (pre)                    | No param change             |
| Optimizer             | AdamW, lr=1e-3, wd=1e-4               | 40B ex. training            |
| NaFlex                | Supported (var. aspect/resolution)    |                             |
| Zero-shot INet@256    | ~83.6%                                |                             |
| Compute@256px         | ~25 GFLOPs                            |                             |

## 7. Context and Significance within Vision–Language Modeling

SigLIP-400M operationalizes recent unification trends in large-scale multimodal modeling: full ViT vision encoders, transformer-based text towers, attention pooling for cross-modal retrieval, and dense-token features for downstream perception tasks. The adoption of LocCa and TIPS losses strengthens dense representation and spatial reasoning, enabling spatial phrase grounding and region-based captioning. The So400m model further distinguishes itself by achieving strong multilingual generalization and fairness on the basis of both corpus diversity and preemptive debiasing, consistent with fairness-by-design strategies. The architecture demonstrates close adherence to ViT scaling laws and offers a configuration space that allows fine-grained performance–cost tradeoffs, largely through NaFlex and MAP mechanisms [2502.14786].

A plausible implication is that SigLIP-400M and its training paradigm represent a converging point for future vision–language pretraining: broad linguistic coverage, spatial generalization, dense feature usability, and architectural modularity supporting both resource- and performance-constrained deployment scenarios.

Source: https://www.emergentmind.com/topics/siglip-400m-architecture