Multilingual SigLIP in Dual-Encoder Models
- Multilingual SigLIP is a dual-encoder vision-language model that aligns images and texts across multiple languages using a unified embedding space and a multilingual tokenizer.
- It employs a SigLIP sigmoid loss combined with self-distillation and masked prediction to enhance semantic understanding, localization, and dense feature extraction.
- Empirical results show improved retrieval benchmarks and fairness metrics, balancing strong English performance with significant multilingual gains on datasets like XM3600.
Searching arXiv for the cited papers and closely related SigLIP work. Searching arXiv for "(Tschannen et al., 20 Feb 2025) SigLIP 2 multilingual vision-language encoders". Multilingual SigLIP denotes SigLIP-style vision-language encoders that align a single shared image encoder and text encoder across multiple languages in a joint image-text space. In the current literature, the most direct formulation is "SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features," which presents multilingual dual encoders trained with the SigLIP sigmoid alignment mechanism, a multilingual tokenizer, and a multilingual WebLI mixture (Tschannen et al., 20 Feb 2025). A related but distinct line is represented by "Behind Maya: Building a Multilingual Vision LLM," which uses a pretrained multilingual SigLIP vision tower inside a LLaVA-style multimodal instruction model rather than training a SigLIP-style multilingual encoder end to end (Alam et al., 13 May 2025).
1. Conceptual scope and taxonomy
Multilingual SigLIP is most precisely understood as a multilingual extension of the SigLIP dual-encoder paradigm. In that paradigm, a vision transformer image tower and a text transformer text tower are trained so that matching image-text pairs score highly and non-matching pairs score poorly. The multilingual extension preserves that encoder-centric formulation while broadening the text space to many languages through tokenizer and data changes rather than through language-specific branches or a separate multilingual alignment head (Tschannen et al., 20 Feb 2025).
A central distinction is between multilingual encoders and multilingual instruction VLMs. SigLIP 2 belongs to the former category: it is a dual-encoder system whose principal outputs are aligned image and text representations for retrieval, zero-shot classification, and transfer. Maya belongs to the latter category: it is a projector-based, LLaVA-style multimodal instruction model in which a multilingual SigLIP encoder supplies frozen visual features to a multilingual decoder LLM (Alam et al., 13 May 2025).
| System | Architecture class | Relevance to multilingual SigLIP |
|---|---|---|
| SigLIP 2 | Dual-encoder vision-language encoder | Direct |
| Maya | LLaVA-style projector-based VLM | Indirect |
This distinction matters because multilinguality can arise from different mechanisms. In SigLIP 2, multilinguality is intrinsic to the encoder pair and the shared embedding space. In Maya, multilinguality is primarily data-driven and decoder-mediated: images are mapped into a multilingual LLM space, and cross-language behavior is inherited from translated supervision and Aya-23’s multilingual prior. A plausible implication is that multilingual SigLIP, in the strict sense, refers to multilingual encoder pretraining rather than merely using a multilingual SigLIP checkpoint as a frozen vision tower.
2. Encoder design and multilingual parameterization
SigLIP 2 follows SigLIP architecturally. It uses a ViT image tower, a text transformer text tower, and MAP-head pooling for both modalities. Fixed-resolution variants use the standard ViT architecture with learned positional embedding. Text length is set to 64 tokens. The released checkpoints span ViT-B (86M), L (303M), So400m (400M), and g (1B), with one explicit scale-specific caveat: the g-sized vision encoder is paired with an So400m-sized text encoder (Tschannen et al., 20 Feb 2025).
The principal multilingual-specific architectural change is the tokenizer. SigLIP 2 uses the multilingual Gemma tokenizer with vocabulary size 256k, and text is lowercased before tokenization. The paper emphasizes backward compatibility: users can swap model weights and the tokenizer, which is now multilingual. No multilingual-specific image branch, language-conditioned projection, or separate alignment head is described. Multilinguality is therefore realized with minimal architectural disruption.
SigLIP 2 also introduces NaFlex variants that support multiple resolutions and preserve native aspect ratio with a single checkpoint for varying sequence lengths. Standard fixed-resolution variants include 224, 256, 384, and 512 px-style configurations, with corresponding sequence lengths such as 196, 256, 576, 729, and 1024. The paper states that NaFlex has potential to improve document understanding and reports retrieval gains on OCR-, document-, and screen-focused datasets, although it does not isolate a multilingual-language benchmark specifically attributable to NaFlex (Tschannen et al., 20 Feb 2025).
The Maya system illustrates a different use of multilingual SigLIP components. It uses [google](https://www.emergentmind.com/topics/service-weaver-google)/siglip-base-patch16-256-multilingual as a frozen vision encoder and Aya-23 8B as the multilingual LLM. The multimodal interface follows LLaVA 1.5: an image is encoded as
then mapped through a trainable projection matrix
and the projected embeddings are consumed together with textual instructions by the multilingual decoder. The projection module is a 2-layer MLP with GELU; 4-layer and 8-layer MLPs were also tested, but the 2-layer MLP achieved the lowest training loss (Alam et al., 13 May 2025).
3. Training recipe, objectives, and multilingual data
The core alignment mechanism in SigLIP 2 remains the SigLIP sigmoid loss. The paper describes it conceptually as forming binary classification problems from every image embedding and every text embedding in the mini-batch and training the embeddings to classify matching and non-matching pairs via logistic regression. This is the defining difference from CLIP-style softmax contrastive training and the defining sense in which SigLIP 2 is a multilingual SigLIP model rather than merely a multilingual VLM with a SigLIP component (Tschannen et al., 20 Feb 2025).
SigLIP 2 extends that core objective with additional stages. In the first step of pretraining, it combines SigLIP with a LocCa-style decoder using equal loss weight. The decoder is trained on image captioning, automatic referring expression prediction, and grounded captioning. Starting at 80% of training completion, the recipe adds self-distillation from SILC and masked prediction from TIPS. The self-distillation stage uses 1 global teacher view and 8 local student views. The masked prediction stage replaces 50% of student image patches with mask tokens. The model-size-dependent reweighting factor for the added objectives is 0.25 for B, 0.5 for L, 1.0 for So400m, and 0.5 for g (Tschannen et al., 20 Feb 2025).
The multilingual data design is equally central. SigLIP 2 trains on WebLI, described as 10 billion images and 12 billion alt-texts covering 109 languages. The sampling is deliberately asymmetric: 90% of training image-text pairs are from English web pages and 10% from non-English web pages. The stated rationale is to strike a good balance between quality on English and multilingual vision-language benchmarks. The paper also applies filtering techniques to mitigate data biases in representation and association with respect to sensitive attributes (Tschannen et al., 20 Feb 2025).
The paper does not state that SigLIP 2 uses synthetic translations, machine-translated captions, multilingual paraphrase augmentation, or OCR-specific multilingual corpora as a separate named source. Its multilinguality comes from the naturally multilingual WebLI mixture, the multilingual tokenizer, and the broader training recipe rather than from an explicit translation pipeline.
By contrast, Maya’s multilingual supervision is explicitly synthetic at the corpus-construction level. Its multilingual pretraining dataset is derived from the original English LLaVA pretraining dataset of 550K samples, expanded into eight languages—English, Chinese, French, Spanish, Russian, Hindi, Japanese, and Arabic—yielding 550K samples per language and 4.4 million samples in total, equally distributed across all eight languages. The translation pipeline includes initial translation using Google Translate, back-translation, final human review with help from Claude, Gemini, and GPT models, prompt-template optimization, and large-scale batch translation with Aya 35B. Prompt engineering uses BLEU and N-gram evaluation, and “Preamble 6” is selected because it yields the best BLEU / n-gram fidelity, typically around 0.4–0.5 BLEU per N-gram view; the framework reportedly achieves more than 0.47 average BLEU scores in seven languages (Alam et al., 13 May 2025).
4. Retrieval performance and the English–multilingual trade-off
The primary multilingual benchmark for SigLIP 2 is Crossmodal-3600 (XM3600), covering 36 languages, evaluated with Recall@1 for text-to-image and image-to-text retrieval. The reported results establish the main trade-off that defines multilingual SigLIP in practice: SigLIP 2 substantially exceeds original SigLIP on multilingual retrieval, while only slightly lagging behind mSigLIP, which performs substantially worse on English-focused benchmarks (Tschannen et al., 20 Feb 2025).
| Model / setting | XM3600 | XM3600 |
|---|---|---|
| SigLIP B/16, 256 px | 22.5 | 29.9 |
| SigLIP 2 B/16, 256 px | 40.7 | 51.0 |
| SigLIP L/16, 256 px | 30.9 | 40.1 |
| SigLIP 2 L/16, 256 px | 46.5 | 56.5 |
| mSigLIP So/16, 256 px | 50.0 | 62.8 |
| SigLIP 2 So/16, 256 px | 48.1 | 57.5 |
| SigLIP 2 g/16, 384 px | 48.6 | 57.9 |
These values show two distinct empirical patterns. First, the jump from original SigLIP to SigLIP 2 is large at multiple scales. Second, the dedicated multilingual baseline mSigLIP still edges out SigLIP 2 on XM3600 in the So/16, 256 px comparison, but SigLIP 2 is explicitly presented as the better-balanced model because it retains substantially stronger English-side behavior (Tschannen et al., 20 Feb 2025).
The same pattern appears in transfer to VLMs. In the appendix’s PaliGemma-style setup, SigLIP 2 improves multilingual indicators such as COCO-35L and XM3600 over SigLIP when used as a frozen vision encoder. For the large-sized comparison, COCO-35L (avg34) increases from 109.9 to 112.2, COCO-35L (en) from 136.7 to 139.4, XM3600 (avg35) from 39.0 to 39.7, and XM3600 (en) from 77.7 to 79.1 (Tschannen et al., 20 Feb 2025). This suggests that the multilingual encoder gains are not confined to pure retrieval evaluation.
Resolution yields smaller but consistent incremental improvements. For example, B/16 XM3600 text-to-image retrieval increases from 40.7 at 256 px to 41.2 at 384 px and 41.4 at 512 px. The paper therefore attributes multilingual gains primarily to the data mixture, multilingual tokenizer, and training recipe rather than to resolution alone.
5. Semantic grounding, dense features, fairness, and cultural diversity
SigLIP 2 is not framed solely as a multilingual retrieval model. It is also presented as improving semantic understanding, localization, and dense features, and these improvements have direct downstream relevance because multilingual vision-language behavior often depends on OCR sensitivity, region grounding, and culturally diverse semantics (Tschannen et al., 20 Feb 2025).
On frozen-feature dense prediction, the reported gains are substantial. For So/14, 224 px, PASCAL segmentation improves from 72.0 to 77.1, ADE20k from 37.6 to 41.8, NYUv2 depth RMSE from 0.576 to 0.493, and NAVI depth from 0.083 to 0.067. For So/14, 384 px, PASCAL improves from 73.8 to 78.1, ADE20k from 40.8 to 45.4, NYUv2 depth from 0.563 to 0.466, and normals NYUv2 from 24.1 to 23.0 (Tschannen et al., 20 Feb 2025). The paper ties these gains to self-distillation, masked prediction, and decoder-based pretraining rather than to multilinguality alone.
Localization gains are similarly large. In referring expression comprehension, L, seq 256 improves on RefCOCO val from 67.33 to 86.04, and So, seq 729 improves on RefCOCO testA from 74.12 to 91.13. The paper attributes this mainly to decoder-based pretraining. It also notes that SigLIP 2 is only outperformed by LocCa in one comparison and hypothesizes that this might be due to SigLIP 2 being pretrained on multilingual data, whereas LocCa is trained on text only from English web sites (Tschannen et al., 20 Feb 2025). A plausible implication is that multilingual pretraining can slightly dilute English-specialized localization performance even while broadening coverage.
The fairness and cultural-diversity results are particularly important for multilingual SigLIP. On Dollar Street, GeoDE, and GLDv2, SigLIP 2 improves both few-shot and zero-shot performance. For L/16, 256 px, Dollar Street 10-shot rises from 18.8 to 26.8, GeoDE country 10-shot from 22.1 to 34.5, GeoDE region 10-shot from 36.2 to 44.4, Dollar Street 0-shot from 52.1 to 55.2, GLDv2 0-shot from 56.7 to 64.5, and GeoDE 0-shot from 93.6 to 94.9 (Tschannen et al., 20 Feb 2025).
The strongest fairness result is on representation bias. For L/16, 256 px, representation bias drops from 35.5 in SigLIP to 7.3 in SigLIP 2. For So400m/14, 224 px, it drops from 33.3 to 7.4, and for So400m/16, 256 px, mSigLIP records 37.3 while SigLIP 2 records 7.2. The paper is careful to note that subgroup benefits in performance broken down by income level or geographic region can be very minor or absent in some cases. The fairness story is therefore strongest for representation bias rather than uniformly across every subgroup metric (Tschannen et al., 20 Feb 2025).
6. Multilingual SigLIP as a backbone for multilingual VLMs: the Maya case
The Maya model is a useful boundary case because it demonstrates how multilingual SigLIP can function as a component in a multilingual VLM while remaining distinct from multilingual SigLIP pretraining proper. Maya is derived from LLaVA 1.5, uses the multilingual SigLIP base patch16-256 vision encoder, and pairs it with Aya-23 8B, an 8B-parameter multilingual LLM with an 8K context window trained across 23 languages. The supported Maya dataset covers 8 of Aya’s 23 languages (Alam et al., 13 May 2025).
Architecturally, Maya is not a CLIP- or SigLIP-style dual encoder. It is a projector-based multimodal instruction model. The vision encoder and LLM are frozen during the projector pretraining stage, and the trainable component is the projection matrix. Images are cropped to 256 × 256. Projector pretraining is run on 8 × H100 GPUs (80GB each), with per-device batch size 32, global batch size 256, learning rate , a cosine scheduler, and duration of approximately 20 hours. Instruction tuning uses the PALO 150K dataset on 8 × H100 GPUs, with per-device batch size 16, global batch size 128, and duration of approximately 48 hours. The paper states that LoRA gave suboptimal results and was discarded in favor of full finetuning, although the wording around frozen versus unfrozen modules remains somewhat ambiguous (Alam et al., 13 May 2025).
Its evaluation also differs from encoder-centric multilingual SigLIP work. Maya is assessed mainly on multilingual instruction-following and VQA-style tasks rather than on retrieval. On the PALO multilingual benchmark, adapted from LLaVA-Bench (In-the-Wild), Maya (8B) is evaluated across English, Chinese, French, Spanish, Russian, Japanese, Arabic, Hindi, Bengali, and Urdu. It reports an average of 60.4, compared with 46.9 for LLaVA-7B, 57.7 for PALO-7B, 49.9 for LLaVA-13B, and 61.9 for PALO-13B. The paper emphasizes especially large gains on Arabic and Hindi relative to English-centric LLaVA baselines: Arabic 63.4 versus 38.9 for LLaVA-7B and 37.2 for LLaVA-13B, and Hindi 64.0 versus 29.4 and 27.8, respectively (Alam et al., 13 May 2025).
For multilingual SigLIP research, the significance of Maya is therefore selective. It does not provide a multilingual sigmoid-loss training recipe, image-text retrieval benchmarks, pairwise batch construction, or temperature/logit-scaling details. Its value lies instead in three areas: a practical recipe for multilingualizing LLaVA pretraining data, an example of integrating a multilingual SigLIP backbone into a multilingual instruction VLM, and a downstream multilingual evaluation protocol that extends beyond English-centric instruction models (Alam et al., 13 May 2025).
7. Limitations, boundaries, and open directions
The current multilingual SigLIP literature defines multilinguality through strong but bounded design choices rather than through exhaustive language coverage or uniform evaluation. SigLIP 2 states that WebLI covers 109 languages, but it does not enumerate them in the provided text. Its principal multilingual benchmark, XM3600, covers 36 languages, yet the paper does not present a standard multilingual zero-shot classification benchmark with language-by-language results. It also does not specify multilingual prompting templates or prompt-engineering guidance for zero-shot usage (Tschannen et al., 20 Feb 2025).
The English–multilingual trade-off remains central. SigLIP 2 is designed to reduce that trade-off rather than eliminate it. On XM3600, mSigLIP still slightly exceeds SigLIP 2 in at least one key comparison, while SigLIP 2 performs substantially better on English-focused benchmarks. This makes SigLIP 2 a balanced multilingual encoder rather than a purely multilingual-optimized one. If the sole objective is maximal multilingual retrieval on XM3600, mSigLIP may remain competitive or slightly better; if the objective is one encoder for English and multilingual workloads, SigLIP 2 is the more general recipe (Tschannen et al., 20 Feb 2025).
The Maya case clarifies a second boundary: multilingual SigLIP should not be conflated with any multilingual VLM that imports a SigLIP checkpoint. Maya uses a multilingual SigLIP encoder, but its training objective is LLaVA-style projector pretraining and multilingual instruction tuning rather than SigLIP’s pairwise sigmoid image-text loss. Its strongest contribution is multilingual data construction and VLM adaptation, not multilingual encoder pretraining mechanics (Alam et al., 13 May 2025).
Taken together, these results position multilingual SigLIP as a specific encoder paradigm built from three coupled ingredients: a multilingual tokenizer, a multilingual data mixture, and a SigLIP-style alignment objective enriched by auxiliary supervision. The available evidence indicates that this combination yields large multilingual retrieval gains over original SigLIP, better cultural-diversity and representation-bias behavior, and stronger transfer as a frozen vision encoder, while preserving much more English-side strength than earlier dedicated multilingual variants (Tschannen et al., 20 Feb 2025).
