MetaCLIP 2: Multilingual Vision-Language Model

• The paper presents a novel contrastive language–image pretraining framework that integrates over 50% non-English with English pairs to boost multilingual performance.
• It introduces scalable substring matching and language-specific metadata curation to balance head and tail concepts across 300+ languages.
• Empirical results show state-of-the-art zero-shot classification and retrieval on both English and multilingual benchmarks without resorting to translation.

MetaCLIP 2 is a contrastive language–image pretraining framework designed to address the curation and scaling challenges inherent in constructing foundation models from global, multilingual web data. It represents a systematic advance over prior approaches by providing an explicit recipe for combining over 50% non-English image–text pairs with English data, using purpose-built metadata, scalable substring matching, and numerically justified balancing. Notably, MetaCLIP 2 not only eliminates the classical "curse of multilinguality"—where performance on English tasks degrades as non-English data is added—but demonstrates mutual benefit for both English and non-English downstream benchmarks. The approach yields state-of-the-art results for both English and multilingual zero-shot classification and retrieval without recourse to proxy translation or non-public filtering systems (Chuang et al., 29 Jul 2025).

1. Motivation and Problem Statement

Earlier vision–language foundation models such as CLIP were trained on large-scale but predominantly English language image–text pairs. Attempts to incorporate multilingual web data have faced notable limitations:

• Absence of robust, scalable curation for non-English data; ad hoc filtering or translation-based pipelines are common but either introduce confounding variables or fail to respect cultural/language diversity.
• English performance typically declines as multilingual data is naively added, a phenomenon labeled the "curse of multilinguality," confounding cross-cultural and global deployment (Chuang et al., 29 Jul 2025).
• Previous works often neglected the long-tail distribution of visual-semantic concepts across languages, with an overrepresentation of high-resource languages and frequent semantic collapse in low-resource ones.

MetaCLIP 2 addresses these issues by constructing a pipeline that allows direct, balanced, and mutually beneficial integration of English and non-English data, preserving both coverage and performance.

2. Multilingual Metadata Construction and Language-Specific Curation

MetaCLIP 2 builds independent metadata sets for over 300 languages, significantly extending previous English-only approaches (Xu et al., 2023). The metadata for each language is assembled from resources such as:

• Wikipedia unigrams and bigrams,
• Wikipedia page titles,
• Multilingual WordNet,
• Additional curated or high-frequency tokens as available.

During data curation, each image–text pair is first processed by automatic language identification of the alt-text (caption). Substring matching is then performed between that alt-text and the appropriate language-specific metadata, using an Aho–Corasick algorithm for efficient, batched, large-scale substring search. This approach ensures that high-priority concepts from all supported languages are considered and that tail entries are not underrepresented.

Counts of matching entries are aggregated globally per language. To control long-tailed distributions and avoid overfitting to head concepts, a balancing strategy is applied: a per-language threshold $t_{\text{lang}}$ is derived such that a fixed percentage of data falls into the tail (empirically, 6%), paralleling the approach in (Xu et al., 2023), but generalized to hundreds of languages via inverse and direct mappings $t\rightarrow p$ and $p\rightarrow t$.

3. Scalable Data Balancing and Sampling

The data balancing algorithm comprises three main phases:

1. Language-Specific Matching: For each language, alt-texts are matched against the corresponding metadata, returning for every image–text pair a set of visual-semantic concept matches.
2. Threshold Computation: Global concept match statistics are used to set per-language thresholds $t_{\text{lang}}$ such that the proportion of head vs. tail concepts is comparable across languages, as dictated by the proportion found in English using English-only threshold $t_{\text{en}}$.
3. Sampling: For each metadata entry, if the occurrence count $C > t_{\text{lang}}$, only $t_{\text{lang}}$ examples are sampled (with probability $t_{\text{lang}}/C$); otherwise, all matching examples are retained. This yields a concept-balanced, language-aware dataset suitable for large-scale contrastive pretraining.

The implementation leverages efficiency optimizations (lazy metadata loading and accelerated substring search) to scale to billions of image–text pairs.

4. Training Framework and Model Scaling

MetaCLIP 2 retains core architectural ingredients of CLIP and MetaCLIP, such as QuickGELU activations, but introduces two crucial innovations:

• Multilingual Tokenizer: The text encoder uses a vocabulary (e.g. XLM-V) covering all included languages, enabling robust text feature extraction across diverse scripts and morphology.
• Scaled Training Budget and Model Capacity: The effective number of seen pairs is increased (from the 12.8B canonical value to 29B pairs) by raising the global batch size, ensuring that English representation is not overwhelmed by the increased presence of non-English data.

Model scaling ablations indicate that only sufficiently large architectures (e.g., ViT-H/14) are able to break the curse of multilinguality. Smaller architectures (e.g., ViT-L/14, ViT-B/32) still exhibit performance decay on English benchmarks if trained on worldwide data, corroborating scaling laws observed in LLMs.

5. Experimental Evaluation

MetaCLIP 2 reports comprehensive evaluations on both English and multilingual tasks:

Model	EN-Only ImageNet (%)	Multilingual Babel-ImageNet (%)	Multilingual Retrieval (XM3600, I2T) (%)	CVQA (%)
MetaCLIP (EN) ViT-H/14	80.5	(not reported)	(not reported)	(not reported)
MetaCLIP 2 (WW) ViT-H/14	81.3	50.2	64.3	57.4
mSigLIP	(lower)	(lower)	(lower)	(lower)

Performance on English-only ImageNet zero-shot classification improves from 80.5% (MetaCLIP EN-only ViT-H/14) to 81.3% (MetaCLIP 2 WW ViT-H/14). The model sets new state-of-the-art results on multilingual benchmarks: 50.2% on Babel-ImageNet, 57.4% on CVQA, and 64.3% on XM3600 image-to-text retrieval, without using translation or altering model architectures.

Ablation studies confirm that:

• Using language identification and language-specific balancing is essential for non-degradation of both English and multilingual results.
• Larger batch sizes and more seen pairs are necessary for models to handle the increased diversity of worldwide data.
• The XLM-V tokenizer yields the best multilingual performance.

6. Significance, Limitations, and Future Prospects

MetaCLIP 2 demonstrates that mutual benefit between English and non-English data is achievable given careful curation, per-language balancing, and sufficient model/data scale. By releasing open-source metadata, curation code, and recipes, it enables direct worldwide pretraining without opaque filtering or translation, preserving cultural and linguistic diversity.

A major implication is that the "curse of multilinguality" in contrastive pretraining is not fundamental, but rather contingent upon data/model scaling and balanced representation across languages. The work highlights the necessity for unbiased global benchmarks (e.g., Babel-ImageNet, CVQA, XM3600) and suggests future research directions including improved tokenization for low-resource languages and continued scaling of both data and model capacity.

A plausible implication is that with further advances in tokenization, metadata curation, and compute scaling, vision–LLMs could exhibit robust, equitable performance across nearly all major world languages and cultural contexts, moving beyond Western/English-centric paradigms (Chuang et al., 29 Jul 2025).