---
title: 'Mass Mining: WikiMatrix and CCMatrix'
url: https://www.emergentmind.com/topics/mass-mining-wikimatrix-ccmatrix
type: topic
---

# Mass Mining: WikiMatrix and CCMatrix

Mass mining refers to the automatic extraction of parallel sentences from large-scale multilingual corpora using language-agnostic sentence embeddings and efficient all-pairs search methodologies. This approach enables the creation of vast bitext resources such as WikiMatrix and CCMatrix, which are widely used for neural machine translation (NMT), cross-lingual retrieval, and related tasks. Both frameworks employ the LASER sentence encoder and a margin-based criterion for identifying high-quality translation pairs in comparable or monolingual corpora, with WikiMatrix sourcing from Wikipedia and CCMatrix from the Common Crawl web corpus [1907.05791, 1911.04944].

## 1. Data Sources and Preprocessing

Mass mining frameworks such as WikiMatrix and CCMatrix depend on large multilingual text collections, sophisticated normalization pipelines, and aggressive deduplication strategies to ensure input data quality.

**WikiMatrix** processes CirrusSearch Wikipedia dumps in over 300 languages, with 182 retained after deduplication and language-ID filtering. Sentence segmentation is performed using SegTok (for 24 languages) or language-specific regex; exact duplicates (approximately 25% boilerplate) are removed. FastText-based sentence-level language ID eliminates content whose detected language does not match the Wikipedia edition, resulting in 595M sentences across 182 languages (e.g., English: 134M, German: 51M, Spanish: 25.2M) [1907.05791].

**CCMatrix** mines from ten monthly snapshots of Common Crawl, totaling 32.7B unique sentences in 38 high-resource languages. Preprocessing involves:
- Removing boilerplate (deduplication eliminates ~70% of content),
- Language identification (document and sentence levels) via fastText,
- Perplexity filtering with Wikipedia-trained language models,
- Sentence splitting by language rules,
- Further sentence-level deduplication in blocks of ~50M,
yielding, for example, English: 8.7B, Russian: 3.0B, Japanese: 2.9B sentences [1911.04944].

| Source       | Languages | Sentences        | Main Preprocessing Steps        |
|--------------|-----------|------------------|---------------------------------|
| Wikipedia    | 182       | 595M             | Deduplication, LID, SegTok      |
| Common Crawl | 38        | 32.7B            | Deduplication, LID, Perplexity  |

## 2. Multilingual Sentence Embeddings and Matching

Both frameworks use the LASER encoder (Artetxe & Schwenk, 2019), a sequence-to-sequence model operating over a shared 50k BPE vocabulary for 93 languages. This encoder produces 1,024-dimensional vectors for each sentence, with representation defined as max-pooling over hidden states [1907.05791, 1911.04944].

Given two sentence embeddings $u, v \in \mathbb{R}^{1024}$, cosine similarity
$$
s(u, v) = \frac{u \cdot v}{\|u\| \cdot \|v\|}
$$
serves as the base metric for sentence similarity. However, both systems adopt margin-based scoring to control for variance in embedding density and enable robust cross-lingual mining [1907.05791].

## 3. Margin-Based Bitext Mining

Extraction of parallel sentences is performed via all-pairs search in the embedding space, without pivoting through English. For a candidate pair $(x, y)$,
the margin score is defined as
$$
\mathrm{margin}(x, y) = \frac{\cos(x, y)}{
    \frac{1}{2k}\sum_{z\in \mathrm{NN}_k(x)} \cos(x, z) +
    \frac{1}{2k}\sum_{z\in \mathrm{NN}_k(y)} \cos(y, z)
}
$$
where $\mathrm{NN}_k(x)$ and $\mathrm{NN}_k(y)$ are the $k$ nearest neighbors of $x$ and $y$, respectively, in the other language. WikiMatrix uses $k=4$; CCMatrix adopts $k=16$ to better accommodate multiple valid translations [1907.05791, 1911.04944].

High-efficiency GPU-based approximate k-NN search with FAISS and product quantization reduces memory and compute demands, enabling searches across tens of millions to billions of sentences per language [1907.05791, 1911.04944].

Candidate pairs are scored in both directions (max-mining), ranked by margin, and greedily aligned while enforcing a 1:1 sentence mapping, discarding already-aligned sentences. Margin thresholds are empirically tuned for optimal precision-recall tradeoff, with τ ≈ 1.02–1.04 for WikiMatrix and τ = 1.06 for CCMatrix [1907.05791, 1911.04944].

## 4. Filtering, Quality Control, and Release Statistics

Several post-processing heuristics and metrics are applied:
- **WikiMatrix** employs length-ratio filtering (accepting pairs with len(x)/len(y) $\in$ [1/1.5, 1.5]), pre-embedding language ID, and releases all pairs with margin ≥1.02 [1907.05791].
- **CCMatrix** performs only LID and margin-based filtering, omitting additional sentence-length or LM filtering after extraction [1911.04944].

Final bitext extraction volumes are as follows:
- **WikiMatrix:** 135M parallel sentences across 1,620 language pairs, with 34M aligned with English (e.g., de–en: 2.3M, en–fr: 8.5M, non-English: ru–uk 2.5M, ca–es 1.6M, ja–ko 222K) [1907.05791].
- **CCMatrix:** 4.5B sentence pairs (margin>1.06), including 661M English alignments and, for example, fr–en: 94.1M, ru–en: 72.4M, de–nl: 33.2M. 20 language pairs exceed 30M pairs, 112 above 10M [1911.04944].

## 5. Benchmarking and Evaluation

Extracted corpora are validated through extensive NMT experiments:
- **WikiMatrix** trains 1,886 Transformer baseline systems (5-layer encoder/decoder, $d_{model}=512$) exclusively on mined bitexts for pairs with ≥25k sentences, evaluated on the TED talks corpus. Sample BLEU scores: Es→En 35.8, De→En 21.9, Fr→En 32.6, Ja→Ko 17.9, Ru→Uk 28.1, En→Hi 25.7. Wikipedia-mined bitexts surpass Europarl on de–en and de–fr with equivalent sentence counts and provide BLEU improvements when combined [1907.05791].
- **CCMatrix** supports MT training for 702 language-pairs on the TED test set, with an average BLEU of 16.3 (with English: 26.9). For WMT’19 news translation, single systems using only CCMatrix mined pairs attain De→En 47.4 BLEU (+3.8 over baseline) and near parity with ensemble/back-translation systems. On low-resource pairs (e.g., Ru–Ja), CCMatrix outperforms previous WAT'19 best submissions [1911.04944].

## 6. Strengths, Limitations, and Applications

Strengths include robust all-pairs coverage (including non-English low-resource pairs), scalability across both curated and noisy web data, and a margin criterion that generalizes across language families and corpora sizes [1907.05791, 1911.04944]. Immediately usable outputs support massively multilingual NMT, cross-lingual retrieval, bilingual lexicon induction, and semantic search.

Limitations are observed in domain mismatch (Wikipedia/web style differs from oral/dialogue domains), lower embedding quality for underrepresented languages, substantial computational/storage requirements, and the constraint of 1:1 sentence alignments. Margin thresholds may require pair-specific tuning for optimal results, which is not always feasible without held-out alignment data [1907.05791, 1911.04944].

## 7. Comparison of WikiMatrix and CCMatrix

Both methodologies are anchored in LASER-derived multilingual sentence embeddings and bidirectional margin-based candidate mining, but differ fundamentally in corpus scope and extraction targets.

| Aspect         | WikiMatrix                       | CCMatrix                      |
|----------------|----------------------------------|-------------------------------|
| Source         | Wikipedia (curated)             | Common Crawl (web/mined)      |
| Language pairs | 1,620 (all-pairs, 85+ languages)| 100+ (mostly English-pivot)   |
| Sentences      | 135M                            | 4.5B                          |
| Coverage       | Broad, includes dialects        | Massive, high-volume pairs    |
| Data structure | Article-based, structured       | Noisy web, domain-mixed       |

WikiMatrix enables all-pairs mining, especially for language pairs without prior public bitexts, while CCMatrix offers superior volume for English-pivot alignments and many high-resource language pairs [1907.05791, 1911.04944].

## References

- Schwenk et al., “WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia” [1907.05791]
- Schwenk et al., “CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB” [1911.04944]

Source: https://www.emergentmind.com/topics/mass-mining-wikimatrix-ccmatrix