Mass Mining: WikiMatrix and CCMatrix

• Mass mining is the automatic extraction of parallel sentences from vast multilingual datasets using language-agnostic embeddings and all-pairs search.
• WikiMatrix and CCMatrix employ LASER encoders and margin-based scoring to ensure robust cross-lingual matching and high-quality bitexts.
• These frameworks enable scalable neural machine translation and cross-lingual retrieval by mining billions of sentence pairs from sources like Wikipedia and Common Crawl.

Mass mining refers to the automatic extraction of parallel sentences from large-scale multilingual corpora using language-agnostic sentence embeddings and efficient all-pairs search methodologies. This approach enables the creation of vast bitext resources such as WikiMatrix and CCMatrix, which are widely used for neural machine translation (NMT), cross-lingual retrieval, and related tasks. Both frameworks employ the LASER sentence encoder and a margin-based criterion for identifying high-quality translation pairs in comparable or monolingual corpora, with WikiMatrix sourcing from Wikipedia and CCMatrix from the Common Crawl web corpus (Schwenk et al., 2019, Schwenk et al., 2019).

1. Data Sources and Preprocessing

Mass mining frameworks such as WikiMatrix and CCMatrix depend on large multilingual text collections, sophisticated normalization pipelines, and aggressive deduplication strategies to ensure input data quality.

WikiMatrix processes CirrusSearch Wikipedia dumps in over 300 languages, with 182 retained after deduplication and language-ID filtering. Sentence segmentation is performed using SegTok (for 24 languages) or language-specific regex; exact duplicates (approximately 25% boilerplate) are removed. FastText-based sentence-level language ID eliminates content whose detected language does not match the Wikipedia edition, resulting in 595M sentences across 182 languages (e.g., English: 134M, German: 51M, Spanish: 25.2M) (Schwenk et al., 2019).

CCMatrix mines from ten monthly snapshots of Common Crawl, totaling 32.7B unique sentences in 38 high-resource languages. Preprocessing involves:

• Removing boilerplate (deduplication eliminates ~70% of content),
• Language identification (document and sentence levels) via fastText,
• Perplexity filtering with Wikipedia-trained LLMs,
• Sentence splitting by language rules,
• Further sentence-level deduplication in blocks of ~50M, yielding, for example, English: 8.7B, Russian: 3.0B, Japanese: 2.9B sentences (Schwenk et al., 2019).

Source	Languages	Sentences	Main Preprocessing Steps
Wikipedia	182	595M	Deduplication, LID, SegTok
Common Crawl	38	32.7B	Deduplication, LID, Perplexity

2. Multilingual Sentence Embeddings and Matching

Both frameworks use the LASER encoder (Artetxe & Schwenk, 2019), a sequence-to-sequence model operating over a shared 50k BPE vocabulary for 93 languages. This encoder produces 1,024-dimensional vectors for each sentence, with representation defined as max-pooling over hidden states (Schwenk et al., 2019, Schwenk et al., 2019).

Given two sentence embeddings $u, v \in \mathbb{R}^{1024}$, cosine similarity

$s(u, v) = \frac{u \cdot v}{\|u\| \cdot \|v\|}$

serves as the base metric for sentence similarity. However, both systems adopt margin-based scoring to control for variance in embedding density and enable robust cross-lingual mining (Schwenk et al., 2019).

3. Margin-Based Bitext Mining

Extraction of parallel sentences is performed via all-pairs search in the embedding space, without pivoting through English. For a candidate pair $(x, y)$, the margin score is defined as

$$\mathrm{margin}(x, y) = \frac{\cos(x, y)}{ \frac{1}{2k}\sum_{z\in \mathrm{NN}_k(x)} \cos(x, z) + \frac{1}{2k}\sum_{z\in \mathrm{NN}_k(y)} \cos(y, z) }$$

where $\mathrm{NN}_k(x)$ and $\mathrm{NN}_k(y)$ are the $k$ nearest neighbors of $x$ and $y$, respectively, in the other language. WikiMatrix uses $k=4$; CCMatrix adopts $s(u, v) = \frac{u \cdot v}{\|u\| \cdot \|v\|}$0 to better accommodate multiple valid translations (Schwenk et al., 2019, Schwenk et al., 2019).

High-efficiency GPU-based approximate k-NN search with FAISS and product quantization reduces memory and compute demands, enabling searches across tens of millions to billions of sentences per language (Schwenk et al., 2019, Schwenk et al., 2019).

Candidate pairs are scored in both directions (max-mining), ranked by margin, and greedily aligned while enforcing a 1:1 sentence mapping, discarding already-aligned sentences. Margin thresholds are empirically tuned for optimal precision-recall tradeoff, with τ ≈ 1.02–1.04 for WikiMatrix and τ = 1.06 for CCMatrix (Schwenk et al., 2019, Schwenk et al., 2019).

4. Filtering, Quality Control, and Release Statistics

Several post-processing heuristics and metrics are applied:

• WikiMatrix employs length-ratio filtering (accepting pairs with len(x)/len(y) $s(u, v) = \frac{u \cdot v}{\|u\| \cdot \|v\|}$1 [1/1.5, 1.5]), pre-embedding language ID, and releases all pairs with margin ≥1.02 (Schwenk et al., 2019).
• CCMatrix performs only LID and margin-based filtering, omitting additional sentence-length or LM filtering after extraction (Schwenk et al., 2019).

Final bitext extraction volumes are as follows:

• WikiMatrix: 135M parallel sentences across 1,620 language pairs, with 34M aligned with English (e.g., de–en: 2.3M, en–fr: 8.5M, non-English: ru–uk 2.5M, ca–es 1.6M, ja–ko 222K) (Schwenk et al., 2019).
• CCMatrix: 4.5B sentence pairs (margin>1.06), including 661M English alignments and, for example, fr–en: 94.1M, ru–en: 72.4M, de–nl: 33.2M. 20 language pairs exceed 30M pairs, 112 above 10M (Schwenk et al., 2019).

5. Benchmarking and Evaluation

Extracted corpora are validated through extensive NMT experiments:

• WikiMatrix trains 1,886 Transformer baseline systems (5-layer encoder/decoder, $s(u, v) = \frac{u \cdot v}{\|u\| \cdot \|v\|}$2) exclusively on mined bitexts for pairs with ≥25k sentences, evaluated on the TED talks corpus. Sample BLEU scores: Es→En 35.8, De→En 21.9, Fr→En 32.6, Ja→Ko 17.9, Ru→Uk 28.1, En→Hi 25.7. Wikipedia-mined bitexts surpass Europarl on de–en and de–fr with equivalent sentence counts and provide BLEU improvements when combined (Schwenk et al., 2019).
• CCMatrix supports MT training for 702 language-pairs on the TED test set, with an average BLEU of 16.3 (with English: 26.9). For WMT’19 news translation, single systems using only CCMatrix mined pairs attain De→En 47.4 BLEU (+3.8 over baseline) and near parity with ensemble/back-translation systems. On low-resource pairs (e.g., Ru–Ja), CCMatrix outperforms previous WAT'19 best submissions (Schwenk et al., 2019).

6. Strengths, Limitations, and Applications

Strengths include robust all-pairs coverage (including non-English low-resource pairs), scalability across both curated and noisy web data, and a margin criterion that generalizes across language families and corpora sizes (Schwenk et al., 2019, Schwenk et al., 2019). Immediately usable outputs support massively multilingual NMT, cross-lingual retrieval, bilingual lexicon induction, and semantic search.

Limitations are observed in domain mismatch (Wikipedia/web style differs from oral/dialogue domains), lower embedding quality for underrepresented languages, substantial computational/storage requirements, and the constraint of 1:1 sentence alignments. Margin thresholds may require pair-specific tuning for optimal results, which is not always feasible without held-out alignment data (Schwenk et al., 2019, Schwenk et al., 2019).

7. Comparison of WikiMatrix and CCMatrix

Both methodologies are anchored in LASER-derived multilingual sentence embeddings and bidirectional margin-based candidate mining, but differ fundamentally in corpus scope and extraction targets.

Aspect	WikiMatrix	CCMatrix
Source	Wikipedia (curated)	Common Crawl (web/mined)
Language pairs	1,620 (all-pairs, 85+ languages)	100+ (mostly English-pivot)
Sentences	135M	4.5B
Coverage	Broad, includes dialects	Massive, high-volume pairs
Data structure	Article-based, structured	Noisy web, domain-mixed

WikiMatrix enables all-pairs mining, especially for language pairs without prior public bitexts, while CCMatrix offers superior volume for English-pivot alignments and many high-resource language pairs (Schwenk et al., 2019, Schwenk et al., 2019).

• Schwenk et al., “WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia” (Schwenk et al., 2019)
• Schwenk et al., “CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB” (Schwenk et al., 2019)