--- title: 'Samanantar: Parallel Corpus for Indic NLP' url: https://www.emergentmind.com/topics/samanantar type: topic --- # Samanantar: Parallel Corpus for Indic NLP Samanantar is the largest publicly available parallel corpus collection for English and 11 Indic languages, providing a substantial resource for machine translation (MT) and multilingual NLP research targeting the Indian subcontinent. Combining both pre-existing and newly mined sentence pairs at massive scale, Samanantar supports the development and benchmarking of both statistical and neural MT systems for low- and mid-resource Indic languages, and enables the induction of parallel data for all pairwise Indic–Indic language combinations. The corpus is openly licensed for research and development, and has supported competitive advances in MT quality for Indic languages in both academic and industry-scale evaluations [2104.05596] [2301.00539] [2401.06398]. ## 1. Goals, Scope, and Linguistic Coverage The primary objective of Samanantar is to provide a large-scale, high-quality parallel corpus supporting English↔Indic and Indic↔Indic MT, addressing persistent data sparsity for most Indian languages. Originally released by Gowtham et al. (2022), the corpus consists of 49.7 million English↔Indic sentence pairs, with per-language increases of up to 8.7× compared to prior public resources. Samanantar covers the following 11 Indic languages: Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil, and Telugu. Domains represented in the data include news, government/legal documents, Wikipedia, religious texts, crowd-sourced dialogues, and software-localization strings [2104.05596] [2301.00539]. Per-language statistics from the core release include (rounded, in millions): | Language | ENG–IL Pairs | Growth Factor | |----------|--------------|--------------| | Assamese | 0.65 | 1.3× | | Bengali | 5.28 | 2.5× | | Gujarati | 8.55 | 3.6× | | Hindi | 10.10 | 8.7× | | Kannada | 0.28 | — | | Malayalam| 0.18 | — | | Marathi | 0.15 | — | | Odia | 0.21 | — | | Punjabi | 0.13 | — | | Tamil | 0.94 | — | | Telugu | 0.93 | — | *Growth factors as reported for languages with existing public parallel data [2104.05596].* By pivoting through English, Samanantar further induces 83.4 million Indic–Indic sentence pairs, covering all 55 directed language combinations, thus yielding a 5.33× expansion over previous cross-Indic bitexts [2104.05596]. ## 2. Data Acquisition, Alignment, and Quality Control Samanantar’s construction pipeline integrates four categories of parallel text: 1. **Existing public corpora:** Sourced from repositories such as OPUS (Tatoeba, Ubuntu, GNOME, JW300, TED, WikiMatrix), WMT, PMIndia, UFAL, TICO-19, and others. 2. **Web-crawled comparable corpora and OCR extraction:** News portals, video subtitles, and scanned PDFs of government/legal documents were processed using OCR (Google Vision API) before sentence alignment. 3. **Multilingual sentence embeddings:** English and Indic sentences are embedded using LaBSE into a 768-dimensional space; top matches are aligned if their cosine similarity exceeds language-specific thresholds (0.75–0.80). 4. **Approximate nearest neighbor retrieval:** FAISS indexing accelerates large-scale alignment by retrieving nearest neighbors in embedding space. Quality control involves: - Automatic sentence-level alignment using similarity filtering (LaBSE, FAISS). - Deduplication and removal of grossly misaligned or truncated sentences. - Stratified human evaluation: 9,566 mined sentence pairs were rated by 38 bilingual annotators, yielding mean semantic textual similarity (STS) scores of 4.27 (accepted), with 90.1% accuracy in the ‘definite accept’ bucket [2104.05596]. - Manual spot-checks and domain balancing in dataset creation [2301.00539]. ## 3. Preprocessing, Curation, and Filtering Raw bitexts are preprocessed through a multi-stage pipeline, typically involving: - **Cleaning:** Stripping non-printable/control characters, filtering tokens outside expected Unicode ranges, punctuation normalization, and whitespace removal. - **Tokenization:** Moses tokenizer with Indic language–specific rules (clitic handling, punctuation separation). - **True-casing:** Training and applying data-driven true-case models to standardize casing. - **Length filtering:** Exclusion of sentence pairs where either side exceeds 80 tokens or whose token ratio exceeds 1.5. Das et al. (2024) document significant noisiness in Samanantar, including misalignments, literal-but-wrong translations, truncated/incomplete segments, and domain drift. They propose automatic mistranslation removal using an NMT-trained filter: sentence pairs are retained if their sentence-level BLEU (measured against the NMT model’s translation) exceeds a threshold θ = (corpus BLEU on validation set) / 4; this empirically removes the bottom quartile of misaligned/mistranslated pairs, improving downstream translation quality, especially for morphologically rich ILs [2401.06398]. ## 4. Integration in Machine Translation Pipelines Samanantar is used in both Statistical Machine Translation (SMT) and Neural MT (NMT) pipelines: - **SMT (Moses-based):** Sentence-aligned, tokenized, and true-cased corpora serve as input to Moses. Word alignments are induced by GIZA++ (IBM Models 1–5; grow-diag-final-and), phrase tables are constructed and binarized, and n-gram LMs are trained using SRILM (Kneser-Ney, typically 5-gram). - **Decoding** maximizes the product of LM and TM probabilities, with distance-based reordering (linear distortion costs on phrase jump magnitude) replacing Moses’s default “msd” reordering to better handle SVO–SOV divergences between English and Indic languages [2301.00539]. - **NMT (IndicTrans):** Samanantar is the primary training resource for Transformer models with 6-layer encoder/decoder, subword vocabularies (BPE; 32K merges), and script normalization (Devanagari unification for Indic languages). Shared dictionaries and language-tag tokens facilitate true multilingual transfer [2104.05596]. - **Evaluation:** BLEU, METEOR, and RIBES are used, with definitions following standard practice. FLORES and WMT test sets provide held-out benchmarks. ## 5. Statistical Characteristics and Performance Metrics The Samanantar corpus exhibits the following domain and statistical characteristics: - **Domains:** Encompasses newswire, legal/government documents (e.g., UN), religious texts, Wikipedia, and crowd/web-sourced content. - **Sentence length distributions:** Vary by language; e.g., Hindi/Bengali present broader length distributions, correlating with higher SMT BLEU, whereas Tamil/Malayalam are skewed towards shorter (≤4 token) sentences, associated with lower translation scores [2301.00539]. - **Observed evaluation metrics (SMT, Das et al., 2023):** - Hindi–English: BLEU ≈ 13–15 (EN↔HI) - Bengali–English: BLEU ≈ 6.4 (EN→BN) - Assamese, Tamil, Malayalam, Sinhala: BLEU < 4 - Filtering out poor-quality pairs via NMT-based BLEU filtering consistently improves scores, with largest relative benefits for Odia–English (up to +1.7 BLEU) and modest gains for Hindi–English (+0.7 BLEU), confirming that “more data is not always better” [2401.06398]. | Language Pair | Size (M) | BLEU (EN→IL) | BLEU (IL→EN) | |:--------------:|:--------:|:------------:|:------------:| | Hindi | 8.56 | 13.09 | 15.41 | | Bengali | 8.52 | 6.41 | 3.06 | | Odia | 1.00 | 6.38 | 14.53 | | Tamil | 5.16 | 1.39 | — | | Assamese | 0.14 | 1.90 | 3.21 | *Exemplars from SMT evaluations; see [2301.00539] and [2401.06398] for model and metric details.* ## 6. Broader Impact and Research Significance Samanantar yields a 4× increase in available English–Indic parallel data and a more than 5× expansion in cross-Indic bitexts compared to prior resources. This scale supports: - Robust training of competitive multilingual NMT models for all major Indian languages. - Creation of true pairwise translation data for all 55 directed Indic–Indic pairs via English-pivoting. - Enhanced representation learning, cross-lingual transfer, multilingual evaluation, and downstream NLP applications for Indian languages. - Public release of both corpus and models under open-data licenses facilitates progress for both academic and industrial research [2104.05596]. The corpus is distributed via GitHub and AI4Bharat, together with scripts, annotation sets, and baseline models, supporting reproducibility and extensibility. This suggests ongoing contributions to the advancement of MT and NLP for low-resource settings. ## 7. Ongoing Challenges and Future Directions Despite Samanantar’s scale and curation, residual noise remains due to the wide domain and automatic alignment. The methods developed for mistranslation detection and removal (e.g., NMT-based filtering) present general strategies for future parallel corpus construction. Planned research by Das et al. includes: - Exploring automated filtering for all remaining ILs. - Comparing alternative thresholding and adequacy/fluency estimation mechanisms. - Examining domain adaptation and fine-tuning to further improve translation robustness on noisy, heterogeneous corpora [2401.06398]. As parallel data mining continues to scale with web-crawling and automatic alignment, such self-supervised data cleaning approaches remain central to constructing reliable MT resources for typologically diverse, high-vocabulary languages. --- **References:** - [2104.05596] Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages - [2301.00539] Statistical Machine Translation for Indic Languages - [2401.06398] An approach for mistranslation removal from popular dataset for Indic MT Task Source: https://www.emergentmind.com/topics/samanantar