Samanantar: Parallel Corpus for Indic NLP

Updated 3 July 2026

Samanantar is a large-scale parallel corpus for English and 11 Indic languages, offering extensive data for machine translation and low-resource NLP research.
It combines public datasets, web-crawled texts, and advanced alignment methods including LaBSE and FAISS for precise sentence pairing.
The corpus enables cross-Indic translation, boosts MT performance through rigorous filtering, and supports reproducible research in multilingual settings.

Samanantar is the largest publicly available parallel corpus collection for English and 11 Indic languages, providing a substantial resource for machine translation (MT) and multilingual NLP research targeting the Indian subcontinent. Combining both pre-existing and newly mined sentence pairs at massive scale, Samanantar supports the development and benchmarking of both statistical and neural MT systems for low- and mid-resource Indic languages, and enables the induction of parallel data for all pairwise Indic–Indic language combinations. The corpus is openly licensed for research and development, and has supported competitive advances in MT quality for Indic languages in both academic and industry-scale evaluations (Ramesh et al., 2021, Das et al., 2023, Das et al., 2024).

1. Goals, Scope, and Linguistic Coverage

The primary objective of Samanantar is to provide a large-scale, high-quality parallel corpus supporting English↔Indic and Indic↔Indic MT, addressing persistent data sparsity for most Indian languages. Originally released by Gowtham et al. (2022), the corpus consists of 49.7 million English↔Indic sentence pairs, with per-language increases of up to 8.7× compared to prior public resources. Samanantar covers the following 11 Indic languages: Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil, and Telugu. Domains represented in the data include news, government/legal documents, Wikipedia, religious texts, crowd-sourced dialogues, and software-localization strings (Ramesh et al., 2021, Das et al., 2023).

Per-language statistics from the core release include (rounded, in millions): | Language | ENG–IL Pairs | Growth Factor | |----------|--------------|--------------| | Assamese | 0.65 | 1.3× | | Bengali | 5.28 | 2.5× | | Gujarati | 8.55 | 3.6× | | Hindi | 10.10 | 8.7× | | Kannada | 0.28 | — | | Malayalam| 0.18 | — | | Marathi | 0.15 | — | | Odia | 0.21 | — | | Punjabi | 0.13 | — | | Tamil | 0.94 | — | | Telugu | 0.93 | — | Growth factors as reported for languages with existing public parallel data (Ramesh et al., 2021).

By pivoting through English, Samanantar further induces 83.4 million Indic–Indic sentence pairs, covering all 55 directed language combinations, thus yielding a 5.33× expansion over previous cross-Indic bitexts (Ramesh et al., 2021).

2. Data Acquisition, Alignment, and Quality Control

Samanantar’s construction pipeline integrates four categories of parallel text:

Existing public corpora: Sourced from repositories such as OPUS (Tatoeba, Ubuntu, GNOME, JW300, TED, WikiMatrix), WMT, PMIndia, UFAL, TICO-19, and others.
Web-crawled comparable corpora and OCR extraction: News portals, video subtitles, and scanned PDFs of government/legal documents were processed using OCR (Google Vision API) before sentence alignment.
Multilingual sentence embeddings: English and Indic sentences are embedded using LaBSE into a 768-dimensional space; top matches are aligned if their cosine similarity exceeds language-specific thresholds (0.75–0.80).
Approximate nearest neighbor retrieval: FAISS indexing accelerates large-scale alignment by retrieving nearest neighbors in embedding space.

Quality control involves:

Automatic sentence-level alignment using similarity filtering (LaBSE, FAISS).
Deduplication and removal of grossly misaligned or truncated sentences.
Stratified human evaluation: 9,566 mined sentence pairs were rated by 38 bilingual annotators, yielding mean semantic textual similarity (STS) scores of 4.27 (accepted), with 90.1% accuracy in the ‘definite accept’ bucket (Ramesh et al., 2021).
Manual spot-checks and domain balancing in dataset creation (Das et al., 2023).

3. Preprocessing, Curation, and Filtering

Raw bitexts are preprocessed through a multi-stage pipeline, typically involving:

Cleaning: Stripping non-printable/control characters, filtering tokens outside expected Unicode ranges, punctuation normalization, and whitespace removal.
Tokenization: Moses tokenizer with Indic language–specific rules (clitic handling, punctuation separation).
True-casing: Training and applying data-driven true-case models to standardize casing.
Length filtering: Exclusion of sentence pairs where either side exceeds 80 tokens or whose token ratio exceeds 1.5.

Das et al. (2024) document significant noisiness in Samanantar, including misalignments, literal-but-wrong translations, truncated/incomplete segments, and domain drift. They propose automatic mistranslation removal using an NMT-trained filter: sentence pairs are retained if their sentence-level BLEU (measured against the NMT model’s translation) exceeds a threshold θ = (corpus BLEU on validation set) / 4; this empirically removes the bottom quartile of misaligned/mistranslated pairs, improving downstream translation quality, especially for morphologically rich ILs (Das et al., 2024).

4. Integration in Machine Translation Pipelines

Samanantar is used in both Statistical Machine Translation (SMT) and Neural MT (NMT) pipelines:

SMT (Moses-based): Sentence-aligned, tokenized, and true-cased corpora serve as input to Moses. Word alignments are induced by GIZA++ (IBM Models 1–5; grow-diag-final-and), phrase tables are constructed and binarized, and n-gram LMs are trained using SRILM (Kneser-Ney, typically 5-gram).
- Decoding maximizes the product of LM and TM probabilities, with distance-based reordering (linear distortion costs on phrase jump magnitude) replacing Moses’s default “msd” reordering to better handle SVO–SOV divergences between English and Indic languages (Das et al., 2023).
NMT (IndicTrans): Samanantar is the primary training resource for Transformer models with 6-layer encoder/decoder, subword vocabularies (BPE; 32K merges), and script normalization (Devanagari unification for Indic languages). Shared dictionaries and language-tag tokens facilitate true multilingual transfer (Ramesh et al., 2021).
Evaluation: BLEU, METEOR, and RIBES are used, with definitions following standard practice. FLORES and WMT test sets provide held-out benchmarks.

5. Statistical Characteristics and Performance Metrics

The Samanantar corpus exhibits the following domain and statistical characteristics:

Domains: Encompasses newswire, legal/government documents (e.g., UN), religious texts, Wikipedia, and crowd/web-sourced content.
Sentence length distributions: Vary by language; e.g., Hindi/Bengali present broader length distributions, correlating with higher SMT BLEU, whereas Tamil/Malayalam are skewed towards shorter (≤4 token) sentences, associated with lower translation scores (Das et al., 2023).
Observed evaluation metrics (SMT, Das et al., 2023):
- Hindi–English: BLEU ≈ 13–15 (EN↔HI)
- Bengali–English: BLEU ≈ 6.4 (EN→BN)
- Assamese, Tamil, Malayalam, Sinhala: BLEU < 4
- Filtering out poor-quality pairs via NMT-based BLEU filtering consistently improves scores, with largest relative benefits for Odia–English (up to +1.7 BLEU) and modest gains for Hindi–English (+0.7 BLEU), confirming that “more data is not always better” (Das et al., 2024).

Language Pair	Size (M)	BLEU (EN→IL)	BLEU (IL→EN)
Hindi	8.56	13.09	15.41
Bengali	8.52	6.41	3.06
Odia	1.00	6.38	14.53
Tamil	5.16	1.39	—
Assamese	0.14	1.90	3.21

Exemplars from SMT evaluations; see (Das et al., 2023) and (Das et al., 2024) for model and metric details.

6. Broader Impact and Research Significance

Samanantar yields a 4× increase in available English–Indic parallel data and a more than 5× expansion in cross-Indic bitexts compared to prior resources. This scale supports:

Robust training of competitive multilingual NMT models for all major Indian languages.
Creation of true pairwise translation data for all 55 directed Indic–Indic pairs via English-pivoting.
Enhanced representation learning, cross-lingual transfer, multilingual evaluation, and downstream NLP applications for Indian languages.
Public release of both corpus and models under open-data licenses facilitates progress for both academic and industrial research (Ramesh et al., 2021).

The corpus is distributed via GitHub and AI4Bharat, together with scripts, annotation sets, and baseline models, supporting reproducibility and extensibility. This suggests ongoing contributions to the advancement of MT and NLP for low-resource settings.

7. Ongoing Challenges and Future Directions

Despite Samanantar’s scale and curation, residual noise remains due to the wide domain and automatic alignment. The methods developed for mistranslation detection and removal (e.g., NMT-based filtering) present general strategies for future parallel corpus construction. Planned research by Das et al. includes:

Exploring automated filtering for all remaining ILs.
Comparing alternative thresholding and adequacy/fluency estimation mechanisms.
Examining domain adaptation and fine-tuning to further improve translation robustness on noisy, heterogeneous corpora (Das et al., 2024).

As parallel data mining continues to scale with web-crawling and automatic alignment, such self-supervised data cleaning approaches remain central to constructing reliable MT resources for typologically diverse, high-vocabulary languages.

References:

(Ramesh et al., 2021) Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages
(Das et al., 2023) Statistical Machine Translation for Indic Languages
(Das et al., 2024) An approach for mistranslation removal from popular dataset for Indic MT Task