IIT Bombay English-Hindi Parallel Corpus
- The IIT Bombay English–Hindi Parallel Corpus is a comprehensive dataset with 1.49 million aligned sentence pairs across 17 diverse sub-corpora.
- The corpus, assembled by CFILT-IITB, involves meticulous preprocessing including Hindi nukta normalization, Moses tokenization, and BPE segmentation.
- Baseline experiments using PBSMT and NMT demonstrate its practical utility in benchmarking English–Hindi machine translation performance.
The IIT Bombay English–Hindi Parallel Corpus is the largest publicly available English–Hindi parallel corpus, comprising 1.49 million aligned sentence pairs drawn from a diverse set of domains and sources. Assembled through the consolidation of publicly available resources as well as new data collection initiatives led by CFILT-IITB, it underpins research in machine translation (MT), especially for resource-scarce language pairs, and has been utilized in shared tasks at the Workshop on Asian Language Translation (2016 and 2017) (Kunchukuttan et al., 2017).
1. Dataset Composition and Domain Distribution
The corpus aggregates 1,492,827 parallel segments in 17 distinct sub-corpora for training, plus a development set of 520 newswire-aligned pairs and a test set of 2,507 newswire pairs. Of the training data, 694,000 segments originate from new sources not previously available in the public domain. The dataset encompasses approximately 20.67 million English tokens and 22.17 million Hindi tokens in training, yielding type counts of 250,782 (English) and 343,601 (Hindi). Out-of-vocabulary (OOV) token-type rates on the dev/test splits are 6.7% for English and 11.4% for Hindi. The average sentence lengths are approximately 13.85 tokens for English and 14.85 tokens for Hindi.
Domain coverage is summarized below:
| Sub-corpus Domain | Segment Count |
|---|---|
| Gyaan-Nidhi encyclopedia | 227,123 |
| HindEnCorp (general-purpose) | 273,885 |
| Tanzil (Qur’an, religious) | 187,080 |
| Hindi–English Linked Wordnets | 175,175 |
| Mahashabdkosh (dictionary) | 159,822 |
| GNOME (software localization) | 145,706 |
| KDE4 (software localization) | 97,227 |
| Indian Govt. websites | 123,360 |
| TED talks | 42,583 |
| OpenSubs2013 (conversational) | 4,222 |
| Tatoeba (conversational) | 4,698 |
| Wikipedia titles | 32,863 |
| Judicial corpus I/II | 5,007/3,727 |
| Indic multi-parallel | 10,349 |
No explicit distributions (e.g., sentence-length histograms, token-frequency curves) are provided, but these can be computed from the released files.
2. Data Acquisition and Pre-processing Pipeline
Data sources include existing OPUS-aligned sub-corpora (GNOME, KDE4, Tanzil, Tatoeba, OpenSubs2013), HindEnCorp, TED-Amara, and new collections (Mahashabdkosh, Judicial I & II, Indian Government corpora, and Gyaan-Nidhi) compiled by CFILT-IITB. Alignment for sources such as Gyaan-Nidhi employs the Moore (2002) method, combining sentence-length and word correspondence statistics, with an estimated precision of 88.6% based on a 300-sentence sample.
Preprocessing comprises:
- Decomposition of Hindi nukta characters into two-codepoint (base+nukta) form throughout using the IndicNLP library.
- English text is left in its original case for distribution, while experimental workflows apply Moses true-casing.
- Non-printable characters are stripped and length filtering applied to remove excessively short or long sentence pairs (using default thresholds in the provided scripts).
- Tokenization uses Moses' tool for English and IndicNLP for Hindi.
No formal alignment-scoring formula is provided; sentence alignment employs established heuristics appropriate for large-scale bitext construction.
3. Licensing, Access, and Distribution Format
The corpus is accessible at http://www.cfilt.iitb.ac.in/iitb_parallel. New sub-corpora are licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 (CC BY-NC-SA 4.0), while legacy datasets are distributed under their original licenses as documented in Table 1 of (Kunchukuttan et al., 2017). The release consists of 17 parallel files (tab- or pipe-separated), with ordering aligned to domain segmentation to enable precise extraction of domain-specific slices.
4. Baseline Machine Translation Benchmarks
Phrase-based and neural approaches were benchmarked using standardized splits. For phrase-based SMT (PBSMT), Moses was utilized with grow-diag-final-and extraction, lexicalized reordering, and batch MIRA tuning. LLMs were trained using KenLM (5-gram, Kneser-Ney): Hindi models on HindMono (≈44M sentences) and English models on WMT News Crawl 2015 (≈23M sentences).
Neural MT (NMT) baselines employed Nematus, operating over 1-layer bidirectional GRU encoder and 1-layer GRU decoder (both 512 units), 256-dimensional embeddings, Adam optimizer (learning rate 0.0001), batch size 50, and a maximum sentence length of 100 tokens. Early stopping with patience of 10 epochs and decoding via a beam of 12, ensemble over four checkpoints (the best and last three) was implemented. BPE (15,500 merges per language) was applied to both source and target separately.
BLEU and METEOR metrics (including METEOR-Indic, which utilizes IndoWordNet for Hindi synonyms and trie-based stemming) were reported on the held-out newswire test set. Results are tabulated below:
| Direction | System | BLEU | METEOR |
|---|---|---|---|
| EN→HI | PBSMT | 11.75 | 0.313 |
| EN→HI | NMT | 12.23 | 0.308 |
| HI→EN | PBSMT | 14.49 | 0.266 |
| HI→EN | NMT | 12.83 | 0.219 |
NMT marginally outperforms PBSMT for EN→HI BLEU, but PBSMT exhibits stronger performance on HI→EN translation. The METEOR-Indic evaluation better captures morphological and synonym correspondences for Hindi.
5. Usage Protocols and Extension Strategies
Recommended practice is to use the train/dev/test splits as provided: 1,492,827/520/2,507 sentences. For pre-processing and replication, the sequential pipeline is:
- Normalize Hindi nukta forms (IndicNLP)
- Tokenize English (Moses) and Hindi (IndicNLP)
- Optionally apply English true-casing
- Apply BPE segmentation (15,500 merges, separately per language)
- Filter sentence pairs by length (e.g., maximum 100 tokens)
Potential extensions proposed in (Kunchukuttan et al., 2017) include harvesting additional government website data (e.g., TDIL), pre-ordering source sentences for PBSMT, and generating synthetic parallel data via back-translation.
6. Research Impact and Further Applications
The IIT Bombay English–Hindi Parallel Corpus establishes a high-coverage, multi-domain foundation for English–Hindi MT and computational linguistics. Its rich domain diversity enables research on generalization, domain adaptation, morphologically rich language translation, and low-resource NMT. The resource's licensing terms and explicit sub-corpus decomposition facilitate both broad and domain-specific evaluation setups, supporting ongoing methodological developments and reproducible benchmarking in English–Hindi machine translation (Kunchukuttan et al., 2017).