AI4Bharat Large-Scale Pretraining

• AI4Bharat’s large-scale pretraining is a comprehensive data-centric initiative advancing ASR and NLU for low-resource Indic languages with extensive, curated, and synthetic datasets.
• The approach employs tailored models like IndicWav2Vec and IndicBERT v2 that leverage multilingual sampling and balanced upsampling to achieve state-of-the-art performance across benchmark tasks.
• Synthetic data generation via the BhashaKritika corpus, combined with rigorous quality filtering and bias mitigation, underpins scalable and culturally nuanced language model development.

AI4Bharat's large-scale pretraining initiatives represent a data-centric and methodologically rigorous effort to advance speech and language technologies for the linguistically diverse populations of the Indian subcontinent. These efforts span automatic speech recognition (ASR), natural language understanding (NLU), and LLM pretraining, targeting low-resource Indic languages using extensive curated, cleaned, and—more recently—synthetic data. This article surveys the architecture and curation workflows of AI4Bharat's key public resources: speech representation learning (IndicWav2Vec) (Javed et al., 2021), multilingual masked language modeling (IndicBERT v2) (Doddapaneni et al., 2022), and synthetic corpus construction at scale (BhashaKritika) (Manoj et al., 13 Nov 2025).

1. Multilingual ASR Pretraining: IndicWav2Vec

AI4Bharat's speech pretraining framework is built upon curating a 17,300-hour raw speech collection in 40 Indian languages—including major and low-resource tongues—across domains such as education, news, technology, and finance. Data curation combines native-speaker manual selection (from YouTube and NewsonAir), strict licensing (CC-BY), speaker diversity, audio normalization (mono, 16 kHz), silence detection (WebRTC-VAD, aggressiveness=2), SNR thresholding (WADA-SNR, SNR>15 dB), and segmentation (≤25 s).

The model architecture follows wav2vec 2.0, comprising a 7-layer convolutional feature encoder (512 channels, kernel widths [10,3,3,3,3,2,2], strides [5,2,2,2,2,2,2]), a Transformer context network (BASE: 12 layers, 768 d, 8 heads; LARGE: 24 layers, 1024 d, 16 heads), and Gumbel-product quantization (2 codebook groups, 320 entries/group) for discretizing encoder outputs. For multilingual sampling, per-minibatch language sampling probability is

$$p_l = \frac{(n_l/N)^\alpha}{\sum_{l'} (n_{l'}/N)^\alpha}$$

where $n_l$ is language hours, $N$ total hours, $\alpha=0.7$ empirically optimal.

Pretraining uses masked reconstruction, combining InfoNCE contrastive loss and codebook diversity loss: $$\mathcal{L} = \mathcal{L}_{\mathrm{contrast}} + \lambda\,\mathcal{L}_{\mathrm{diversity}}$$ Fine-tuned IndicWav2Vec yields state-of-the-art ASR results on nine Indic languages, notably outperforming baselines including on severely underresourced languages like Sinhala and Nepali. Analysis of learned representations reveals phoneme-sharing in codebooks, layer-wise language-family clustering, and attention patterns focusing on local context (Javed et al., 2021).

2. Large-Scale Monolingual Pretraining: IndicBERT v2

For NLU, AI4Bharat constructed IndicCorp v2, a 20.9 billion-token cleaned monolingual corpus covering 24 Indic languages and Indian English. The pipeline combines web crawling of news, Wikipedia/OSCAR data, and strict post-processing including paragraph-level LID (cld3, langdetect), script filers (≥75% native Unicode), minimum span length (≥10 words), offensive term blacklisting (~90/language), and aggressive de-duplication.

IndicBERT v2 adopts a standard BERT-Base encoder (12 layers, 768 hidden d, 12 heads, 3072 intermed.), with a 250,000 WordPiece vocabulary trained on the upsampled corpus. Script and language-specific document prepending (<lang-id>) yield improved morphological fertility over mBERT/XLM-R.

The model is pretrained for 1M steps on 128 TPU v3 cores using the Masked Language Modeling objective with a sequence length of 512 and batch size 4096: $$L_{\mathrm{MLM}}(\theta) = -\sum_{i \in M} \log P(x_i | x_{[1,n]\setminus\{i\}}; \theta)$$ High-quality parallel data enables additional TLM ablations; next-sentence prediction is omitted. Upsampling of low-resource languages ($T=0.3$ temperature) mitigates data imbalance.

Evaluation on the IndicXTREME benchmark (nine tasks—classification, structure prediction, QA, retrieval—across 20 languages, all human-supervised) demonstrates that IndicBERT v2 achieves an average +2 point gain over MuRIL in accuracy/F1 across seven of nine tasks. Substantial improvements are observed in translation-based tasks (FLORES: +17 points vs MuRIL). Zero-shot transfer via Hindi as the fine-tuning language (in place of English) yields notable performance gains, affirming the advantage of in-family resource sharing (Doddapaneni et al., 2022).

3. Synthetic Pretraining Data: BhashaKritika Corpus

BhashaKritika addresses the scarcity of high-quality Indic pretraining data through large-scale synthetic corpus generation—540 billion tokens across Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil, and Telugu. Five principal strategies are employed:

• Document-grounded generation: LLMs generate in the target Indic language, prompted by web extracts (from FineWeb/C4, FineWeb2), with style templates for blogs, encyclopedias, stories, and textbooks.
• Persona-based generation: ~164M English and 50K Indic personas condition the LLM on speaker role.
• Math & Reasoning: LLMs generate textbook-like concept explanations and solutions from verified question–answer pairs.
• Topic-aware retrieval-augmented generation: Underrepresented India-related topics (Wikipedia-based) trigger generation grounded in freshly retrieved web docs.
• Translation: 25B tokens of English (Cosmopedia) translated using Sarvam-Translate, maximizing cross-lingual diversity.

Document-grounded and persona-based strategies yield the lowest discard rates after cleaning (∼1.5%), while translation and reasoning have higher discard (∼4–5%). Human evaluations and downstream results consistently favor native-language generation over translation for quality and diversity.

Corpus-wide distribution by language is shown in the table:

Language	Tokens (B)	Share (%)
Hindi	145.8	27
Punjabi	70.2	13
Bengali	64.8	12
Marathi	59.4	11
Gujarati	48.6	9
Tamil	43.2	8
Telugu	43.2	8
Malayalam	27.0	5
Kannada	21.6	4
Oriya	16.2	3

(Manoj et al., 13 Nov 2025)

4. Quality Filtering and Bias Mitigation in Synthetic Data

BhashaKritika's quality control pipeline integrates several mechanisms:

• LID/Script Detection: Samples must pass language and script identification, else are rejected.
• Heuristic Rules: Constraints on length ([100,2500] words), NSFW keywords (ratio=0.0), AI-related terms, script ratios, and n-gram repetition (≤0.3).
• Perplexity Filtering: Per-language Kneser–Ney KenLM 5-gram models compute $$PPL(x_{1…N}) = 2^{-(1/N)\sum_{i=1}^N \log_2 p(x_i | x_{1…i-1})}$$; samples at the 80th percentile threshold are filtered.
• FastText Classifier: A model (98.9% accuracy, 384K Gemini-1.5 annotations) removes low-quality generations.
• Bias Evaluation: WEAT (Word Embedding Association Test) quantifies gender, caste, race, religion, and regional bias. Effect size reductions are achieved with anti-bias augmentation (e.g., religion bias 1.34 to 1.29).

Empirical model runs on LLaMA variants reveal that BhashaKritika-pretrained models converge faster and perform on par or better on Indic downstream tasks compared to web-only data. Native, context-grounded generation and English instruction prompts are recommended for optimal quality. (Manoj et al., 13 Nov 2025)

5. Empirical Insights and Strategic Recommendations

Comparative ablations identify key determinants of corpus and model quality:

• Direct generation outperforms translation: Both human assessment and benchmark scores indicate higher overall quality, diversity, and lower discard rates for natively generated Indic content versus translated English.
• Persona and context-grounding improve cultural/semantic parity: Outputs conditioned on Indic personas, especially with matched documents, achieve minimal discard and enhanced authenticity.
• Balanced language sampling/uniform script mapping mitigate resource skew: Upsampling and script-sharing (romanization to Devanagari) enhance representation for low-resource languages and closely related scripts.
• Training resource allocation is more efficient in family-specific models: Indic-only pretraining is empirically superior to massive multilingual approaches for downstream Indic benchmarks, confirming better parameter capacity utilization.

Downstream evaluation across ASR and NLU tasks demonstrates that these strategies deliver consistent and sometimes state-of-the-art performance in both resource-rich and underrepresented Indic languages, positioning AI4Bharat's frameworks as foundational for scalable, inclusive LLM pretraining in the region (Javed et al., 2021, Doddapaneni et al., 2022, Manoj et al., 13 Nov 2025).

6. Limitations and Directions for Expansion

Urgent areas for further research include targeted corpus expansion and benchmark creation for left-behind languages—such as Santali and Sindhi—which remain underrepresented due to both lack of large-scale clean data and script/linguistic isolation. Continued integration of parallel data, script-sharing ablations, and adaptive filtering strategies could further enhance cross-lingual transfer. The publicly available datasets and codebases enable reproducible research and future extension to typologically related language families using the established pipeline of verified crawling, robust synthetic generation, balanced upsampling, and human-supervised evaluation (Doddapaneni et al., 2022, Manoj et al., 13 Nov 2025).