IndicLLMSuite: Blueprint for Indic LLM Data

• IndicLLMSuite is an open blueprint featuring a modular architecture with four components that support large-scale, multilingual LLM data construction.
• It employs a stratified quality approach by segregating data into manually verified, mined unverified, and synthetic sets to ensure transparency and reusability.
• The suite integrates holistic pipelines for pretraining, instruction tuning, and safety alignment, paving the way for scalable and commercially viable Indic LLMs.

IndicLLMSuite is an open blueprint for building LLM data resources for Indian languages. It is presented not as a single benchmark or a single model, but as a full data stack spanning pre-training, instruction fine-tuning, and safety alignment for the 22 languages in the 8th Schedule of the Indian Constitution. In its released form, the suite covers 22 languages, contains 251B tokens and 74.8M instruction-response pairs, and is organized around four named components: Sangraha, Setu, IndicAlign-Instruct, and IndicAlign-Toxic (Khan et al., 2024).

1. Conceptual scope and historical placement

IndicLLMSuite is motivated by a dual scarcity: Indian languages lack both sufficiently large, high-quality corpora and open, reusable pipelines for constructing such corpora from heterogeneous sources such as websites, PDFs, and videos. The suite therefore addresses a methodology gap as much as a data gap. Its stated goal is to make Indic LLM development less dependent on ad hoc collection, opaque filtering, or licenses that block commercial use, while keeping both the resources and the construction process open (Khan et al., 2024).

Within the broader AI4Bharat trajectory, IndicLLMSuite follows earlier ecosystem-building work that coupled IndicCorp v2, IndicXTREME, and IndicBERT v2 for multilingual Indic NLU (Doddapaneni et al., 2022). The earlier effort emphasized monolingual corpora, encoder-style multilingual pretraining, and human-supervised evaluation. IndicLLMSuite moves that program toward the generative-LLM regime by adding large-scale instruction data, synthetic conversational data, transliteration pipelines, and toxic-alignment data (Khan et al., 2024).

A useful way to characterize the suite is as infrastructure rather than as a finished model family. Sangraha supplies pretraining text, Setu supplies the Spark-based curation pipeline, IndicAlign-Instruct supplies supervised instruction data, and IndicAlign-Toxic supplies safety-oriented alignment pairs. This division of labor is central to how later papers interpret the suite.

2. Internal organization

The suite’s architecture is compactly described by its four components.

Component	Function	Scale
Sangraha	Pretraining corpus	251.3B tokens across 22 languages
Setu	Data processing and curation pipeline	Spark-based, with crawling, cleaning, filtering, and deduplication
IndicAlign-Instruct	Instruction fine-tuning collection	roughly 74.7M prompt-response pairs across 20 languages
IndicAlign-Toxic	Safety-alignment data	123K toxic-prompt/non-toxic-response pairs

Sangraha itself is stratified into three quality tiers. Sangraha Verified contains 64.3B tokens of manually verified high-quality data. Sangraha Unverified contains 24.3B tokens mined from existing multilingual web corpora and filtered using perplexity models derived from the verified split. Sangraha Synthetic contains 162.7B tokens, largely from machine translation of English Wikimedia into Indic languages and subsequent transliteration into Roman script (Khan et al., 2024).

This three-way split encodes the suite’s explicit quality philosophy: manually verified data, filtered but not manually audited data, and synthetic data are all treated as useful, but they are not conflated. That is a defining property of IndicLLMSuite’s design.

3. Sangraha and the Setu pipeline

Sangraha Verified is built from web, PDF, and speech sources, with 48.6B web tokens, 14.55B PDF tokens, and 1.20B speech tokens. A core methodological choice is manual verification of source URLs before crawling. Candidate sites were rejected if they were non-Indic or non-English, adult, gambling, generally toxic, or clearly machine translated. Crawling used the open-source webcorpus toolkit, and HTML extraction used Trafilatura (Khan et al., 2024).

The PDF pipeline is especially important because much Indic text remains locked in scans. OCR was performed with Google Cloud Vision, and the final OCR’d corpus covers 507,419 PDFs and 82M pages. Setu then performs layout-aware PDF processing using bounding boxes at block, paragraph, word, and character levels, followed by page-level filters such as bounding-box suppression, horizontally sparse page removal, vertically sparse page removal, high-overlap page removal, sparse-block page removal, and low-script-confidence page removal (Khan et al., 2024).

Setu itself has four main stages: Document Preparation, Cleaning and Analysis, Flagging and Filtering, and Deduplication. The cleaning layer includes code-span removal for web data, symbol-heavy filtering, terminal-punctuation filtering, repeated-chunk filtering for PDFs, and chunk-length filtering. The flagging layer uses an ensemble of IndicLID, cld3, and NLLB for language identification, plus document-level metrics such as NSFW-word count, non-Latin/non-Indic character count, a 10-gram character repetition score, and a 5-gram word repetition score (Khan et al., 2024).

Several thresholds are explicit. Script-based cleaning removes a sentence if

$$\frac{\#\text{native script characters}}{\#\text{total characters}} < 0.75$$

and deduplication uses MinHashLSH with 5-grams and a Jaccard-similarity threshold of 0.7. For Sangraha Unverified, document filtering is based on language-model perplexity. If $V_\ell$ is the verified validation set for language $\ell$, the threshold is the 80th percentile of validation perplexities,

$$T_\ell = Q_{0.8}\big(\{\mathrm{PPL}(d): d \in V_\ell\}\big),$$

and a candidate document is kept when $\mathrm{PPL}(d) \le T_\ell$ (Khan et al., 2024).

Synthetic expansion is handled by Setu-Translate and Setu-Transliterate. Using IndicTrans2, the suite translates the entirety of English Wikimedia into 14 Indian languages, producing nearly 90B tokens. A transliterated version of the translated Wikimedia corpus contributes about 72B tokens via IndicXlit, motivated by the prevalence of Romanized Indic in informal digital use (Khan et al., 2024).

4. Instruction tuning and toxicity alignment

IndicAlign-Instruct aggregates several heterogeneous instruction sources rather than depending on one template family. The components include Indic-ShareLlama, Dolly-Translated, OpenAssistant-Translated, WikiHow, IndoWordNet, Anudesh, Wiki-Conv, and Wiki-Chat (Khan et al., 2024).

The numerically dominant component is IndoWordNet, built from 21 intents and 105 prompt-response templates over the IndoWordNet lexicon, yielding about 74M pairs for 18 Indic languages. This makes IndicAlign-Instruct large not only because of translated chat data, but because of lexical-semantic supervision at scale. Other components target different interaction regimes: Wiki-Conv and Wiki-Chat create grounded conversations from India-centric English Wikipedia; Anudesh collects human interaction data through a consent-driven platform; and translated instruction datasets provide additional cross-lingual breadth (Khan et al., 2024).

Anudesh is notable because it explicitly accommodates English, native Indic languages, Romanized Indic, and English-Indic code-mixed forms. Because Llama2-70B-Chat had limitations in Indic languages, the platform used a translate-test pipeline in which user prompts were translated into English, answered by the model, and translated back to the target Indic language. Participants consented to release conversations under CC BY 4.0 (Khan et al., 2024).

IndicAlign-Toxic contains two datasets: HH-RLHF-Translated and Toxic Matrix, for a total of about 123K toxic-prompt/non-toxic-response examples. Toxic Matrix is generated from a three-axis taxonomy of content type, target group, and prompt style, after which toxic prompts are answered by Llama2-70B-Chat to obtain refusal responses with rationale. The suite explicitly notes that this taxonomy is not exhaustive and that synthetic toxicity alignment is promising but not foolproof (Khan et al., 2024).

5. Language coverage, imbalance, and stated limits

The suite centers on the 22 constitutionally recognized Indian languages, with English also present inside several components because Indian English is part of the resource mixture. Coverage is broad, but not uniform. High-resource languages such as Hindi, Bengali, Tamil, Telugu, Urdu, Gujarati, Kannada, Malayalam, Marathi, and Nepali receive tens of billions of pretraining tokens once verified, synthetic, and unverified splits are combined. By contrast, total Sangraha tokens are only 1.5M for Bodo, 0.06M for Dogri, 0.5M for Kashmiri, 10.1M for Konkani, 14.6M for Maithili, 7.4M for Manipuri, 0.3M for Santali, and 258.2M for Sindhi, while Hindi has 34.5B and Bengali 30.0B (Khan et al., 2024).

The suite explicitly acknowledges that some languages remain poorly represented because of limited digital presence and because OCR and other tooling did not yet support all scripts well enough for every source. It also acknowledges demographic skew in the crowdsourced Anudesh data, possible artifacts from synthetic data, incomplete dialect and sociolect coverage, and the ambiguous legal status of some web-sourced training data. Safety filtering in Sangraha relies on NSFW-word detection and can miss toxicity or create false positives, and the paper states that broad downstream evaluation of models trained on the suite is beyond its scope (Khan et al., 2024).

This makes IndicLLMSuite a blueprint rather than a closed solution. Its strongest claim is not that it solves Indic LLM training, but that it makes the full data-construction workflow reproducible and openly extensible.

6. Role in the later Indic benchmark ecosystem

Later work repeatedly treats IndicLLMSuite as a broad backbone to which specialist benchmarks can attach. In that literature, it functions less as a benchmark in its own right than as a reference architecture for what a comprehensive Indic LLM ecosystem should contain.

This role is explicit in code-generation work. “IndicEval-XL” positions itself as a specialist coding benchmark and states that, if an IndicLLMSuite is intended to cover broad LLM abilities, IndicEval-XL would naturally serve as the code-generation component or companion benchmark (Singh et al., 26 Feb 2025). Other papers make similar claims for different capability slices: IndicParam for low- and extremely low-resource exam-style multiple-choice evaluation (Maheshwari et al., 29 Nov 2025), IndicMMLU-Pro for multilingual MMLU-style knowledge and reasoning (KJ et al., 27 Jan 2025), MILU for culturally grounded, India-first multiple-choice evaluation (Verma et al., 2024), IndicIFEval for verifiable instruction following in 14 Indic languages (Jayakumar et al., 25 Feb 2026), IndicRAGSuite for retrieval and RAG infrastructure (Prasanjith et al., 2 Jun 2025), IndicDB for multilingual Text-to-SQL over Indian administrative databases (Dawar et al., 15 Apr 2026), and Indi-RomCoM for Romanized Indic-English code-mixed instruction following (Das et al., 29 Jun 2026).

A plausible implication is that “IndicLLMSuite” has come to denote two related things. In the narrow sense, it is the 2024 data-and-pipeline release. In the broader sense used by later papers, it is a modular vision of Indic LLM infrastructure in which pretraining resources, instruction data, safety alignment, and multiple specialist benchmarks are composed rather than collapsed into a single monolithic leaderboard.

7. Significance

IndicLLMSuite’s main contribution is to turn Indic LLM data creation into a documented, reusable workflow. It supplies a large multilingual pretraining mixture, an openly described Spark-based curation pipeline, translation and transliteration infrastructure, instruction-tuning data built from translated, synthetic, lexical-semantic, and crowdsourced sources, and an initial safety-alignment layer. It also does so under permissive licenses for code and with explicit concern for commercially usable data sources (Khan et al., 2024).

Its limits are equally clear. Coverage remains uneven across languages, synthetic data plays a very large role, safety alignment is preliminary, and the paper does not itself report broad downstream model results. Yet later work’s repeated use of IndicLLMSuite as a point of reference suggests that its enduring importance lies precisely in this incompleteness: it established the open, modular resource stack on top of which specialist Indic benchmarks and models could be organized (Khan et al., 2024).