IndicSUPERB Benchmark: Indian Speech Research
- IndicSUPERB is a universal benchmark for Indian speech research that evaluates six key tasks using the diverse, demographically balanced Kathbath corpus.
- It standardizes evaluation similar to GLUE and SUPERB, enabling robust assessment across tasks such as ASR, SID, LID, QbE, and keyword spotting.
- The benchmark has advanced research in supervised ASR, continual learning, and deepfake detection by providing a rich, scalable dataset for model evaluation.
IndicSUPERB is a speech processing universal performance benchmark for Indian languages built around the Kathbath corpus and introduced to provide a standardized evaluation suite for Indic speech research analogous to the role played by GLUE in NLU and SUPERB in speech for English-centric settings (Javed et al., 2022). Its original release combines 1,684 hours of labelled speech data across 12 Indian languages from 1,218 contributors located in 203 districts in India with benchmarks spanning six speech tasks: Automatic Speech Recognition, Speaker Verification, Speaker Identification, Language Identification, Query By Example, and Keyword Spotting (Javed et al., 2022). Subsequent work has treated IndicSUPERB not merely as an evaluation package for supervised ASR, but as a reusable substrate for data scaling, continual learning under no-replay constraints, and generation of real speech corpora for deepfake detection benchmarks (Bhogale et al., 2022, T et al., 8 Aug 2025, Girish et al., 21 Apr 2026).
1. Origins and design rationale
IndicSUPERB was proposed in response to a specific benchmarking gap: Indian languages lacked a comprehensive, diverse speech benchmark comparable to SUPERB, despite the central role that standardized training and test datasets had already played in accelerating English NLU and SLU research (Javed et al., 2022). The benchmark was framed explicitly in the lineage of GLUE and language-specific GLUE variants for text, and of SUPERB for speech, but targeted Indian speech technology, where prior datasets either covered a single task or only a small number of tasks and languages (Javed et al., 2022).
The original contribution was organized around three elements. First, the creators collected Kathbath, containing 1,684 hours of labelled speech data across 12 Indian languages from 1,218 contributors located in 203 districts in India. Second, they used Kathbath to create benchmarks across six speech tasks for 12 languages. Third, they trained and evaluated different self-supervised models alongside a commonly used FBANK baseline on the released benchmarks (Javed et al., 2022).
A common reduction is to describe IndicSUPERB as an ASR benchmark. That description is incomplete. The benchmark was designed as a multi-task evaluation framework covering recognition, speaker-centric, language-centric, retrieval, and command-detection problems, with the intent of enabling systematic comparison across heterogeneous speech processing objectives rather than only transcription (Javed et al., 2022).
2. Kathbath corpus and benchmark construction
The data foundation of IndicSUPERB is Kathbath, a corpus of labelled read speech covering Kannada, Malayalam, Tamil, Telugu, Gujarati, Marathi, Bengali, Odia, Hindi, Punjabi, Sanskrit, and Urdu (Javed et al., 2022). The corpus includes 1,684 hours of labelled read speech data across these 12 languages, together with additional noisy data in the benchmark construction described in the paper’s details (Javed et al., 2022).
Kathbath was assembled from 1,218 participants distributed across 203 districts and 22 states, with speaker diversity extending across region, age, and gender (Javed et al., 2022). For most languages the gender distribution is close to 50-50 male/female, and some languages intentionally oversampled females to allow bias analysis (Javed et al., 2022). The data collection procedure used the Karya mobile app with a “maker-checker” verification flow. Quality was validated by human scoring on accuracy, volume, and quality, each on a 0–2 scale, with only 2’s accepted; the data were further filtered to retain high scoring, non-offensive, noise-free samples, and all participants provided informed consent through the app (Javed et al., 2022).
The textual prompts were drawn from IndicCorp, a web-crawled monolingual corpus, with 100k sentences per language restricted to 8–15 words per sentence for readability (Javed et al., 2022). This design matters because it couples controlled read-speech acquisition with a large lexical base and broad demographic coverage. A plausible implication is that the benchmark’s utility derives not only from aggregate hours, but from the interaction of linguistic diversity, speaker diversity, and carefully defined validation regimes.
3. Task suite, splits, and evaluation protocol
IndicSUPERB defines six benchmark tasks. Their construction and primary evaluation criteria are summarized below (Javed et al., 2022).
| Task | Construction | Metric |
|---|---|---|
| ASR | Audio-transcript pairs from Kathbath | WER |
| ASV | 50,000 same/different-speaker pairs per split per language | EER |
| SID-mono / SID-multi | Speaker-ID classification within one language or across all languages | Accuracy |
| LID | Pool audio from all languages with language labels | Accuracy |
| QbE | Spoken-query retrieval over curated keyword collections | MTWV |
| KS | Keyword classification using LDCIL command lists | Accuracy |
The ASR benchmark directly uses the aligned audio-transcript pairs from Kathbath, with train, validation, and test splits balanced for speaker and gender diversity and including unknown speakers and noisy variants (Javed et al., 2022). ASV is defined as a binary same-speaker/different-speaker task, with 50,000 pairs in each split and each language, balanced 50% positive and 50% negative (Javed et al., 2022). SID appears in mono-language and multi-language forms, distinguishing between closed-set speaker recognition within a language and across pooled multilingual speaker sets (Javed et al., 2022). LID pools audio from all 12 languages and predicts language identity (Javed et al., 2022).
QbE and KS extend the benchmark beyond conventional recognition. For QbE, each language uses 50 popular entities, 20 sentences per entity mined from the web, and recordings from 20 volunteers, producing a retrieval collection of approximately 1k utterances per query (Javed et al., 2022). For KS, the benchmark uses the LDCIL dataset’s command or keyword list, taking the top 50 per language and defining train, validation, and test splits accordingly (Javed et al., 2022).
The split structure is important. For each language and task, train, validation, and test splits are organized by speaker identity with Test-Known, Test-Unknown, and Noisy conditions; test splits are balanced for gender, while some training splits are intentionally imbalanced to study bias (Javed et al., 2022). This makes robustness to unseen speakers, acoustic degradation, and demographic skew a first-class property of the benchmark rather than an afterthought.
4. Baseline systems and empirical findings in the original release
The original evaluation compares a standard acoustic baseline, FBANK, against self-supervised encoders including IndicWav2Vec and XLS-R (Javed et al., 2022). IndicWav2Vec was pretrained on 17,000 hours of raw audio from 40 Indian languages, while XLS-R was pretrained on 128 languages over 500,000 hours and includes 11 of the 12 target Indic languages (Javed et al., 2022). For ASR, the benchmark fine-tunes pretrained models per language using CTC loss, also studies joint multilingual fine-tuning with a union character vocabulary, and integrates a 6-gram KenLM LLM trained on large monolingual corpora for decoding (Javed et al., 2022). For SID, LID, and KS, representations are mean-pooled and fed to a linear classifier with cross-entropy loss; for ASV, an X-Vector model with cosine similarity is evaluated by EER; for QbE, Dynamic Time Warping is applied using the best-performing validation distance metric (Javed et al., 2022).
The reported findings establish a clear hierarchy. Self-supervised representations outperform FBANK across all classification and recognition tasks, and language-specific fine-tuned models are more accurate than baseline on most tasks, including a large gap of 76\% for Language Identification (Javed et al., 2022). In the detailed results, LID accuracy on unknown speakers and clean speech rises from 14.10% with FBANK to 90.78% with IndicWav2Vec, while XLS-R reaches 79.96% (Javed et al., 2022). For SID-multi, accuracy improves from 36.79% with FBANK to 79.26% with IndicWav2Vec and 70.71% with XLS-R; for KS, average accuracy rises from 21.5% with FBANK to 96.9% with IndicWav2Vec and 97.1% with XLS-R; for QbE, MTWV improves from 0.001 with FBANK to 0.022 with IndicWav2Vec and 0.012 with XLS-R; for ASV on Test-Known, average EER drops from 3.0 with FBANK to 2.1 with both IndicW2V and XLS-R (Javed et al., 2022).
The ASR experiments further show that language-specific fine-tuning gives better performance than jointly fine-tuning multilingual models, although the gap is small (Javed et al., 2022). This is significant because it complicates a simple monolingual-versus-multilingual dichotomy: the benchmark results do not imply that multilingual fine-tuning is ineffective, only that language-specific optimization remains advantageous under the reported setup. Performance also drops on unknown speakers and noisy data across tasks, especially for ASV, and gender-balanced evaluation reveals that the more represented gender in training can achieve better performance, as illustrated by lower WER for female speakers in Hindi and Gujarati when training data contains more female samples (Javed et al., 2022).
The Kathbath resource also has extrinsic value beyond the benchmark itself. Adding Kathbath data to existing ASR training improves WER on public benchmarks such as MSR, MUCS, and OpenSLR, with an average 1.5% absolute reduction in WER despite domain differences (Javed et al., 2022). This suggests that IndicSUPERB’s underlying corpus is useful both as an evaluation standard and as a transferable training resource.
5. Use in ASR scaling and continual learning
IndicSUPERB became an early testbed for measuring the effect of newly mined supervision in low-resource Indic ASR. In work on mining audio-text pairs from public All India Radio archives, the benchmark was used as the primary means to evaluate ASR performance across seven languages: Bengali, Gujarati, Hindi, Marathi, Odia, Tamil, and Telugu (Bhogale et al., 2022). The comparison was between baseline ASR systems trained on prior public labelled data and systems trained with the addition of Shrutilipi, a mined dataset containing over 6,400 hours of labelled audio across 12 Indian languages totalling 4.95M sentences and yielding on average a 2.3x increase over publicly available labelled data (Bhogale et al., 2022). On IndicSUPERB, Wav2Vec2.0 models evaluated on Kathbath Test Unknown improved from an average WER of 21.1% to 15.3%, while on the MUCS Blind Set the average WER improved from 17.3% to 15.5%; the overall average reduction in WER was 5.82% (Bhogale et al., 2022). For Hindi, whose evaluation included seven benchmarks, average WER fell from 18.8% to 13.5%, and a Conformer model that was approximately 10x smaller than Wav2Vec improved from 21.2% to 18.9% (Bhogale et al., 2022). The same study reports that models trained with Shrutilipi are more resilient to noisy test conditions, with WER increase reduced by 5.9% under synthetic-noise benchmarks relative to the improvement on clean benchmarks (Bhogale et al., 2022).
A later line of work used a subset of IndicSUPERB for continual learning in Indic ASR under no-replay and privacy-conscious constraints (T et al., 8 Aug 2025). That study defines each language as a sequential task and uses nine languages—Hindi, Bengali, Marathi, Telugu, Tamil, Urdu, Gujarati, Kannada, and Odia—with 3,000 training utterances per language, split into 2,000 clean and 1,000 noisy examples, and 400 utterances each for validation and test, with 200 clean and 200 noisy samples per split (T et al., 8 Aug 2025). Evaluation is performed after each task on all previously seen tasks using WER on both RNN-T and CTC decoding paths, together with average WER and Backward Transfer:
where and (T et al., 8 Aug 2025).
The study compares naive fine-tuning with Elastic Weight Consolidation, Memory Aware Synapses, and Learning without Forgetting using an indicconformer hybrid RNN-T/CTC model (T et al., 8 Aug 2025). Its findings are relevant to the interpretation of IndicSUPERB as a benchmark: LwF performs best overall in mitigating catastrophic forgetting, MAS offers a trade-off, EWC sometimes achieves low WER but worse BWT, naive fine-tuning degrades on longer task horizons, increasing epochs worsens forgetting, and noisy training improves BWT while worsening absolute WER (T et al., 8 Aug 2025). These results show that IndicSUPERB can function not only as a fixed supervised benchmark but also as a controlled sequential-task environment for studying the stability-plasticity dilemma in multilingual ASR.
6. Reuse in deepfake detection and broader significance
IndicSUPERB has also been repurposed outside conventional ASR. The Indic-CodecFake dataset uses IndicSUPERB as its real speech corpus, taking genuine speech from the official train, validation, and test splits to build the first large-scale benchmark of real and Neural Audio Codec-synthesized speech across multiple Indic languages (Girish et al., 21 Apr 2026). The languages listed in that work are Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, Sanskrit, Tamil, Telugu, and Urdu, and the benchmark explicitly relies on IndicSUPERB’s speaker, accent, speaking-style, and regional diversity (Girish et al., 21 Apr 2026). Fake speech is generated by resynthesizing each real waveform through multiple neural audio codecs, and evaluation includes seen-codec and unseen-codec settings (Girish et al., 21 Apr 2026).
This reuse exposes a limitation of assuming that strong English-centric models transfer cleanly to Indic speech domains. In the deepfake setting, AASIST trained on English-based CodecFake achieves 94.21% ACC on its original domain but only 48.0% ACC on Indic-CodecFake, while zero-shot ALMs such as Qwen2-audio, AudioFlamingo-2/3, and Pengi perform poorly, all at or below 15% accuracy with high EER (Girish et al., 21 Apr 2026). By contrast, SATYAM, a hyperbolic ALM that combines semantic representations from Whisper and prosodic representations from TRILLsson with Bhattacharya-distance-based alignment, reaches 98.3% ACC and 3.3% EER on the ICF test set (Girish et al., 21 Apr 2026). The specific architectural details belong to the deepfake study rather than to IndicSUPERB itself, but the benchmark’s role is foundational: its authentic multilingual speech data make it possible to expose cross-lingual distribution shift and evaluate robustness in a realistic Indic setting.
Taken together, the original benchmark paper and later derivative studies establish several durable points. First, IndicSUPERB is intrinsically multi-task, not ASR-only (Javed et al., 2022). Second, it is useful both as an evaluation benchmark and as a source corpus whose diversity improves downstream systems when added to training, as seen with Kathbath and Shrutilipi (Javed et al., 2022, Bhogale et al., 2022). Third, strong pretraining does not eliminate the need for benchmark-specific analysis under unseen speakers, noise, continual learning, or synthetic-speech distribution shift (Javed et al., 2022, T et al., 8 Aug 2025, Girish et al., 21 Apr 2026). This suggests that IndicSUPERB’s central contribution is methodological as much as empirical: it provides a standardized, multilingual, and stress-tested substrate for studying robustness, transfer, and failure modes in Indian speech technology.