Vaani Dataset: Indian Speech & Multimodal Data

• Vaani Dataset is a family of interconnected resources comprising diverse Indian speech, audio–image pairs, and Hindi benchmark subsets.
• It employs district-centric, geo-representative collection methods with image-based elicitation and robust metadata-driven quality control.
• The datasets drive advances in self-supervised ASR, transcription-free adaptation, and multilingual spoken language identification in low-resource settings.

The term Vaani Dataset does not denote a single immutable artifact in the recent literature. Instead, it refers to a set of related but distinct speech and multimodal resources used in Indian language technology research, most prominently the large-scale VAANI multimodal collection, the GramVaani/Vaani Hindi ASR challenge corpus, and the later Vaani Benchmark V1.0 for inclusive Hindi evaluation. Across these works, Vaani data support low-resource ASR, self-supervised and continued pretraining, spoken language identification, and audio–image representation learning, with recurring design choices centered on spontaneous speech, geographic breadth, multimodal pairing, and realistic acoustic conditions (Pulikodan et al., 30 Mar 2026, Seth et al., 2022, Pulikodan et al., 23 Jun 2026).

1. Terminological scope and dataset family

Published usage of the name Vaani spans several resources that are connected by theme and, in some cases, by collection methodology, but are not interchangeable.

Resource	Reported composition	Primary role
VAANI project dataset	~289,838 images; 24,009,427 audio segments; ~31,270 hours audio; ~2,067 hours transcribed; 112 languages and dialects; 165 districts	Multimodal speech–image–text collection
Vaani in audio–image alignment study	~31,255 hours; 105 languages; 1,894 hours transcribed; 11,848,593 audio–image pairs; 287K unique images	Pretraining, alignment, and ASR fine-tuning corpus
GramVaani / Vaani challenge corpus	1108 hours Hindi telephone speech; 1000 hours unlabeled; 100 hours labeled train; 5 hours dev; 3 hours blind test	Hindi ASR benchmark and continued-pretraining target
Vaani Benchmark V1.0	20.64 hours; 3,252 speakers; 104 districts; three independent transcriptions per segment	Inclusive Hindi ASR evaluation benchmark
Balanced Vaani LID subset	10 hours per language for 42 languages; 8:1:1 speaker-disjoint split	Spoken language identification benchmark

This multiplicity of usages is itself a defining property of the term in current research. One line of work uses Vaani to denote a large multilingual, district-centric, multimodal corpus for India-wide modeling; another uses it for a Hindi telephone-speech challenge dataset; and a third derives evaluation-focused Hindi benchmark subsets from a broader collection (Pulikodan et al., 30 Mar 2026, Seth et al., 2022, Pulikodan et al., 19 Jun 2026, Basu et al., 8 Jun 2026).

Outside machine-learning benchmark literature, the closely related names Gram Vaani and Mobile Vaani refer to voice-driven participatory media systems built around IVR, moderation, community mobilization, and district-level organization. Those systems provide socio-technical background for some Vaani-named resources, but they are not themselves presented as standardized ASR benchmark corpora in the same way as the later dataset papers (Seth, 2019, Moitra et al., 2021).

2. Corpus design, collection strategy, and multimodal structure

The large-scale VAANI dataset is explicitly framed as an India-representative multimodal resource designed to capture linguistic diversity through a district-centric, geo-centric collection strategy rather than a purely language-centric one. In one published description, it open-sources around 289K images, approximately 31,270 hours of audio recordings, and around 2,067 hours of transcribed speech, covering 112 languages from 165 districts from 31 States and Union territories (Pulikodan et al., 30 Mar 2026). A later paper reports a closely related Vaani resource with approximately 31,255 hours of speech covering 105 languages, of which 1,894 hours are transcribed (Pulikodan et al., 23 Jun 2026). This suggests that the label Vaani dataset may denote closely related releases or accounting conventions rather than a single fixed snapshot.

A central design choice is image-based elicitation. Speakers are shown an image and asked to describe it in their own words, producing spontaneous responses rather than read speech. The images were physically captured for the project, include district-specific as well as more general content, and were constrained to be .jpg, 640×400 pixels, under 500 KB, clear, unique, free of PII, and not sourced from online or third-party media. The capture date was required to be no earlier than July 1, 2023 in the project description (Pulikodan et al., 30 Mar 2026). The audio–image alignment paper emphasizes the same collection principle and treats the resulting corpus as naturally containing paired audio–image examples because each utterance is elicited in response to a specific image (Pulikodan et al., 23 Jun 2026).

The speech collection protocol is designed to preserve authenticity while maintaining quality control. Speakers were recruited locally, had to be native residents of the recorded pincode, and identity was verified using documents such as PAN or Aadhaar. Participants were between 20 and 70 years old; gender balance was maintained; each district had at least 800 speakers; no speaker contributed more than 15 minutes of effective speech; and speakers were asked to produce 10–20 seconds of effective speech per image in the language or dialect they speak at home (Pulikodan et al., 30 Mar 2026). Recordings were 16 kHz, 16-bit, single channel / mono, and raw, with no transcoding or post-processing.

Metadata are an important part of the corpus design. Reported fields include State, District, Gender, Pincode, Asserted Language, and Languages Spoken. Earlier collection descriptions also mention age, education, socio-economic background, location, and duration of stay in that location. This metadata structure makes the dataset relevant not only for ASR, but also for region-aware modeling, fairness analysis, and district-level performance studies (Pulikodan et al., 30 Mar 2026).

The quality-control pipeline is multi-stage and combines automated and manual validation. Automated audio checks include metadata validation, file-format checks, duration validation, duplicate detection, silence checks, and Signal-to-Noise Ratio (SNR) screening for manual review. Automated transcription checks include ID validation, script consistency, word/duration ratio checks, LM log-likelihood assessment, and language/geographic plausibility checks. Manual validation is district-local and expert-led; if more than 10% of a batch is problematic, the batch is escalated to additional experts (Pulikodan et al., 30 Mar 2026). The transcription guidelines require verbatim transcription, explicit marking of [unintelligible], [inaudible], <UNKNOWN_SEGMENT>, and <PAUSE>, retention of slang, written-out numbers, and markup for incomplete utterances and foreground sounds.

3. Benchmark variants for Hindi ASR evaluation

Two distinct Hindi evaluation resources appear under the Vaani name in the literature.

The earlier GramVaani / Vaani challenge corpus is described as a corpus of 1108 hours of real-world, spontaneous telephone speech recordings in multiple dialects of Hindi, released as part of the Interspeech GramVaani ASR Challenge 2022. Of this total, 1000 hours is unlabeled, 100 hours is labeled training data, 5 hours is development data, and 3 hours of blind test data was released for evaluation. The audio is recorded at different sampling rates ranging from 8 kHz to 48 kHz and stored in mp3 format. For wav2vec 2.0 continued pretraining, 991 hours of unlabeled audio were used for pretraining and 9 hours were held out for validation. The paper explicitly notes that it does not further define the dataset’s exact annotation protocol, speaker composition, or official split filenames (Seth et al., 2022).

The later Vaani Benchmark V1.0 is an evaluation-focused Hindi ASR benchmark dataset built to address limited geographic coverage, weak demographic diversity, and the inadequacy of single-reference scoring. It was collected from 104 districts across 22 states and Union Territories of India, contains 20.64 hours of audio from 3,252 speakers, and was sampled from a broader collection of about 200 hours gathered across 165 districts and 31 states/UTs, with up to 15 minutes per district retained for the benchmark (Pulikodan et al., 19 Jun 2026). Unlike the telephone-speech challenge corpus, it is explicitly multimodal, consisting of aligned image, speech, and text.

The benchmark’s defining annotation feature is that each audio segment has three independent transcriptions. The pipeline begins with a human transcription checked by two ASR systems, followed by consistency checks and automated validation, independent review and correction by separate transcribers, random supervisor audits of about 10% of transcriptions, and further manual validation by three independent transcribers when discrepancies are large. Transcribers are selected from the same district as the speech source when possible. The dataset also annotates code-switched content in both the native script and the original script, and marks non-speech audio events/noise (Pulikodan et al., 19 Jun 2026).

This design motivates three evaluation protocols. Approach 1 computes WER against each reference separately and reports the mean. Approach 2 takes the minimum error count across references per utterance before aggregation. Approach 3 is a multi-reference alignment-based WER that accounts for alternative realizations across the three transcriptions. The paper defines

$\text{WER} = \frac{S + I + D}{N}$

with utterance-level deletion and effective-length terms

$$D_u = \bigcap_{i=1}^{3} deleted\_per\_ref[i], \qquad N_u = |H| + |D_u| - I_u.$$

The reported pairwise inter-set WERs between transcription sets—10.51%, 13.62%, and 12.91%—are used to argue that transcription variation remains substantial even after quality control, making single-reference evaluation inadequate (Pulikodan et al., 19 Jun 2026).

4. Role in self-supervised ASR and continued pretraining

The GramVaani/Vaani Hindi corpus became a central testbed for analyzing when self-supervised pretraining is useful for low-resource ASR. In that setting, the downstream recognizer is a joint CTC/attention encoder-decoder with a 12-block conformer encoder, a 6-block transformer decoder, and CTC weight = 0.4. Upstream SSL representations are 1024-dimensional and passed through a linear layer to obtain 80-dimensional features; the setup uses layer-wise weighted sums of upstream features and beam search with beam size 20. The paper compares an Fbank-Pitch baseline, W2V-GV trained from scratch on GramVaani unlabeled data, and several wav2vec 2.0 LARGE upstreams: LL, LL+CV+SF, XLSR-128, and IndicW2V (Seth et al., 2022).

Without continued pretraining, the Hindi results are 34.2 dev / 33.7 test for Fbank-Pitch, 32.4 / 32.3 for W2V-GV, 35.0 / 34.4 for LL, 34.3 / 34.2 for LL+CV+SF, 32.7 / 32.5 for XLSR-128, and 33.6 / 33.1 for IndicW2V. After continued pre-training (CP) on GramVaani Hindi unlabeled audio, the same upstreams reach 29.7 / 29.8 for LL, 29.1 / 28.9 for LL+CV+SF, 27.3 / 27.1 for XLSR-128, and 31.4 / 31.5 for IndicW2V. The paper identifies XLSR-128 as the best-performing upstream for Hindi and states that, after CP, it improves over the FBank baseline by 6.9% WER on dev and 6.6% WER on test; even before CP, XLSR-128 is already better than FBank by 1.5% dev and 1.2% test (Seth et al., 2022).

The same corpus is also used to probe an extreme low-resource condition with only 10 hours of labeled GramVaani data. In that regime, the reported Hindi WERs are 75.7 dev / 74.5 test for Fbank-Pitch, 50.4 / 50.3 for W2V-GV, 56.8 / 55.9 for XLSR-128, and 46.2 / 46.0 for XLSR-128 (Continued pre-trained). The paper summarizes this as an absolute 29.5% WER improvement over the FBank baseline and 4.2% over W2V-GV, with roughly 29% absolute WER improvement from the FBank baseline when averaging dev and test (Seth et al., 2022).

The central interpretation is that ASR performance improves with increasing similarity and volume of pretraining data, and that continued pretraining on in-domain Hindi audio materially improves downstream recognition. At the same time, the study reports catastrophic forgetting for multilingual upstreams when they are continued-pretrained on Hindi and then evaluated on other languages: for XLSR-128, average WER degradation after Hindi CP is 3.5% for Gujarati, 2.8% for Tamil, and 3.2% for Telugu. Vaani therefore serves not only as a benchmark, but also as an instrument for studying domain adaptation, language match, domain diversity, and cross-language interference in SSL-based ASR (Seth et al., 2022).

5. Audio–image continued pretraining and transcription-free adaptation

In later multimodal work, Vaani is used not merely as an evaluation set, but as the enabling substrate for a transcription-free adaptation stage based on naturally aligned audio–image pairs. The reported pipeline has three stages: (1) pretraining of a FastConformer audio encoder on Vaani speech, (2) continued pretraining / representation alignment using audio–image pairs, and (3) supervised ASR fine-tuning using transcribed speech. The audio backbone is a 17-layer FastConformer encoder, and all 17 layers are trainable during alignment. The image side is frozen and instantiated with SigLIP2-base / patch16-256 with a 768-dimensional embedding space, SigLIP2-large / patch16-384 with a 1024-dimensional space, or Qwen3-VL-Embedding-2B / visual stack with a 2048-dimensional space (Pulikodan et al., 23 Jun 2026).

The multimodal supervision comes from a large alignment set of 11,848,593 audio–image pairs, spanning 287K unique images and 16,580.36 hours of audio. The audio encoder is coupled to an MLP alignment head that maps audio representations into the image embedding space. Three alignment configurations are reported. In the single-vector SigLIP variant, audio uses single-query attention pooling and images use one pooled vector. In SigLIP-MT and Qwen-MT, the audio side uses multi-query attention pooling with $K_a = 16$ audio queries. On the image side, SigLIP-MT selects the top 16 patch tokens by L2 norm from 576 patches, while Qwen-MT performs a 2×2 spatial merge to yield up to 16 image tokens. Pair scoring uses cosine similarity in the single-vector case and a MaxSim late-interaction operator in the multi-token case:

$$\text{MaxSim } \tfrac{1}{K_a}\sum_a\max_v\langle q_a, V_v\rangle .$$

Training uses a SigLIP-style sigmoid contrastive loss, written in the paper as

$-\log\sigma(y_{ij}(t\,s_{ij}+b)),$

where $s_{ij}$ is the pair score, $y_{ij}$ is the positive/negative label, and $t$ and $b$ are learnable temperature and bias terms. The alignment stage is optimized with in-batch negatives gathered across all GPUs, batch size 64, AdamW, weight decay 0.01, gradient clipping at 1.0, bfloat16, 1,000-step warmup, and 200,000 optimization steps. Newly initialized layers use learning rate $3\times 10^{-4}$, while pretrained encoder weights use a smaller rate scaled by 0.05 (Pulikodan et al., 23 Jun 2026).

For supervised ASR fine-tuning, the paper uses 1,636 hours of Vaani speech for training, 177.6 hours for validation, and 81.05 hours for testing, with evaluation on a 48-language test partition. In a separate out-of-domain setting, it fine-tunes and evaluates on FLEURS South Asia with 124.35 hours train, 16.44 hours dev, and 36.88 hours test. Evaluation is by WER, with paired utterance-level bootstrap confidence intervals and p-values (Pulikodan et al., 23 Jun 2026).

The empirical finding is that audio–image alignment consistently improves ASR relative to direct fine-tuning from the pretrained audio encoder. On the in-domain Vaani test set, baseline WER 0.2809 is reduced to 0.2768 with Qwen3-VL, 0.2771 with SigLIP2-base, and 0.2740 with SigLIP2-large; the best relative improvement is 2.47% for SigLIP2-large. On FLEURS South Asia, the baseline 0.6778 drops to 0.5683 with Qwen3-VL, 0.5358 with SigLIP2-large, and 0.5338 with SigLIP2-base. The paper reports that the best model improves 13 of 14 languages, with only one language degrading and no statistically significant harmful regressions, and further notes that the benefit of alignment shrinks as more labeled fine-tuning data becomes available (Pulikodan et al., 23 Jun 2026).

6. Spoken language identification and cross-corpus generalization

Vaani is also used as the central training corpus for large-scale Indic spoken language identification (LID). In that study, the authors derive a balanced subset by selecting 10 hours of speech per language for 42 languages, aiming to maximize diversity in districts and speakers, and split it into train/validation/test in an 8:1:1 ratio with speaker-disjoint splits. The 42 languages span four linguistic families: Indo-Aryan, Dravidian, Sino-Tibetan, and European represented by English (Basu et al., 8 Jun 2026).

The task is explicitly cross-corpus. Models are trained on Vaani and evaluated on Vaani-Test as the in-domain set, and on FLEURS and Kathbath as out-of-domain sets. The paper compares Whisper and FastConformer encoders, each paired with a linear classifier, under three objectives: cross-entropy (CE), CE + supervised contrastive loss, and hierarchical softmax (HSM). The HSM model uses a tree Root → Family → Sub-family → Language, with

$$D_u = \bigcap_{i=1}^{3} deleted\_per\_ref[i], \qquad N_u = |H| + |D_u| - I_u.$$0

and loss

$$D_u = \bigcap_{i=1}^{3} deleted\_per\_ref[i], \qquad N_u = |H| + |D_u| - I_u.$$1

The reported findings are strongly Vaani-dependent. With CE, Whisper benefits substantially from fine-tuning on Vaani, moving from 56.0 → 71.8 on Vaani-Test, 61.9 → 72.7 on FLEURS, and 57.7 → 68.3 on Kathbath. FastConformer, by contrast, is already strong when frozen, with 67.4 on Vaani-Test, 94.2 on FLEURS, and 90.9 on Kathbath; fine-tuning changes Vaani-Test only marginally (67.4 → 67.6) and hurts out-of-domain performance to 89.9 and 87.4 on FLEURS and Kathbath respectively. HSM consistently outperforms CE and CE+SupCon for both encoders, with the largest gains on out-of-domain benchmarks. The paper also reports that CE+SupCon degrades FastConformer’s cross-corpus generalization, which it interprets as over-specialization to Vaani’s in-domain conditions (Basu et al., 8 Jun 2026).

The linguistic analysis enabled by Vaani is equally important. Family-level results show that Central Indo-Aryan is the hardest subgroup for both encoders. The dominant confusion pairs are Hindi–Urdu and the Sadri–Chhattisgarhi–Surgujia cluster, while Bajjika and Halbi are relatively better identified. In this role, Vaani functions as a taxonomy-aware benchmark for fine-grained discrimination among closely related Indic varieties, not merely as a source of raw audio (Basu et al., 8 Jun 2026).

7. Documentation limits, ambiguities, and nearby names

The Vaani literature contains several documentation limits that materially affect interpretation. The GramVaani/Vaani challenge paper gives split hours, domain description, and downstream usage, but explicitly does not define the exact annotation protocol, speaker composition, or official split filenames (Seth et al., 2022). Conversely, Vaani Benchmark V1.0 provides extensive detail on transcription multiplicity and evaluation methodology, but it is a small evaluation set rather than a full-scale pretraining corpus (Pulikodan et al., 19 Jun 2026). The large-scale VAANI project paper is rich on collection design and QC, yet the exact counts reported there do not numerically match the later audio–image alignment paper (Pulikodan et al., 30 Mar 2026, Pulikodan et al., 23 Jun 2026).

A common misconception is to assume that all Vaani-named resources are the same dataset. The published evidence does not support that simplification. The Hindi telephone-speech challenge corpus, the district-centric multimodal VAANI collection, the Hindi benchmark subset with three references, and the balanced 42-language LID subset serve different experimental roles and are described with different statistics. A plausible implication is that Vaani functions as a dataset family label in the literature rather than a single canonical release.

Another potential confusion concerns Vedavani. Although the name is similar, the paper "Vedavani: A Benchmark Corpus for ASR on Vedic Sanskrit Poetry" presents a separate 54+ hour Sanskrit speech corpus for Vedic Sanskrit poetry and explicitly does not identify it as the Vaani dataset. It is therefore a distinct benchmark with a related name, not a Vaani release in the sense used by the VAANI and GramVaani papers (Kumar et al., 30 May 2025).

Finally, the broader Gram Vaani / Mobile Vaani literature shows that some Vaani-named resources originate in a larger ecosystem of voice-based participatory media, IVR interaction, moderation, district-level organization, and community mobilization. That background is relevant because it highlights selection effects, editorial mediation, and deployment context; however, those papers describe a socio-technical platform rather than the standardized benchmark datasets used in recent ASR, LID, and multimodal representation-learning studies (Seth, 2019, Moitra et al., 2021).