Project VAANI: India Multilingual Speech Data
- Project VAANI is an initiative that builds an India-representative multimodal dataset capturing spontaneous, image-prompted speech across 112 languages and 165 districts.
- The project employs a district-centric collection strategy and strict quality protocols to preserve natural dialectal variation and regional nuances.
- Vaani Benchmark V1.0, a subset of the corpus, provides a Hindi ASR benchmark using multi-reference evaluation to reflect transcription variability and improve model accuracy.
Project VAANI is an initiative to create an India-representative multi-modal dataset that maps India’s linguistic diversity through image-prompted spontaneous speech, associated images, and a transcribed subset. In its first two phases, it covers 165 districts across 31 States and Union Territories and releases around 289K images, approximately 31,270 hours of audio recordings, and around 2,067 hours of transcribed speech spanning 112 languages. Subsequent work derived Vaani Benchmark V1.0, a Hindi automatic speech recognition benchmark constructed from the broader corpus and designed for inclusive, multi-reference evaluation under geographic and demographic variation (Pulikodan et al., 30 Mar 2026, Pulikodan et al., 19 Jun 2026).
1. Scope and conceptual orientation
Project VAANI is framed as a response to a structural limitation of much Indian language technology: existing Indic speech resources often focus on a relatively small set of major languages, rely heavily on read speech, and do not adequately capture geographic, dialectal, and sociolinguistic variation. Its organizing principle is therefore not purely language-centric. Instead, the project adopts a geo-centric, district-centric collection strategy intended to reflect how speech varies across districts, regions, communities, genders, education levels, and social backgrounds. This suggests that the project treats linguistic representation as inseparable from regional and social variation rather than as a flat inventory of named languages (Pulikodan et al., 30 Mar 2026).
The initiative is also explicitly multimodal. Each speech sample is elicited through an image prompt, creating image–speech pairs for the full corpus and image–speech–text triplets where transcription is available. The image-prompt design is intended to encourage spontaneous responses rather than constrained reading, thereby preserving natural lexical variation, local description strategies, and district-specific expression. The project’s broader motivation is inclusive digital access: the paper argues that speech interfaces are especially important where text literacy, device familiarity, or language standardization remain barriers, and positions VAANI as infrastructure for ASR, language identification, multilingual speech technology, and multimodal learning in Indian settings (Pulikodan et al., 30 Mar 2026).
A recurrent source of confusion is the relationship between Project VAANI and Vaani Benchmark V1.0. They are related but not identical. Project VAANI is the broader multimodal corpus; Vaani Benchmark V1.0 is a Hindi ASR benchmark subset derived from that corpus for systematic model evaluation (Pulikodan et al., 30 Mar 2026, Pulikodan et al., 19 Jun 2026).
2. Corpus scale, geographic coverage, and linguistic composition
The released corpus spans 165 districts, 28 states and 3 Union Territories, 112 languages, 158,441 speakers, 24,009,427 audio segments, approximately 31,270 hours of audio, around 2,067 hours of transcription, and 289,838 images. Many low-resource and under-documented languages are represented, including Angika, Khortha, Malvani, Shekhawati, Duruwa, Jaipuri, Bearybashe, Kurumuli, Kudukh or Kurukh, Bajjika, Agariya, and Halbi. The paper further notes that 34 rare languages were collected from North-Eastern India alone, and that many of these do not even have their own scripts (Pulikodan et al., 30 Mar 2026).
The broader corpus and its benchmark derivative can be summarized as follows.
| Artifact | Scale | Function |
|---|---|---|
| Project VAANI release | 165 districts; 31 States and Union Territories; 112 languages; 158,441 speakers; 24,009,427 audio segments; 31,270 hours audio; 2,067 hours transcribed; 289,838 images | India-representative multimodal corpus |
| Vaani Benchmark V1.0 | 104 districts; 22 States and Union Territories; 3,252 speakers; 20.64 hours; 8,315 images; three independent transcriptions per utterance | Hindi ASR benchmark derived from the broader project |
The significance of this scale is not only numerical. The corpus is meant to be India-representative in the sense of broad geographic and linguistic coverage, not in the narrower sense of population-proportional sampling. District-level collection is used as a proxy for capturing accental and dialectal variation within named languages. The paper’s empirical analysis later supports this design choice: region-specific Hindi fine-tuning showed that a model trained on Hindi from one state performed better on geographically proximate states than on distant states, even when the language label remained “Hindi” (Pulikodan et al., 30 Mar 2026).
3. Collection design and multimodal methodology
The collection pipeline has three major components: image capture, speech collection, and transcription. Images are not incidental prompts but a central elicitation mechanism. District vendors were asked to gather 1,700 to 2,000 images per district, including both district-specific images and generic images. Images had to be physically captured for the initiative rather than scraped, stored in .jpg format at 640 × 400 resolution and under 500 KB, and screened to avoid personally identifiable information, recognizable logos, and privately owned objects. This suggests that the visual modality was designed both for prompt diversity and for later multimodal research use (Pulikodan et al., 30 Mar 2026).
Speaker recruitment was coordinated district by district. The protocol required at least 800 speakers per district, age between 20 and 70 with a uniform distribution across this range, gender balance, and no more than 15 minutes of effective speech per speaker. Speakers had to be native residents of the recorded pincode, with residence validated through identity documents such as PAN or Aadhaar, and were encouraged to speak the language or dialect used at home with family. During onboarding, speakers registered on the vendor platform, recorded sample responses to sample images, passed a quality screen, and then described district-specific and generic images in their own words (Pulikodan et al., 30 Mar 2026).
Audio was collected through vendor mobile applications under tightly specified technical constraints: 16 kHz, 16-bit, mono, raw audio, with no transcoding or post-processing. Recording instructions specified a quiet indoor environment, avoidance of television, kitchen, traffic, bird noise, and echo, and placement of the recording device between one and two feet from the speaker’s mouth. This design reflects an attempt to standardize capture quality while still preserving spontaneous speech. It also indicates that the corpus is not simply a collection of read prompts transferred into audio, but a structured elicitation of free-form descriptions grounded in images and local context (Pulikodan et al., 30 Mar 2026).
The benchmark paper preserves this same logic at evaluation scale. Vaani Benchmark V1.0 consists of spontaneous Hindi speech elicited by image descriptions and paired with image and text, making it suitable not only for ASR but also for audio-based image retrieval, text-based image retrieval, speaker identification, district identification, and future multimodal ASR and visual grounding tasks (Pulikodan et al., 19 Jun 2026).
4. Transcription protocol and quality assurance
Only a subset of the corpus was transcribed, but the transcription process was highly structured. Segments selected for transcription were distributed nearly evenly across the 165 districts, and transcribers were required to come from the same district as the audio being transcribed. This district-local design was intended to preserve familiarity with local dialects and speech patterns. Transcription followed detailed verbatim guidelines: words were transcribed only if properly heard and understood; otherwise [unintelligible] or [inaudible] was used; unknown-language content was marked with <UNKNOWN_SEGMENT>; pauses longer than 0.5 seconds were marked with <PAUSE>; speech errors, slang, repetitions, false starts, fillers, and stutters were preserved; numbers were normalized to word form rather than numerals; and incomplete utterances ended with -- (Pulikodan et al., 30 Mar 2026).
The quality-control pipeline combined automated and manual stages. Automated audio QC checked metadata integrity, filename consistency, format compliance, duration constraints, duplicate identifiers, silence at the start or end of segments, and speaker-level duration limits. A later stage computed signal-to-noise ratio and flagged files for manual review. Manual validation then sampled all flagged files, at least one segment from each speaker, and a random sample such that 10% of data passing the first two automated stages also underwent human inspection. Validators from the corresponding districts checked whether the audio contained human speech, whether only one person was speaking, whether language and gender labels matched, whether the content was coherent and relevant to the image, and whether no personally identifiable information was present (Pulikodan et al., 30 Mar 2026).
Transcription QC used both structural and linguistic checks. These included duration-to-word-count consistency, script uniformity, repetition ratios, language consistency, pincode relevance, language-model loglikelihood thresholding, and WER thresholding. Manual transcription validation checked exact audio-text correspondence, script-language compatibility, naturalness of the speech, and absence of personally identifiable information. A further escalation rule required that if quality issues exceeded a 10% threshold, the entire dataset was sent for further validation by additional experts. The overall design indicates that VAANI treats annotation quality as a core research variable rather than a downstream cleanup step (Pulikodan et al., 30 Mar 2026).
5. Vaani Benchmark V1.0 and multi-reference evaluation
Vaani Benchmark V1.0 operationalizes the corpus as a Hindi ASR benchmark for inclusive evaluation. It contains 20.64 hours of spontaneous speech from 3,252 speakers across 104 districts in 22 states and Union Territories, elicited using 8,315 images and recorded under real-world acoustic conditions. Each utterance has three independent human transcriptions. This three-reference design is central to the benchmark’s critique of standard single-reference ASR evaluation for Hindi spontaneous speech (Pulikodan et al., 19 Jun 2026).
The benchmark quantifies inter-transcriber disagreement directly. Pairwise disagreement, measured as WER between transcription sets, is 10.51% for Set 1 vs Set 2, 13.62% for Set 1 vs Set 3, and 12.91% for Set 2 vs Set 3. The paper interprets the residual 10–15% disagreement not as annotation failure but as evidence of orthographic variation, lexical variation, listener-dependent perception, dialectal diversity, and code-switching representation differences. This is one of the project’s clearest methodological claims: multiple valid transcriptions may exist for the same audio segment in Hindi spontaneous speech, so single-reference WER systematically over-penalizes ASR systems (Pulikodan et al., 19 Jun 2026).
To address this, the benchmark defines three scoring approaches. Approach 1 averages single-reference WER over the three transcription sets. Approach 2 computes segment-level errors against all three references and keeps the minimum error count. Approach 3 is an alignment-based multi-reference WER that reconciles word-level alternatives across the three references. The benchmark also reports district-wise performance using mean WER and standard deviation across districts, making it possible to inspect geographic bias in addition to average accuracy (Pulikodan et al., 19 Jun 2026).
The benchmark evaluates 21 systems, including open and proprietary models. The best model reported is Vaani Fast Conformer, an open model, with WER 17.5 under Approach 1, 14.0 under Approach 2, 10.6 under Approach 3, and district mean ± standard deviation of 15.2 ± 4.1. The next best systems include Gemini-3.1-Pro at 18.8 / 15.1 / 11.9 and Sarvam Saaras v3 at 20.3 / 16.9 / 13.7. The benchmark’s main empirical finding is that multi-reference scoring materially lowers measured WER for nearly every system; for example, Vaani Fast Conformer drops from 17.5 to 10.6, and Gemini-3.1-Pro from 18.8 to 11.9. This demonstrates that evaluation protocol changes can alter conclusions about model quality on regionally varied Hindi speech (Pulikodan et al., 19 Jun 2026).
6. Research uses, empirical findings, and limitations
The broader VAANI corpus has already been used for several downstream tasks. The resource paper reports language-specific ASR fine-tuning with Whisper-small on Hindi, Kannada, Telugu, and Bengali; multilingual ASR fine-tuning with Gemma-3n-2B, Whisper-large-v3-turbo, and parakeet-tdt-0.6b-v2 on Hindi, Kannada, Telugu, Bengali, Chakma, and Bhojpuri; region-specific Hindi fine-tuning across Bihar, Uttar Pradesh, Chhattisgarh, Maharashtra, Jharkhand, Uttarakhand, Rajasthan, Goa, and Andhra Pradesh; spoken language identification across 42 languages; and image retrieval using SigLIP2 over Hindi image–speech–text data (Pulikodan et al., 30 Mar 2026).
The language-identification results provide the clearest quantitative cross-task evidence. Using a Whisper-medium encoder with a pooled attention layer and a two-layer classifier, the paper reports 74.8% accuracy on the VAANI test set, 76.1% on FLEURS, and 65.8% on Kathbath. Per-language F1 values were high for languages such as Assamese, Garo, Malayalam, and Wancho, but much weaker for closely related varieties such as Hindi, Khariboli, Maithili, Urdu, Bhojpuri, and Magahi. This suggests that the corpus is not merely large, but sufficiently fine-grained to expose the difficulty of discriminating among closely related Indo-Aryan varieties (Pulikodan et al., 30 Mar 2026).
At the same time, the project has explicit limitations. Collection at this scale is described as operationally challenging, resource intensive, and time consuming. The target of roughly 200 hours per district was not met uniformly, so geographic and language coverage remain long-tailed and imbalanced. Only around 2,067 hours of the 31,270 hours of audio are transcribed, which limits fully supervised work for many languages. Some represented languages lack scripts or have weakly standardized orthographies, complicating asserted-language labels and transcription practice. The benchmark derivative is also narrower than the parent corpus: it evaluates only ASR, even though the underlying design is multimodal and could support broader tasks (Pulikodan et al., 30 Mar 2026, Pulikodan et al., 19 Jun 2026).
The most important conceptual implication is that Project VAANI is not only a dataset release but a methodological argument. It advances a district-centric view of linguistic representation, treats spontaneous image-prompted speech as preferable to read-speech dominance, and shows that both model training and model evaluation change when regional variation and multiple valid transcriptions are taken seriously. In that sense, the project functions simultaneously as corpus infrastructure, evaluation reform, and a speech-first account of linguistic inclusion in Indian AI systems (Pulikodan et al., 30 Mar 2026, Pulikodan et al., 19 Jun 2026).