---
title: 'Vaani Dataset: Indian Speech & Multimodal Data'
url: https://www.emergentmind.com/topics/vaani-dataset
type: topic
---

# Vaani Dataset: Indian Speech & Multimodal Data

The term **Vaani Dataset** does not denote a single immutable artifact in the recent literature. Instead, it refers to a set of related but distinct speech and multimodal resources used in Indian language technology research, most prominently the large-scale **VAANI** multimodal collection, the **GramVaani/Vaani** Hindi ASR challenge corpus, and the later **Vaani Benchmark V1.0** for inclusive Hindi evaluation. Across these works, Vaani data support low-resource ASR, self-supervised and continued pretraining, spoken language identification, and audio–image representation learning, with recurring design choices centered on spontaneous speech, geographic breadth, multimodal pairing, and realistic acoustic conditions [2603.28714][2203.16973][2606.24080].

## 1. Terminological scope and dataset family

Published usage of the name *Vaani* spans several resources that are connected by theme and, in some cases, by collection methodology, but are not interchangeable.

| Resource | Reported composition | Primary role |
|---|---|---|
| **VAANI project dataset** | ~289,838 images; 24,009,427 audio segments; ~31,270 hours audio; ~2,067 hours transcribed; 112 languages and dialects; 165 districts | Multimodal speech–image–text collection |
| **Vaani in audio–image alignment study** | ~31,255 hours; 105 languages; 1,894 hours transcribed; 11,848,593 audio–image pairs; 287K unique images | Pretraining, alignment, and ASR fine-tuning corpus |
| **GramVaani / Vaani challenge corpus** | 1108 hours Hindi telephone speech; 1000 hours unlabeled; 100 hours labeled train; 5 hours dev; 3 hours blind test | Hindi ASR benchmark and continued-pretraining target |
| **Vaani Benchmark V1.0** | 20.64 hours; 3,252 speakers; 104 districts; three independent transcriptions per segment | Inclusive Hindi ASR evaluation benchmark |
| **Balanced Vaani LID subset** | 10 hours per language for 42 languages; 8:1:1 speaker-disjoint split | Spoken language identification benchmark |

This multiplicity of usages is itself a defining property of the term in current research. One line of work uses *Vaani* to denote a large multilingual, district-centric, multimodal corpus for India-wide modeling; another uses it for a Hindi telephone-speech challenge dataset; and a third derives evaluation-focused Hindi benchmark subsets from a broader collection [2603.28714][2203.16973][2606.21408][2606.09317].

Outside machine-learning benchmark literature, the closely related names **Gram Vaani** and **Mobile Vaani** refer to voice-driven participatory media systems built around IVR, moderation, community mobilization, and district-level organization. Those systems provide socio-technical background for some Vaani-named resources, but they are not themselves presented as standardized ASR benchmark corpora in the same way as the later dataset papers [1907.06360][2104.07901].

## 2. Corpus design, collection strategy, and multimodal structure

The large-scale **VAANI** dataset is explicitly framed as an India-representative multimodal resource designed to capture linguistic diversity through a **district-centric, geo-centric collection strategy** rather than a purely language-centric one. In one published description, it open-sources around **289K images**, approximately **31,270 hours of audio recordings**, and around **2,067 hours of transcribed speech**, covering **112 languages from 165 districts from 31 States and Union territories** [2603.28714]. A later paper reports a closely related Vaani resource with approximately **31,255 hours of speech covering 105 languages**, of which **1,894 hours are transcribed** [2606.24080]. This suggests that the label *Vaani dataset* may denote closely related releases or accounting conventions rather than a single fixed snapshot.

A central design choice is **image-based elicitation**. Speakers are shown an image and asked to describe it in their own words, producing spontaneous responses rather than read speech. The images were **physically captured** for the project, include district-specific as well as more general content, and were constrained to be **.jpg**, **640×400 pixels**, **under 500 KB**, clear, unique, free of **PII**, and not sourced from online or third-party media. The capture date was required to be **no earlier than July 1, 2023** in the project description [2603.28714]. The audio–image alignment paper emphasizes the same collection principle and treats the resulting corpus as naturally containing paired audio–image examples because each utterance is elicited in response to a specific image [2606.24080].

The speech collection protocol is designed to preserve authenticity while maintaining quality control. Speakers were recruited locally, had to be **native residents** of the recorded pincode, and identity was verified using documents such as **PAN or Aadhaar**. Participants were between **20 and 70 years old**; gender balance was maintained; each district had at least **800 speakers**; no speaker contributed more than **15 minutes of effective speech**; and speakers were asked to produce **10–20 seconds of effective speech per image** in the language or dialect they speak at home [2603.28714]. Recordings were **16 kHz**, **16-bit**, **single channel / mono**, and raw, with no transcoding or post-processing.

Metadata are an important part of the corpus design. Reported fields include **State**, **District**, **Gender**, **Pincode**, **Asserted Language**, and **Languages Spoken**. Earlier collection descriptions also mention **age**, **education**, **socio-economic background**, **location**, and **duration of stay in that location**. This metadata structure makes the dataset relevant not only for ASR, but also for region-aware modeling, fairness analysis, and district-level performance studies [2603.28714].

The quality-control pipeline is multi-stage and combines automated and manual validation. Automated audio checks include metadata validation, file-format checks, duration validation, duplicate detection, silence checks, and **Signal-to-Noise Ratio (SNR)** screening for manual review. Automated transcription checks include ID validation, script consistency, word/duration ratio checks, LM log-likelihood assessment, and language/geographic plausibility checks. Manual validation is district-local and expert-led; if more than **10%** of a batch is problematic, the batch is escalated to additional experts [2603.28714]. The transcription guidelines require verbatim transcription, explicit marking of **[unintelligible]**, **[inaudible]**, `<UNKNOWN_SEGMENT>`, and `<PAUSE>`, retention of slang, written-out numbers, and markup for incomplete utterances and foreground sounds.

## 3. Benchmark variants for Hindi ASR evaluation

Two distinct Hindi evaluation resources appear under the Vaani name in the literature.

The earlier **GramVaani / Vaani** challenge corpus is described as a **corpus of 1108 hours of real-world, spontaneous telephone speech recordings in multiple dialects of Hindi**, released as part of the **Interspeech GramVaani ASR Challenge 2022**. Of this total, **1000 hours is unlabeled**, **100 hours is labeled training data**, **5 hours is development data**, and **3 hours of blind test data** was released for evaluation. The audio is recorded at **different sampling rates ranging from 8 kHz to 48 kHz** and stored in **mp3 format**. For wav2vec 2.0 continued pretraining, **991 hours** of unlabeled audio were used for pretraining and **9 hours** were held out for validation. The paper explicitly notes that it does **not** further define the dataset’s exact annotation protocol, speaker composition, or official split filenames [2203.16973].

The later **Vaani Benchmark V1.0** is an evaluation-focused **Hindi ASR benchmark dataset** built to address limited geographic coverage, weak demographic diversity, and the inadequacy of single-reference scoring. It was collected from **104 districts across 22 states and Union Territories of India**, contains **20.64 hours** of audio from **3,252 speakers**, and was sampled from a broader collection of about **200 hours** gathered across **165 districts** and **31 states/UTs**, with up to **15 minutes per district** retained for the benchmark [2606.21408]. Unlike the telephone-speech challenge corpus, it is explicitly **multimodal**, consisting of aligned **image, speech, and text**.

The benchmark’s defining annotation feature is that each audio segment has **three independent transcriptions**. The pipeline begins with a human transcription checked by two ASR systems, followed by consistency checks and automated validation, independent review and correction by separate transcribers, random supervisor audits of about **10%** of transcriptions, and further manual validation by **three independent transcribers** when discrepancies are large. Transcribers are selected from the **same district** as the speech source when possible. The dataset also annotates **code-switched content** in both the **native script** and the **original script**, and marks **non-speech audio events/noise** [2606.21408].

This design motivates three evaluation protocols. **Approach 1** computes WER against each reference separately and reports the mean. **Approach 2** takes the minimum error count across references per utterance before aggregation. **Approach 3** is a **multi-reference alignment-based WER** that accounts for alternative realizations across the three transcriptions. The paper defines
$$
\text{WER} = \frac{S + I + D}{N}
$$
with utterance-level deletion and effective-length terms
$$
D_u = \bigcap_{i=1}^{3} deleted\_per\_ref[i], \qquad N_u = |H| + |D_u| - I_u.
$$
The reported pairwise inter-set WERs between transcription sets—**10.51%**, **13.62%**, and **12.91%**—are used to argue that transcription variation remains substantial even after quality control, making single-reference evaluation inadequate [2606.21408].

## 4. Role in self-supervised ASR and continued pretraining

The **GramVaani/Vaani** Hindi corpus became a central testbed for analyzing when self-supervised pretraining is useful for low-resource ASR. In that setting, the downstream recognizer is a **joint CTC/attention encoder-decoder** with a **12-block conformer encoder**, a **6-block transformer decoder**, and **CTC weight = 0.4**. Upstream SSL representations are **1024-dimensional** and passed through a linear layer to obtain **80-dimensional features**; the setup uses **layer-wise weighted sums of upstream features** and **beam search with beam size 20**. The paper compares an **Fbank-Pitch** baseline, **W2V-GV** trained from scratch on GramVaani unlabeled data, and several wav2vec 2.0 LARGE upstreams: **LL**, **LL+CV+SF**, **XLSR-128**, and **IndicW2V** [2203.16973].

Without continued pretraining, the Hindi results are **34.2 dev / 33.7 test** for Fbank-Pitch, **32.4 / 32.3** for W2V-GV, **35.0 / 34.4** for LL, **34.3 / 34.2** for LL+CV+SF, **32.7 / 32.5** for XLSR-128, and **33.6 / 33.1** for IndicW2V. After **continued pre-training (CP)** on GramVaani Hindi unlabeled audio, the same upstreams reach **29.7 / 29.8** for LL, **29.1 / 28.9** for LL+CV+SF, **27.3 / 27.1** for XLSR-128, and **31.4 / 31.5** for IndicW2V. The paper identifies **XLSR-128** as the best-performing upstream for Hindi and states that, after CP, it improves over the FBank baseline by **6.9% WER on dev** and **6.6% WER on test**; even before CP, XLSR-128 is already better than FBank by **1.5% dev** and **1.2% test** [2203.16973].

The same corpus is also used to probe an extreme low-resource condition with only **10 hours of labeled GramVaani data**. In that regime, the reported Hindi WERs are **75.7 dev / 74.5 test** for Fbank-Pitch, **50.4 / 50.3** for W2V-GV, **56.8 / 55.9** for XLSR-128, and **46.2 / 46.0** for **XLSR-128 (Continued pre-trained)**. The paper summarizes this as an **absolute 29.5% WER improvement over the FBank baseline** and **4.2% over W2V-GV**, with roughly **29% absolute WER improvement** from the FBank baseline when averaging dev and test [2203.16973].

The central interpretation is that ASR performance improves with increasing **similarity** and **volume** of pretraining data, and that continued pretraining on in-domain Hindi audio materially improves downstream recognition. At the same time, the study reports **catastrophic forgetting** for multilingual upstreams when they are continued-pretrained on Hindi and then evaluated on other languages: for **XLSR-128**, average WER degradation after Hindi CP is **3.5%** for Gujarati, **2.8%** for Tamil, and **3.2%** for Telugu. Vaani therefore serves not only as a benchmark, but also as an instrument for studying domain adaptation, language match, domain diversity, and cross-language interference in SSL-based ASR [2203.16973].

## 5. Audio–image continued pretraining and transcription-free adaptation

In later multimodal work, Vaani is used not merely as an evaluation set, but as the enabling substrate for a transcription-free adaptation stage based on naturally aligned audio–image pairs. The reported pipeline has three stages: **(1) pretraining** of a FastConformer audio encoder on Vaani speech, **(2) continued pretraining / representation alignment** using audio–image pairs, and **(3) supervised ASR fine-tuning** using transcribed speech. The audio backbone is a **17-layer FastConformer encoder**, and **all 17 layers are trainable** during alignment. The image side is frozen and instantiated with **SigLIP2-base / patch16-256** with a **768-dimensional** embedding space, **SigLIP2-large / patch16-384** with a **1024-dimensional** space, or **Qwen3-VL-Embedding-2B / visual stack** with a **2048-dimensional** space [2606.24080].

The multimodal supervision comes from a large alignment set of **11,848,593 audio–image pairs**, spanning **287K unique images** and **16,580.36 hours of audio**. The audio encoder is coupled to an MLP **alignment head** that maps audio representations into the image embedding space. Three alignment configurations are reported. In the single-vector **SigLIP** variant, audio uses single-query attention pooling and images use one pooled vector. In **SigLIP-MT** and **Qwen-MT**, the audio side uses **multi-query attention pooling** with **$K_a = 16$** audio queries. On the image side, SigLIP-MT selects the **top 16 patch tokens by L2 norm** from 576 patches, while Qwen-MT performs a **2×2 spatial merge** to yield up to 16 image tokens. Pair scoring uses cosine similarity in the single-vector case and a MaxSim late-interaction operator in the multi-token case:
$$
\text{MaxSim } \tfrac{1}{K_a}\sum_a\max_v\langle q_a, V_v\rangle .
$$

Training uses a **SigLIP-style sigmoid contrastive loss**, written in the paper as
$$
-\log\sigma(y_{ij}(t\,s_{ij}+b)),
$$
where $s_{ij}$ is the pair score, $y_{ij}$ is the positive/negative label, and $t$ and $b$ are learnable temperature and bias terms. The alignment stage is optimized with in-batch negatives gathered across all GPUs, **batch size 64**, **AdamW**, **weight decay 0.01**, **gradient clipping at 1.0**, **bfloat16**, **1,000-step warmup**, and **200,000 optimization steps**. Newly initialized layers use learning rate **$3\times 10^{-4}$**, while pretrained encoder weights use a smaller rate scaled by **0.05** [2606.24080].

For supervised ASR fine-tuning, the paper uses **1,636 hours** of Vaani speech for training, **177.6 hours** for validation, and **81.05 hours** for testing, with evaluation on a **48-language** test partition. In a separate out-of-domain setting, it fine-tunes and evaluates on **FLEURS** South Asia with **124.35 hours train**, **16.44 hours dev**, and **36.88 hours test**. Evaluation is by **WER**, with paired utterance-level bootstrap confidence intervals and p-values [2606.24080].

The empirical finding is that audio–image alignment consistently improves ASR relative to direct fine-tuning from the pretrained audio encoder. On the in-domain Vaani test set, baseline WER **0.2809** is reduced to **0.2768** with Qwen3-VL, **0.2771** with SigLIP2-base, and **0.2740** with SigLIP2-large; the best relative improvement is **2.47%** for SigLIP2-large. On **FLEURS** South Asia, the baseline **0.6778** drops to **0.5683** with Qwen3-VL, **0.5358** with SigLIP2-large, and **0.5338** with SigLIP2-base. The paper reports that the best model improves **13 of 14 languages**, with only one language degrading and no statistically significant harmful regressions, and further notes that the benefit of alignment shrinks as more labeled fine-tuning data becomes available [2606.24080].

## 6. Spoken language identification and cross-corpus generalization

Vaani is also used as the central training corpus for large-scale Indic **spoken language identification (LID)**. In that study, the authors derive a balanced subset by selecting **10 hours of speech per language for 42 languages**, aiming to maximize diversity in districts and speakers, and split it into **train/validation/test in an 8:1:1 ratio** with **speaker-disjoint splits**. The 42 languages span **four linguistic families**: **Indo-Aryan**, **Dravidian**, **Sino-Tibetan**, and **European** represented by **English** [2606.09317].

The task is explicitly cross-corpus. Models are trained on Vaani and evaluated on **Vaani-Test** as the in-domain set, and on **FLEURS** and **Kathbath** as out-of-domain sets. The paper compares **Whisper** and **FastConformer** encoders, each paired with a linear classifier, under three objectives: **cross-entropy (CE)**, **CE + supervised contrastive loss**, and **hierarchical softmax (HSM)**. The HSM model uses a tree **Root → Family → Sub-family → Language**, with
$$
P(y_i \mid \mathbf{z}_i) = \prod_{\ell=1}^{L_i} P(c_\ell \mid n_\ell, \mathbf{z}_i)
$$
and loss
$$
\mathcal{L}_{\mathrm{HSM}} = - \sum_{\ell=1}^{L_i} \log P(c_\ell^{(i)} \mid n_\ell^{(i)}, \mathbf{z}_i).
$$

The reported findings are strongly Vaani-dependent. With CE, **Whisper** benefits substantially from fine-tuning on Vaani, moving from **56.0 → 71.8** on Vaani-Test, **61.9 → 72.7** on FLEURS, and **57.7 → 68.3** on Kathbath. **FastConformer**, by contrast, is already strong when frozen, with **67.4** on Vaani-Test, **94.2** on FLEURS, and **90.9** on Kathbath; fine-tuning changes Vaani-Test only marginally (**67.4 → 67.6**) and hurts out-of-domain performance to **89.9** and **87.4** on FLEURS and Kathbath respectively. **HSM** consistently outperforms CE and CE+SupCon for both encoders, with the largest gains on out-of-domain benchmarks. The paper also reports that **CE+SupCon** degrades FastConformer’s cross-corpus generalization, which it interprets as over-specialization to Vaani’s in-domain conditions [2606.09317].

The linguistic analysis enabled by Vaani is equally important. Family-level results show that **Central Indo-Aryan** is the hardest subgroup for both encoders. The dominant confusion pairs are **Hindi–Urdu** and the **Sadri–Chhattisgarhi–Surgujia** cluster, while **Bajjika** and **Halbi** are relatively better identified. In this role, Vaani functions as a taxonomy-aware benchmark for fine-grained discrimination among closely related Indic varieties, not merely as a source of raw audio [2606.09317].

## 7. Documentation limits, ambiguities, and nearby names

The Vaani literature contains several documentation limits that materially affect interpretation. The **GramVaani/Vaani** challenge paper gives split hours, domain description, and downstream usage, but explicitly does **not** define the exact annotation protocol, speaker composition, or official split filenames [2203.16973]. Conversely, **Vaani Benchmark V1.0** provides extensive detail on transcription multiplicity and evaluation methodology, but it is a small evaluation set rather than a full-scale pretraining corpus [2606.21408]. The large-scale **VAANI** project paper is rich on collection design and QC, yet the exact counts reported there do not numerically match the later audio–image alignment paper [2603.28714][2606.24080].

A common misconception is to assume that all Vaani-named resources are the same dataset. The published evidence does not support that simplification. The Hindi telephone-speech challenge corpus, the district-centric multimodal VAANI collection, the Hindi benchmark subset with three references, and the balanced 42-language LID subset serve different experimental roles and are described with different statistics. A plausible implication is that *Vaani* functions as a dataset family label in the literature rather than a single canonical release.

Another potential confusion concerns **Vedavani**. Although the name is similar, the paper **"Vedavani: A Benchmark Corpus for ASR on Vedic Sanskrit Poetry"** presents a separate **54+ hour Sanskrit speech corpus** for **Vedic Sanskrit poetry** and explicitly does **not** identify it as the Vaani dataset. It is therefore a distinct benchmark with a related name, not a Vaani release in the sense used by the VAANI and GramVaani papers [2506.00145].

Finally, the broader **Gram Vaani / Mobile Vaani** literature shows that some Vaani-named resources originate in a larger ecosystem of voice-based participatory media, IVR interaction, moderation, district-level organization, and community mobilization. That background is relevant because it highlights selection effects, editorial mediation, and deployment context; however, those papers describe a socio-technical platform rather than the standardized benchmark datasets used in recent ASR, LID, and multimodal representation-learning studies [1907.06360][2104.07901].

Source: https://www.emergentmind.com/topics/vaani-dataset