Vaani Benchmark V1.0: An Inclusive Multimodal Benchmark Dataset for Hindi

Abstract: Benchmarking is critical for the systematic evaluation and comparison of automatic speech recognition (ASR) systems. While several open-source datasets are available for Hindi ASR, existing benchmarks remain limited in geographic diversity, demographic representation, and transcription robustness. We introduce an inclusive, multimodal Hindi ASR benchmark collected from 104 districts across India. The dataset consists of spontaneous speech elicited using image prompts and recorded in real-world acoustic conditions across diverse demographic groups. Each audio segment is annotated with three independent transcriptions, enabling multi-reference evaluation that accounts for permissible orthographic and lexical variations. This design supports more robust, inclusive, and realistic ASR evaluation. We benchmark multiple open-source and proprietary ASR models and report their comparative performance on the benchmark dataset.

• The paper introduces an inclusive, multimodal Hindi benchmark enabling robust ASR evaluation through multi-reference transcriptions and rich metadata.
• It employs a rigorous, multi-stage quality control pipeline and district-level speaker sampling across 104 districts to minimize transcription bias.
• Benchmark evaluations reveal significant WER variations among models, highlighting the need for realistic, diverse data in Hindi ASR research.

An Authoritative Overview of the Vaani Benchmark V1.0: An Inclusive Multimodal Benchmark Dataset for Hindi

Motivation and Context

Research in ASR for Indian languages has been impeded by limitations in benchmark datasets—chiefly with respect to geographic and demographic diversity, transcription robustness, and the support for multimodal evaluation. The Vaani Benchmark V1.0 directly addresses these deficiencies for Hindi by constructing a dataset that integrates: (1) wide geographic and demographic coverage; (2) multiple, independently produced transcriptions per utterance for robust evaluation; (3) multimodal and metadata-rich design supporting a range of evaluation scenarios; and (4) standardized processes for minimizing transcription bias and ensuring data quality.

Existing Hindi ASR benchmarks, including IndicSUPERB, LAHAJA, Vistaar, and various smaller corpora, predominantly provide single-reference transcriptions and limited code-switching annotations, and often lack systematic metadata for evaluating regional biases. The Vaani Benchmark sets itself apart by encompassing data from 3,252 speakers spanning 104 districts within 22 states and Union Territories, with spontaneous speech elicited via image prompts and transcribed with three independent human references per utterance. This architecture enables much-needed multi-reference evaluation protocols for both ASR benchmarking and linguistic analysis.

Data Collection, Quality Control, and Benchmark Construction

The Vaani Benchmark comprises 20.64 hours of spontaneous Hindi speech elicited through image descriptions, aligning audio with corresponding images and textual transcriptions. The dataset is constructed with a rigorous methodology to ensure the integrity and representativeness of the data:

• Speakers, distributed across 104 districts, were shown up to 50 images (locally and regionally relevant), and asked to describe them in their home language using a mobile application.
• The audio undergoes a multistage quality control pipeline, which includes both automated and manual validation of audio and initial transcriptions. Code-switched speech is explicitly annotated in both native and original script.
• Each segment is transcribed via a three-tier process: an initial human transcription, corrections and consistency checks, and two additional independent human transcriptions. These are subsequently validated for consistency; disagreements invoke further adjudication steps.
• District-local transcribers are employed to ensure dialectal consistency and minimize inter-dialect transcription variance.
• Metadata includes district, state, gender, known languages, and speaker identifiers to facilitate bias and subgroup analysis, speaker recognition benchmarks, and downstream geographical or dialectal studies.

The rigorous data preparation workflow is schematized below:

Figure 1: The data preparation process, detailing the multistage annotation, QA, and adjudication pipeline for robust multi-reference transcription.

Geographic and Demographic Representation

A core innovation in Vaani is the explicit effort toward geographic balance, with sampling and speaker selection processes constrained at the district level and further stratified by speaker. The benchmark carefully ensures non-overlapping benchmark and training portions by removing selected segments from the main Vaani corpus once admitted to the benchmark. This enables fine-grained analysis and accurate district-level generalization studies.

Figure 2: Geographic distribution of the speakers and datasets across 104 districts in 22 states and Union Territories.

Multi-Reference Transcription and Evaluation Protocols

Vaani defines three evaluation paradigms for WER:

• Approach 1: Computes WER for each reference set independently, reporting mean values across references.
• Approach 2: For each segment, selects the minimal error among the three references at the segment level, thus mitigating penalization arising from orthographic/lexical ambiguity.
• Approach 3: Aggregates errors by aligning each hypothesis with all reference transcriptions at the word level. This alignment-based protocol permits robust estimation of deletions (requiring deletion across all references for an error to be counted) and models alternative word-level realizations due to dialect or transcription ambiguity.

Algorithmically, Approach 3 provides the most principled multi-reference WER metric, as it directly integrates the linguistic and orthographic variability intrinsic to Hindi as spoken across India. Pairwise inter-reference WER is measured at 10–15% even after several quality control passes, illustrating the irreducible subjectivity of human transcription in real-world, diverse settings. The multi-reference design is thus essential for realistic and fair ASR evaluation.

Model Benchmarking Results

Vaani Benchmark is used to evaluate both open-source and proprietary ASR systems, including specialized Indic models, large-scale multilingual ASR, self-supervised architectures, and commercial APIs. Models are assessed exclusively on the core ASR task, although the data architecture also enables future image-speech retrieval and multimodal grounding tasks.

Key results include:

• Best WER (Approach 3): Vaani Fast Conformer (10.6%), Gemini-3.1-Pro (11.9%), and Sarvam Saaras v3 (13.7%).
• Lower-tier models (e.g., Whisper-large-v3, Vakyansh-wav2vec2, GPT-4o-Transcribe) display substantially higher WERs, highlighting the challenge presented by the diversity and spontaneous nature of the benchmark.
• Significant performance variation across districts is observed for all models, with standard deviations (e.g., 4.1 to 12.7) reflecting persistent regional and socio-linguistic bias that is not addressed by generic large-scale training.
• Multi-reference penalization: A consistent drop in WER is seen when moving from single-reference to multi-reference protocols, quantifying the inflation effect single references have on reported errors due to permissible orthographic/phrase variability.

These results empirically demonstrate the necessity of evaluation paradigms that capture actual language use—including dialectal, code-switched, and spontaneous speech—rather than relying on idealized, overly constrained test sets.

Implications and Future Directions

The Vaani Benchmark has immediate implications for Hindi ASR system development and evaluation:

• Robustness to Linguistic Variation: By providing multi-reference, regionally annotated transcriptions for spontaneous and diverse speech, the benchmark directly evaluates models' real-world robustness and surfaces bias/failure cases unobservable in prior benchmarks.
• Benchmark for Multimodal and Code-Switching Tasks: The image-text-audio architecture permits evaluation of state-of-the-art multimodal and code-switching ASR systems, a critical direction given current trends in LMM and multimodal LLM design.
• Dataset for Downstream Tasks: Metadata supports benchmarking for speaker identification, district identification, audio and text-based image retrieval, and code-switching detection/generation.
• Generalization Diagnostics: Explicit reporting of per-district performance enables quantitative analysis of regional bias, vital for ethical deployment of ASR in India's sociolinguistic landscape.
• Future Work: Plans include further language extension, increased coverage, more granular noise tagging, and multimodal task benchmarking.

Conclusion

The Vaani Benchmark V1.0 constitutes a methodologically rigorous, geographically and demographically diverse, multi-reference, and multimodal Hindi benchmark. Its construction and evaluation protocols expose substantial limitations in existing ASR systems' capabilities to generalize across realistic, varied speech. The benchmark provides essential tools for evaluating and building the next generation of robust, inclusive, and context-aware ASR models for Indic languages, and establishes a new standard for benchmark dataset construction moving forward.