IndicContextEval: AudioLLM Context Benchmark

Updated 5 July 2026

IndicContextEval is a benchmark that evaluates contextual utilization in AudioLLMs across 8 Indic languages using a seven-level prompting taxonomy.
It covers 55.93 hours of natural speech from 555 speakers across 23 professional domains, combining both structured metadata and free-form descriptions to test transcription behavior.
The benchmark measures performance changes using WER and NEER, highlighting model strengths and pitfalls under correct versus adversarial contextual signals.

Searching arXiv for the specified and related papers to ground the article in current literature. {"query":"arXiv (Joshi et al., 17 Jun 2026) IndicContextEval audio LLMs 8 Indic languages", "max_results": 5} {"query":"(Joshi et al., 17 Jun 2026)", "max_results": 10} Searching for paper (Joshi et al., 17 Jun 2026) on arXiv. IndicContextEval is a benchmark for evaluating context utilisation in Audio LLMs across eight Indic languages. It was introduced to test whether prompt-conditioned speech systems genuinely ground their transcriptions in supplied contextual signals—such as domain descriptions or entity lists—or instead rely primarily on parametric knowledge acquired during pretraining. The benchmark consists of 55.93 hours of natural speech from 555 native-speaker participants, spans 23 professional domains, and operationalizes context through a controlled seven-level prompting framework, denoted $L0$ through $L6$ , that isolates the effect of individual prompt components on transcription behaviour (Joshi et al., 17 Jun 2026).

1. Problem setting and benchmark objective

IndicContextEval addresses a specific evaluation gap in contextual automatic speech recognition for Indic languages. AudioLLMs increasingly accept textual prompts at inference time, including domain metadata and lexicons of likely entities, but standard ASR benchmarks generally evaluate transcription under fixed prompting conditions and rarely expose models to explicit contextual inputs. IndicContextEval was designed to determine whether improvements under prompting reflect genuine contextual grounding rather than memorized prior knowledge (Joshi et al., 17 Jun 2026).

The benchmark formalizes this question through controlled prompt variation while keeping the acoustic input fixed. In the benchmark’s framing, the critical variable is not only absolute transcription accuracy but also the direction and magnitude of performance changes as different contextual signals are introduced. This makes the benchmark diagnostic rather than purely leaderboard-oriented. Its central claim is that context use should be measured explicitly, including under negative controls in which the supplied context is wrong.

A notable feature of the benchmark is its multilingual and multi-domain scope. It covers Hindi, Bengali, Telugu, Marathi, Gujarati, Malayalam, Odia, and Urdu, and it includes 23 professional domains rather than a single topical setting. The paper characterizes this as the first multilingual, multi-domain, natural-speech benchmark for systematically evaluating context utilisation in AudioLLMs (Joshi et al., 17 Jun 2026).

2. Dataset composition and annotation structure

IndicContextEval comprises 55.93 hours of natural speech from 555 speakers. Each language contributes at least 3 hours, ranging from 3.37 hours for Urdu to 13.70 hours for Telugu, with an average of approximately 7 hours per language. Speakers are drawn from students and professionals, and regional metadata is recorded to capture geographic variation (Joshi et al., 17 Jun 2026).

The benchmark spans 23 professional domains, including Core Engineering, Data Science, Medical Sciences, Robotics, Forensics & Legal, Business & Defense, Arts & Media, Culinary Arts, and Linguistics. Each domain is further subdivided into domain-specific subfields. The speech material combines two styles: read recordings, based on domain-pertinent sentences generated via Gemini 3 Pro and translated by Sarvam-Translate, and extempore recordings, elicited through question-driven narratives. This pairing broadens both lexical coverage and acoustic variability (Joshi et al., 17 Jun 2026).

All utterances undergo a multi-stage quality-control process. Native-speaker annotators produce verbatim transcriptions in the target script, preserving code-mixing and transliterating English entities. Each utterance is associated with structured metadata comprising domain label and one-sentence description, speech style, speaker region, named-entity lists containing 20–30 terms in both English and native script, and natural-language “audio descriptions” generated from metadata via Gemini 3 Flash (Joshi et al., 17 Jun 2026).

The dataset’s comparison against prior contextual-ASR resources clarifies its design position. In the reported benchmark comparison, IndicContextEval is described as having 56 hours, 23 domains, 8 Indic languages, and natural audio, whereas ProfASR has 8.6 hours, 4 domains, 1 language, and synthetic audio; ContextASR has 838 hours, 10+ domains, 2 languages, and synthetic audio; and Earnings-22 has 119 hours, 1 domain, 1 language, and natural audio (Joshi et al., 17 Jun 2026). This suggests that IndicContextEval prioritizes multilinguality, domain breadth, and natural speech over scale alone.

3. Seven-level prompting taxonomy

The benchmark’s core methodological innovation is a seven-level prompting taxonomy in which exactly one new contextual signal is introduced at each step. This controlled progression is intended to support causal attribution of performance changes to specific prompt features rather than to prompt length or uncontrolled prompt engineering (Joshi et al., 17 Jun 2026).

At $L0$ , the model receives no context beyond a bare transcription instruction: “Transcribe the following audio.” At $L1$ , a language hint is added, specifying the target language and script. At $L2$ , structured domain metadata is introduced, including speech style, geographic region, and a one-sentence domain description. At $L3$ , the model instead receives a short free-form audio description summarizing the topic and style. At $L4$ , the prompt includes 20–30 domain entity terms in English while requiring output in the native script. At $L5$ , the same entity list is provided in the native script. At $L6$ , the model is given unrelated-domain entities in native script as an adversarial negative control (Joshi et al., 17 Jun 2026).

This design partitions different notions of context into distinct experimental interventions. $L1$ tests whether explicit language specification reduces script ambiguity. $L6$ 0 and $L6$ 1 contrast structured metadata against natural-language descriptions. $L6$ 2 and $L6$ 3 test lexical biasing, with $L6$ 4 specifically measuring the effect of script alignment. $L6$ 5 tests whether a model uses context selectively or over-relies on it.

The benchmark imposes a uniform transcription instruction across levels: output must be in the native script, numbers must be spelled out, hesitations must be omitted, and English words must be transliterated. Because the acoustic input is constant across $L6$ 6– $L6$ 7, differences across levels are interpretable as prompt effects rather than data effects (Joshi et al., 17 Jun 2026).

4. Evaluation protocol and metrics

IndicContextEval evaluates models primarily through Word Error Rate and Named Entity Error Rate. Word Error Rate is defined as

$L6$ 8

where $L6$ 9 is the number of substitutions, $L0$ 0 the number of deletions, $L0$ 1 the number of insertions, and $L0$ 2 the total number of words in the reference. Named Entity Error Rate is defined over the reference entities for levels $L0$ 3– $L0$ 4 as the percentage of missing or mis-recognized entities (Joshi et al., 17 Jun 2026).

The paper does not introduce an additional scalar “context sensitivity” metric. Instead, contextual grounding is inferred from deltas in WER and NEER between prompt levels. In particular, the benchmark treats improvements under correct entity prompts and degradations under adversarial prompts as evidence about how the model integrates textual context with acoustics.

Five systems are evaluated: IndicConformer as a standalone ASR baseline at $L0$ 5, and four AudioLLMs—GPT-4o Transcribe, Gemini 3 Flash, Sarvam Audio, and Gemma-3N—across $L0$ 6– $L0$ 7. At $L0$ 8, the reported baseline performance is: IndicConformer, WER 18.81 and NEER 29.58; Sarvam Audio, 16.86 and 25.93; Gemini 3 Flash, 18.90 and 25.85; GPT-4o Transcribe, 28.61 and 35.59; and Gemma-3N, 38.73 and 35.50 (Joshi et al., 17 Jun 2026).

The adversarial comparison between $L0$ 9 and $L1$ 0 is especially important in the benchmark’s methodology. GPT-4o Transcribe changes from WER 28.61 at $L1$ 1 to 28.47 at $L1$ 2, Gemini 3 Flash from 18.90 to 19.67, Sarvam Audio from 16.86 to 16.69, and Gemma-3N from 38.73 to 47.95. On NEER, GPT-4o changes from 35.59 to 34.55, Gemini 3 Flash from 25.85 to 25.60, Sarvam Audio from 25.93 to 25.62, and Gemma-3N from 35.50 to 36.25 (Joshi et al., 17 Jun 2026). These deltas are used to distinguish robustness, limited sensitivity, invariance, and failure modes.

5. Empirical findings and behavioural typology

The reported results show that context utilisation varies substantially across models. The benchmark identifies a “Language Identification Tax” between $L1$ 3 and $L1$ 4: adding a language hint sharply reduces WER for script-ambiguous languages. The reported gains are $L1$ 5 points for Gemma-3N, $L1$ 6 for Gemini 3 Flash, $L1$ 7 for Sarvam Audio, and $L1$ 8 for GPT-4o Transcribe (Joshi et al., 17 Jun 2026). This establishes language specification as a strong but model-dependent contextual signal.

Prompt format also matters. The benchmark reports that natural-language audio descriptions at $L1$ 9 produce larger WER reductions than structured metadata at $L2$ 0. For GPT-4o, the paper reports $L2$ 1 points at $L2$ 2 versus $L2$ 3 at $L2$ 4. For Gemma-3N, $L2$ 5 causes a sharp degradation of $L2$ 6 points, whereas $L2$ 7 leads to only $L2$ 8 (Joshi et al., 17 Jun 2026). A plausible implication is that free-form descriptions are easier for some AudioLLMs to integrate than rigid metadata blocks.

The strongest positive effects occur with native-script entity prompts. From $L2$ 9 to $L3$ 0, the benchmark reports the largest gains in NEER and WER: GPT-4o improves by $L3$ 1 NEER points and $L3$ 2 WER points, Gemini 3 Flash by $L3$ 3 NEER points and $L3$ 4 WER points, Gemma-3N by $L3$ 5 NEER points, and Sarvam Audio by $L3$ 6 NEER points (Joshi et al., 17 Jun 2026). The paper notes that the $L3$ 7 gap, interpreted as script mismatch cost, can exceed 10 NEER points.

Adversarial prompting at $L3$ 8 yields a behavioural typology. GPT-4o is described as exhibiting robust selective use, because it nearly recovers its $L3$ 9 performance under wrong entities. Gemini 3 Flash shows sensitive integration, with modest WER degradation of $L4$ 0 points under incorrect entities. Gemma-3N exhibits blind reliance, suffering a $L4$ 1-point WER regression and producing hallucinations and repeated segments. Sarvam Audio is categorized as context-blind ASR, remaining essentially invariant across prompt levels (Joshi et al., 17 Jun 2026).

The benchmark also reports language-specific variation. A per-language heatmap at $L4$ 2 identifies Malayalam and Telugu as the most challenging languages for most models, and the paper notes cross-language variation of up to 20 WER points in Gemma-3N (Joshi et al., 17 Jun 2026). This indicates that context utilisation and baseline recognition quality are not uniform across the eight-language setting.

The paper interprets model differences in terms of architectural and training behaviour. GPT-4o Transcribe’s pattern is taken to suggest explicit internal mechanisms for validating prompt relevance against acoustics. Gemini 3 Flash is described as strong at entity recognition and WER reduction but limited in adversarial robustness. Gemma-3N’s instability is associated with weak context–acoustic alignment, while Sarvam Audio is characterized as being optimized more for acoustic modeling than for prompt-conditioned inference (Joshi et al., 17 Jun 2026). These interpretations remain model-behaviour hypotheses rather than direct mechanistic analyses.

IndicContextEval belongs to a broader landscape of Indic evaluation resources but targets a distinct capability. L3Cube-IndicQuest evaluates whether multilingual LLMs capture region-specific factual knowledge across English and 19 Indic languages through 4,000 factual QA pairs spanning literature, history, geography, politics, and economics (Rohera et al., 2024). Vyākarana evaluates multilingual LLMs on Indic morphosyntax using Colorless Green sentences and four tasks—PoS Tagging, Syntax Tree-depth Prediction, Grammatical Case Marking, and Subject-Verb Agreement—under standard probing protocols (Patil et al., 2021). By contrast, IndicContextEval focuses on prompt-conditioned transcription and the causal role of auxiliary context in AudioLLMs. This suggests a broader movement toward modality-specific evaluation in Indic NLP, with separate benchmarks for factual knowledge, morphosyntax, and contextual speech grounding.

The name “IndicContextEval” also requires a terminological clarification. A formalized three-part context-based indicator derived from Bouramoul et al.’s information-retrieval evaluation framework combines system context, query context, and user context into a single score using

$L4$ 3

with $L4$ 4 (Bouramoul et al., 2011). In that case, however, the original paper did not provide a single closed-form name; the label is a later formalization. In current arXiv usage, the term primarily denotes the 2026 AudioLLM benchmark rather than the earlier IR evaluation construct (Joshi et al., 17 Jun 2026).

The benchmark’s stated future implications include more low-resource languages, finer-grained context types such as user profiles, and training objectives that explicitly penalize over-reliance on prompts (Joshi et al., 17 Jun 2026). These directions follow directly from its main empirical result: contextual prompting is neither uniformly beneficial nor uniformly ignored, and robust context utilisation must therefore be evaluated as a first-class capability rather than treated as an incidental property of transcription quality.