IndicContextEval: A Benchmark for Evaluating Context Utilisation in Audio Large Language Models Across 8 Indic Languages

Abstract: AudioLLMs enable speech recognition conditioned on textual prompts such as domain descriptions or entity lists. However, it remains unclear whether these models genuinely utilise such context or rely on parametric knowledge learned during pretraining. Existing benchmarks cannot answer this question because they evaluate transcription under fixed prompting conditions and rarely include explicit contextual inputs. We introduce IndicContextEval, a 56-hour multilingual benchmark of natural speech from 555 speakers across 8 Indian languages and 23 professional domains. We design a 7-level prompting framework that progressively introduces contextual signals, including metadata, natural-language descriptions, entity lists in English and native script, and adversarial prompts with incorrect entities. Evaluating five models reveals substantial differences in context utilisation behaviour, highlighting the need for explicit evaluation of contextual grounding in AudioLLMs.

• The paper introduces IndicContextEval, a benchmark with 56 hours of natural speech from 555 speakers across 8 Indic languages and 23 domains.
• The paper demonstrates that native-script entity prompts significantly improve transcription accuracy, with Gemini 3 Flash reaching 17.39% NEER.
• The paper finds that AudioLLMs show varied sensitivity to contextual and adversarial prompts, underlining challenges in robust context utilisation.

Authoritative Summary of "IndicContextEval: A Benchmark for Evaluating Context Utilisation in Audio LLMs Across 8 Indic Languages" (2606.19157)

Motivation and Benchmark Design

The "IndicContextEval" benchmark addresses a critical gap in AudioLLM evaluation: systematic assessment of context utilisation in transcription tasks across linguistically diverse and domain-rich Indian speech. Existing ASR benchmarks either disregard contextual prompting or are limited to synthetic, monolingual settings and a restricted set of context types. IndicContextEval introduces 56 hours of natural speech (from 555 speakers) spanning 8 Indian languages and 23 professional domains. Each utterance is paired with granular contextual metadata, including domain descriptions, entity lists (in both English and native scripts), and adversarial prompts. This enables attribution of transcription performance to specific context signals, controlling for confounding factors.

The benchmark implements a seven-level prompt taxonomy (L0–L6), where each level incrementally introduces context: from no context (L0), language specification (L1), metadata (L2), natural-language descriptions (L3), entity lists (L4/L5), to adversarial prompts (L6). All transcriptions are evaluated in the native script, ensuring real-world applicability.

Experimental Protocol and Evaluation Metrics

IndicContextEval assesses context sensitivity in five state-of-the-art models:

• Proprietary AudioLLMs: GPT-4o Transcribe, Gemini 3 Flash, Sarvam Audio
• Open models: Gemma-3N
• Non-LLM ASR: IndicConformer (baseline, evaluated only at L1)

All models receive prompts structured identically, isolating each context type’s contribution. Output requirements explicitly specify script, format, and transliteration policies, reducing evaluation ambiguity. Performance is measured via Word Error Rate (WER) and Named Entity Error Rate (NEER), with NEER as the primary metric for entity biasing.

Key Findings and Numerical Results

Baseline and Contextual Performance

• Sarvam Audio achieves the lowest baseline WER at L1 (16.86%), surpassing IndicConformer (18.81%). Gemini 3 Flash matches IndicConformer (18.90%), while GPT-4o Transcribe (28.61%) and Gemma-3N (38.73%) lag substantially.
• Contextual prompts delivered in the native script (L5) yield the most pronounced improvements in NEER across all models: Gemini 3 Flash reaches 17.39% NEER at L5, outperforming all baselines and other AudioLLMs. The script-mismatch penalty for entity lists (L4 vs. L5) exceeds 11 points in NEER.
• Malayalam consistently proves to be the most challenging language for all models, with elevated WER across the board.

Context Utilisation and Robustness

• Balanced Model (GPT-4o Transcribe): Gains from relevant entity prompts ($-\Delta$2.57 WER at L5) but unaffected by adversarial entities (L6 ≈ L1), indicating selective grounding and cross-validation of acoustic and textual signals.
• Sensitive Model (Gemini 3 Flash): Demonstrates consistent improvements with context, securing best entity recognition. Slight degradation when exposed to adversarial entities (+0.77 WER from L1 to L6) confirms reliance on prompt content rather than parametric memory alone.
• Unstable Model (Gemma-3N): Benefits from correct context (NEER drop from 35.5% to 26.9% at L5) but exhibits instability and hallucinations, reflected in WER increase and transcript corruption.
• Context-Blind Model (Sarvam Audio): Minimal response to context prompts; transcription heavily determined by acoustic input with weak contextual integration.

Adversarial Testing

• L6 prompts (wrong entities) reveal divergent model behaviours: GPT-4o Transcribe and Sarvam Audio are robust ($|\Delta|$ < 0.2 WER), Gemini 3 Flash is moderately sensitive (+0.77 WER), whereas Gemma-3N exhibits catastrophic degradation (+9.22 WER).
• IndicConformer (non-LLM) remains competitive at entity recognition but is outperformed by context-aware AudioLLMs when native-script prompts are supplied.

Practical and Theoretical Implications

IndicContextEval decisively shows that context grounding in AudioLLMs is not guaranteed by prompt interfaces alone. Model behaviour is highly variable: some architectures systematically exploit context, while others ignore or misuse it. The benefit of native-script entity prompts confirms a fundamental cross-lingual challenge with script alignment and prompt integration, relevant for diverse multilingual deployments. Adversarial sensitivity quantifies the risk of prompt injection and hallucination in LLM-based transcription systems.

For practical deployments, accurate entity biasing is now demonstrably possible with AudioLLMs (when properly prompted), closing the gap with domain-tailored conventional ASR. However, reliance on parametric memorisation and instability in prompt handling pose reliability and trustworthiness concerns, demanding further research into robust entity grounding, prompt parsing, and multilingual prompt engineering.

Speculation and Future Directions

The major finding—that context utilisation varies substantially across AudioLLMs and is script-sensitive—implies that future research should address prompt-token grounding, adversarial robustness, and prompt-architecture co-design. IndicContextEval enables fine-grained evaluation of both prompt-driven and memory-driven transcription, offering a blueprint for more general benchmarks across other low-resource languages and domains.

Further advancements may include fine-tuned multilingual LLMs with explicit context weighting mechanisms, context-provenance tracking in outputs, and adversarial prompt resistance at the architecture or training-data level. Enhanced evaluations with user-specific entity lists and dynamic context sources (e.g., in dialog or voice assistants) will be crucial for downstream applications.

Conclusion

IndicContextEval establishes a rigorous benchmark for context-sensitive evaluation of AudioLLMs, spanning 8 Indic languages and 23 domains. It reveals that model performance is highly contingent upon both the form and content of contextual prompts. Entity biasing in native script delivers the most significant transcription gains, but adversarial prompts expose inconsistent or unreliable behaviour. Contextual grounding in AudioLLMs remains an unsolved challenge, and further research is warranted to achieve robust, interpretable, and trustworthy contextual ASR in multilingual, domain-rich environments.