VoiceAgentBench: Multilingual Speech Agent Benchmark

• VoiceAgentBench is a benchmark for evaluating SpeechLMs beyond transcription by testing structured tool selection, workflow chaining, and safety behaviors.
• It utilizes 5,757 synthetic spoken queries across multiple languages and scenarios, emphasizing Indian-context applications and adversarial robustness.
• Empirical findings show a significant performance gap between end-to-end SpeechLMs and ASR→LLM pipelines, especially in parameter filling and safety for multilingual tasks.

VoiceAgentBench is a benchmark for evaluating SpeechLMs in realistic spoken agentic settings, targeting capabilities that lie beyond transcription or speech question answering: structured tool selection, parameter filling, workflow sequencing, multi-turn state tracking, and safety-critical refusal behavior. It comprises over 5,500 synthetic spoken queries—reported as 5,757 items in the benchmark comparison table—and supports English, Hindi, Bengali, Marathi, Tamil, Telugu, and Malayalam, with explicit emphasis on multilingual generalization, Indian-context grounding, and adversarial robustness (Jain et al., 9 Oct 2025).

1. Motivation and problem setting

VoiceAgentBench was introduced from the premise that existing speech benchmarks largely isolate component skills such as ASR, speech translation, SLU, or generic audio reasoning, but do not systematically measure whether a voice assistant can act as an agent over structured tool environments. The benchmark therefore targets the operational requirements of agentic voice systems: selecting the right tool from a catalog, filling arguments robustly from speech, planning over dependent tool chains, preserving conversational state across turns, and refusing unsafe actions even when the spoken input includes adversarial hints (Jain et al., 9 Oct 2025).

The benchmark organizes these requirements into six scenario classes. Single-tool invocation measures direct mapping from spoken intent to one API with correct arguments. Single-tool with retrieval adds tool selection under distractors. Parallel multi-tool workflows require multiple independent calls. Dependent multi-tool chains require output-to-input propagation across calls. Multi-turn interactions test state maintenance before the final tool invocation. Safety tasks test refusal under harmful or adversarially framed requests. In the paper’s framing, these are the capabilities that determine whether a speech model is usable as a voice assistant for agentic tasks rather than as a speech interface to isolated QA (Jain et al., 9 Oct 2025).

This design also makes multilingual and culturally grounded evaluation first-class rather than peripheral. Approximately 30% of the benchmark comprises Indian-context scenarios such as local travel, food ordering, banking, and entertainment references, and the multilingual setup extends beyond English and Hindi to five additional Indic languages. That choice is central to the paper’s claim that tool-use evaluation from speech must account for language, culture, and adversarial conditions simultaneously rather than sequentially (Jain et al., 9 Oct 2025).

2. Corpus structure, task inventory, and multilingual coverage

VoiceAgentBench reports a total of 5,757 spoken queries. The corpus combines Indian-context subsets with English source-native subsets, and the task categories differ in both linguistic coverage and interaction format. Multi-turn dialogue is English-only, whereas single-tool, retrieval, parallel, dependent, and safety subsets include English, Hindi, and a five-language Indic aggregate (Jain et al., 9 Oct 2025).

Category	Coverage	Source/task family
Single Tool	English 142, Hindi 134, 5 Indic 710, English source-native 400	BFCL-simple
Single Tool with retrieval	English 179, Hindi 177, 5 Indic 895, English source-native 200	BFCL-multiple
Parallel multi-APIs	English 125, Hindi 120, 5 Indic 625, English source-native 200	BFCL-parallel
Dependent multi-APIs	English 40, Hindi 40, 5 Indic 200	Novel dependent chains
Multi-turn dialogue with tool use	English 398 dialogues	API-Bank Level-2
Safety/adversarial	English 80, Hindi 80, 5 Indic 400, English source-native 176	AgentHarm-derived

The benchmark also reports average audio durations by subset. Single-tool prompts are relatively short, with averages such as 4.50 s for English and 7.32 s for the five-language Indic aggregate. Parallel and safety subsets are much longer: parallel averages reach 14.80 s for the five-language aggregate, and safety prompts reach 35.19 s for Hindi. Multi-turn dialogue is represented by 398 English dialogues with an average of 15.23 s per user audio in turns (Jain et al., 9 Oct 2025).

The language distribution is not merely an add-on to an English core. The five-language Indic aggregate is numerically substantial in each supported category; for example, the retrieval subset contains 895 Indic queries versus 179 English, and the safety subset contains 400 Indic queries versus 80 English. This makes VoiceAgentBench unusual among tool-use benchmarks in that multilingual speech evaluation is embedded into the benchmark’s scale rather than into a small diagnostic split (Jain et al., 9 Oct 2025).

3. Data generation pipeline and speaker-diversity construction

VoiceAgentBench is synthetic in speech form but procedurally layered in its construction. Tool schemas are drawn from several sources: BFCL-derived APIs for single, retrieval, and parallel tasks; a novel set of 21 dependent tools spanning cab booking, food ordering, and payments; API-Bank Level-2 for multi-turn tasks; and AgentHarm tools across 11 harm categories for safety. Query generation then depends on the task type: Indian-context single, retrieval, and parallel prompts are generated with Gemma3 27B; dependent chains are generated with GPT-4o-mini conditioned on tool schemas and explicit dependency structure; multi-turn dialogues are rewritten into Indian context with GPT-4o-mini while preserving API correctness; and safety prompts are contextualized with Indian entities, with half of the prompts carrying adversarial hints toward misuse (Jain et al., 9 Oct 2025).

The multilingual extension begins from English Indian-context prompts. These are translated into Bengali, Marathi, Tamil, Telugu, and Malayalam with Gemma3 27B, except Malayalam, which uses Llama-3.3 70B, followed by lightweight QC for script mixing and token issues. English speech is synthesized with ElevenLabs and then passed to Coqui-TTS for voice conversion. Hindi and the other Indic languages use Krutrim-TTS for both TTS and voice conversion conditioned on selected reference audio. The paper does not specify sampling rate or file format, but it does state that ground-truth transcripts are available for ASR ablations (Jain et al., 9 Oct 2025).

A distinctive technical contribution is the speaker-reference sampling method. Candidate reference audios come from IndicSUPERB and IndicST, and are embedded with ECAPA-TDNN trained on VoxLingua107. Diversity is then optimized in embedding space. The paper states two common diversity objectives,

$$\max_{S \subseteq V,\ |S|=k} \sum_{i \in S} \sum_{j \in S,\ j \neq i} d(e_i,e_j)$$

and

$$\max_{S \subseteq V,\ |S|=k} \min_{i \neq j;\ i,j \in S} d(e_i,e_j),$$

with Euclidean distance $d(e_i,e_j)=\|e_i-e_j\|_2$. The implemented selection procedure is Farthest Point Sampling (FPS): initialize with a random seed, compute each candidate’s minimum distance to the current set, and iteratively add the farthest point. The selection step is

$$x^*=\arg\max_{x \notin S}\min_{r \in S}\|e_x-e_r\|_2.$$

The reported ablation finds that FPS yields the highest diversity metric relative to DPP and a density-based probabilistic method, so it is used in the final benchmark construction (Jain et al., 9 Oct 2025).

4. Task formalization, tool environment, and evaluation protocol

VoiceAgentBench evaluates two inference paradigms: speech-only end-to-end SpeechLMs, and ASR→LLM pipelines using Whisper v3 Large followed by a text LLM. For both, the expected output is a strict Python-style function-call representation,

[tool1(param1=..., ...), tool2(...)]

with Pydantic schemas automatically derived from JSON tool specifications for structural validation. In multilingual settings, spoken queries may be non-English, but the tool calls themselves must remain in English, mirroring English API schemas in production tool environments (Jain et al., 9 Oct 2025).

The task taxonomy distinguishes between independent and dependent orchestration. Parallel workflows are evaluated as unordered sets of independent calls. Dependent workflows are evaluated as chains in which later arguments reference prior outputs, as in cab-booking pipelines that geocode locations before estimating cost and checking availability. Multi-turn dialogue tasks are grounded in conversational history and require final tool calls that correctly incorporate the accumulated slot values (Jain et al., 9 Oct 2025).

The benchmark uses four primary metrics. Tool Selection Accuracy is exact-match over tool names or tool sets:

$$TS=\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}[\hat{S}_i=S_i].$$

Tool Call Structure measures schema adherence over correctly selected tools:

$$TCS=\frac{\#\text{ schema-valid calls among correctly selected}}{\#\text{ correctly selected calls}}.$$

Parameter Filling is a judge-based semantic accuracy, where GPT-4o-mini returns a binary verdict per query:

$PF=\frac{1}{N}\sum y_i,\quad y_i\in\{0,1\}.$

Safety is measured as refusal rate on harmful prompts:

$$RR=\frac{\#\text{ refusals}}{\#\text{ harmful requests}}.$$

The paper explicitly notes that VoiceAgentBench uses judge-based parameter correctness rather than slotwise precision/recall or F1. Safety evaluation is likewise judge-based, with GPT-4o-mini deciding whether the model refused to call tools under harmful or adversarial prompts (Jain et al., 9 Oct 2025).

5. Empirical findings and dominant failure modes

The reported baseline systems include three end-to-end SpeechLMs—KimiAudio 7B, Qwen2.5-Omni 7B, and AudioFlamingo3 7B—and three Whisper v3 Large–based ASR→LLM pipelines paired with Qwen3 8B, Gemma3 27B, and LLaMA 3.3 70B. On English Indian-context tasks, KimiAudio 7B is the strongest SpeechLM, with approximate PF values of 68% on single-tool, 66% on retrieval, 69% on parallel, 5% on dependent chains, and 61% on multi-turn dialogue, for an average PF of about 54%. The ASR→LLM pipelines reach average PF values of about 56%, with none exceeding 70% on any category. On English source-native subsets, however, PF is much higher: KimiAudio 7B reaches about 80% PF averaged over single, retrieval, and parallel tasks, while the ASR→LLM pipelines reach about 82–84% (Jain et al., 9 Oct 2025).

The multilingual gap is pronounced. In Hindi Indian-context tasks, KimiAudio 7B falls to an average PF of about 42%, and dependent tasks remain around 8%. The ASR→LLM pipelines remain near 54–58% average PF. Across the five-language Indic average, KimiAudio 7B falls further to about 25% PF, while the ASR→LLM pipelines reach about 30–35%, with Whisper v3 + LLaMA 3.3 70B reported as the strongest. The paper’s central conclusion is that end-to-end SpeechLMs remain substantially weaker than strong ASR→LLM pipelines on complex tool orchestration, especially outside English (Jain et al., 9 Oct 2025).

Dependent multi-tool chains are the hardest category for all systems. The best reported PF on dependent chains in English is about 15% for Whisper v3 + Qwen3 8B, with lower results in Hindi and the five-language Indic average. This finding is paired with a diagnostic ASR ablation: replacing Whisper outputs with gold transcripts yields PF gains of at least 24% on average across non-Hindi Indic languages, whereas English gains are much smaller, around 8–16% in single-tool settings. That pattern isolates ASR quality as a major bottleneck in multilingual tool use, but not the only one, because even gold-transcript settings do not close the orchestration gap (Jain et al., 9 Oct 2025).

Safety performance is also uneven. In English, ASR→LLM pipelines reach refusal rates of about 53–56%, with Whisper v3 + Gemma3 27B highest at 56%, and KimiAudio 7B near 51%. In Hindi and the five-language Indic average, the ASR→LLM pipelines remain substantially higher, reaching up to about 59% in Hindi and about 50% in the Indic average, whereas SpeechLM safety collapses: KimiAudio 7B drops to about 1–3% RR in Hindi and Indic evaluation. The paper further reports that removing refusal scaffolding drastically lowers RR and that adversarial hints depress RR across all models. This suggests that multilingual safety alignment is substantially less stable than English-only refusal behavior (Jain et al., 9 Oct 2025).

The error analysis identifies several recurring failure modes: retrieval of semantically adjacent but wrong tools, low TCS due to missing required fields or datatype errors, PF failures from argument misbinding, failure to propagate outputs through dependent chains, and multilingual safety violations under adversarial cues. Concrete examples include mapping “Mumbai” to the broader state “Maharashtra,” omitting nested references in dependent cab-booking chains, and executing harmful tool sequences when hints explicitly nudge the model toward unsafe actions (Jain et al., 9 Oct 2025).

6. Position in the voice-agent benchmark landscape

Subsequent benchmark papers consistently treat VoiceAgentBench as a reference point for speech-grounded tool use, while also delimiting its scope. Full-Duplex-Bench-v3 states that VoiceAgentBench tests complex multi-tool workflows but uses synthetic audio and lacks full-duplex, real-time dynamics and natural disfluencies; accordingly, it does not measure latency, interruption or endpointing behavior, or mid-utterance corrections (Lin et al., 6 Apr 2026). EVA-Bench similarly characterizes it as a benchmark for multi-tool workflows driven by static TTS-synthesized text queries rather than live bot-to-bot audio conversation, and therefore as not covering conversational dynamics or audio-level articulation checks (Bogavelli et al., 13 May 2026).

Other later works sharpen the same distinction along different axes. IHBench treats VoiceAgentBench as a task-oriented spoken-dialogue benchmark that measures end-to-end task completion but not controlled mid-speech interruptions or post-interruption workflow recovery (Salimi et al., 17 Jun 2026). ProVoice-Bench positions it as a reactive benchmark, in contrast to proactive monitoring tasks that require deciding when to intervene, when to remain silent, and how to ground intervention in digital context and environmental sound (Xu et al., 16 Apr 2026).

Taken together, these later characterizations suggest that VoiceAgentBench established one of the core evaluation layers for voice agents: multilingual, culturally grounded, speech-to-tool execution under structured APIs. Later benchmarks then extended the evaluation frontier along dimensions that VoiceAgentBench either did not target or abstracted away—full-duplex timing, overlap handling, interruption recovery, proactive triggering, and audio-first user-experience metrics. In that sense, VoiceAgentBench occupies a foundational position in the benchmark landscape: it asks whether voice assistants can act as agents over tools at all, and later suites ask whether they can do so robustly under the temporal, conversational, and acoustic constraints of live voice interaction (Jain et al., 9 Oct 2025).