EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents

Abstract: Voice agents, artificial intelligence systems that conduct spoken conversations to complete tasks, are increasingly deployed across enterprise applications. However, no existing benchmark jointly addresses two core evaluation challenges: generating realistic simulated conversations, and measuring quality across the full scope of voice-specific failure modes. We present EVA-Bench, an end-to-end evaluation framework that addresses both. On the simulation side, EVA-Bench orchestrates bot-to-bot audio conversations over dynamic multi-turn dialogues, with automatic simulation validation that detects user simulator error and appropriately regenerates conversations before scoring. On the measurement side, EVA-Bench introduces two composite metrics: EVA-A (Accuracy), capturing task completion, faithfulness, and audio-level speech fidelity; and EVA-X (Experience), capturing conversation progression, spoken conciseness, and turn-taking timing. Both metrics apply to different agent architectures, enabling direct cross-architecture comparison. EVA-Bench includes 213 scenarios across three enterprise domains, a controlled perturbation suite for accent and noise robustness, and pass@1, pass@k, pass^k measurements that distinguish peak from reliable capability. Across 12 systems spanning all three architectures, we find: (1) no system simultaneously exceeds 0.5 on both EVA-A pass@1 and EVA-X pass@1; (2) peak and reliable performance diverge substantially (median pass@k - pass^k gap of 0.44 on EVA-A); and (3) accent and noise perturbations expose substantial robustness gaps, with effects varying across architectures, systems, and metrics (mean up to 0.314). We release the full framework, evaluation suite, and benchmark data under an open-source license.

• The paper introduces EVA-Bench, an open-source framework that simulates multi-turn conversations and measures voice agent performance with composite accuracy and experience metrics.
• It employs a robust simulation and perturbation suite to evaluate voice agents under realistic conditions, highlighting sensitivities to accents, noise, and transcription accuracy.
• The paper reveals a significant gap between peak and reliable performance across architectures, emphasizing the need for comprehensive, cross-architecture benchmarking for deployment readiness.

Authoritative Summary of "EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents" (2605.13841)

Motivation and Scope

The evaluation of voice agents—AI systems that engage in spoken dialogue to complete task-oriented workflows—presents unique challenges not addressed by traditional text-based agent benchmarks. These challenges arise from the temporal, auditory, and interactive complexities intrinsic to spoken conversations, the architecture heterogeneity among voice agents (cascade, hybrid, end-to-end S2S), and highly context- and policy-sensitive downstream applications in enterprise settings (e.g., airline customer service, healthcare HR, IT service management). Existing benchmarks typically lack (a) realistic, multi-turn, validated audio simulations; (b) metrics that expose the full range of voice-specific error modes, especially at the audio entity level; and (c) comprehensive measures of both accuracy and user experience that permit cross-architecture comparison.

EVA-Bench Framework Overview

EVA-Bench is introduced as an extensible, open-source end-to-end framework designed to rigorously benchmark voice agents under realistic conversational, acoustic, and behavioral conditions. Key pillars of the framework are:

• Validated, multi-turn bot-to-bot audio simulation: Automated conversation generation between a domain-parameterized user simulator and the system under evaluation, with robust validation to ensure fidelity to intended user goals, persona, and acoustic characteristics. Regeneration on simulator error decouples agent evaluation from simulation artifacts.
• Perturbation suite: Systematic perturbations encompassing accent (e.g., French-accented speaker), background noise (e.g., coffee shop), and caller behavioral styles (e.g., impatience, slow speech), supporting analysis of robustness to real-world deployment variables.
• Composite architecture-agnostic metrics: Two headline metrics—EVA-A (accuracy) and EVA-X (experience)—decompose the evaluation into well-defined subdimensions for task completion, faithfulness, speech fidelity, conversation progression, conciseness, and turn-taking, operationalized via deterministic computation and LLM- or LALM-as-judge scoring on per-conversation artifacts.
• Multi-trial reliability statistics: Pass@1, pass@k, and pass^k (probability of passing all k independent trials) quantify the gap between best-case and reliable system behavior across repeated runs, exposing variance not visible in single-shot evaluation.
• Open benchmark corpus: 213 multi-turn task scenarios across airline, healthcare HR, and ITSM domains, designed to stress entity recognition, workflow branching, and strict policy adherence.

Metrics and Measurement Methodology

EVA-A: Accuracy

• Task Completion: Deterministic SHA-256 hash comparison of the agent's final scenario database against ground truth—binary metric sensitive to any deviation from correct execution. No partial credit is awarded for near-correct outcomes, enforcing strict correctness.
• Faithfulness: LLM-as-judge evaluation of adherence to instructions, policy, and tool outputs. Pipeline-aware: in cascade systems, only the LLM’s outputs are attributed to the agent; in S2S/audio-native, mishearing is charged to the agent.
• Speech Fidelity: LALM-as-judge scoring of whether the agent’s spoken output, at the audio level, faithfully renders the intended entities, with particular attention to high-stakes items (codes, dates, amounts). Distinct from transcript-level evaluation, this ensures detection of speech errors otherwise undetectable from text alone.

EVA-X: Experience

• Conversation Progression: LLM-as-judge evaluation of forward conversational movement, absence of unnecessary tool calls/restatements, and avoidance of stalls.
• Conciseness: Measures per-turn brevity appropriate for auditory delivery, penalizing verbose, densely packed, or enumerative agent responses.
• Turn-Taking: Deterministically computed from aligned audio logs, classifying turns as on-time versus late/early and scoring interruptions and yielding behavior with tool-call-aware latency thresholds.

Reliability Aggregates

• Pass@1: Fraction of all trials passing thresholds on all relevant metrics; measures average performance.
• Pass@k: Fraction of scenarios where at least one of the k trials passes; measures theoretical peak performance.
• Pass^k: Fraction that all k trials pass in a scenario; measures practical reliability.

Empirical Findings Across Architectures

Major Results

• No evaluated system simultaneously exceeds 0.5 on both EVA-A pass@1 and EVA-X pass@1. The only system above 0.4 on both axes is GPT-Realtime-1.5 (0.47, 0.57). The accuracy-experience frontier is thus sparsely populated despite diverse architectures, indicating that high accuracy does not co-occur with a consistently strong user experience.
• Peak vs. reliable performance gap is substantial in all systems: The median gap between pass@k and pass^k is 0.44 on EVA-A, confirming that trial variance and fragility are significant—single-run scores significantly overstate operational reliability.
• Perturbation analysis shows architecture-dependent degradation: Cascade systems (STT/LLM/TTS) are strongly degraded in accuracy by accent (+10 points drop for task completion, up to 31 in worst case under accent+noise); S2S systems are largely unaffected in accuracy but degrade on experience (turn-taking, −0.16 under background noise). Within class, robustness varies: the best cascades more closely resemble S2S systems under perturbation.
• Named entity transcription is an accuracy bottleneck in cascade systems: Correlation r=0.93 between entity-level transcription accuracy and task completion highlights the centrality of accurate transcription of codes and identifiers. Systems below 70% entity transcription accuracy show 39% lower task completion.
• Faithfulness and task completion are decoupled: 72% of conversations with perfect task completion harbor at least one faithfulness deviation, necessitating both metrics for comprehensive accuracy assessment.

Fine-grained Observations

• Within cascade systems, a clear trade-off emerges: high-accuracy models incur significantly higher mean latencies in tool-call turns, while lower-latency systems exhibit sharply reduced accuracy.
• S2S models lead on user experience, particularly turn-taking (mean scores: S2S 0.82–0.83; cascade 0.28–0.58), technical responsiveness, and conversation completion. However, policy adherence (faithfulness) remains weaker in S2S than in the best cascades (+24.6 percentage points more violations).
• Hybrid architectures evaluated (AudioLLM+TTS) cluster with their input-side (audio) lineage on accuracy robustness—highlighting the nuanced effects of architecture composition.
• Perturbation sensitivity is not uniform: e.g., turn-taking is adversely affected in 81% of significant cases under perturbed conditions across all architectures.

Methodology Rigor and Validation

Extensive validation ensures that metric outcomes reflect genuine system behavior rather than judge stochasticity or simulation artifacts:

• Simulation validation: 12% of evaluated trials required regeneration due to user simulator drift; speech fidelity check errors were below 4%. LLM-as-judge validation achieves quadratic-weighted Cohen’s K of 0.777 to 0.845, matching or exceeding human annotator agreement.
• Variance decomposition: Main sources of score variance are scenario difficulty and per-trial stochasticity; judge variance is several times smaller in all metrics. Multi-architecture scenario interaction is significant: scenario-level ICCs for faithfulness and task completion are high (0.17–0.31), but model×scenario interaction explains 4–18% of variance, indicating model-specific sensitivity.

Implications and Future Prospects

EVA-Bench sets a new evaluative paradigm by (a) enabling principled, architecture-agnostic measurement; (b) surfacing latent failure modes (especially in the audio entity pipeline) missed by prior benchmarks; and (c) providing detailed actionable diagnostics for system improvement. Direct practical implications include:

• Deployment assurance: Reliance on pass@1 (single evaluation) is insufficient; reliable deployment requires robust pass^k performance, particularly in enterprise and safety-critical regimes.
• Robustness requirements: Accent and noise robustness remain broad open problems, particularly for STT-based pipelines. The current performance landscape indicates significant real-world risk for voice agent deployments across diverse user populations and telephony conditions.
• Compositional metrics: Joint measurement of task accuracy and conversation experience will drive agent development strategies that explicitly balance latency, naturalness, and instruction-following.

From a broader AI perspective, EVA-Bench’s modular extensibility establishes a platform for measuring progress on end-to-end spoken language tasks, supporting new architectures (e.g., audio foundation models, multilingual agents) and new evaluation axes (e.g., safety, user preference, fairness) as research advances.

Conclusion

EVA-Bench advances the state of evaluation for voice agents by combining rigorous simulation, architecture-independent metrics, trial-level reliability statistics, and domain-rich, entity-centric task design. The benchmark reveals that no current system meets even moderate adequacy on both accuracy and experience under realistic multi-turn audio simulation, that real-world reliability is materially lower than peak capability, and that robustness to heterogenous input conditions remains unsolved. EVA-Bench’s open, extensible design is well-positioned to become an indispensable resource for the development and assessment of robust, production-ready voice agents in the era of increasingly capable multimodal AI.