EVA-Bench Evaluation Framework

• EVA-Bench is a benchmark that distinguishes between voice-agent evaluation and embodied video anticipation, offering tailored metrics for each domain.
• In the voice-agent context, it employs bot-to-bot audio simulations, composite accuracy and experience scores, and rigorous repeated-trial validation.
• For embodied video anticipation, it integrates multimodal tasks with advanced language and video quality metrics across diverse real and simulated scenarios.

EVA-Bench is a benchmark name used in at least two distinct research contexts. In the exact-title usage, it denotes an end-to-end framework for evaluating voice agents through bot-to-bot audio conversations, simulator validation, composite accuracy and experience metrics, controlled perturbations, and repeated-trial reliability statistics (Bogavelli et al., 13 May 2026). In an earlier embodied-world-model work, EVA-Bench denotes the “Embodied Video Anticipation Benchmark,” a multimodal benchmark for task-level future prediction in egocentric human, real-robot, and simulated-robot scenarios (Chi et al., 2024). The shared label therefore does not identify a single benchmark family.

1. Terminological scope and disambiguation

The term “EVA-Bench” is not monosemous in recent arXiv literature. It refers most directly to the voice-agent benchmark titled “EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents” (Bogavelli et al., 13 May 2026), but it also appears as the short form of “Embodied Video Anticipation Benchmark” in “EVA: An Embodied World Model for Future Video Anticipation” (Chi et al., 2024).

Usage	Paper	Domain
EVA-Bench	“EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents”	Voice agents
EVA-Bench	“EVA: An Embodied World Model for Future Video Anticipation”	Embodied video anticipation

Several similarly named benchmarks are separate entities rather than variants of EVA-Bench. “EvasionBench” is a financial Q&A benchmark for evasive-answer detection (Ma et al., 14 Jan 2026), “EvaLearn” evaluates sequential problem solving in LLMs (Dou et al., 3 Jun 2025), “EVMbench” evaluates AI agents on smart contract security (Wang et al., 5 Mar 2026), and “EBench” diagnoses mobile manipulation policies (Gao et al., 16 Jun 2026). A plausible implication is that “EVA-Bench” should be interpreted strictly from paper context rather than as a stable umbrella brand.

2. EVA-Bench for voice agents: end-to-end framework

In the voice-agent setting, EVA-Bench is introduced as an end-to-end evaluation framework for AI systems that conduct spoken conversations to complete tasks. Its central claim is that prior work did not jointly solve two problems: generating realistic spoken multi-turn simulations and measuring quality across the full scope of voice-specific failure modes (Bogavelli et al., 13 May 2026).

The framework orchestrates parallel per-scenario bot-to-bot audio sessions over WebSocket. Each scenario specifies a user goal, user persona, scenario database, and ground-truth final state. A decision tree accompanies the user goal so that repeated trials remain interpretable and converge toward a fixed intended outcome under correct system behavior. The benchmark is designed for domains where spoken entities are operationally critical, including confirmation codes, employee IDs, names, dates, monetary amounts, and other domain-specific identifiers.

A major design feature is architecture coverage. EVA-Bench supports three classes of voice agents: Cascade, formalized as $$\text{STT} \rightarrow \text{LLM} \rightarrow \text{TTS}$$; S2S, where the system is audio-native end to end; and Hybrid, where audio input is handled by an AudioLLM or LALM and output is passed through a TTS stage. This architecture-aware framing is important because different systems expose different intermediate signals, and the benchmark adapts metric attribution accordingly.

The benchmark includes 213 scenarios across three enterprise domains: Airline Customer Service Management with 50 scenarios, Healthcare Human Resources Service Delivery with 83 scenarios, and Enterprise Information Technology Service Management with 80 scenarios. Within domains, scenarios span Single-Intent, Multi-Intent, and Adversarial variants. Multi-intent scenarios can combine one to four concurrent workflows, while adversarial scenarios stress hard policy boundaries under social pressure.

Simulation validity is treated as a first-class benchmark component. Before scoring, conversations are checked for Conversation Finished, User Behavioral Fidelity, and User Speech Fidelity. Invalid simulations are regenerated rather than scored. This validation layer is one of the framework’s most distinctive features, since it attempts to ensure that failures reflect the agent under test rather than simulator corruption.

3. Voice-agent metrics, thresholds, and repeated-trial statistics

The voice-agent EVA-Bench is organized around two composite dimensions: EVA-A (Accuracy) and EVA-X (Experience) (Bogavelli et al., 13 May 2026). EVA-A comprises Task Completion, Faithfulness, and Speech Fidelity. EVA-X comprises Conversation Progression, Conciseness, and Turn-Taking. All component scores are normalized to $[0,1]$ before aggregation.

Task Completion is deterministic and binary. The framework compares the actual final backend state with the expected final state via canonical serialization and SHA-256 hashing. Faithfulness is an LLM-as-judge metric using Claude Opus 4.6, scored over fabricating_tool_parameters, misrepresenting_tool_result, violating_policies, failing_to_disambiguate, and hallucination. The dimension-level rating is aggregated by minimum and normalized as

$\text{Faithfulness}=\frac{r_{\text{faith}}-1}{2}.$

Speech Fidelity is a LALM-as-judge metric using Gemini 3 Flash and is defined as the mean of per-turn binary correctness scores:

$$\text{SpeechFidelity}=\frac{1}{|\mathcal{T}|}\sum_{t\in\mathcal{T}} s_t.$$

A conversation passes EVA-A iff all three conditions hold simultaneously:

$$(\text{task completion}=1.0)\ \land\ (\text{faithfulness}\ge 0.5)\ \land\ (\text{speech fidelity}\ge 0.95).$$

On the experience side, Conversation Progression is an LLM-as-judge metric using GPT-5.2, with dimensions unnecessary_tool_calls, information_loss, redundant_statements, and question_quality, normalized as

$$\text{ConversationProgression}=\frac{r_{\text{prog}}-1}{2}.$$

Conciseness is also judged by GPT-5.2, but per assistant turn and then averaged:

$$\text{Conciseness}=\frac{1}{|\mathcal{T}_A|}\sum_{t\in\mathcal{T}_A}\frac{r_t-1}{2}.$$

Turn-Taking is deterministic and timing-based, with separate cases for uninterrupted turns, agent interruptions, user interruptions, and simultaneous interruption. The interruption formulas given in the paper include

$$S_{\text{agent}}=\min(S_{\text{overlap}},S_{\text{count}},S_{\text{post}})$$

and

$$S_{\text{yield}}=\max\left(0,1-\frac{\Delta t}{\Delta t_{\max}}\right), \qquad \Delta t_{\max}=2000\text{ ms}.$$

A conversation passes EVA-X iff

$$(\text{turn-taking}\ge 0.8)\ \land\ (\text{conversation progression}\ge 0.5)\ \land\ (\text{conciseness}\ge 0.5).$$

The framework also defines repeated-trial statistics. With $[0,1]$0 scenarios and $[0,1]$1 trials per scenario, pass@1 is the fraction of all trials that pass,

$[0,1]$2

pass@k is the fraction of scenarios with at least one successful trial,

$[0,1]$3

and pass$[0,1]$4 estimates reliable capability,

$[0,1]$5

This distinction between peak and reliable capability is a defining feature of the benchmark.

A controlled perturbation suite extends the clean evaluation. The paper focuses experimentally on French-accented user speech, coffee shop background noise, and accent + noise combined, while the framework itself supports additional accent, noise, connection-degradation, and persona perturbations.

4. Empirical findings and limitations of the voice-agent benchmark

The voice-agent EVA-Bench evaluates 12 systems spanning 7 cascade, 2 hybrid, and 3 S2S architectures (Bogavelli et al., 13 May 2026). The headline result is negative: no system simultaneously exceeds 0.5 on both EVA-A pass@1 and EVA-X pass@1. The strongest balanced point reported is GPT-Realtime-1.5 at roughly $[0,1]$6 on $[0,1]$7.

The benchmark also reports a large divergence between peak and reliable performance. Across the 12 systems, the median $[0,1]$8 gap is 0.44 on EVA-A and 0.24 on EVA-X. This means that one successful attempt on a scenario is often a poor proxy for repeatable competence. The result is central to the paper’s argument that single-trial evaluation overstates deployment-readiness.

Architecture effects are strongest on turn-taking. Mean Turn-Taking ranges are 0.28–0.58 for cascade systems and 0.82–0.83 for S2S systems. By contrast, the most accurate cascade systems frequently underperform on experience metrics, suggesting a capability–latency tension. For example, Nova-3 + GPT-5.4 + Sonic reaches 0.504 EVA-A pass@1 but only 0.007 EVA-X pass@1, whereas Gemini-3.1-Flash-Live leads EVA-X pass@1 at 0.589.

Perturbation results are substantial and architecture-dependent. Accent causes the largest accuracy degradation in cascade pipelines, with mean 10-point task-completion drops and a 17-point worst-case drop. Under combined accent + noise, cascade task completion drops by a mean 19 points, with worst systems dropping 31 points. S2S systems are more stable on accuracy under accent but experience degradation under noise, with EVA-X effects up to $[0,1]$9. The abstract reports perturbation effects with mean up to 0.314.

The benchmark includes validation evidence for its judges. Reported judge–human agreement values are 0.836 for Faithfulness, 0.845 for Conversation Progression, 0.823 for Conciseness, and 0.777 for Speech Fidelity. The framework also quantifies simulator instability: 12.0% of trials required regeneration due to user simulator error in one main analysis, while a broader compute-accounting appendix reports 24.1% of trials required regeneration overall, roughly half from simulator error and half from infrastructure failures or timeouts.

Limitations are explicit. The benchmark is English only, simulator speech is cleaner than real human speech, harmful outputs and PII leakage are not directly evaluated, and full reproduction depends on commercial APIs. The paper also cautions that only 12 systems are evaluated and that architecture-level conclusions should therefore not be overgeneralized.

5. EVA-Bench as the Embodied Video Anticipation Benchmark

A different use of the same name appears in “EVA: An Embodied World Model for Future Video Anticipation,” where EVA-Bench abbreviates Embodied Video Anticipation Benchmark (Chi et al., 2024). Here the benchmark is not about spoken dialogue, but about task-level video prediction in embodied scenes. The benchmark is motivated by the claim that prior video-generation and robotics evaluations do not adequately capture embodied anticipation as a video-language to video-language problem.

This EVA-Bench is built around four meta-tasks: Action-Description, Finish-Thinking, How-To, and Next-Step. The benchmark’s general formulation is

$\text{Faithfulness}=\frac{r_{\text{faith}}-1}{2}.$0

where $\text{Faithfulness}=\frac{r_{\text{faith}}-1}{2}.$1 is visual observation, $\text{Faithfulness}=\frac{r_{\text{faith}}-1}{2}.$2 is a question or instruction, $\text{Faithfulness}=\frac{r_{\text{faith}}-1}{2}.$3 is a predicted future video, and $\text{Faithfulness}=\frac{r_{\text{faith}}-1}{2}.$4 is a text response. The evaluation therefore covers both semantic understanding and future-video generation.

The benchmark is curated from the broader EVA-Instruct dataset and contains 125 high-quality samples spanning egocentric human daily activities, real-world robots, and simulated robots. Scenarios include pick-and-place tasks, cooking, bike repair, COVID testing, and indoor organization. The larger EVA-Instruct corpus contains 500K QA pairs distributed across How-To-200K, Action-Description-200K, Finish-Think-50K, and Next-Step-50K.

Its scoring framework is EVA-Score, with EVAS-L for language, EVAS-V for visual quality, and a combined EVA-Score. The normalized task score is given as

$\text{Faithfulness}=\frac{r_{\text{faith}}-1}{2}.$5

Language metrics include BLEU-1, BLEU-2, METEOR, ROUGE-L, CIDEr, SPICE, CLIPScore, and GPT-4o score. Video metrics include Subject Consistency, Background Consistency, Motion Smoothness, Dynamic Degree, Aesthetic Quality, Fréchet Video Distance, and the benchmark-specific Goal Completion Estimation.

The reported benchmark results are structured by meta-task. On How-To, EVA reaches EVAS-L 85.51, EVAS-V 69.93, and EVA-Score 77.72. On Next-Step, EVA reaches EVAS-L 73.02, EVAS-V 65.34, and EVA-Score 69.18. In the Action-Description table, EVA reports EVAS-L 47.50 and a GPT-4o score of 62.63%. These results are used to argue that embodied anticipation requires stronger semantic grounding than generic VLM plus generator pipelines provide.

The paper also notes limitations. EVA-Bench only considers interleaved video and language, has a relatively small benchmark size of 125 samples, and does not fully specify a formal in-domain versus OOD benchmark split. Long-horizon generation is emphasized conceptually through self-ask extension, but a standardized benchmark-wide long-horizon rollout protocol is not fully specified in the available description.

6. Conceptual significance of the two EVA-Bench usages

The two EVA-Bench instances share a structural idea even though their domains differ. Both evaluate systems end to end rather than through isolated component scores. The voice-agent EVA-Bench evaluates spoken, tool-using, real-time conversational systems with architecture-aware observability, simulator validation, and repeated-trial statistics (Bogavelli et al., 13 May 2026). The embodied-video EVA-Bench evaluates multimodal anticipation systems that must jointly understand current context and generate semantically appropriate future video and text (Chi et al., 2024).

Their metric philosophies are also parallel. The voice benchmark combines backend correctness with faithfulness, speech fidelity, progression, conciseness, and timing. The embodied-video benchmark combines language metrics, multimodal judges, and video-quality or goal-completion metrics. In both cases, the benchmark rejects a single narrow scalar such as transcript accuracy or raw FVD as sufficient.

At the same time, the name overlap should not obscure their incompatibility as benchmark objects. One is centered on enterprise voice agents across 213 scenarios and three architecture classes; the other is centered on embodied video anticipation across 125 curated samples and four anticipation meta-tasks. This suggests that “EVA-Bench” functions as a local benchmark name within separate research programs rather than as a single standardized benchmark family.

For citation and technical discussion, the most precise usage is therefore paper-specific: EVA-Bench for voice-agent evaluation when referring to the 2026 end-to-end voice benchmark (Bogavelli et al., 13 May 2026), and EVA-Bench as the Embodied Video Anticipation Benchmark when referring to the 2024 embodied-world-model benchmark (Chi et al., 2024).