Voice Evaluation of Reasoning Ability (VERA)

• VERA is a benchmark designed to assess real-time reasoning in voice systems by comparing performance on text versus voice modalities, quantifying the Voice Reasoning Gap.
• It comprises 2,931 voice-native episodes adapted from established benchmarks and organized into five tracks, offering diverse reasoning challenges.
• The testbed emphasizes practical constraints such as low time-to-first-response, interruptibility, and full-duplex interaction that impact system accuracy in live conversations.

Voice Evaluation of Reasoning Ability (VERA) is a benchmark and testbed purpose-built to measure how well voice-interactive systems reason under real-time conversational constraints, and to diagnose the modality-induced performance gap between voice and text. It comprises 2,931 voice-native episodes derived from established text benchmarks and organized into five tracks—Math, Web, Science, Long-Context, and Factual—so that the same model families can be compared across text and voice conditions. VERA is centered on “thinking while speaking”: streaming input and output, full-duplex behavior, barge-in, interruptibility, and low time-to-first-response all force systems to commit to spoken reasoning before iterative computation has converged. The benchmark formalizes this discrepancy as the Voice Reasoning Gap, or VRG, and reports large, consistent cross-modal deficits that increase with task complexity (Lin et al., 30 Sep 2025).

1. Conceptual framing and the Voice Reasoning Gap

VERA begins from a specific problem formulation: modern voice systems are strong at audio understanding, including transcription and acoustic scene analysis, but users increasingly expect them to think while speaking—solve competition mathematics, synthesize web evidence, reason over long contexts, and answer factual questions in live conversation. The benchmark therefore asks how much reasoning performance is lost when the same model families operate through a low-latency, streaming voice interface rather than text.

Its central construct is the Voice Reasoning Gap for a task distribution $\mathcal{T}$:

$$\mathrm{VRG}(\mathcal{T}) = \mathbb{E}_{t \sim \mathcal{T}}\left[P_{\text{text}}(t) - P_{\text{voice}}(t)\right].$$

Here, $P_{\text{text}}(t)$ and $P_{\text{voice}}(t)$ denote the best achievable accuracies for task $t$ in text and voice modalities, measured with top models from the same family where possible. This definition makes VERA more than a task collection: it is a controlled protocol for cross-modal comparison.

The benchmark’s real-time constraints are not ancillary. Streaming input and output, full-duplex interaction, interruptions, and low time-to-first-response are treated as constitutive properties of voice reasoning. Once a system begins speaking, its reasoning path is partially externalized and cannot be revised without conversational cost. A plausible implication is that VERA is designed to isolate not only whether a model can solve a task, but whether it can do so under irreversible, latency-constrained speech generation (Lin et al., 30 Sep 2025).

2. Corpus design and voice-native construction

VERA contains 2,931 voice-native episodes. The episodes are adapted from established text benchmarks while preserving reasoning difficulty and making prompts speakable, interruptible, and suitable for conversational audio delivery. The average spoken prompt duration is 22.6 s, and the average speaking rate is 172.9 WPM.

Track	Episodes	Source benchmark/task
Math	115	AIME competition problems (2020–2025)
Web	1,107	BrowseComp multi-hop web QA episodes
Science	161	GPQA Diamond graduate-level science QA
Context (Long-Context)	548	MRCR multi-round coreference episodes
Factual	1,000	SimpleQA short factual questions

The source tasks are adapted in track-specific ways. AIME items use integer answers in $[0,999]$ for speakability, with algebra verbalized as expressions such as “x squared plus three x minus two.” BrowseComp episodes retain multi-hop reasoning, while URL citations are converted into speakable attributions such as “according to a 2014 journal article.” GPQA Diamond items verbalize scientific notation and chemistry, for example “H two S O four.” MRCR preserves long-context retrieval and coreference by pairing short spoken prompts, approximately 4.2 s, with associated long textual evidence up to 100K characters. SimpleQA contributes unambiguous short factual questions optimized for voice.

The adaptation pipeline has four stages. First, a voice suitability filter rejects items with visual dependencies, excessive audio memory load, non-interruptible reasoning, or poor TTS articulation. Second, TTS-aware rewriting expands speakable numbers and symbols, adds prosodic sentence segmentation, and introduces natural openings such as “Can you help me figure out…,” while preserving semantics. Third, structured validation uses an LLM validator that scores TTS readiness, conversational naturalness, and reasoning preservation; episodes are retained only if $Q_{\text{overall}} \ge \tau$ and $Q_{\text{reason}} \ge 7.0$, and accepted episodes average $Q_{\text{overall}} = 9.0$. Fourth, speech generation renders the prompts to 24 kHz audio with Higgs-Audio v2; automatic variation in timbre, tone, and emotion is validated via WeSpeaker, with $\sigma = 0.120$ and mean $$\mathrm{VRG}(\mathcal{T}) = \mathbb{E}_{t \sim \mathcal{T}}\left[P_{\text{text}}(t) - P_{\text{voice}}(t)\right].$$0 (Lin et al., 30 Sep 2025).

3. Interaction model, scoring protocol, and measured quantities

VERA episodes simulate live, streaming dialogs. Models receive a spoken prompt and are expected to respond in speech with low latency; many systems are full-duplex and can produce audio while listening. The benchmark explicitly evaluates streaming input and output, interruptions, and the pressure to begin speaking quickly even when deeper reasoning would benefit from delayed commitment.

Accuracy is defined as

$$\mathrm{VRG}(\mathcal{T}) = \mathbb{E}_{t \sim \mathcal{T}}\left[P_{\text{text}}(t) - P_{\text{voice}}(t)\right].$$1

with track-specific scoring semantics: Math uses strict final numeric or choice correctness; Web and Science judge the synthesized answer against ground truth; Context uses exact substring retrieval when specified; Factual uses entity or value match with minor spelling tolerated. Macro-averaging assigns equal weight to the five tracks:

$$\mathrm{VRG}(\mathcal{T}) = \mathbb{E}_{t \sim \mathcal{T}}\left[P_{\text{text}}(t) - P_{\text{voice}}(t)\right].$$2

This prevents the larger Web track from dominating the aggregate.

Latency is reported as time-to-first-response (TTFR). For streaming voice systems it is the first audio byte after user speech end; for non-streaming voice it is the first audio token; for text it is the first text token:

$$\mathrm{VRG}(\mathcal{T}) = \mathbb{E}_{t \sim \mathcal{T}}\left[P_{\text{text}}(t) - P_{\text{voice}}(t)\right].$$3

Speech fidelity is measured by Word Error Rate after LLM-based normalization of math notation. Scoring uses LLM-as-a-judge with majority vote across three independent GPT-4o judgments. Judge reliability is reported as 97.8% overall agreement with humans, with 95% CI 96.8–98.7%; agreement is perfect on Math, and cross-vendor agreement between GPT-4o and Gemini-2.5-Flash is 98.1%.

The evaluation covers native streaming voice APIs—GPT-realtime, Gemini-2.5-Flash-audio, and Nova-Sonic—alongside open voice stacks and audio-LLMs such as Qwen2-Audio, UltraVox, Audio Flamingo 3, Audio Flamingo 3 (thinking mode), and Phi-4-multimodal; end-to-end speech-in/speech-out systems such as Moshi, Freeze-Omni, and Qwen2.5-Omni; the LiveAnswer cascade baseline, which decouples reasoning from narration; and text-only upper bounds including GPT-4o, GPT-5, and Gemini-2.5 Pro/Flash (Lin et al., 30 Sep 2025).

4. Quantitative results and the empirical shape of the gap

VERA’s headline result is a large cross-modal gap. The best text upper bound is 54.0% macro-averaged accuracy, attained by GPT-5 with effort=high, whereas the best native voice system is GPT-realtime at 11.3%, with TTFR 2.69 s and WER 9.6%. On competition mathematics, a leading text model reaches 74.8% while its voice counterpart reaches 6.1%, a gap of 68.7 points. Even on the Factual track, where voice performance is relatively stronger, the reported comparison is 48.3% for text versus 27.4% for voice.

The latency–accuracy frontier is a core empirical observation. Systems with TTFR $$\mathrm{VRG}(\mathcal{T}) = \mathbb{E}_{t \sim \mathcal{T}}\left[P_{\text{text}}(t) - P_{\text{voice}}(t)\right].$$4 s cluster around approximately 10% accuracy, forming a low-latency plateau. The “fast-and-accurate” upper-left corner is empty. Approaching text-level performance requires substantially relaxing latency budgets: text high-effort GPT-5 is reported at TTFR 35.9 s, and Gemini Pro at 21.1 s. This suggests that the conflict between conversational immediacy and deep reasoning is not incidental but structural under current architectures.

The LiveAnswer cascade baseline provides a different trade-off. By separating reasoning from narration it reaches 27.0% macro average, with TTFR 10.50 s and WER 7.5%. On Math it reaches 59.1%, still 15.7 points below GPT-5 text at 74.8%; on Context it reaches only 0.2%, reflecting narration and exact-match failures. Within the GPT and Gemini families, voice variants trail text across all tracks. In the Qwen family, diverse voice architectures remain below 5% on reasoning tracks. Variance analysis reports that within-voice variance across Math, $$\mathrm{VRG}(\mathcal{T}) = \mathbb{E}_{t \sim \mathcal{T}}\left[P_{\text{text}}(t) - P_{\text{voice}}(t)\right].$$5, is 171× smaller than between-modalities, $$\mathrm{VRG}(\mathcal{T}) = \mathbb{E}_{t \sim \mathcal{T}}\left[P_{\text{text}}(t) - P_{\text{voice}}(t)\right].$$6, indicating that the dominant effect is modality rather than model-family idiosyncrasy.

The reported differences are statistically validated. McNemar tests give $$\mathrm{VRG}(\mathcal{T}) = \mathbb{E}_{t \sim \mathcal{T}}\left[P_{\text{text}}(t) - P_{\text{voice}}(t)\right].$$7 across tracks. For GPT-5 versus GPT-realtime, the macro gap is 40.4 points with 95% CI 37.7–43.2. Track-level gaps are 68.7 for Math, 71.5 for Context, 11.5 for Web, 29.2 for Science, and 20.9 for Factual (Lin et al., 30 Sep 2025).

5. Diagnostic experiments and architecture-specific failure modes

VERA is designed not only to rank systems but to diagnose why they fail. One diagnostic experiment increases “thinking time” in Audio Flamingo 3 thinking mode. Latency rises from 2.40 s to 15.14 s, a +530% increase, yet overall accuracy does not improve, moving from 1.7% to 1.5%; Context performance declines from 3.8% to 1.8%. In the benchmark’s interpretation, longer pre-speech deliberation alone does not resolve VRG.

A second diagnostic tests decoupling reasoning from narration through the LiveAnswer cascade. Accuracy improves relative to native voice systems, from approximately 10–11% to 27.0%, but remains far below text upper bounds and introduces new errors. The latency profile is dominated by speech-to-text at 9.68 s and LLM+TTS completion at 63.40 s, with partial TTFR from STT completion to first audio of 0.83 s. The resulting failures are not simply slower versions of native streaming failures; they are characterized by grounding and consistency problems, especially on exact-match retrieval.

The benchmark formalizes error signatures through a deviation score,

$$\mathrm{VRG}(\mathcal{T}) = \mathbb{E}_{t \sim \mathcal{T}}\left[P_{\text{text}}(t) - P_{\text{voice}}(t)\right].$$8

which measures per-model deviation from the global baseline for an error category $$\mathrm{VRG}(\mathcal{T}) = \mathbb{E}_{t \sim \mathcal{T}}\left[P_{\text{text}}(t) - P_{\text{voice}}(t)\right].$$9. Native streaming systems show a completion bias: fewer off_target and no_final_answer errors, but more fluent-but-wrong outputs. The cascade shows grounding and consistency issues, including unsupported_fact $P_{\text{text}}(t)$0, off_target $P_{\text{text}}(t)$1, and logical_contradiction $P_{\text{text}}(t)$2. End-to-end systems display different extremes: Moshi shows off_target $P_{\text{text}}(t)$3, while Qwen2.5-Omni shows no_final_answer $P_{\text{text}}(t)$4 and a strong negative deviation for unsupported_fact $P_{\text{text}}(t)$5.

The concrete failure analyses align with these aggregate patterns. Math failures include premature answers and arithmetic slips. Web failures include hallucinated references and misattributed sources. Context failures include drift and failure to retrieve exact strings. Cascade failures include mis-grounded narration in which the final spoken statement is inconsistent with the internal reasoning trace. A common misconception is that voice reasoning failures are primarily a transcription problem; VERA’s combination of WER reporting, latency analysis, and architecture-specific error taxonomies suggests a broader systems problem involving commitment timing, state management, and cross-stage consistency (Lin et al., 30 Sep 2025).

6. Relation to adjacent benchmarks and follow-on systems

VERA sits within a rapidly expanding literature on voice-native evaluation, but its emphasis is narrower and more diagnostic than several adjacent benchmarks. The related benchmark "$P_{\text{text}}(t)$6-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains" evaluates full-duplex voice agents on 278 grounded customer-service tasks requiring policy adherence and environment interaction. It reports GPT-5 at 85% pass@1 in text, while voice agents reach 31–51% under clean conditions and 26–38% under realistic conditions, retaining only 30–45% of text capability; manual review attributes 79–90% of failures to agent behavior rather than simulator artifacts. This complements VERA by shifting from reasoning tracks such as Math and GPQA Diamond toward grounded task completion with realistic accents, noise, and turn-taking (Ray et al., 14 Mar 2026).

VERA has also been used as an evaluation target for streaming architectures. "LTS-VoiceAgent: A Listen-Think-Speak Framework for Efficient Streaming Voice Interaction via Semantic Triggering and Incremental Reasoning" evaluates a cascaded ASR–LLM–TTS system on VERA’s AIME and GPQA-Diamond settings. On VERA it reports Acc 6.88, Lat 230 ms, TTFS 415 ms, NFE 2.37, NIT 0.13, and IR 5.46%, and attributes these results to a Dynamic Semantic Trigger and a Dual-Role Stream Orchestrator that separate background state maintenance from foreground response generation (Zou et al., 26 Jan 2026).

Other contemporaneous benchmarks broaden the notion of spoken reasoning. "SpeechR: A Benchmark for Speech Reasoning in Large Audio-LLMs" organizes 3,366 multimodal instances across factual retrieval, procedural inference, and normative judgment, and evaluates them in multiple-choice, generative, and acoustic-feature settings; its central finding is that high transcription accuracy does not translate into strong reasoning capabilities (Yang et al., 4 Aug 2025). "SpeechIQ: Speech Intelligence Quotient Across Cognitive Levels in Voice Understanding LLMs" structures voice evaluation around Remembering, Understanding, and Application, argues that WER is necessary but insufficient, and treats Application-level QA as the closest proxy to reasoning while explicitly leaving higher-order levels such as Analyze, Evaluate, and Create for future work (Wan et al., 25 Jul 2025).

7. Reproducibility, limitations, and research directions

VERA is released with code and data at https://github.com/linyueqian/VERA. Its evaluation harness requires a voice interface that accepts spoken prompt audio and returns spoken output, precise TTFR capture, ASR over model output, normalization to canonical math and notation, majority-vote grading with three judgments, and reporting of per-track accuracy, macro-average, WER, and TTFR. The repository includes filtering, adaptation, and quality-check prompts, grading and failure-taxonomy prompts, and normalization recipes.

The benchmark’s scope is deliberately bounded. It covers five domains selected through voice feasibility filters; visual reasoning and non-speech noise or accents are not the focus. The prompts use synthesized speech with validated acoustic diversity rather than broad real-world speaker variation. The Context track uses long textual evidence but short spoken prompts, so it is not yet a benchmark for long spoken context. Future work is identified in the paper itself: richer speaker and accent variation, noisy conditions, and multi-turn long-spoken-context interaction.

The broader implication is architectural. The persistence of VRG under native streaming, end-to-end, and cascade designs suggests that incremental patches are insufficient. The paper argues for decoupling thinking from speaking through an editable internal state that is separate from the speech buffer; asynchronous backends and chunked parallel reasoning that use playback time to compute are identified as promising directions. Managing consistency across stages and confidence-gated handoffs is described as critical. In practical terms, VERA recommends reporting macro-averaged accuracy, TTFR, WER, and explicit modality-gap measures such as $P_{\text{text}}(t)$7 and $P_{\text{text}}(t)$8. This suggests that progress in voice reasoning will be measured not only by better answers, but by better mechanisms for deferring, revising, and grounding reasoning under the irreversible timing constraints of speech (Lin et al., 30 Sep 2025).