MultiVox: Benchmark for Omni-modal Voice Assistants

• MultiVox is a diagnostic benchmark integrating speech paralinguistics and visual cues to assess omni-modal voice assistants.
• It uses 1,000 paired scenarios with confounders to rigorously test speech grounding, visual grounding, and multimodal fusion.
• Empirical results reveal models excel in visual tasks but struggle with nuanced speech features, underscoring the need for improved integration.

MultiVox is a diagnostic benchmark for evaluating omni-modal voice assistants (OVAs) in multimodal, paralinguistically rich settings. It addresses the deficiencies of unimodal benchmarks and vision-centric multimodal evaluations by explicitly testing the ability of voice assistants to integrate spoken input—including fine-grained paralinguistic speech features—and visual cues such as images and videos. The MultiVox dataset comprises 1,000 human-recorded, professionally annotated speech+visual scenarios with exactly paired confounder samples, supporting systematic assessment of speech and visual grounding, and their contextual integration (Selvakumar et al., 14 Jul 2025).

1. Rationale and Position within Benchmarking Ecosystem

Prevailing benchmarks assess either vision–language (e.g., VQA, OmniBench, OmniXR) or unimodal voice assistant capabilities (e.g., VoXDialogue, SD-Eval, S2S-Arena), but not their integration. Vision-centric datasets focus on visual reasoning from text, omitting spoken input. Most voice-assistant datasets train and evaluate unimodal (speech-only) skills, typically on TTS-synthesized speech lacking natural paralinguistic richness such as pitch, timbre, and background noise. These approaches produce benchmarks that are blind to context-aware, multimodal reasoning and insensitive to finer attributes of natural speech required for robust OVA deployment.

MultiVox is the first benchmark designed to systematically evaluate:

• Speech grounding: fine-grained understanding of emotion, volume, ambient sounds, and speaker profile.
• Visual grounding: reliable perception of objects, scene text, and actions from images/videos.
• Multimodal fusion: requirement that both speech and visual modalities be integrated for correct, contextually appropriate response generation.
• Robustness to modality shortcutting: use of confounder-paired samples (identical text+visual input, flipped speech feature) to foil unimodal-only strategies (Selvakumar et al., 14 Jul 2025).

2. Dataset Composition and Annotation Protocol

MultiVox consists of 1,000 unique scenarios, each provided with a confounder (speech attribute flipped), creating 2,000 distinct prompt variants. Eight professional voice actors (age and gender diverse) recorded speech prompts, ensuring authentic paralinguistic variation. Visual content comprises 560 images and 440 videos, each paired with speech such that one detail in each modality is critical for the correct answer.

Taxonomy of Paralinguistic Features:

• Acoustic Scene: e.g., restaurant noise vs. library silence.
• Speaker Profile: e.g., child vs. adult vs. elderly, gender.
• Paralanguage: e.g., angry vs. happy tone, loudness, timbre.

Annotation Pipeline:

• Expert-defined category and subcategory definitions.
• Task “specification cards” with attribute definition, illustrative scenarios, templates, and answer formats.
• Expert annotation of scenarios, confounders, correct answers, and rationales.
• Voice prompts recorded under audio supervision, with background sound overlays as necessary.
• Dual-verification for cue perceptibility, with reject-and-re-record if verification fails.

Visual Feature Coverage:

• Object recognition, scene understanding, scene text reading, and video-specific cues (e.g., object motion direction). Each scenario contains one visual “hook” determinative for the answer, paired with a critical speech “hook.”

3. Formal Task and Modality Integration Definitions

Let $s$ denote the recorded speech waveform, $v$ the visual input (image or video), and $y$ the target textual response. The evaluation decomposes into three core tasks:

• Visual Grounding (VG): Infer the critical visual attribute (classification/regression) from $(s,v)$.
• Speech Grounding (SG): Infer the relevant paralinguistic attribute from $(s,v)$.
• Contextual Appropriateness (CA): Rate the model’s open-ended natural language response to the user’s query, contingent upon both $s$ and $v$.

The core multimodal capability is captured by learning a function $f: (s,v) \mapsto y$, with performance measured on how well $f$ fuses information across speech and vision to generate contextually appropriate, grounded answers. The confounder-pairing design ensures that unimodal reliance—ignoring cross-modal cues—is empirically detectable and penalized (Selvakumar et al., 14 Jul 2025).

4. Evaluation Metrics

Three principal evaluation axes are used:

• Visual and Speech Grounding Accuracy: For classification tasks, accuracy $(TP + TN)/(TP + TN + FP + FN)$. Macro-F1 is computed if multiple labels exist per category.
• Contextual Appropriateness (CA): Model output is human-judged (GPT-4 judge) on a 1–5 scale, evaluating content correctness, integration of speech/visual hooks, and naturalness. Mean appropriateness score:

$v$0

• Confounder Sensitivity (R_flip): Fraction of confounder pairs for which the model’s answer changes when a speech property is inverted. $v$1 implies strong speech sensitivity; lower flip rates signal unimodal or shortcut reliance.

5. Empirical Results

Performance is starkly differentiated between human subjects and state-of-the-art OVAs, captured by three axes: VG (visual grounding), SG (speech grounding), and CA (contextual appropriateness). A summary of results is provided below.

Model	Avg. VG (%)	Avg. SG (%)	Avg. CA (1–5)
Human Baseline	90.77	90.03	4.35
Qwen 2.5 Omni (7B)	86.52	23.68	2.91
VITA 1.5 (1.6B)	83.87	22.32	2.66
Gemini 2.0 Flash	85.70	23.61	2.91
phi4 multimodal (5.6B)	85.86	22.68	2.79

• Human contextual appropriateness (CA) score is 4.35/5; best OVA model scores reach 2.91 (e.g., Qwen 2.5 Omni, Gemini 2.0 Flash).
• Visual grounding (VG) is relatively robust in models ($v$280–85%), whereas speech grounding (SG) is severely limited (20–33%).
• Confounder analysis: over 45% of paired samples receive identical model answers despite flipped speech cues, with $v$377% of “correct” answers on these pairs attributable to chance or modality bias rather than genuine speech grounding.

6. Failure Analysis and Implications for OVA Research

Dominant failure mode (68.3%) is speech perception error, i.e., inability to correctly interpret paralinguistic cues such as tone, volume, timbre, or environmental noise. Multimodal reasoning failure (15.6%) indicates inability to integrate perceived speech/visual information, while 16.1% of remaining errors reflect failures in following instructions or question templates.

Breakdown by feature/domain highlights:

• Models avoid firm speaker profiling (refusal 35%, ambiguity 31%, default “adult” 35%) and rely on visual context infrequently (17%).
• SG accuracy for acoustic scenes is low (23.7%); music recognition is especially poor (successful in only ~11% of relevant cases).
• When text meaning and paralinguistic tone conflict, model’s predictions are correct only 48% of the time, indicating over-reliance (71%) on lexical/visual cues.

Models perform best on vision-only tasks (object/scene recognition), and on multimodal queries where the answer is redundant across modalities. Performance on “hard” fusion tasks—where the critical cue is speech or requires integration—is consistently weak.

A plausible implication is that future OVA progress will depend on architectural and corpus advances in paralinguistic learning, as well as on fusion mechanisms that enforce joint reasoning rather than unimodal shortcuts.

7. Directions for Benchmarking and Model Development

MultiVox demonstrates essential gaps in OVA capabilities:

• Speech grounding of paralinguistic content is the principal bottleneck.
• Current model architectures allow for shortcutting via unimodal cues; confounder-based protocols are necessary for robust evaluation.
• Open-source, human-recorded multimodal evaluation datasets are a necessary catalyst.
• Future work should expand to multilingual and dialectal variants, and address output quality for voice synthesis (prosody, naturalness)—tasks orthogonal to but complementary with current MultiVox objectives (Selvakumar et al., 14 Jul 2025).