XModBench: Tri-Modal Consistency Benchmark

• XModBench is a large-scale tri-modal benchmark that assesses cross-modal consistency and modality-invariant reasoning with 60,828 multiple-choice items across six modality pairings.
• It systematically varies context-to-candidate modalities—audio, vision, and text—to diagnose modality disparities, directional imbalances, and robustness under controlled semantic shifts.
• Empirical findings highlight superior performance in text and vision tasks while revealing audio as a significant bottleneck, underscoring challenges in achieving true multimodal integration.

XModBench is a large-scale, tri-modal benchmark for evaluating cross-modal consistency and modality-invariant reasoning in omni-modal LLMs (OLLMs). It is designed to test whether a model produces stable, correct answers when semantically identical content is presented through audio, vision, or text, and whether performance remains symmetric when context and answer modalities are exchanged. The benchmark comprises 60,828 multiple-choice question–answer pairs derived from 10,138 unique semantic instances, each instantiated across all six context-to-candidate modality compositions: Audio→Text, Audio→Vision, Text→Audio, Text→Vision, Vision→Audio, and Vision→Text (Wang et al., 16 Oct 2025).

1. Conceptual scope and motivation

XModBench was introduced to address a specific limitation in prior multimodal evaluation: many existing benchmarks measure task accuracy for particular modality pairs, but do not determine whether reasoning is invariant to modality or whether models exhibit systematic modality-specific biases. In the benchmark’s framing, cross-modal competence is not reducible to general question-answering performance. Instead, it requires that the same semantic content, expressed through different modalities, should yield consistent model behavior.

The benchmark therefore focuses explicitly on cross-modal consistency. Each unique semantic instance is realized in all six modality directions, allowing direct comparison while semantics are held constant. This design supports diagnosis along three axes: modality-invariant reasoning, modality disparity, and directional imbalance. Modality-invariant reasoning is reflected by high accuracy together with robustness to modality shifts. Modality disparity refers to accuracy gaps caused by replacing one modality with another while keeping semantic content fixed. Directional imbalance refers to asymmetry when swapping context and candidate modalities, such as Text→Vision versus Vision→Text (Wang et al., 16 Oct 2025).

Relative to prior benchmarks, XModBench is positioned as broader in modality coverage and more diagnostic in structure. Vision–text-centric benchmarks such as MMBench, SEED-Bench-2, OCRBench v2, Video-MME, and LLaVA-Bench variants provide broad task coverage, but do not systematically evaluate tri-modal consistency across all context→candidate directions. Audio–text and AVQA-style benchmarks such as AudioBench, AVQA, Pano-AVQA, Music-AVQA, WorldSense, AV-Reasoner, AV-Odyssey Bench, and OmniBench are valuable for audiovisual understanding, but typically fix one side to text or do not balance all six directions. Recent consistency studies are described as focusing mainly on image–text settings. XModBench generalizes the consistency question to audio, vision, and text in a balanced design.

A common misconception is that strong aggregate multimodal accuracy is sufficient evidence of modality-agnostic reasoning. XModBench is constructed to challenge that assumption. Its design suggests that a model may perform well in selected modality pairings while still failing to preserve reasoning behavior under controlled modality substitutions.

2. Dataset design and benchmark composition

The benchmark is evaluation-only; no train/dev/test split is defined. Each item consists of a context and four candidates in a multiple-choice format, with a 25% chance baseline. Scoring is 0/1 accuracy per item, and aggregate accuracy is averaged over items. The benchmark is balanced across all six modality directions for every task and subtask (Wang et al., 16 Oct 2025).

The six modality compositions are:

• Audio→Text (A→T)
• Audio→Vision (A→V)
• Text→Audio (T→A)
• Text→Vision (T→V)
• Vision→Audio (V→A)
• Vision→Text (V→T)

These six permutations operationalize controlled cross-modal evaluation by varying both context modality and candidate modality while preserving semantics. This permits direct observation of modality effects and directionality effects.

XModBench covers five task families and 17 subtasks spanning perception, spatial reasoning, temporal reasoning, linguistic understanding, and external knowledge.

Task family	Goal	Representative subtasks
Perception	Recognize the same object/activity/scene across modalities	General activity recognition, natural environments, musical instruments, instrument compositions
Spatial reasoning	Interpret positions and motion in 2D/3D across modalities	2D arrangement, 3D localization, 3D movement understanding
Temporal reasoning	Understand event order and frequency over time	Temporal order, temporal counting, temporal calculation
Linguistic understanding	Unify OCR/ASR-type recognition and affective meaning cross-modally	Linguistic recognition, translation, emotion classification in dialogues
External knowledge	Link multimodal content to world knowledge	Movie recognition, music genre classification, singer identification

The benchmark’s semantic core is a set of curated cross-modal triplets. These text–image–audio triplets are assembled from re-annotated or extended public datasets, synthetic or model-generated modalities used to complete missing channels, and targeted web collection. Examples given in the benchmark description include VGG-Sound for perception, STARSS23 for spatial reasoning, Urbansas for movement, FireRedTTS for speech synthesis, rendered text images for translation tasks, and web-collected singer portraits, songs, movie posters, trailers, and plot descriptions.

Candidate generation is designed to make distractors semantically challenging yet unambiguous. Question wording is refined with large LLMs, specifically GPT-5, and includes both human-written and LLM-assisted variations to avoid stylistic biases. Quality control combines LLM-based filtering using systems such as Gemini and GPT with human double-checking and iterative correction. The paper does not report inter-annotator agreement figures.

The benchmark also incorporates modality-specific processing. For spatial audio, it uses FOA encoding, 2D rotation on Ambisonics channels, decoding, and HRTF-based binauralization using SOFA/CIPIC for realistic azimuth changes. For 2D arrangement, stereo panning is implemented with vector-base amplitude panning. OCR-rendered images and TTS-synthesized speech are used in language-oriented tasks.

3. Evaluation methodology and diagnostic metrics

Let $\mathrm{Acc}(X \to Y)$ denote accuracy when the context is modality $X$ and candidates are in modality $Y$, with $X,Y \in \{A, V, T\}$ and $X \ne Y$. XModBench evaluates performance at both aggregate and diagnostic levels (Wang et al., 16 Oct 2025).

Overall task competence is summarized by the mean accuracy across the six directions:

$$\mathrm{Acc}_{\mathrm{avg}} = \frac{1}{6} \sum_{m \in D} \mathrm{Acc}(m),$$

where $$D = \{A \to T, A \to V, T \to A, T \to V, V \to A, V \to T\}$$.

Robustness to modality shifts is summarized by the standard deviation across the six directions:

$$\sigma = \sqrt{ \frac{1}{6} \sum_{m \in D} \big(\mathrm{Acc}(m) - \mathrm{Acc}_{\mathrm{avg}}\big)^2 }.$$

Within this framework, high $\mathrm{Acc}_{\mathrm{avg}}$ together with low $\sigma$ indicates better modality invariance and greater stability across modality changes.

Modality disparity is defined through paired substitutions while holding the opposite side fixed. For example, text versus vision is measured as:

$X$0

Analogous definitions apply to $X$1 and $X$2. Large negative values indicate that the substituted modality degrades performance.

Directional imbalance captures asymmetry under reversal of context and candidate modalities:

$X$3

for $X$4.

The benchmark operationalizes cross-modal consistency by comparing accuracies across the six instantiations of the same semantics and by reporting $X$5 together with the gap metrics above. The paper does not report a separate per-instance agreement rate, but it states that the benchmark design allows such a quantity to be computed if desired. This suggests that XModBench is not limited to aggregate benchmarking; it can also support finer-grained analyses of semantic stability.

A concrete example is the semantic instance “A dog is barking.” In the six instantiations, the same semantics are presented as barking audio, a barking-dog image, or the text “A dog is barking,” and the candidates appear either as text labels, images, or audio clips. Cross-modal consistency is achieved if the model selects the dog-barking answer across all six variants. In the benchmark’s evaluation logic, this contributes 6/6 correct to per-direction accuracies.

4. Models evaluated and empirical findings

The benchmark evaluates both closed-source and open-source OLLMs. The closed-source models are Gemini 1.5 Pro, Gemini 2.0 Flash, Gemini 2.5 Flash, and Gemini 2.5 Pro. The open-source models are Qwen2.5-Omni, Baichuan Omni 1.5, EchoInk-R1, VideoLLaMA 2, VITA, the Unified-IO 2 family (Large, XL, XXL), and PandaGPT. GPT-series models are omitted because of API limitations for joint audio–vision input (Wang et al., 16 Oct 2025).

The strongest reported model is Gemini 2.5 Pro, with overall average accuracy $X$6 and $X$7 across directions. Human performance is reported as $X$8 with $X$9. This establishes a sizable gap not only in mean accuracy but also in stability across modalities.

For Gemini 2.5 Pro, the task-family averages are:

• Perception: $Y$0
• Spatial reasoning: $Y$1
• Temporal reasoning: $Y$2
• Linguistic understanding: $Y$3
• External knowledge: $Y$4

These results identify spatial reasoning and temporal reasoning as the most difficult families, with spatial reasoning at $Y$5 and temporal reasoning at $Y$6. The benchmark description emphasizes that even the strongest system achieves less than 60% accuracy on spatial and temporal reasoning in the abstract-level summary.

Per-direction performance for Gemini 2.5 Pro is:

• A→T: $Y$7
• A→V: $Y$8
• T→A: $Y$9
• T→V: $X,Y \in \{A, V, T\}$0
• V→A: $X,Y \in \{A, V, T\}$1
• V→T: $X,Y \in \{A, V, T\}$2

The corresponding human per-direction results are:

• A→T: $X,Y \in \{A, V, T\}$3
• A→V: $X,Y \in \{A, V, T\}$4
• T→A: $X,Y \in \{A, V, T\}$5
• T→V: $X,Y \in \{A, V, T\}$6
• V→A: $X,Y \in \{A, V, T\}$7
• V→T: $X,Y \in \{A, V, T\}$8

These figures support three benchmark-level findings. First, text functions as the strongest anchor and audio as the weakest link. Second, vision–text settings outperform audio–text settings, and audio–vision pairings without text are typically weakest. Third, performance is often better when text is the candidate space, especially in Vision→Text relative to Text→Vision.

The reported modality disparity values for Gemini 2.5 Pro reinforce this pattern. $X,Y \in \{A, V, T\}$9 is approximately $X \ne Y$0, indicating the strongest negative disparity when substituting text with audio. $X \ne Y$1 is approximately $X \ne Y$2, while $X \ne Y$3 is approximately $X \ne Y$4, suggesting that text and vision are relatively closer to each other than either is to audio.

Directional imbalance is also systematic. For text–vision asymmetry, Gemini 2.5 Pro has $X \ne Y$5. Qwen2.5-Omni shows an even larger gap of approximately $X \ne Y$6. Audio–text asymmetry is described as around 6–8 points, while audio–vision is more symmetric but weaker in both directions. A plausible implication is that many current OLLMs are better aligned to text-form answer spaces than to non-text candidate spaces.

Among open-source systems, Qwen2.5-Omni and EchoInk-R1 are reported as strongest, with overall scores of $X \ne Y$7 and $X \ne Y$8, and $X \ne Y$9 and $$\mathrm{Acc}_{\mathrm{avg}} = \frac{1}{6} \sum_{m \in D} \mathrm{Acc}(m),$$0, respectively. They nevertheless trail Gemini 2.5 Pro, particularly in spatial reasoning, temporal reasoning, and external knowledge. Baichuan Omni 1.5 and earlier Unified-IO models show higher $$\mathrm{Acc}_{\mathrm{avg}} = \frac{1}{6} \sum_{m \in D} \mathrm{Acc}(m),$$1, indicating weaker robustness to modality shifts.

5. Task-level behavior and failure modes

The perception family shows particularly strong text–vision performance and pronounced degradation when audio is involved. For Gemini 2.5 Pro, T→V reaches $$\mathrm{Acc}_{\mathrm{avg}} = \frac{1}{6} \sum_{m \in D} \mathrm{Acc}(m),$$2 and V→T reaches $$\mathrm{Acc}_{\mathrm{avg}} = \frac{1}{6} \sum_{m \in D} \mathrm{Acc}(m),$$3, whereas A→T is $$\mathrm{Acc}_{\mathrm{avg}} = \frac{1}{6} \sum_{m \in D} \mathrm{Acc}(m),$$4, A→V is $$\mathrm{Acc}_{\mathrm{avg}} = \frac{1}{6} \sum_{m \in D} \mathrm{Acc}(m),$$5, T→A is $$\mathrm{Acc}_{\mathrm{avg}} = \frac{1}{6} \sum_{m \in D} \mathrm{Acc}(m),$$6, and V→A is $$\mathrm{Acc}_{\mathrm{avg}} = \frac{1}{6} \sum_{m \in D} \mathrm{Acc}(m),$$7 (Wang et al., 16 Oct 2025). This contrast indicates that successful recognition in one modality pair does not imply equivalent grounding across all modality pairs.

Spatial reasoning exposes a different weakness: directional asymmetry and poor handling of audio spatial cues. For Gemini 2.5 Pro, V→T is $$\mathrm{Acc}_{\mathrm{avg}} = \frac{1}{6} \sum_{m \in D} \mathrm{Acc}(m),$$8 whereas T→V is $$\mathrm{Acc}_{\mathrm{avg}} = \frac{1}{6} \sum_{m \in D} \mathrm{Acc}(m),$$9, showing directional imbalance even within relatively strong vision–text settings. Audio-involving directions remain around the low 30s, despite the use of explicit spatialization procedures. This supports the benchmark’s characterization of audio as a bottleneck, especially for spatially structured tasks.

Temporal reasoning, with an average of $$D = \{A \to T, A \to V, T \to A, T \to V, V \to A, V \to T\}$$0 for Gemini 2.5 Pro, shows a mixed profile. The model is described as relatively strong on temporal order, approaching ceiling when text is the candidate modality, but weaker on temporal counting and temporal calculation, particularly when audio is the candidate. This suggests that sequence discrimination and symbolic counting impose different burdens on cross-modal representations.

Linguistic understanding is one of the stronger families. Gemini 2.5 Pro performs well on recognition and translation, with examples including V→T up to $$D = \{A \to T, A \to V, T \to A, T \to V, V \to A, V \to T\}$$1, A→T at $$D = \{A \to T, A \to V, T \to A, T \to V, V \to A, V \to T\}$$2, and T→A at $$D = \{A \to T, A \to V, T \to A, T \to V, V \to A, V \to T\}$$3. External knowledge is stronger still, with T→V at $$D = \{A \to T, A \to V, T \to A, T \to V, V \to A, V \to T\}$$4, V→T at $$D = \{A \to T, A \to V, T \to A, T \to V, V \to A, V \to T\}$$5, and A→T at $$D = \{A \to T, A \to V, T \to A, T \to V, V \to A, V \to T\}$$6. These results indicate that current models can perform strongly when semantic retrieval or label-space alignment is well supported, even though cross-modal invariance remains incomplete.

The benchmark’s error analysis identifies several recurrent failure modes. One is inconsistent grounding across modalities: a model may correctly identify an object in audio→text, such as didgeridoo, but fail in audio→image, implying fragile mapping from auditory semantics to visual evidence. Another is direction reversal and motion confusion in spatial movement tasks, where swapping audio→text to text→audio can produce inversion of directionality. A third is the especially poor performance of audio–vision settings without text, suggesting reliance on textual candidates to normalize noisy perceptual inputs.

A frequent misunderstanding in interpreting multimodal benchmarks is to treat weak audio performance as merely a consequence of harder datasets. XModBench does not eliminate that possibility, but its paired, semantics-controlled design suggests a stronger claim: the same content, when expressed through audio rather than text, can induce a substantial drop in performance even under otherwise matched conditions.

6. Evaluation protocol, availability, and limitations

The practical evaluation protocol is straightforward. Each item provides a context, represented as a pointer to an audio file, image, or text string, and four candidates, represented as files or strings, together with a gold label. Every unique semantic instance appears in six files, one for each modality configuration (Wang et al., 16 Oct 2025).

The recommended procedure is:

1. For each item and each of the six modality directions, pass the context and four candidates to the model using a fixed instruction template asking for one choice among A/B/C/D.
2. Collect the predicted option and compute per-direction accuracy.
3. Aggregate results by reporting $$D = \{A \to T, A \to V, T \to A, T \to V, V \to A, V \to T\}$$7 and $$D = \{A \to T, A \to V, T \to A, T \to V, V \to A, V \to T\}$$8 across the six directions, per-task family averages, and the diagnostic metrics for modality disparity and directional imbalance.

The authors state that scoring scripts and evaluation tools are provided. Reproducibility guidance includes fixing model decoding parameters such as temperature across modalities, using the same prompt template for all directions to avoid instruction bias, and evaluating all six directions for every item rather than subsetting directions. Data and tools are made available at the project page: https://xingruiwang.github.io/projects/XModBench/. The paper does not specify a license and advises consulting the project page for licensing and versioning details.

The benchmark also includes an optional dual-context setting in which audio and visual context are combined with text candidates. In this setting, Gemini 2.5 Pro improves modestly over the stronger unimodal baseline V→T, with A+V→T at $$D = \{A \to T, A \to V, T \to A, T \to V, V \to A, V \to T\}$$9 versus V→T at $$\sigma = \sqrt{ \frac{1}{6} \sum_{m \in D} \big(\mathrm{Acc}(m) - \mathrm{Acc}_{\mathrm{avg}}\big)^2 }.$$0. This suggests that current models do not yet fully exploit multi-cue synergy.

Several limitations are stated explicitly. Some modalities are synthesized, including TTS-generated speech and rendered text, which may introduce distribution shifts relative to in-the-wild data. The paper reports accuracies and standard deviations, but not confidence intervals or statistical tests, leaving uncertainty quantification to future work. The benchmark identifies directional bias toward text as a candidate space, suggesting that training data and objectives may overweight text outputs. The paper also notes that audio remains a bottleneck and that better architectures or training methods for cross-modal fusion are needed. On ethics, the benchmark uses data collected from public datasets and the web, emphasizes filtering for quality and unambiguous content, and advises that source licenses and privacy should be respected when extending the benchmark.

Taken together, these properties define XModBench as a diagnostic benchmark rather than only a leaderboard instrument. Its balanced six-direction design, semantics-controlled triplets, and explicit metrics for disparity and directional imbalance make it suitable for evaluating whether OLLMs approach genuinely modality-invariant reasoning, or whether they continue to rely on asymmetric and modality-specific internal representations.