---
title: 'MCR-BENCH: Modal Conflict Resolution'
url: https://www.emergentmind.com/topics/mcr-bench
type: topic
---

# MCR-BENCH: Modal Conflict Resolution

Searching arXiv for the benchmark to verify the correct paper and avoid conflating similarly named benchmarks.
MCR-BENCH, short for **Modal Conflict Resolution Benchmark**, is a curated evaluation benchmark for **Large Audio-Language Models (LALMs)** that measures how models behave when **audio input** and **textual description** disagree. It was introduced to test **modality prioritization / text bias** in settings where the correct answer should be determined from what is heard rather than from auxiliary text. The benchmark is built around **audio-centric tasks**, systematically varying the text while keeping the audio fixed, and asks a central question: when audio and text conflict, which modality does the model actually trust? The benchmark and its empirical study show that current LALMs frequently prioritize text over audio, with substantial performance degradation under contradictory text and persistent overconfidence even when the input modalities are inconsistent [2508.15407].

## 1. Definition and motivation

MCR-BENCH was created because existing LALM benchmarks such as **AudioBench** and **AIR-Bench** nearly always assume **aligned** or **complementary** audio-text pairs. In those settings, captions match audio, questions match content, and transcriptions are accurate. Such evaluations reveal basic capability, but not **robustness**. MCR-BENCH instead targets **cross-modal inconsistency**, defined as cases in which audio and text cannot both be accurate because one must be wrong or irrelevant [2508.15407].

The benchmark’s formal setting is straightforward. Each sample contains an **audio input** and a **textual description**. A model is then asked to perform an **audio-centric task** such as question answering, classification, or emotion recognition on the basis of both inputs. The benchmark systematically varies the text while holding the audio fixed, allowing direct observation of whether the model relies on audio evidence or defers to text.

This design is motivated by realistic deployment conditions. In voice assistants, accessibility tools, multi-speaker conversations, and systems affected by upstream errors, the text supplied to a model may be **wrong**, **misleading**, or **irrelevant**. A false transcript, manipulated metadata, or noisy contextual description can therefore become a test of whether a LALM preserves modality balance or collapses into text-dominant behavior. MCR-BENCH is intended to expose and quantify that failure mode.

## 2. Benchmark construction and task coverage

MCR-BENCH spans three tasks, each drawn from a distinct source dataset and chosen specifically because the ground-truth label is determined by the audio rather than by the text.

| Task | Source | Data and role |
|---|---|---|
| Audio Question Answering (AQA) | ClothoAQA | 1,991 audio clips; 6 questions and answers each; 35,838 QA pairs total |
| Speech Emotion Recognition (SER) | MELD | ~13,000 utterances from 1,433 dialogues in *Friends*; one of seven emotions and sentiment |
| Vocal Sound Classification (VSC) | VocalSound | Non-verbal human vocalizations in various acoustic conditions |

In **AQA**, the model answers questions about an audio clip, such as whether people are speaking or whether music is present. In **SER**, the model classifies the emotion conveyed by a speech segment. In **VSC**, the model classifies non-verbal human vocal sounds such as laughs, shouts, and cries. Together, these tasks cover general audio scene comprehension, affective speech understanding, and vocal sound recognition beyond conventional speech.

For each audio sample \(a\), the benchmark constructs four text conditions \(t \in \{\text{neu}, \text{fth}, \text{adv}, \text{irr}\}\). The **neutral** condition provides no text and serves as audio-only input. The **faithful** condition supplies text that accurately describes the audio. The **adversarial** condition supplies text that deliberately contradicts the audio. The **irrelevant** condition supplies fluent but semantically unrelated text. Conceptually, each evaluation instance can be viewed through audio \(a_i\), text condition \(t_i\), task label \(y_i\), and model prediction \(\hat{y}_i(a_i,t_i)\).

The benchmark’s conflict construction is controlled rather than incidental. For AQA, **GPT-4o** is used in a one-shot prompt to convert each question-answer pair into a **factual statement** and a **non-factual statement**. The factual statement becomes the faithful description, and the contradictory non-factual statement becomes the adversarial description. For the irrelevant condition, text is selected from **AudioCaps** by computing sentence similarity against the true caption and choosing captions with **minimal semantic overlap** while remaining grammatically plausible. This yields a spectrum of accurate, contradictory, and orthogonal text over the same underlying audio [2508.15407].

## 3. Evaluation protocol, prompt, and metrics

MCR-BENCH evaluates models with a single standardized prompt that explicitly warns that the text may be unreliable:

```text
The following text provides descriptions derived from audio content. This information may be accurate, inaccurate, or irrelevant. Answer the question with discretion regarding how you use these details.
Description: <DESCRIPTION>
Question: <QUESTION>
```

Under the **neutral** condition, the description is omitted and the model receives only the audio and the question. This protocol turns the benchmark into a direct probe of whether a LALM can selectively use text rather than treating text as authoritative by default [2508.15407].

The reported experiments cover seven LALMs: **Qwen-Audio-Chat**, **Qwen2-Audio-Instruct**, **Gazelle**, **SALMONN-7B**, **SALMONN-13B**, **Audio-Flamingo2**, and **SeaLLMs-Audio-7B**. Although these models differ in architecture and training pipelines, they share the core structure of an audio encoder feeding into or alongside a large language model.

The benchmark uses several metrics. For a given task subset with \(N\) samples, **Accuracy** under text condition \(t\) is \(\text{Acc}_t = C_t/N\), where \(C_t\) is the number of correct predictions. **Normalized Accuracy** is \(\text{Norm}_t = C_t/C_{\text{neu}}\), measuring the effect of text relative to audio-only performance. **Macro Accuracy** averages \(\text{Acc}_{\text{fth}}\), \(\text{Acc}_{\text{adv}}\), and \(\text{Acc}_{\text{irr}}\), using neutral as the baseline.

Two benchmark-specific quantities are central. **Text Influence Rate (TIR)** measures how often text changes the answer relative to the neutral condition. Under faithful text it counts incorrect-to-correct shifts; under adversarial text it counts correct-to-incorrect shifts; under irrelevant text it counts both types of perturbation. High \(\text{TIR}_{\text{adv}}\) is therefore a direct indicator that misleading text can overturn correct audio-based predictions. **Modal Robustness Score (MRS)** summarizes how much performance under adversarial and irrelevant text remains relative to audio-only performance, with \(\alpha = 0.8\) to emphasize adversarial robustness. For confidence analysis, the study uses the **maximum token probability** in the model’s output sequence as a scalar proxy for certainty.

A common misconception is that instruction-level warnings alone adequately test modality balance. MCR-BENCH was designed precisely because aligned-pair benchmarks do not reveal whether a model can resist contradictory or irrelevant text. Its prompt warns about unreliability, but the evaluation asks whether the model’s behavior actually changes under that warning.

## 4. Empirical findings: text bias as a modality-prioritization failure

The central empirical result is that, across all seven evaluated LALMs, **text dominates audio when the modalities disagree** [2508.15407]. In **Audio Question Answering**, **Qwen-Audio-Chat** achieves **87.8%** accuracy in the neutral condition and **100.0%** with faithful text, but only **1.7%** with adversarial text; its \(\text{TIR}_{\text{adv}}\) is **98.3%**. **Qwen2-Audio-Instruct** follows nearly the same pattern: **87.5%** neutral, **100.0%** faithful, **1.5%** adversarial, again with \(\text{TIR}_{\text{adv}} = 98.3\%\). In these cases, almost every correct audio-based answer is overturned by wrong text.

The effect is even more severe in **Speech Emotion Recognition**. Several models collapse to **0.0%** adversarial accuracy: **Qwen2-Audio-Instruct** falls from **41.8%** neutral to **0.0%** adversarial, **SALMONN-13B** from **46.9%** to **0.0%**, and **Gazelle** from **44.9%** to **0.0%**. In practical terms, every adversarial text sample yields a wrong prediction for those model-task combinations. In **Vocal Sound Classification**, the pattern remains strong: **Qwen-Audio-Chat** drops from **60.1%** neutral to **3.0%** adversarial, with \(\text{TIR}_{\text{adv}} \approx 96.7\%\).

Task-wise vulnerability is not uniform. **SER** is the most vulnerable, with average adversarial accuracy across models of about **2.3%**. **AQA** is also highly fragile, with average adversarial accuracy around **6.7%**. **VSC** is slightly more robust at about **10.4%**, and the **SALMONN** models retain around **25%** adversarial accuracy there. This suggests that tasks involving subtle, high-level semantics such as emotion are especially easy for text to hijack, whereas more concrete classification tasks retain somewhat more anchoring in audio.

The **irrelevant** condition is more complex. Irrelevant text sometimes helps and sometimes hurts. For example, **SeaLLMs-Audio-7B** on AQA reaches **112.4% normalized accuracy** with irrelevant text, corresponding to **81.8%** accuracy compared with **72.8%** in the neutral condition. Yet \(\text{TIR}_{\text{irr}}\) values are often in the teens or higher, showing that unrelated text still perturbs decisions. The study reports **no clear correlation between textual relevance to the audio and the model’s susceptibility**. A plausible implication is that the failure is not limited to semantically plausible contradictions; even off-topic language can redirect prediction behavior.

## 5. Confidence analysis and representation-level evidence

The benchmark’s reliability analysis shows that wrong answers under contradictory text are not accompanied by an appropriate uncertainty signal. Using **Qwen2-Audio-Instruct** and **SeaLLMs-Audio-7B**, the study computes confidence as the **maximum token probability** in the output sequence and finds that confidence under **adversarial text** is often **comparable to or higher than** confidence under **faithful text**. Confidence under **irrelevant text** decreases slightly, but the reduction is small relative to the performance drop. The result is a calibration failure: the models do not reliably become uncertain when modalities disagree [2508.15407].

The paper also analyzes hidden states to determine whether the models internally distinguish conflicting inputs even when their outputs remain text-biased. For **Qwen2-Audio-Instruct** on **VSC**, the last-layer hidden state of the final token is collected for adversarial and faithful inputs, yielding matrices \(A \in \mathbb{R}^{N \times d}\) and \(F \in \mathbb{R}^{N \times d}\). After centralization, singular value decomposition is applied:
$$
A = U_A \Sigma_A V_A^\top, \quad F = U_F \Sigma_F V_F^\top
$$
The singular values decay rapidly. About **6 dimensions** explain **95%** of the variance for adversarial representations, and **5 dimensions** explain **95%** for faithful ones. This indicates that the representations occupy compact low-dimensional subspaces within the high-dimensional embedding space.

Separability analysis then asks whether these internal states encode the conflict distinction. With a 3:1 train/test split, simple **SVM** and **Random Forest** classifiers are trained to discriminate adversarial from faithful hidden states at different layers. At **layer 32**, the **SVM** reaches **Acc 95.8%, F1 95.9%, AUC 98.8**, while the **Random Forest** reaches **Acc 98.0%, F1 98.0%, AUC 99.8**. Even at **layer 16**, accuracy remains above **93%**. The internal representation therefore appears to encode whether the text is adversarial or faithful, yet this recognition does not translate into corrected reasoning or output moderation. This suggests a disconnect between conflict representation and modality weighting during generation.

## 6. Mitigation, limitations, and research significance

The paper evaluates two mitigation classes: **prompting** and **supervised finetuning**. Prompting is tested on **Qwen2-Audio-Instruct** with three strategies: **Zero-Shot Chain-of-Thought**, **Audio Priority Prompt**, and **Bias Awareness Prompt**. The base adversarial accuracy on the AQA subset is **1.5%** with \(\text{TIR}_{\text{adv}} \approx 98.3\%\). The **Bias Awareness Prompt** improves adversarial accuracy to **6.9%**, and in some subset experiments to **17.4%**, while reducing \(\text{TIR}_{\text{adv}}\) to **92.2%** or about **79.7%** in subset analysis. The other prompts yield smaller improvements. A common misconception is that prompt engineering can solve the benchmark; the reported results show that prompting helps modestly but does not eliminate text bias [2508.15407].

The stronger intervention is **supervised finetuning (SFT)** on conflict-rich data. Starting from **Qwen2-Audio-Instruct**, the study uses **1,000 samples** from **ClothoAQA** and **MELD** with deliberately mismatched audio-text pairs and trains with **LoRA** at rank 8 and \(\alpha = 32\), targeting all linear layers while freezing ViT components. Training runs for **2 epochs** with learning rate \(1 \times 10^{-4}\), warmup ratio 0.05, per-device batch size 1, gradient accumulation 16, **bfloat16** precision, and maximum sequence length 2048.

The gains are substantial but not free of tradeoffs. On **ClothoAQA**, adversarial accuracy rises from **1.5%** to about **42.1%**, \(\text{Acc}_{\text{irr}}\) increases from **75.5%** to about **89.2%**, and \(\text{TIR}_{\text{adv}}\) drops from **98.3%** to about **18.7%**; however, faithful accuracy declines from **100.0%** to about **90.9%**. On **MELD**, adversarial accuracy rises from **0.0%** to about **43.8%**, while faithful accuracy drops from **100.0%** to about **60.6%**. On **VocalSound**, evaluated as out-of-distribution, adversarial accuracy improves from **11.8%** to about **17.7%**, faithful accuracy remains close at about **96.2%**, and \(\text{TIR}_{\text{adv}}\) declines from **86.2%** to about **76.9%**. The pattern indicates that SFT can materially reduce the power of adversarial text, especially in-domain, but often at the cost of lower performance when text is trustworthy.

The benchmark’s stated limitations are explicit. It currently covers only **three tasks**—AQA, SER, and VSC—and does not include all audio-language scenarios such as long-form dialogue, music, or complex audio-visual interactions. Its conflicts are **synthetic**, because adversarial and irrelevant texts are generated or selected algorithmically. The mitigation study is limited to prompting and SFT rather than deeper architectural changes such as **modality-aware gating** or **conflict detectors**. It also focuses on English and Western audio content. These constraints do not weaken the benchmark’s diagnostic purpose, but they delimit its current scope.

MCR-BENCH therefore functions as a **reliability benchmark** for audio-text conflict handling. Its practical use is direct: researchers can obtain audio data and text variants from **ClothoAQA**, **MELD**, and **VocalSound**, run a LALM with the standardized prompt, compute \(\text{Acc}_t\), \(\text{Norm}_t\), \(\text{TIR}_t\), and **MRS**, and inspect confidence behavior under contradictory inputs. The project is available at **https://github.com/WangCheng0116/MCR-BENCH**. The benchmark’s broader significance lies in establishing that current LALMs often *detect* inconsistency internally yet still generate confident, text-dominated outputs. This suggests that future work must address modality balance during training and develop fusion mechanisms that can respond to cross-modal conflict rather than merely encode it.

Source: https://www.emergentmind.com/topics/mcr-bench