MCR-BENCH: Modal Conflict Resolution

• MCR-BENCH is a benchmark evaluating large audio-language models by testing their ability to handle conflicting audio and textual inputs.
• It systematically varies text conditions (neutral, faithful, adversarial, irrelevant) across audio tasks like question answering, emotion recognition, and sound classification.
• Empirical results reveal strong text bias in models, with interventions like supervised finetuning partially mitigating but not eliminating modal prioritization failures.

Searching arXiv for the benchmark to verify the correct paper and avoid conflating similarly named benchmarks. MCR-BENCH, short for Modal Conflict Resolution Benchmark, is a curated evaluation benchmark for Large Audio-LLMs (LALMs) that measures how models behave when audio input and textual description disagree. It was introduced to test modality prioritization / text bias in settings where the correct answer should be determined from what is heard rather than from auxiliary text. The benchmark is built around audio-centric tasks, systematically varying the text while keeping the audio fixed, and asks a central question: when audio and text conflict, which modality does the model actually trust? The benchmark and its empirical study show that current LALMs frequently prioritize text over audio, with substantial performance degradation under contradictory text and persistent overconfidence even when the input modalities are inconsistent (Wang et al., 21 Aug 2025).

1. Definition and motivation

MCR-BENCH was created because existing LALM benchmarks such as AudioBench and AIR-Bench nearly always assume aligned or complementary audio-text pairs. In those settings, captions match audio, questions match content, and transcriptions are accurate. Such evaluations reveal basic capability, but not robustness. MCR-BENCH instead targets cross-modal inconsistency, defined as cases in which audio and text cannot both be accurate because one must be wrong or irrelevant (Wang et al., 21 Aug 2025).

The benchmark’s formal setting is straightforward. Each sample contains an audio input and a textual description. A model is then asked to perform an audio-centric task such as question answering, classification, or emotion recognition on the basis of both inputs. The benchmark systematically varies the text while holding the audio fixed, allowing direct observation of whether the model relies on audio evidence or defers to text.

This design is motivated by realistic deployment conditions. In voice assistants, accessibility tools, multi-speaker conversations, and systems affected by upstream errors, the text supplied to a model may be wrong, misleading, or irrelevant. A false transcript, manipulated metadata, or noisy contextual description can therefore become a test of whether a LALM preserves modality balance or collapses into text-dominant behavior. MCR-BENCH is intended to expose and quantify that failure mode.

2. Benchmark construction and task coverage

MCR-BENCH spans three tasks, each drawn from a distinct source dataset and chosen specifically because the ground-truth label is determined by the audio rather than by the text.

Task	Source	Data and role
Audio Question Answering (AQA)	ClothoAQA	1,991 audio clips; 6 questions and answers each; 35,838 QA pairs total
Speech Emotion Recognition (SER)	MELD	~13,000 utterances from 1,433 dialogues in Friends; one of seven emotions and sentiment
Vocal Sound Classification (VSC)	VocalSound	Non-verbal human vocalizations in various acoustic conditions

In AQA, the model answers questions about an audio clip, such as whether people are speaking or whether music is present. In SER, the model classifies the emotion conveyed by a speech segment. In VSC, the model classifies non-verbal human vocal sounds such as laughs, shouts, and cries. Together, these tasks cover general audio scene comprehension, affective speech understanding, and vocal sound recognition beyond conventional speech.

For each audio sample $a$, the benchmark constructs four text conditions $$t \in \{\text{neu}, \text{fth}, \text{adv}, \text{irr}\}$$. The neutral condition provides no text and serves as audio-only input. The faithful condition supplies text that accurately describes the audio. The adversarial condition supplies text that deliberately contradicts the audio. The irrelevant condition supplies fluent but semantically unrelated text. Conceptually, each evaluation instance can be viewed through audio $a_i$, text condition $t_i$, task label $y_i$, and model prediction $\hat{y}_i(a_i,t_i)$.

The benchmark’s conflict construction is controlled rather than incidental. For AQA, GPT-4o is used in a one-shot prompt to convert each question-answer pair into a factual statement and a non-factual statement. The factual statement becomes the faithful description, and the contradictory non-factual statement becomes the adversarial description. For the irrelevant condition, text is selected from AudioCaps by computing sentence similarity against the true caption and choosing captions with minimal semantic overlap while remaining grammatically plausible. This yields a spectrum of accurate, contradictory, and orthogonal text over the same underlying audio (Wang et al., 21 Aug 2025).

3. Evaluation protocol, prompt, and metrics

MCR-BENCH evaluates models with a single standardized prompt that explicitly warns that the text may be unreliable:

$t_i$3

Under the neutral condition, the description is omitted and the model receives only the audio and the question. This protocol turns the benchmark into a direct probe of whether a LALM can selectively use text rather than treating text as authoritative by default (Wang et al., 21 Aug 2025).

The reported experiments cover seven LALMs: Qwen-Audio-Chat, Qwen2-Audio-Instruct, Gazelle, SALMONN-7B, SALMONN-13B, Audio-Flamingo2, and SeaLLMs-Audio-7B. Although these models differ in architecture and training pipelines, they share the core structure of an audio encoder feeding into or alongside a LLM.

The benchmark uses several metrics. For a given task subset with $N$ samples, Accuracy under text condition $t$ is $\text{Acc}_t = C_t/N$, where $C_t$ is the number of correct predictions. Normalized Accuracy is $$t \in \{\text{neu}, \text{fth}, \text{adv}, \text{irr}\}$$0, measuring the effect of text relative to audio-only performance. Macro Accuracy averages $$t \in \{\text{neu}, \text{fth}, \text{adv}, \text{irr}\}$$1, $$t \in \{\text{neu}, \text{fth}, \text{adv}, \text{irr}\}$$2, and $$t \in \{\text{neu}, \text{fth}, \text{adv}, \text{irr}\}$$3, using neutral as the baseline.

Two benchmark-specific quantities are central. Text Influence Rate (TIR) measures how often text changes the answer relative to the neutral condition. Under faithful text it counts incorrect-to-correct shifts; under adversarial text it counts correct-to-incorrect shifts; under irrelevant text it counts both types of perturbation. High $$t \in \{\text{neu}, \text{fth}, \text{adv}, \text{irr}\}$$4 is therefore a direct indicator that misleading text can overturn correct audio-based predictions. Modal Robustness Score (MRS) summarizes how much performance under adversarial and irrelevant text remains relative to audio-only performance, with $$t \in \{\text{neu}, \text{fth}, \text{adv}, \text{irr}\}$$5 to emphasize adversarial robustness. For confidence analysis, the study uses the maximum token probability in the model’s output sequence as a scalar proxy for certainty.

A common misconception is that instruction-level warnings alone adequately test modality balance. MCR-BENCH was designed precisely because aligned-pair benchmarks do not reveal whether a model can resist contradictory or irrelevant text. Its prompt warns about unreliability, but the evaluation asks whether the model’s behavior actually changes under that warning.

4. Empirical findings: text bias as a modality-prioritization failure

The central empirical result is that, across all seven evaluated LALMs, text dominates audio when the modalities disagree (Wang et al., 21 Aug 2025). In Audio Question Answering, Qwen-Audio-Chat achieves 87.8% accuracy in the neutral condition and 100.0% with faithful text, but only 1.7% with adversarial text; its $$t \in \{\text{neu}, \text{fth}, \text{adv}, \text{irr}\}$$6 is 98.3%. Qwen2-Audio-Instruct follows nearly the same pattern: 87.5% neutral, 100.0% faithful, 1.5% adversarial, again with $$t \in \{\text{neu}, \text{fth}, \text{adv}, \text{irr}\}$$7. In these cases, almost every correct audio-based answer is overturned by wrong text.

The effect is even more severe in Speech Emotion Recognition. Several models collapse to 0.0% adversarial accuracy: Qwen2-Audio-Instruct falls from 41.8% neutral to 0.0% adversarial, SALMONN-13B from 46.9% to 0.0%, and Gazelle from 44.9% to 0.0%. In practical terms, every adversarial text sample yields a wrong prediction for those model-task combinations. In Vocal Sound Classification, the pattern remains strong: Qwen-Audio-Chat drops from 60.1% neutral to 3.0% adversarial, with $$t \in \{\text{neu}, \text{fth}, \text{adv}, \text{irr}\}$$8.

Task-wise vulnerability is not uniform. SER is the most vulnerable, with average adversarial accuracy across models of about 2.3%. AQA is also highly fragile, with average adversarial accuracy around 6.7%. VSC is slightly more robust at about 10.4%, and the SALMONN models retain around 25% adversarial accuracy there. This suggests that tasks involving subtle, high-level semantics such as emotion are especially easy for text to hijack, whereas more concrete classification tasks retain somewhat more anchoring in audio.

The irrelevant condition is more complex. Irrelevant text sometimes helps and sometimes hurts. For example, SeaLLMs-Audio-7B on AQA reaches 112.4% normalized accuracy with irrelevant text, corresponding to 81.8% accuracy compared with 72.8% in the neutral condition. Yet $$t \in \{\text{neu}, \text{fth}, \text{adv}, \text{irr}\}$$9 values are often in the teens or higher, showing that unrelated text still perturbs decisions. The study reports no clear correlation between textual relevance to the audio and the model’s susceptibility. A plausible implication is that the failure is not limited to semantically plausible contradictions; even off-topic language can redirect prediction behavior.

5. Confidence analysis and representation-level evidence

The benchmark’s reliability analysis shows that wrong answers under contradictory text are not accompanied by an appropriate uncertainty signal. Using Qwen2-Audio-Instruct and SeaLLMs-Audio-7B, the study computes confidence as the maximum token probability in the output sequence and finds that confidence under adversarial text is often comparable to or higher than confidence under faithful text. Confidence under irrelevant text decreases slightly, but the reduction is small relative to the performance drop. The result is a calibration failure: the models do not reliably become uncertain when modalities disagree (Wang et al., 21 Aug 2025).

The paper also analyzes hidden states to determine whether the models internally distinguish conflicting inputs even when their outputs remain text-biased. For Qwen2-Audio-Instruct on VSC, the last-layer hidden state of the final token is collected for adversarial and faithful inputs, yielding matrices $a_i$0 and $a_i$1. After centralization, singular value decomposition is applied:

$a_i$2

The singular values decay rapidly. About 6 dimensions explain 95% of the variance for adversarial representations, and 5 dimensions explain 95% for faithful ones. This indicates that the representations occupy compact low-dimensional subspaces within the high-dimensional embedding space.

Separability analysis then asks whether these internal states encode the conflict distinction. With a 3:1 train/test split, simple SVM and Random Forest classifiers are trained to discriminate adversarial from faithful hidden states at different layers. At layer 32, the SVM reaches Acc 95.8%, F1 95.9%, AUC 98.8, while the Random Forest reaches Acc 98.0%, F1 98.0%, AUC 99.8. Even at layer 16, accuracy remains above 93%. The internal representation therefore appears to encode whether the text is adversarial or faithful, yet this recognition does not translate into corrected reasoning or output moderation. This suggests a disconnect between conflict representation and modality weighting during generation.

6. Mitigation, limitations, and research significance

The paper evaluates two mitigation classes: prompting and supervised finetuning. Prompting is tested on Qwen2-Audio-Instruct with three strategies: Zero-Shot Chain-of-Thought, Audio Priority Prompt, and Bias Awareness Prompt. The base adversarial accuracy on the AQA subset is 1.5% with $a_i$3. The Bias Awareness Prompt improves adversarial accuracy to 6.9%, and in some subset experiments to 17.4%, while reducing $a_i$4 to 92.2% or about 79.7% in subset analysis. The other prompts yield smaller improvements. A common misconception is that prompt engineering can solve the benchmark; the reported results show that prompting helps modestly but does not eliminate text bias (Wang et al., 21 Aug 2025).

The stronger intervention is supervised finetuning (SFT) on conflict-rich data. Starting from Qwen2-Audio-Instruct, the study uses 1,000 samples from ClothoAQA and MELD with deliberately mismatched audio-text pairs and trains with LoRA at rank 8 and $a_i$5, targeting all linear layers while freezing ViT components. Training runs for 2 epochs with learning rate $a_i$6, warmup ratio 0.05, per-device batch size 1, gradient accumulation 16, bfloat16 precision, and maximum sequence length 2048.

The gains are substantial but not free of tradeoffs. On ClothoAQA, adversarial accuracy rises from 1.5% to about 42.1%, $a_i$7 increases from 75.5% to about 89.2%, and $a_i$8 drops from 98.3% to about 18.7%; however, faithful accuracy declines from 100.0% to about 90.9%. On MELD, adversarial accuracy rises from 0.0% to about 43.8%, while faithful accuracy drops from 100.0% to about 60.6%. On VocalSound, evaluated as out-of-distribution, adversarial accuracy improves from 11.8% to about 17.7%, faithful accuracy remains close at about 96.2%, and $a_i$9 declines from 86.2% to about 76.9%. The pattern indicates that SFT can materially reduce the power of adversarial text, especially in-domain, but often at the cost of lower performance when text is trustworthy.

The benchmark’s stated limitations are explicit. It currently covers only three tasks—AQA, SER, and VSC—and does not include all audio-language scenarios such as long-form dialogue, music, or complex audio-visual interactions. Its conflicts are synthetic, because adversarial and irrelevant texts are generated or selected algorithmically. The mitigation study is limited to prompting and SFT rather than deeper architectural changes such as modality-aware gating or conflict detectors. It also focuses on English and Western audio content. These constraints do not weaken the benchmark’s diagnostic purpose, but they delimit its current scope.

MCR-BENCH therefore functions as a reliability benchmark for audio-text conflict handling. Its practical use is direct: researchers can obtain audio data and text variants from ClothoAQA, MELD, and VocalSound, run a LALM with the standardized prompt, compute $t_i$0, $t_i$1, $t_i$2, and MRS, and inspect confidence behavior under contradictory inputs. The project is available at https://github.com/WangCheng0116/MCR-BENCH. The benchmark’s broader significance lies in establishing that current LALMs often detect inconsistency internally yet still generate confident, text-dominated outputs. This suggests that future work must address modality balance during training and develop fusion mechanisms that can respond to cross-modal conflict rather than merely encode it.