MMKC-Bench: Multimodal Conflict Benchmark

• MMKC-Bench is a benchmark suite that rigorously evaluates multimodal models for detecting conflicts between internal parametric knowledge and external contextual inputs.
• It operationalizes three conflict types—entity-recognition, entity-knowledge, and visual-semantic—across context-memory and inter-context scenarios.
• The suite uses automated pipelines combined with human verification to generate diverse conflict instances, guiding refined retrieval-augmented generation systems.

MMKC-Bench is a benchmark suite designed to rigorously evaluate the ability of large multimodal models (LMMs) to recognize, detect, and respond to factual knowledge conflicts that arise when multimodal input and retrieval-augmented generation (RAG) frameworks confront parametric memory. MMKC-Bench operationalizes three primary categories of multimodal knowledge conflict—entity-recognition, entity-knowledge, and visual-semantic—across both context-memory (C–M) and inter-context (C–C) discordance scenarios. The benchmark encompasses 1,573 conflict instances and 3,381 images covering 23 conflict types, using automated LLM pipelines and comprehensive human verification for dataset integrity (2505.19509). MMKC-Bench facilitates both behavioral outcome analysis and explicit conflict detection tasks across leading LMM architectures, providing a unified evaluation protocol that exposes critical tendencies, notably the bias toward parametric knowledge.

1. Motivation and Scope

Large multimodal models, especially those operating in RAG settings, often encounter situations wherein their internal parametric knowledge ($M$) contradicts either the contextual evidence ($C$) supplied as input or different retrieved evidence sources among themselves. MMKC-Bench addresses limitations in prior benchmarks, which focused predominantly on intra-memory or purely textual conflicts, by:

• Incorporating multimodal (vision+language) context-memory and inter-context conflicts.
• Targeting fine-grained factual knowledge, including entities, attributes, and visual semantics.
• Structuring both behavioral outcome evaluation and direct conflict detection challenges.
• Utilizing an automated LLM-driven adversarial editing pipeline with post-hoc human verification to assure data fidelity.

This design enables comprehensive analysis of LMMs’ tendencies in multimodal RAG workflows, where realistic contradiction scenarios are under-represented by existing resources such as ConflictBank and WikiConflict.

2. Data Collection and Corpus Structure

MMKC-Bench is constructed via a programmatic pipeline that systematically identifies entities and concepts, harvests images and descriptive text, and induces counterfactual conflicts:

• Entity/Concept Enumeration: LLMs produce high-frequency exemplars for entity-recognition (ER) and entity-knowledge (EK), with visual-semantic (VS) types drawn from gestures, actions, emotions, and symbols.
• Image and Description Harvesting: For each entity, 3–5 images from Google Images are paired with LLM-generated summaries from Wikipedia text, ensuring factual coverage.
• Conflict Generation: Conflicting instances $k_c^1$, $k_c^2$ are synthesized by substituting entity names, factual tail entities, or semantic labels within (image, description) pairs.
• Question/Answer Creation: LLMs generate both multiple-choice (one parametric, two conflicting, one distractor) and open-ended VQA-style questions.
• Human Verification: All content is subject to manual multi-pass validation for relevance, factual accuracy, and clarity.

Corpus Statistics

Conflict Type	Instances	Images
Visual Entity Conflict	757	2,271
Entity Knowledge Conflict	669	669
Visual Semantic Conflict	147	441
Total	1,573	3,381

The dataset comprises 23 knowledge subtypes distributed across these conflict categories.

3. Conflict Taxonomy and Formalization

MMKC-Bench delineates conflict at three formal levels:

• Intra-memory Conflict: $$E_\mathrm{intra} = \{ (m_j, m_k) \in M \times M : m_j \neq m_k \}$$
• Context-memory Conflict: $$E_\mathrm{ctx} = \{ (c_i, m_j) \in C \times M : c_i \neq m_j \}$$
• Inter-context Conflict: $$E_\mathrm{inter} = \{ (c_i^1, c_i^2) \in C^1 \times C^2 : c_i^1 \neq c_i^2 \}$$

The benchmark places primary emphasis on context-memory and inter-context scenarios, as intra-memory discrepancies are difficult to meaningfully isolate in the multimodal setting. Context types encompass:

• Visual Entity Conflict: Contradictory assignment of entity labels to images.
• Entity Knowledge Conflict: Contradictions about attributes or relations (e.g., “birth year”).
• Visual Semantic Conflict: Conflicting interpretation of gestures, actions, or symbols.

4. Tasks and Evaluation Methodology

Two principal task families are supported:

A. Model Behavior Analysis

Given context-memory or inter-context scenarios, models answer either:

• Multiple-choice: Four options—original (non-conflict parametric), two directly conflicting, one unrelated distractor.
• Open-ended: Free-form response in line with VQA frameworks.

Outcomes per trial:

• Original Answer (OAR): Agrees with underlying parametric knowledge.
• Counter Answer (CAR): Agrees with (a) conflicting context(s).
• Irrelevant Answer (IAR): Unrelated to conflict framing.

Let $N$ denote total instances. The primary metrics:

• $OAR = \#\text{original}/N$
• $CAR = \#\text{counter}/N$
• $C$0, where $C$1.

B. Conflict Detection

• Coarse-grained: Given full evidence, the model outputs “yes”/“no” to the presence of conflict.
• Fine-grained: Binary decision given evidence fragments.

Metrics (standard binary classification):

$C$2

5. Experimental Results and Model Behavior

Evaluations include three series of LMMs—Qwen2.5-VL (3B--72B), InternVL3 (8B--78B), and GPT-4o mini. Representative results for model behavior analysis (multiple choice, averaged over ER, EK, VS) are:

Model	OAR (Ctx-Mem)	CAR (Ctx-Mem)	OAR (Int-Ctx)	CAR (Int-Ctx)	Coarse Acc.
Qwen2.5-VL-7B	0.67	0.30	0.70	0.28	0.79
InternVL3-8B	0.49	0.41	0.46	0.50	0.75
GPT-4o mini	0.66	0.27	0.62	0.34	0.76

Major findings include:

• Parametric Dominance: OAR consistently exceeds CAR, indicating LMMs often prefer their internal memory over retrieved evidence.
• Conflict-Type Sensitivity: OAR is lowest for knowledge-level conflicts (EK), suggesting models override internal knowledge more readily when faced with attribute/value factual discordance, as opposed to entity or semantic recognition.
• Inter-context Effects: Introducing two mutually contradictory contexts modestly increases CAR (ΔCAR up to +0.21), but does not reverse the bias toward parametric answers.
• Model Scaling: OAR increases monotonically with model size, indicating larger models have stronger—but less adaptable—internal knowledge stores.
• Conflict Detection: Models reliably flag the presence of conflicts in coarse-grained settings ($C$3–$C$4), with lower but still substantive performance (around $C$5–$C$6) in fine-grained detection.

6. Analysis and Implications

MMKC-Bench reveals persistent tendencies and open challenges for LMMs in the context of RAG and external evidence integration:

• Scenario Bias: Unlike pure LLMs, LMMs trained in multimodal environments show a strong propensity to defer to parametric memory, even when presented with contradictory and explicit visual or textual context.
• Resilience to Recognition Errors: Recognition conflicts, such as entity misidentification, are particularly resistant to correction by context.
• Need for Specialized Training: The data suggest that scaling perceptual pre-training alone is insufficient to enhance external evidence integration. Balanced curriculum incorporating multi-source and contradiction-heavy tasks is required.
• RAG System Design: Implementation of explicit conflict-resolution modules and uncertainty quantification is identified as a priority to mitigate the fallback-to-memory pattern.

A plausible implication is that LMMs may pose a reliability challenge in applied RAG scenarios that surface real-world knowledge conflicts, especially as model scale increases.

7. Open Research Directions

MMKC-Bench is explicitly constructed for extensibility and cites several priority frontiers:

• Collecting and curating naturally occurring multimodal conflicts beyond the scope of adversarial/counterfactual edits.
• Extending the benchmark to video and audio modalities to probe temporal and cross-modal contradictions.
• Designing systematic evaluations for conflict resolution (beyond detection), including downstream trust and decision impact.
• Mechanistically, MMKC-Bench motivates revisiting model architectures and training pipelines to better normalize, arbitrate, and potentially recalibrate the locus of truth between parametric memory and compositional, conflict-laden retrieval.

The resource is positioned as a core asset for the emergent research community seeking robust, aware, and resolvable multimodal retrieval-augmented generation systems (2505.19509).