REAL: Resolving Knowledge Conflicts in Knowledge-Intensive Visual Question Answering via Reasoning-Pivot Alignment

Abstract: Knowledge-intensive Visual Question Answering (KI-VQA) frequently suffers from severe knowledge conflicts caused by the inherent limitations of open-domain retrieval. However, existing paradigms face critical limitations due to the lack of generalizable conflict detection and intra-model constraint mechanisms to handle conflicting evidence. To address these challenges, we propose the REAL (Reasoning-Pivot Alignment) framework centered on the novel concept of the Reasoning-Pivot. Distinct from reasoning steps that prioritize internal self-derivation, a reasoning-pivot serves as an atomic unit (node or edge) in the reasoning chain that emphasizes knowledge linkage, and it typically relies on external evidence to complete the reasoning. Supported by our constructed REAL-VQA dataset, our approach integrates Reasoning-Pivot Aware SFT (RPA-SFT) to train a generalizable discriminator by aligning conflicts with pivot extraction, and employs Reasoning-Pivot Guided Decoding (RPGD), an intra-model decoding strategy that leverages these pivots for targeted conflict mitigation. Extensive experiments across diverse benchmarks demonstrate that REAL significantly enhances discrimination accuracy and achieves state-of-the-art performance, validating the effectiveness of our pivot-driven resolution paradigm.

• The paper introduces REAL, a framework that localizes contradictions to indispensable reasoning pivots rather than relying on unreliable entity or keyword mismatches.
• REAL combines the 4,149-sample REAL-VQA dataset, pivot-aware supervised fine-tuning, and gated orthogonal contrastive decoding to identify and suppress conflicting evidence.
• The best model reaches 44.1% on InfoSeek, 41.4% on E-VQA, and 80.3% multiple-choice accuracy on A-OKVQA, while remaining within 1.3× greedy-decoding latency.

Motivation and problem statement

Retrieval-augmented multimodal LLMs (MLLMs) for knowledge-intensive visual question answering (KI-VQA) depend on external evidence whose quality cannot be guaranteed: open-domain retrieval routinely returns noisy or contradictory passages, producing knowledge conflicts that degrade answer accuracy. The authors argue that existing mitigation paradigms have two structural deficiencies. First, conflict detection is typically defined by entity or keyword mismatch, which is a poor proxy in KI-VQA because multi-hop reasoning chains necessarily introduce intermediate entities and shared property types that differ from the visual anchor without constituting contradictions. Second, prevailing solutions operate externally—reorganizing candidate evidence or applying contrastive prompts—without intra-model constraints, so the same conflict type yields divergent resolution behaviors across instances.

The paper's central conceptual contribution is the Reasoning-Pivot: an atomic node or edge in the reasoning chain (formalized as the set $\mathcal{P} = \{e_1, p_1, e_2, p_2, y\}$ for a two-hop query) that is indispensable for reaching the answer under zero prior knowledge. A valid knowledge conflict exists only when mutually exclusive assertions arise within the information set of a single pivot; mismatches elsewhere are treated as non-conflicting noise. This definition directly addresses the two failure modes of keyword matching: intermediate entities in multi-hop chains and same-property-type keywords at different chain stages.

The REAL framework

REAL comprises three components: a purpose-built dataset, a supervised fine-tuning strategy, and a training-free decoding algorithm.

REAL-VQA dataset. Built on Wikipedia sources aligned with E-VQA and InfoSeek, each sample contains reasoning-pivots and five ground-truth paragraphs. Conflicts are generated by substituting the ground-truth pivot with a counterfactual counterpart from the same Wikidata category, then having GPT-4o rewrite the paragraph anchored in the actual Wikipedia context of the substituted entity. This retrieval-grounded rewriting avoids the "hybrid hallucination" problem of naive entity swapping. Quality control uses a vote-of-confidence filter (ten stochastic GPT-4o scorings, cumulative sum ≥ 80, no single score < 6) followed by expert verification, yielding 4,149 training and 629 test samples. A limitation worth noting: the scale is modest relative to E-VQA (~221k QA pairs), so the discriminator's generalization claims rest heavily on cross-dataset transfer rather than in-domain coverage.

Reasoning-Pivot Aware SFT (RPA-SFT). To prevent shortcut learning from binary labels alone, RPA-SFT structures supervision as staged reasoning: special tokens <RPivot>/</RPivot> wrap pivots in inputs as semantic anchors, and target outputs sequentially perform question-pivot extraction, paragraph-pivot extraction, and only then conflict verification. This aligns the optimization objective with logical comparison within pivot sets rather than surface patterns.

Reasoning-Pivot Guided Decoding (RPGD). RPGD contrasts standard logits against a conflict-dominant pathway induced by patch shuffling of visual embeddings, which destroys object-level topology while preserving feature magnitude—forcing reliance on conflicting text without discarding information as masking would. An adaptive gating mechanism maps discriminator-identified pivot spans to subword tokens $\mathcal{K}$ and scales suppression strength by the sigmoid of conflict-pathway logits, keeping penalties near baseline ($\varepsilon = 0.1$) when conflict evidence is weak. Finally, Gram-Schmidt orthogonalization decomposes the standard logits into components parallel to and independent of the conflict logits, subtracting only the projected component $L_{\mathrm{proj}} = c \cdot L_{\mathrm{conf}}$ scaled by the gate. The authors contend this geometric decomposition avoids the excessive penalties and language degradation of linear subtraction schemes such as VCD and CAD—an empirical claim supported by their ablations but not analyzed theoretically.

Main results

On KI-VQA benchmarks with a fixed retriever (EVA-CLIP-8B, $k=5$), REAL achieves state-of-the-art accuracy:

Model	InfoSeek (All)	E-VQA (All)
ReflectiVA (LLaMA3.1-8B)	40.2	35.5
mKG-RAG (LLaMA3-8B)	40.5	36.3
VLM-PRF (InternVL3-8B)	42.5	39.2
REAL (InternVL3.5-8B)	43.8	39.2
REAL (Qwen3-VL-8B)	44.1	41.4

The best configuration surpasses prior work by +3.8% absolute on E-VQA and +1.6% on InfoSeek, and generalizes to commonsense QA with a +3.6% gain on A-OKVQA (80.3 MC / 68.3 DA with LLaVA-1.5-7B).

Conflict discrimination results are the strongest quantitative claim in the paper. Fine-tuning exclusively on REAL-VQA, RPA-SFT improves average discrimination over Qwen3-VL-8B baselines by 14.68%, reaching MCC/F1 of 98.1/99.1 in-domain, 93.4/95.5 on E-VQA, 87.9/95.4 on ScienceQA, and 52.9/74.8 on the unseen MMKC benchmark. Pivot-level diagnostics confirm the gains stem from genuine localization: RPA-SFT raises reasoning-pivot detection F1 from 61.6 to 79.4 and conflict-pivot F1 from 46.6 to 74.7 on Qwen3-VL-8B, whereas few-shot baselines score near chance on these subtasks. Scaling experiments across Qwen3-VL (2B–32B) show RPA-SFT corrects opposing prediction biases present at different model sizes, though MMKC remains the weakest transfer target (MCC 68.2 even at 32B), indicating that entirely novel conflict taxonomies are still substantially harder than in-domain or structurally similar conflicts.

Ablations attribute the decoding gains to all three RPGD components: removing patch shuffle costs 3.1 points on E-VQA single-hop, removing adaptive gating 1.4, and removing orthogonalization 2.0. Against decoding baselines applied to the same RPA-SFT models, RPGD outperforms greedy, VCD, and CAD—for example, 45.5 vs. 42.4 (greedy) on E-VQA single-hop with Qwen3-VL-8B. Notably, VCD actually degrades performance relative to greedy in this setting, suggesting generic contrastive objectives misfire when the "negative" distribution encodes genuine evidential conflict rather than visual bias. RPGD also maintains latency within 1.3× of greedy decoding while delivering 2.4–3.3% accuracy gains, whereas VCD and CAD add latency without reliable improvement. Supplementary experiments replacing the Wikipedia corpus with GPT-4o as a generative knowledge engine preserve the gains, supporting source-agnosticism of the conflict-resolution mechanism.

Limitations and open questions

The authors concede three dependencies. First, effectiveness is contingent on retrieval quality and pivot-extraction precision; sparse, biased, or noisy evidence can still compromise both detection and resolution, meaning REAL mitigates rather than eliminates retrieval-induced error. Second, dual-stream contrastive decoding incurs computational overhead beyond the reported 1.3× factor in longer generation regimes. Third, performance may vary across domains and languages—the evaluation covers English Wikipedia-centric corpora, and MMKC results suggest degraded transfer to unfamiliar conflict types. Two questions remain open: whether the reasoning-pivot formalization extends to deeper chains where pivot sets grow combinatorially, and whether the Gram-Schmidt projection retains its advantage over learned intervention heads at larger model scales where zero-shot discrimination is already strong.

Conclusion

REAL reframes knowledge conflicts in KI-VQA as inconsistencies localized to indispensable reasoning-pivots rather than global entity or keyword mismatches. Combining a verified synthetic dataset, multi-stage pivot-aware fine-tuning, and gated orthogonal contrastive decoding, it delivers state-of-the-art KI-VQA accuracy with strong cross-benchmark discrimination generalization at modest inference cost. The approach's dependence on retrieval quality and its reduced efficacy on unseen conflict taxonomies delineate the boundary conditions of the proposed paradigm.