X-MADAM-RAG: Diagnosing and Handling Chinese-English Evidence Conflict in Retrieval-Augmented Generation

Abstract: Retrieval-augmented generation (RAG) systems may receive evidence that is not merely noisy but mutually contradictory. This issue becomes particularly salient in multilingual settings, where retrieved Chinese and English evidence may support incompatible answer candidates. We study this problem through X-RAMDocs-ZHEN, a controlled Chinese-English benchmark derived from RAMDocs for diagnosing evidence conflict in RAG. The benchmark contains 300 examples across six balanced conditions, including monolingual support, bilingual agreement, reversed conflict directions, and conflict with optional noise. We further examine X-MADAM-RAG, an interpretable pipeline that decomposes evidence handling into per-document candidate extraction, visible-evidence repair, deterministic candidate grouping, and conflict-aware aggregation. On the original controlled benchmark with Qwen2.5-7B-Instruct, X-MADAM-RAG achieves 0.9667 strict accuracy and 0.9767 conflict-aware success, outperforming an evidence-normalized single-call baseline. However, a zero-call rule-only extractor reaches 1.0000 on the same benchmark, revealing strong template regularity. To probe this limitation, we construct a deterministic naturalized stress test that removes explicit answer templates while preserving candidate strings. On its 100-sample subset, rule-only extraction falls to 0.0000, but X-MADAM-RAG also drops to 0.3000 strict accuracy, below both naive and evidence-normalized baselines. A privileged oracle remains perfect, indicating that document-level extraction is the main bottleneck. These findings position X-RAMDocs-ZHEN and X-MADAM-RAG as diagnostic tools for controlled evidence conflict rather than as evidence of general hallucination detection or robustness to natural retrieval.

• The paper introduces X-MADAM-RAG, an interpretable pipeline that decomposes conflict handling into extraction, repair, normalization, grouping, and aggregation, achieving 0.9767 conflict-aware success.
• It presents X-RAMDocs-ZHEN, a leakage-aware benchmark covering six conflict scenarios to rigorously evaluate Chinese-English evidence discrepancies in RAG systems.
• Ablation studies confirm the critical role of candidate repair, with significant accuracy drops when extraction mechanisms fail under naturalized stress tests.

Diagnosing and Handling Evidence Conflict in Multilingual Retrieval-Augmented Generation Systems

Introduction

Retrieval-Augmented Generation (RAG) systems must frequently resolve cases where retrieved evidence is not simply noisy, but fundamentally contradictory—especially in multilingual contexts. "X-MADAM-RAG: Diagnosing and Handling Chinese-English Evidence Conflict in Retrieval-Augmented Generation" (2606.12903) directly investigates this issue by proposing X-RAMDocs-ZHEN, a controlled Chinese-English benchmark, and X-MADAM-RAG, an interpretable conflict-handling pipeline. The work systematically diagnoses RAG behavior under Chinese-English evidence conflict, quantifies the role of per-document extraction, and provides actionable insights for advancing robust, auditable RAG architectures.

The X-RAMDocs-ZHEN Benchmark

Construction and Leakage Control

X-RAMDocs-ZHEN is a leakage-aware diagnostic dataset comprising 300 questions, each associated with Chinese and/or English evidence snippets. The benchmark is constructed to evaluate how systems handle monolingual support, bilingual agreement, reversed conflict directions, and conflict with optional noise. Crucially, the construction process prevents leakage of answer or conflict metadata into model inputs, ensuring that deployment-viable methods cannot exploit construction artifacts, while permitting privileged oracle evaluations to estimate upper bounds.

Figure 1: Construction and leakage-control flow of X-RAMDocs-ZHEN ensures only prompt-relevant fields are available to standard systems; privileged oracle evaluations are used for upper-bound diagnostics.

Benchmark Design and Statistical Composition

The benchmark balances entity, date, and numeric conflicts. Evidence is derived from RAMDocs such that questions, answers, and evidence are normalized and assigned one of six precise conflict scenarios. Notably, the dataset is not directly human-audited and is based on controlled snippets rather than naturally retrieved passages.

A deterministic "naturalized" stress test is introduced, wherein explicit answer templates are removed from evidence snippets while candidate strings and metadata are preserved. This isolates systems’ reliance on surface templates for extraction.

The X-MADAM-RAG Pipeline

The X-MADAM-RAG pipeline decomposes RAG conflict handling into distinct, auditable stages: document-level candidate extraction, visible-evidence repair, candidate normalization, deterministic candidate grouping, and conflict-aware aggregation. Each evidence document is independently processed, with missing or unusable extraction output addressed by a label-blind repair component operating solely on visible evidence. Candidate normalization and deterministic grouping are used to yield either direct answers, explicit conflict enumerations, or abstentions.

Figure 2: X-MADAM-RAG processes each evidence document independently, applies candidate repair, normalization, deterministic grouping, and outputs conflict-aware responses.

Baseline comparisons include naive concatenation (all evidence to a single prompt), evidence-normalized single-call (evidence grouped by language), a zero-call rule-only extractor leveraging surface patterns, and a privileged oracle where extraction is replaced by private supported-answer metadata.

Experimental Results

Controlled Benchmark Performance

On the standardized benchmark, X-MADAM-RAG achieves strict accuracy 0.9667 and conflict-aware success 0.9767 using Qwen2.5-7B-Instruct, outperforming both naive and evidence-normalized baselines. Unsupported-candidate rate is notably reduced to 0.0200. However, a rule-only extractor obtains perfect accuracy—a strong indication that the original benchmark is highly template-regular, minimizing its value as a measure of semantic extraction or generalization.

Component Contribution

Ablation analysis demonstrates the importance of both repair and deterministic grouping within X-MADAM-RAG. The omission of repair leads to the most significant accuracy degradation, confirming that the ability to recover candidates from incomplete or noisy extractions is vital for conflict diagnosis.

Naturalized Stress Test

The naturalized stress test is pivotal in demonstrating extraction bottlenecks. Upon removal of explicit answer templates, the rule-only extractor's strict accuracy collapses from 1.0000 to 0.0000. X-MADAM-RAG strict accuracy also drops sharply to 0.3000, underperforming both naive (0.6500) and evidence-normalized (0.6800) baselines. In contrast, the oracle—replacing only the extraction step—remains perfect, establishing that extraction robustness, not grouping or aggregation, is the principal unresolved issue.

Figure 3: Strict accuracy drops dramatically on naturalization for both rule-only extraction and X-MADAM-RAG; the oracle remains unaffected, indicating extraction is the fundamental bottleneck under varied evidence phrasing.

Implications and Future Directions

This research identifies robust document-level extraction as the central open problem in conflicting multilingual RAG. Conflict-aware aggregation mechanisms are not sufficient for reliable performance when confronted with less regular, more naturalistic evidence phrasing. The presented diagnostic framework is valuable for comparative evaluation and model auditing, but the benchmark’s heavy template regularity constrains its relevance to general-purpose robustness.

Practically, findings motivate the development of extraction modules that are resilient to paraphrase and heterogeneous surface forms, leveraging more sophisticated answer normalization and semantic candidate verification. The approach invites future work in scaling to naturally retrieved multilingual corpora, extending to additional language pairs, refining conflict-awareness metrics, and advancing toward end-to-end systems that jointly optimize extraction and aggregation in a conflict-conscious manner.

Conclusion

X-MADAM-RAG offers an interpretable, pipeline-based diagnostic tool for Chinese-English evidence conflict in RAG settings, with X-RAMDocs-ZHEN as a controlled benchmark. While the pipeline achieves strong results under controlled, template-regular conditions, both empirical and ablation analyses reveal that document-level extraction from naturalized evidence forms remains the chief performance bottleneck. These insights establish a clear research agenda: robust extraction and semantic verification are prerequisites for progress in multilingual RAG under adversarial or ambiguous evidence, with broader impacts expected for auditability and trustworthy QA system development.