X-MADAM-RAG: Conflict-Aware RAG Pipeline

• X-MADAM-RAG is a diagnostic RAG pipeline that decomposes evidence handling into four stages: candidate extraction, visible-evidence repair, deterministic grouping, and conflict-aware aggregation.
• It addresses Chinese-English evidence conflict by preserving multilingual candidate support and explicitly signaling incompatible answer candidates.
• Evaluations on the X-RAMDocs-ZHEN benchmark reveal that while its aggregation logic is robust, document-level extraction remains the critical bottleneck.

Searching arXiv for the cited work and closely related RAG diagnostics/conflict papers to ground the article. X-MADAM-RAG is an interpretable retrieval-augmented generation pipeline for diagnosing and handling Chinese-English evidence conflict. It is studied together with X-RAMDocs-ZHEN, a controlled benchmark derived from RAMDocs, and it decomposes evidence handling into four explicit stages: per-document candidate extraction, visible-evidence repair, deterministic candidate grouping, and conflict-aware aggregation. Its stated aim is not general multilingual QA, but the narrower problem of preserving evidence-supported answer candidates and explicitly signaling conflict when Chinese and English documents support incompatible answers (Kang et al., 11 Jun 2026).

1. Problem formulation and scope

X-MADAM-RAG addresses a failure mode that standard RAG pipelines often leave implicit: retrieved evidence may be not merely noisy but mutually contradictory. In the setting studied here, contradiction is defined at the level of answer candidates rather than full logical inconsistency. A conflict arises when different retrieved documents support different discrete candidate strings—such as competing entities, dates, or numbers—and these candidates cannot all be correct simultaneously.

The problem is particularly salient in multilingual retrieval. A Chinese snippet may support one answer candidate while an English snippet supports another, and a single-call RAG system may silently choose one, average across them, or generate an unsupported answer. X-MADAM-RAG is designed to make that behavior auditable. It treats visible evidence as the sole basis for candidate extraction and aggregation, preserves competing candidates instead of collapsing them prematurely, and reports conflict when multiple normalized candidate groups remain after document-level processing.

The system is explicitly scoped as a diagnostic framework for controlled Chinese-English evidence conflict. The associated paper states that it should not be read as evidence of general hallucination detection or robustness to natural retrieval. That limitation is central rather than peripheral: the benchmark is controlled, the evaluation is lexical, and later stress tests show that document-level extraction rather than aggregation is the main bottleneck (Kang et al., 11 Jun 2026).

2. X-RAMDocs-ZHEN: benchmark design and evidence conditions

X-RAMDocs-ZHEN v0.1 is derived from RAMDocs and is constructed as a balanced diagnostic benchmark rather than an open-domain corpus. The benchmark contains 300 samples and 534 evidence documents split evenly between Chinese and English. It is built from 68 usable RAMDocs records and 54 unique source records. Questions are bilingual, with 175 Chinese questions and 125 English questions. Answer types are distributed across 144 entity cases, 101 date cases, and 55 number cases (Kang et al., 11 Jun 2026).

A key design feature is the separation between public and private fields. Standard methods receive only the question text, visible evidence snippets, public evidence IDs, and language tags. Gold answers, document roles, conflict labels, language roles, expected behavior, and supported-answer metadata are hidden and used only for construction, automatic evaluation, and the non-deployable oracle diagnostic.

The six balanced evidence conditions define the benchmark’s diagnostic structure.

Condition	Label	Brief description
1	C1_ZH_ONLY_CORRECT	Only Chinese evidence supports the reference answer
2	C2_EN_ONLY_CORRECT	Only English evidence supports the reference answer
3	C3_BILINGUAL_CONSISTENT	Chinese and English support the same candidate
4	C4_ZH_TRUE_EN_FALSE	Chinese supports the reference; English supports a competing candidate
5	C5_EN_TRUE_ZH_FALSE	English supports the reference; Chinese supports a competing candidate
6	C6_CONFLICT_WITH_NOISE	Conflict plus optional noise evidence

Each condition contains 50 examples. Conditions C4 and C5 encode reversed conflict directions, while C6 adds distracting evidence in 34 of 50 cases. This construction isolates whether a system can distinguish monolingual support, bilingual agreement, and bilingual disagreement without access to hidden labels.

The benchmark’s original evidence snippets are strongly template-regular. They often contain explicit answer templates such as “the answer is …” or Chinese equivalents including “答案是”. This regularity becomes important in interpreting the results, because it allows simple lexical rules to solve the original controlled benchmark almost perfectly.

3. Pipeline architecture

X-MADAM-RAG decomposes evidence handling into four explicit stages. The decomposition is the defining architectural feature.

Per-document candidate extraction

Each evidence document is processed independently by an extraction agent. The input to that agent is the question, the public evidence ID, the language tag, and the visible evidence text for that document only. The output is one or more candidate answers supported by that document. In the reported experiments, the extraction agent is Qwen2.5-7B-Instruct run with deterministic decoding.

Because extraction is document-local, the system does not permit early cross-document reconciliation. This preserves provenance: each candidate can be traced back to a specific snippet and language. It also means that extraction quality is structurally upstream of all later conflict handling.

Visible-evidence repair

When extraction returns no candidate or an unusable candidate, X-MADAM-RAG invokes a label-blind repair module. Repair may search parsed fields from the extractor output, the extractor’s raw text, and the visible evidence text itself. It is constrained to visible evidence only and cannot access gold answers, document roles, or condition labels.

This stage is not a minor convenience. In ablations on the original benchmark, removing repair reduces strict accuracy from 0.9667 to 0.8033 and conflict-aware success from 0.9767 to 0.6967, while increasing unsupported-candidate rate from 0.0200 to 0.0933. The system’s reported performance on the controlled benchmark therefore depends materially on repair (Kang et al., 11 Jun 2026).

Deterministic candidate grouping

After extraction and optional repair, candidates are normalized lexically and grouped deterministically. Candidates that are identical after normalization are assigned to the same group. The grouping is string-based rather than embedding-based or entailment-based. The stated purpose is auditability and reproducibility: one can inspect exactly which documents support which candidate string.

This design favors transparent candidate preservation over semantic generalization. A plausible implication is that the method is strong when answer strings are explicit and stable, but fragile when equivalent answers are paraphrased rather than repeated lexically. The naturalized stress test is consistent with that interpretation.

Conflict-aware aggregation

Aggregation operates over the grouped candidate sets. If no candidate groups remain, the system abstains. If exactly one group remains, it returns that candidate. If multiple groups remain, the system explicitly reports evidence conflict and enumerates the competing candidates rather than forcing a single resolution.

This aggregation logic makes conflict detection an artifact of grouped candidate multiplicity rather than a latent judgment produced by a single end-to-end call. It is therefore interpretable by construction. The same logic is also used in the oracle diagnostic, where document-level extraction is replaced by hidden supported-answer metadata. The oracle’s perfect performance on both the original and naturalized settings indicates that the grouping and aggregation design is not the limiting factor in the reported experiments (Kang et al., 11 Jun 2026).

4. Evaluation metrics and controlled-benchmark results

The evaluation uses five automatic lexical diagnostics. Strict accuracy measures whether the normalized returned answer string matches the reference answer. Conflict-aware success reduces to strict correctness on non-conflict samples, but on conflict samples it additionally requires the system to express conflict and preserve the reference-supported candidate among its reported candidates. Conflict F1 evaluates binary conflict prediction. Conflict awareness measures how often expected conflict is explicitly expressed. Unsupported-candidate rate measures how often the system generates at least one candidate not lexically supported by any evidence document according to private metadata.

On the original 300-sample benchmark, the reported results are as follows.

Method	Strict accuracy	Conflict-aware success
Naive RAG baseline	0.8700	0.8667
Evidence-normalized RAG baseline	0.8900	0.8867
Rule-only extractor	1.0000	1.0000
X-MADAM-RAG	0.9667	0.9767
Oracle extraction	1.0000	1.0000

The broader diagnostic profile is equally informative. X-MADAM-RAG reaches conflict F1 of 0.9804, conflict awareness of 1.0000, and unsupported-candidate rate of 0.0200. The naive baseline reaches unsupported-candidate rate 0.0800, and the evidence-normalized single-call baseline reaches 0.1067. Relative to the evidence-normalized baseline, X-MADAM-RAG improves strict accuracy by +0.0767, with 95% bootstrap interval $[0.0367, 0.1167]$ and paired sign-test $p = 0.0004$ (Kang et al., 11 Jun 2026).

These numbers establish two separate facts. First, among deployable LLM-based methods on the original controlled benchmark, X-MADAM-RAG is the strongest reported method. Second, the rule-only zero-call extractor reaches 1.0000 on all metrics, revealing that the original benchmark is heavily template-regular. The latter point limits how much can be inferred from the former. High performance on the original benchmark does not by itself demonstrate robust semantic conflict handling; it may instead reflect the ease of extracting templated answer strings.

5. Naturalized stress test and extraction bottleneck

To probe the benchmark’s template dependence, the authors construct a deterministic naturalized stress test. The transformation removes explicit answer templates such as “the answer is [CANDIDATE]” and “答案是 [CANDIDATE]” while preserving candidate strings and all hidden metadata. The transformed corpus is still controlled prose rather than naturally retrieved text, but it removes the surface regularities that the rule-only baseline exploits. Evaluation is reported on a 100-sample subset approximately balanced across conditions (Kang et al., 11 Jun 2026).

The change in system behavior is large.

Method	Strict accuracy	Conflict-aware success
Rule-only extractor	0.0000	0.0000
Naive RAG baseline	0.6500	0.6300
Evidence-normalized RAG baseline	0.6800	0.6600
X-MADAM-RAG	0.3000	0.2600
Oracle extraction	1.0000	1.0000

The rule-only extractor collapses from perfect performance to zero, confirming that its success depended entirely on surface templates. More importantly, X-MADAM-RAG drops from 0.9667 strict accuracy to 0.3000 and from 0.9767 conflict-aware success to 0.2600, falling below both single-call baselines. Its unsupported-candidate rate on the naturalized subset is 0.0500, which remains lower than the naive baseline’s 0.1900 and the evidence-normalized baseline’s 0.3000, but the core answer-selection performance degrades sharply.

The oracle remains perfect. Since the oracle shares the same grouping and aggregation logic as X-MADAM-RAG but replaces document-level extraction with private supported-answer metadata, the paper concludes that document-level extraction is the main bottleneck. This is the central technical result of the stress test. It implies that the architecture’s downstream logic is effective once document-level candidate support is known, but the implemented extractor is not robust to less templated phrasing.

This also fixes the proper interpretation of the system. X-RAMDocs-ZHEN and X-MADAM-RAG are positioned as diagnostic tools for controlled evidence conflict, not as evidence of general hallucination detection or robustness to natural retrieval. The stress test is not a peripheral negative result; it is the paper’s main safeguard against overclaiming (Kang et al., 11 Jun 2026).

6. Position within RAG diagnostics and conflict research

X-MADAM-RAG sits at the intersection of multi-stage RAG orchestration, conflict-aware evidence handling, and diagnostic benchmarking, but its scope is narrower and more controlled than several adjacent lines of work.

MA-RAG treats RAG as a multi-agent reasoning pipeline with Planner, Step Definer, Extractor, and QA Agents for ambiguous and multi-hop tasks, emphasizing decomposition and collaborative chain-of-thought reasoning without fine-tuning (Nguyen et al., 26 May 2025). X-MADAM-RAG shares the general commitment to decomposition, but the decomposition is different in kind: it is not plan-based task solving, but document-local candidate extraction followed by deterministic grouping and conflict-aware aggregation over Chinese-English evidence.

RAG-X argues that medical RAG evaluation must separate retrieval from generation and introduces Context Utilization Efficiency to distinguish grounded success from lucky guesses in clinical QA (Sivakumar et al., 3 Mar 2026). X-MADAM-RAG is analogous in spirit rather than in metric design. Its oracle extraction diagnostic serves a similar diagnostic purpose: it isolates whether the bottleneck lies in extraction or in later-stage evidence handling. In the reported experiments, that bottleneck is extraction.

MetaRAG approaches RAG reliability through metamorphic hallucination detection: it decomposes answers into atomic factoids, generates synonym and antonym mutations, verifies them against retrieved context, and aggregates span-level hallucination scores in a black-box setting (Sok et al., 11 Sep 2025). By contrast, X-MADAM-RAG does not score hallucination spans or entailment relations. Its concern is whether visible Chinese and English evidence supports competing candidate strings and whether the system can preserve and report that conflict explicitly.

A separate poisoning study comparing vanilla RAG, agentic RAG, MADAM-RAG, and Recursive LLMs reports that a MADAM-RAG reimplementation exhibits the highest apparent contradiction detection rate, but that the corresponding judge over-identifies this behavior and that the system cannot resolve contradictions reliably, producing a 41.4% non-answer rate even on clean inputs (Korn, 7 May 2026). That result concerns a different threat model—knowledge base poisoning rather than controlled bilingual evidence conflict—but it reinforces a general lesson consistent with X-MADAM-RAG: explicit disagreement handling and contradiction signaling do not, by themselves, solve the harder problem of reliable resolution.

Taken together, these comparisons suggest a specific place for X-MADAM-RAG in the RAG literature. It is best understood as a controlled diagnostic framework for answer-candidate conflict in multilingual evidence, with an interpretable architecture whose downstream aggregation is strong under perfect extraction, but whose deployable performance is dominated by the quality of document-level candidate extraction. A plausible implication is that future work on this line will need stronger multilingual extraction under paraphrase, richer normalization beyond lexical equivalence, and evaluation on natural retrieval rather than controlled prose if the approach is to move from diagnostic instrument to robust general-purpose system.