X-RAMDocs-ZHEN: Evidence Conflict Diagnostic Benchmark
- X-RAMDocs-ZHEN is a controlled Chinese-English benchmark that isolates evidence conflict in retrieval-augmented generation by assessing candidate extraction, conflict detection, and answer preservation.
- It incorporates 300 samples evenly split between Chinese and English, derived from RAMDocs, to diagnose bilingual disagreement and pinpoint document-level extraction failures.
- The benchmark evaluates systems using metrics such as strict accuracy, conflict-aware success, and unsupported-candidate rate to measure effective evidence handling.
X-RAMDocs-ZHEN is a controlled Chinese-English benchmark for diagnosing evidence conflict in retrieval-augmented generation (RAG), designed to test what a system does when retrieved Chinese and English evidence supports incompatible answer candidates for the same question rather than merely introducing ordinary noise (Kang et al., 11 Jun 2026). It is derived from RAMDocs, but its stated role is narrower: not to measure general multilingual question answering ability, broad hallucination detection, or robustness to natural retrieval, but to isolate whether a system can preserve evidence-supported candidates, recognize disagreement, avoid unsupported candidates, and provide traceable evidence use when evidence conflicts across languages (Kang et al., 11 Jun 2026).
1. Scope, task definition, and diagnostic target
The benchmark is motivated by evidence conflict rather than standard retrieval error. In the formulation used for X-RAMDocs-ZHEN, retrieved context is not simply incomplete, irrelevant, or noisy; it is mutually contradictory across languages. The central question is therefore not whether cross-lingual evidence is helpful, but how a RAG system behaves when Chinese and English evidence disagrees (Kang et al., 11 Jun 2026).
The task is explicitly constrained and evidence-bounded:
“Given a question and a set of Chinese and/or English evidence snippets, the system must produce an answer or a conflict-aware response using only the visible evidence.”
Within that framing, the benchmark tests four behaviors: extraction of candidate answers from each evidence snippet, detection of conflict, avoidance of unsupported candidates, and auditability of the final response. This design makes X-RAMDocs-ZHEN a diagnostic benchmark for multilingual evidence handling rather than a general-purpose multilingual QA corpus (Kang et al., 11 Jun 2026).
A recurrent misconception is to interpret the benchmark as measuring overall multilingual reasoning quality. The paper explicitly rejects that interpretation. X-RAMDocs-ZHEN is structured to examine whether systems preserve supported alternatives instead of collapsing conflict into a single unsupported answer. A plausible implication is that its utility is greatest when studying failure modes in evidence aggregation rather than end-to-end retrieval quality.
2. Derivation from RAMDocs and controlled construction
X-RAMDocs-ZHEN v0.1 is derived from RAMDocs and contains 300 samples across 534 evidence documents, split evenly between Chinese and English (Kang et al., 11 Jun 2026). Its construction begins from RAMDocs records and proceeds through a controlled pipeline: questions, answers, and document fields are normalized; Chinese-English evidence snippets are rendered under controlled conditions; each sample is assigned one of six evidence conditions; and private source identifiers are replaced with public evidence IDs. RAMDocs itself was introduced to simulate ambiguity, misinformation, and noise in retrieved evidence (Wang et al., 17 Apr 2025).
The benchmark uses 54 unique source records drawn from 68 usable RAMDocs records, with 100% provenance coverage and zero automatic severe errors or warnings. At the same time, the paper notes that no completed human audit is available (Kang et al., 11 Jun 2026). That qualification matters because it places the benchmark squarely in the category of controlled automatic diagnostics rather than human-validated multilingual evaluation.
A key design feature is the separation between visible prompt content and hidden evaluation metadata. Standard methods receive only the question, visible evidence text, public evidence IDs, and language tags. They do not receive gold answers, document roles, conflict labels, language roles, expected behavior, or supported-answer metadata. Those hidden fields are reserved for construction and automatic evaluation. This leakage-aware design prevents a model from inferring the answer from metadata rather than from evidence itself (Kang et al., 11 Jun 2026).
The paper also distinguishes a privileged “oracle diagnostic” from deployable methods. The oracle accesses hidden document-level metadata and is therefore explicitly described as non-deployable. This distinction is central to the benchmark’s interpretability: it allows the study to separate limitations of extraction from limitations of grouping and aggregation.
3. Six balanced conditions and the structure of conflict
X-RAMDocs-ZHEN contains 300 samples total, evenly divided across six balanced conditions, with 50 examples per condition (Kang et al., 11 Jun 2026). These conditions are the benchmark’s main structural contribution because they disentangle monolingual support, bilingual agreement, reversed conflict direction, and conflict with added noise.
| Condition | Description |
|---|---|
| C1 | Chinese-only correct |
| C2 | English-only correct |
| C3 | Bilingual consistent |
| C4 | Chinese true, EN false |
| C5 | English true, ZH false |
| C6 | Conflict with optional noise |
Condition C1 provides correct evidence in Chinese only. Condition C2 provides correct evidence in English only. Condition C3 presents bilingual agreement, where both languages support the same candidate. Condition C4 assigns the reference candidate to Chinese evidence and a competing candidate to English evidence. Condition C5 reverses that direction. Condition C6 introduces bilingual conflict with optional noise, and the paper specifies that noise is present in 34 of 50 C6 cases (Kang et al., 11 Jun 2026).
This balanced design makes it possible to diagnose not merely whether a system notices disagreement, but also which language it trusts and whether noise changes that behavior. That is a distinctive feature of X-RAMDocs-ZHEN relative to broader conflict-oriented RAG benchmarks. RAMDocs, for example, jointly studies ambiguity, misinformation, noise, and evidence imbalance, whereas X-RAMDocs-ZHEN isolates Chinese-English contradiction under controlled conditions (Wang et al., 17 Apr 2025).
The benchmark’s conflict logic is procedural rather than axiomatically formalized. The paper does not present a full symbolic theory of contradiction, but it does encode conflict structure in the hidden metadata and condition labels. A plausible implication is that the benchmark privileges observable system behavior under controlled contradictions over abstract formalization of inconsistency.
4. Evidence handling pipeline and evaluation protocol
The benchmark is paired with X-MADAM-RAG, an interpretable pipeline that decomposes evidence handling into document-level extraction, visible-evidence repair, candidate normalization, deterministic grouping, and conflict-aware aggregation (Kang et al., 11 Jun 2026). Operationally, the aggregation rule is simple: a single group yields a direct answer, multiple groups yield conflict enumeration, and no group yields abstention.
This decomposition is important because it renders the failure surface auditable. Candidate extraction is performed per document, repaired only from visible evidence, normalized into comparable forms, grouped deterministically, and then aggregated under explicit conflict awareness. The oracle diagnostic replaces extraction with private supported_answer metadata while keeping the same downstream grouping and aggregation, which isolates extraction as a separable component (Kang et al., 11 Jun 2026).
All LLM experiments use Qwen2.5-7B-Instruct with local inference, deterministic decoding (temperature = 0), bfloat16 weights, SDPA attention, and max 512 new tokens. The benchmark is evaluated on the original 300-sample set and on a 100-sample naturalized stress subset. Ablations are run only on the original benchmark (Kang et al., 11 Jun 2026).
The paper reports five automatic metrics:
- Strict accuracy: normalized lexical match between returned answer and reference.
- Conflict-aware success: requires conflict recognition plus preservation of the reference candidate on conflict samples; reduces to strict correctness on non-conflict samples.
- Conflict F1: compares expected vs. predicted conflict.
- Conflict awareness: measures explicit conflict expression among expected-conflict cases.
- Unsupported-candidate rate: measures whether generated candidates fail to lexically match any evidence-supported candidate in private metadata.
The authors explicitly caution that these are not semantic entailment or human judgment metrics. That caveat is substantive rather than incidental: the benchmark is intended as an automatic lexical diagnostic of controlled evidence conflict, not as a substitute for semantic faithfulness evaluation (Kang et al., 11 Jun 2026).
5. Empirical results, template regularity, and the naturalized stress test
On the original controlled benchmark, the main reported results are as follows (Kang et al., 11 Jun 2026):
| Method | Strict accuracy | Conflict-aware success |
|---|---|---|
| Naive RAG | 0.8700 | 0.8667 |
| Evidence-normalized RAG | 0.8900 | 0.8867 |
| Rule-only extractor | 1.0000 | 1.0000 |
| X-MADAM-RAG | 0.9667 | 0.9767 |
| Oracle extraction | perfect | perfect |
The full original-benchmark table also reports Conflict F1, Conflict awareness, and Unsupported-candidate rate. For Naive RAG, those values are 0.9579, 0.7400, and 0.0800. For Evidence-normalized RAG, they are 0.9766, 0.7533, and 0.1067. For Rule-only extractor, all reported metrics are perfect and the unsupported-candidate rate is 0.0000. For X-MADAM-RAG, they are 0.9804, 1.0000, and 0.0200. Oracle extraction is perfect across all reported metrics (Kang et al., 11 Jun 2026).
Among deployable LLM methods, X-MADAM-RAG is best on the original benchmark. The paper states that it improves strict accuracy over evidence-normalized RAG by 0.0767 and reduces unsupported-candidate rate from 0.1067 to 0.0200. The reported 95% bootstrap interval for the strict-accuracy gap is [0.0367, 0.1167], with an exploratory sign-test p-value of 0.0004 (Kang et al., 11 Jun 2026).
At the same time, the perfect score of the zero-call rule-only extractor reveals that the original benchmark is strongly template-regular. This is a central interpretive result, not a peripheral ablation. It shows that high performance on the controlled set can arise from surface regularity rather than genuine semantic extraction.
To probe that limitation, the paper constructs a deterministic naturalized stress test that removes explicit answer templates while preserving candidate strings and private metadata. The transformed set covers all 300 samples and 534 documents, with zero remaining templates and perfect candidate and metadata preservation, but experiments are reported on a 100-sample approximately balanced subset (Kang et al., 11 Jun 2026).
On that stress subset, performance changes sharply:
- Rule-only extractor falls from 1.0000 to 0.0000.
- Naive reaches 0.6500 strict accuracy.
- Evidence-normalized reaches 0.6800 strict accuracy.
- X-MADAM-RAG falls to 0.3000 strict accuracy.
- Oracle extraction remains 1.0000.
The oracle’s continued perfection while both rule-only extraction and X-MADAM-RAG collapse indicates that the principal bottleneck is document-level extraction, not downstream grouping or aggregation (Kang et al., 11 Jun 2026). This is arguably the paper’s most consequential finding. It shifts attention from conflict-aware aggregation alone to the fragility of candidate extraction under even deterministic naturalization.
6. Interpretive significance, limitations, and appropriate use
The paper is explicit about what X-RAMDocs-ZHEN can and cannot diagnose. It is presented as a controlled diagnostic tool for evidence conflict handling, preservation of candidate answers, conflict awareness, language-directional disagreement, and extraction robustness under template and near-template phrasing (Kang et al., 11 Jun 2026). It does not establish human-validated faithfulness, broad hallucination detection, robustness to natural retrieval, semantic entailment, or general multilingual retrieval performance.
Several limitations are stated directly. The benchmark uses controlled snippets derived from RAMDocs fields, not naturally retrieved multilingual passages. The original benchmark is template-regular. The naturalized stress test remains deterministic controlled prose, not fully natural text. Only one model, Qwen2.5-7B-Instruct, is evaluated. The benchmark covers only Chinese and English. All metrics are automatic lexical diagnostics. No completed human audit is available. The oracle is privileged and non-deployable (Kang et al., 11 Jun 2026).
These constraints also delimit the benchmark’s significance. X-RAMDocs-ZHEN is not evidence that multilingual RAG systems can robustly handle contradiction “in the wild.” Rather, it provides a tightly controlled environment for determining whether a method preserves supported candidates under bilingual disagreement and whether apparent success depends on templates. This suggests a specific research use: controlled diagnosis of failure modes before claims are made about natural multilingual retrieval robustness.
Its broader contribution lies in making Chinese-English evidence conflict a first-class evaluation object. Earlier work such as RAMDocs emphasized joint ambiguity, misinformation, and noise in conflicting-evidence RAG (Wang et al., 17 Apr 2025). X-RAMDocs-ZHEN narrows the lens to bilingual contradiction and makes extraction, grouping, and aggregation separately inspectable. The resulting picture is technically precise: conflict-aware aggregation can help on controlled inputs, but robust document-level extraction remains the central unresolved problem (Kang et al., 11 Jun 2026).
