---
title: 'X-RAMDocs-ZHEN: Evidence Conflict Diagnostic Benchmark'
url: https://www.emergentmind.com/topics/x-ramdocs-zhen
type: topic
---

# X-RAMDocs-ZHEN: Evidence Conflict Diagnostic Benchmark

X-RAMDocs-ZHEN is a controlled Chinese-English benchmark for diagnosing evidence conflict in retrieval-augmented generation (RAG), designed to test what a system does when retrieved Chinese and English evidence supports incompatible answer candidates for the same question rather than merely introducing ordinary noise [2606.12903]. It is derived from RAMDocs, but its stated role is narrower: not to measure general multilingual question answering ability, broad hallucination detection, or robustness to natural retrieval, but to isolate whether a system can preserve evidence-supported candidates, recognize disagreement, avoid unsupported candidates, and provide traceable evidence use when evidence conflicts across languages [2606.12903].

## 1. Scope, task definition, and diagnostic target

The benchmark is motivated by **evidence conflict** rather than standard retrieval error. In the formulation used for X-RAMDocs-ZHEN, retrieved context is not simply incomplete, irrelevant, or noisy; it is **mutually contradictory across languages**. The central question is therefore not whether cross-lingual evidence is helpful, but how a RAG system behaves when Chinese and English evidence disagrees [2606.12903].

The task is explicitly constrained and evidence-bounded:

> “Given a question and a set of Chinese and/or English evidence snippets, the system must produce an answer or a conflict-aware response using only the visible evidence.”

Within that framing, the benchmark tests four behaviors: extraction of candidate answers from each evidence snippet, detection of conflict, avoidance of unsupported candidates, and auditability of the final response. This design makes X-RAMDocs-ZHEN a diagnostic benchmark for multilingual evidence handling rather than a general-purpose multilingual QA corpus [2606.12903].

A recurrent misconception is to interpret the benchmark as measuring overall multilingual reasoning quality. The paper explicitly rejects that interpretation. X-RAMDocs-ZHEN is structured to examine whether systems preserve supported alternatives instead of collapsing conflict into a single unsupported answer. A plausible implication is that its utility is greatest when studying failure modes in evidence aggregation rather than end-to-end retrieval quality.

## 2. Derivation from RAMDocs and controlled construction

X-RAMDocs-ZHEN v0.1 is **derived from RAMDocs** and contains **300 samples across 534 evidence documents, split evenly between Chinese and English** [2606.12903]. Its construction begins from RAMDocs records and proceeds through a controlled pipeline: questions, answers, and document fields are normalized; Chinese-English evidence snippets are rendered under controlled conditions; each sample is assigned one of six evidence conditions; and private source identifiers are replaced with public evidence IDs. RAMDocs itself was introduced to simulate ambiguity, misinformation, and noise in retrieved evidence [2504.13079].

The benchmark uses **54 unique source records** drawn from **68 usable RAMDocs records**, with **100% provenance coverage** and **zero automatic severe errors or warnings**. At the same time, the paper notes that **no completed human audit is available** [2606.12903]. That qualification matters because it places the benchmark squarely in the category of controlled automatic diagnostics rather than human-validated multilingual evaluation.

A key design feature is the separation between visible prompt content and hidden evaluation metadata. Standard methods receive only the **question**, **visible evidence text**, **public evidence IDs**, and **language tags**. They do **not** receive **gold answers**, **document roles**, **conflict labels**, **language roles**, **expected behavior**, or **supported-answer metadata**. Those hidden fields are reserved for construction and automatic evaluation. This leakage-aware design prevents a model from inferring the answer from metadata rather than from evidence itself [2606.12903].

The paper also distinguishes a privileged “oracle diagnostic” from deployable methods. The oracle accesses hidden document-level metadata and is therefore explicitly described as non-deployable. This distinction is central to the benchmark’s interpretability: it allows the study to separate limitations of extraction from limitations of grouping and aggregation.

## 3. Six balanced conditions and the structure of conflict

X-RAMDocs-ZHEN contains **300 samples total**, evenly divided across **six balanced conditions**, with **50 examples per condition** [2606.12903]. These conditions are the benchmark’s main structural contribution because they disentangle monolingual support, bilingual agreement, reversed conflict direction, and conflict with added noise.

| Condition | Description |
|---|---|
| C1 | Chinese-only correct |
| C2 | English-only correct |
| C3 | Bilingual consistent |
| C4 | Chinese true, EN false |
| C5 | English true, ZH false |
| C6 | Conflict with optional noise |

Condition C1 provides correct evidence in Chinese only. Condition C2 provides correct evidence in English only. Condition C3 presents bilingual agreement, where both languages support the same candidate. Condition C4 assigns the reference candidate to Chinese evidence and a competing candidate to English evidence. Condition C5 reverses that direction. Condition C6 introduces bilingual conflict with optional noise, and the paper specifies that noise is present in **34 of 50** C6 cases [2606.12903].

This balanced design makes it possible to diagnose not merely whether a system notices disagreement, but also **which language it trusts** and whether noise changes that behavior. That is a distinctive feature of X-RAMDocs-ZHEN relative to broader conflict-oriented RAG benchmarks. RAMDocs, for example, jointly studies ambiguity, misinformation, noise, and evidence imbalance, whereas X-RAMDocs-ZHEN isolates Chinese-English contradiction under controlled conditions [2504.13079].

The benchmark’s conflict logic is procedural rather than axiomatically formalized. The paper does not present a full symbolic theory of contradiction, but it does encode conflict structure in the hidden metadata and condition labels. A plausible implication is that the benchmark privileges observable system behavior under controlled contradictions over abstract formalization of inconsistency.

## 4. Evidence handling pipeline and evaluation protocol

The benchmark is paired with **X-MADAM-RAG**, an interpretable pipeline that decomposes evidence handling into **document-level extraction**, **visible-evidence repair**, **candidate normalization**, **deterministic grouping**, and **conflict-aware aggregation** [2606.12903]. Operationally, the aggregation rule is simple: **a single group yields a direct answer, multiple groups yield conflict enumeration, and no group yields abstention**.

This decomposition is important because it renders the failure surface auditable. Candidate extraction is performed per document, repaired only from visible evidence, normalized into comparable forms, grouped deterministically, and then aggregated under explicit conflict awareness. The oracle diagnostic replaces extraction with private `supported_answer` metadata while keeping the same downstream grouping and aggregation, which isolates extraction as a separable component [2606.12903].

All LLM experiments use **Qwen2.5-7B-Instruct** with **local inference**, **deterministic decoding (`temperature = 0`)**, **bfloat16 weights**, **SDPA attention**, and **max 512 new tokens**. The benchmark is evaluated on the **original 300-sample set** and on a **100-sample naturalized stress subset**. Ablations are run only on the original benchmark [2606.12903].

The paper reports five automatic metrics:

- **Strict accuracy**: normalized lexical match between returned answer and reference.
- **Conflict-aware success**: requires conflict recognition plus preservation of the reference candidate on conflict samples; reduces to strict correctness on non-conflict samples.
- **Conflict F1**: compares expected vs. predicted conflict.
- **Conflict awareness**: measures explicit conflict expression among expected-conflict cases.
- **Unsupported-candidate rate**: measures whether generated candidates fail to lexically match any evidence-supported candidate in private metadata.

The authors explicitly caution that these are **not semantic entailment or human judgment metrics**. That caveat is substantive rather than incidental: the benchmark is intended as an automatic lexical diagnostic of controlled evidence conflict, not as a substitute for semantic faithfulness evaluation [2606.12903].

## 5. Empirical results, template regularity, and the naturalized stress test

On the original controlled benchmark, the main reported results are as follows [2606.12903]:

| Method | Strict accuracy | Conflict-aware success |
|---|---:|---:|
| Naive RAG | 0.8700 | 0.8667 |
| Evidence-normalized RAG | 0.8900 | 0.8867 |
| Rule-only extractor | 1.0000 | 1.0000 |
| X-MADAM-RAG | 0.9667 | 0.9767 |
| Oracle extraction | perfect | perfect |

The full original-benchmark table also reports **Conflict F1**, **Conflict awareness**, and **Unsupported-candidate rate**. For **Naive RAG**, those values are **0.9579**, **0.7400**, and **0.0800**. For **Evidence-normalized RAG**, they are **0.9766**, **0.7533**, and **0.1067**. For **Rule-only extractor**, all reported metrics are perfect and the unsupported-candidate rate is **0.0000**. For **X-MADAM-RAG**, they are **0.9804**, **1.0000**, and **0.0200**. **Oracle extraction** is perfect across all reported metrics [2606.12903].

Among deployable LLM methods, X-MADAM-RAG is best on the original benchmark. The paper states that it improves strict accuracy over evidence-normalized RAG by **0.0767** and reduces unsupported-candidate rate from **0.1067** to **0.0200**. The reported **95% bootstrap interval** for the strict-accuracy gap is **[0.0367, 0.1167]**, with an exploratory **sign-test p-value of 0.0004** [2606.12903].

At the same time, the perfect score of the **zero-call rule-only extractor** reveals that the original benchmark is **strongly template-regular**. This is a central interpretive result, not a peripheral ablation. It shows that high performance on the controlled set can arise from surface regularity rather than genuine semantic extraction.

To probe that limitation, the paper constructs a **deterministic naturalized stress test** that removes explicit answer templates while preserving candidate strings and private metadata. The transformed set covers all **300 samples and 534 documents**, with **zero remaining templates** and **perfect candidate and metadata preservation**, but experiments are reported on a **100-sample approximately balanced subset** [2606.12903].

On that stress subset, performance changes sharply:

- **Rule-only extractor** falls from **1.0000** to **0.0000**.
- **Naive** reaches **0.6500** strict accuracy.
- **Evidence-normalized** reaches **0.6800** strict accuracy.
- **X-MADAM-RAG** falls to **0.3000** strict accuracy.
- **Oracle extraction** remains **1.0000**.

The oracle’s continued perfection while both rule-only extraction and X-MADAM-RAG collapse indicates that the principal bottleneck is **document-level extraction**, not downstream grouping or aggregation [2606.12903]. This is arguably the paper’s most consequential finding. It shifts attention from conflict-aware aggregation alone to the fragility of candidate extraction under even deterministic naturalization.

## 6. Interpretive significance, limitations, and appropriate use

The paper is explicit about what X-RAMDocs-ZHEN can and cannot diagnose. It is presented as a **controlled diagnostic tool** for **evidence conflict handling**, **preservation of candidate answers**, **conflict awareness**, **language-directional disagreement**, and **extraction robustness under template and near-template phrasing** [2606.12903]. It does **not** establish **human-validated faithfulness**, **broad hallucination detection**, **robustness to natural retrieval**, **semantic entailment**, or **general multilingual retrieval performance**.

Several limitations are stated directly. The benchmark uses **controlled snippets derived from RAMDocs fields**, not naturally retrieved multilingual passages. The original benchmark is **template-regular**. The naturalized stress test remains **deterministic controlled prose**, not fully natural text. Only **one model**, **Qwen2.5-7B-Instruct**, is evaluated. The benchmark covers only **Chinese and English**. All metrics are **automatic lexical diagnostics**. No completed human audit is available. The oracle is **privileged and non-deployable** [2606.12903].

These constraints also delimit the benchmark’s significance. X-RAMDocs-ZHEN is not evidence that multilingual RAG systems can robustly handle contradiction “in the wild.” Rather, it provides a tightly controlled environment for determining whether a method preserves supported candidates under bilingual disagreement and whether apparent success depends on templates. This suggests a specific research use: controlled diagnosis of failure modes before claims are made about natural multilingual retrieval robustness.

Its broader contribution lies in making **Chinese-English evidence conflict** a first-class evaluation object. Earlier work such as RAMDocs emphasized joint ambiguity, misinformation, and noise in conflicting-evidence RAG [2504.13079]. X-RAMDocs-ZHEN narrows the lens to bilingual contradiction and makes extraction, grouping, and aggregation separately inspectable. The resulting picture is technically precise: conflict-aware aggregation can help on controlled inputs, but **robust document-level extraction remains the central unresolved problem** [2606.12903].

Source: https://www.emergentmind.com/topics/x-ramdocs-zhen