MultiWikiAtomic: Multilingual Contextual Benchmark

• MultiWikiAtomic is a multilingual benchmark that separates contextual knowledge (CK) and parametric knowledge (PK) in open-ended generation.
• It constructs datasets by atomizing Wikipedia content in English, Spanish, and Danish to measure sentence-level grounding via the CoPE framework.
• The benchmark reveals a 'lost-in-the-later' bias in models, demonstrating that prompted strategies and context positioning significantly impact grounding performance.

to=arxiv_search.query _日本一级特黄大片 {"^^^^^^^^3search_query3 OR ti:\3^^^^^^^^"Lost-in-the-Later\" OR all:\3^^^^^^^^"MultiWikiAtomic\"","start":^^^^3search_query3 to=arxiv_search.query code 彩神争霸官网ing? wait tool name maybe arxiv_search? Let's inspect? The developer said arXiv search tool. Need know actual function name. It might be "arxiv". Hmm not shown. Let's list? No tool list in system. Maybe function is unavailable. Need infer. In prior tasks maybe tool name arxiv_search with search_arxiv? Let's see available tools? Not given. We can attempt? to=arxiv_search.query 天天彩票提现json {"query":"(&&&^^^^3search_query3^^^^&&&) MultiWikiAtomic","max_results":5} to=arxiv.search 彩票娱乐注册json {"query":"(&&&^^^^3search_query3^^^^&&&) MultiWikiAtomic","max_results":5} to=search_arxiv 大发展有限公司官网json {"query":"(&&&^^^^3search_query3^^^^&&&) MultiWikiAtomic","max_results":5} MultiWikiAtomic is a multilingual benchmark for analyzing contextual grounding in open-ended generation. Introduced as the multilingual extension of WikiAtomic within the CoPE framework, it is designed to separate contextual knowledge (CK), defined as content entailed by the provided input context, from parametric knowledge (PK), defined as content not entailed by that context, even when the latter is factually correct in the real world. Its central role is to support controlled measurement of how LLMs integrate supplied context, prioritize information across input positions, and drift between grounded and parametric generation in English, Spanish, and Danish (&&&^^^^3search_query3^^^^&&&).

^^^^3id:(Tao et al., 7 Jul 2025) OR ti:\3^^^^. Conceptual scope and problem setting

MultiWikiAtomic addresses a specific evaluation problem: existing methods could not precisely quantify CK versus PK in multilingual, open-ended generation settings. The benchmark therefore operationalizes contextual grounding as a behavioral property of model outputs relative to a supplied context, rather than as a truth-assessment task. In this formulation, a sentence counts as CK if it is entailed by the context, and as PK if it is not entailed by the context, regardless of whether it is objectively true or false.

This distinction is foundational. A response can be factually correct yet still be PK if it is unsupported by the input. Conversely, a response can be grounded in a false context and still count as CK because it follows the provided text. The dataset is thus tailored to attribution analysis, knowledge-source disambiguation, and positional-bias measurement, not unrestricted factuality evaluation.

The benchmark is coupled to CoPE, which uses MultiWikiAtomic as a controlled environment for multilingual open-ended question answering. Wikipedia-derived contexts are atomized into single-proposition sentences, models generate free-form responses to topic prompts, and those responses are then atomized and classified sentence-by-sentence as CK or PK. This yields a finer-grained measurement than conventional accuracy-style evaluation because it quantifies not only whether an answer is acceptable, but whether it is grounded in the input (&&&^^^^3search_query3^^^^&&&).

^^^^3 OR all:\3^^^^. Dataset construction and atomicization protocol

The dataset construction is deliberately simple and controlled. The original WikiAtomic English data were extended by adding ^^^^3id:(Tao et al., 7 Jul 2025) OR ti:\3search_query3search_query3^^^^ Wikipedia articles each in Spanish and Danish, yielding 5,^^^^3search_query3search_query3search_query3^^^^ atomic sentences per added language and ^^^^3id:(Tao et al., 7 Jul 2025) OR ti:\3^^^^5,^^^^3search_query3search_query3search_query3^^^^ total atomic sentences across English, Spanish, and Danish. For each topic or article, the benchmark constructs contexts of six sizes, ranging from ^^^^3search_query3^^^^ up to 5^^^^3search_query3^^^^ sentences, so that the same topic can be probed under progressively richer contextual conditions (&&&^^^^3search_query3^^^^&&&).

The task format is open-ended QA. For each topic, the prompt takes the form “With this information, tell me about [Topic].” The main benchmark uses knowledge-consistent contexts rather than contradictory ones, although a separate contradiction-rich experiment is also included. This design isolates how models exploit aligned contextual evidence before testing behavior under conflict.

A defining feature of MultiWikiAtomic is sentence atomization. Both source articles and model responses are converted into atomic sentences through a prompt-based procedure inspired by prior atomic factual evaluation. The atomization prompt requests multiple passes: remove commas, split on “and/or,” replace indirect references with direct topic references, separate temporal details, and ensure each atomic sentence contains exactly one factual proposition. The purpose is to reduce semantic overlap and ambiguity across languages.

This construction choice gives the benchmark its “atomic” granularity. Rather than scoring a whole response monolithically, MultiWikiAtomic decomposes both context and output into proposition-level units, enabling sentence-level entailment testing and segment-level recall analysis.

3. CoPE scoring framework and formal measures

MultiWikiAtomic is evaluated through CoPE, which formalizes contextual grounding at the level of atomic sentences. If a response PRESERVED_PLACEHOLDER_^^^^3search_query3^^^^ contains PRESERVED_PLACEHOLDER_^^^^3id:(Tao et al., 7 Jul 2025) OR ti:\3^^^^ atomic sentences PRESERVED_PLACEHOLDER_^^^^3 OR all:\3^^^^, contextual knowledge is defined as

$$\text{CK} = \frac{\sum_{i=1}^{n} \mathbb{I}(S_i \in C)}{n} \times 100$$

where $C$ is the set of entailed context sentences and $\mathbb{I}$ indicates whether a response sentence is supported by the context. PK is defined complementarily as

$PK = 100 - CK.$

CoPE also introduces a context-recall measure over context segments. If the input context is partitioned into $k$ segments $\{Q_1,\dots,Q_k\}$, recall from segment $q$ is

PRESERVED_PLACEHOLDER_^^^^3id:(Tao et al., 7 Jul 2025) OR ti:\3search_query3^^^^

This metric is used to identify where in the input the model actually sources its grounded information.

Sentence classification is implemented via a bidirectional NLI procedure adapted from INFUSE, aggregating entailment in both directions: context-to-output and output-to-context. The model used is mDeBERTa-v3-base-xnli-multilingual-nli, with the CK/PK threshold set to PRESERVED_PLACEHOLDER_^^^^3id:(Tao et al., 7 Jul 2025) OR ti:\3id:(Tao et al., 7 Jul 2025) OR ti:\3^^^^. Controlled checks with synthetic examples found classification error under ^^^^3search_query3^^^^.5%, and filtering borderline entailment scores between ^^^^3search_query3^^^^.4 and ^^^^3search_query3^^^^.8 did not materially change the main trends (&&&^^^^3search_query3^^^^&&&).

These definitions make MultiWikiAtomic a proposition-level attribution benchmark rather than a retrieval dataset or a contradiction-only stress test. Its unit of analysis is the atomic factual statement, and its main observable is the provenance of generated content.

4. Empirical findings on grounding and positional bias

The experiments on MultiWikiAtomic reveal that models do not fully exploit available context. CK generally peaks around 7^^^^3search_query3^^^^–75% for the best models, implying that roughly 3^^^^3search_query3^^^^% of the generated output remains PK even when relevant context is available. Larger models such as Gemini ^^^^3id:(Tao et al., 7 Jul 2025) OR ti:\3^^^^.5 Pro and GPT-4o perform better than smaller ones like Llama 3.^^^^3 OR all:\3^^^^ 3B, while reasoning-oriented models GPT-o3 and Qwen 3 ^^^^3 OR all:\33^^^^5B show the lowest CK scores, around 55% even with more context (&&&^^^^3search_query3^^^^&&&).

The paper identifies a positional bias termed “lost-in-the-later.” Across English, Spanish, and Danish, models strongly favor the beginning of the context: the first quartile contributes the most grounded information, and the last quartile the least. The claim is not that models merely exhibit the known long-context “lost in the middle” effect. Rather, later context is systematically underused even for relatively short inputs of up to 5^^^^3search_query3^^^^ atomic sentences.

A further control test randomized sentence order for 45^^^^3search_query3^^^^ questions. Recall changed only slightly, about 5%, which indicates that the bias is not merely an artifact of original Wikipedia ordering. This suggests that the positional effect is intrinsic to model behavior under the evaluated prompting regime.

The benchmark also localizes PK within outputs. PK tends to rise toward the end of responses, so earlier response segments are more likely to be grounded, whereas later segments increasingly drift into parametric content. Under contradiction-rich conditions, CK drops as expected but not to zero, reinforcing that CK measures adherence to the provided context rather than factual truth. In mixed settings where factual and counterfactual sentences are interleaved, models perform better when true information appears first than when it appears later, again echoing the lost-in-the-later pattern.

5. Multilingual behavior and prompting effects

MultiWikiAtomic is explicitly multilingual, and the benchmark shows that language affects grounding difficulty. Danish generally yields lower CK than English, and Spanish and Danish are more challenging overall, reflecting the difficulty of grounding in lower-resource languages. However, the main positional pattern persists across all three languages and across all tested model families (&&&^^^^3search_query3^^^^&&&).

The benchmark also probes the effect of chain-of-thought prompting. Contrary to the assumption that CoT should improve careful context use, the reported results show that CoT does not mitigate the lost-in-the-later effect and often degrades contextual grounding. Reasoning models, as well as non-reasoning models prompted with CoT, use context less than non-reasoning models without CoT. CoT prompting in particular results in lower recall and shorter responses, leaving less room to recover evidence from the input.

Prompting interventions are therefore evaluated directly against MultiWikiAtomic. A simple CK prompt, which explicitly instructs the model to use only the provided context and to draw evenly from all parts of it, performs best. This CK prompt raises CK scores across languages, improves balance in context recall, and outperforms both a strict “context only” prompt and a balanced prompt alone. CoT+CK improves relative to plain CoT by recovering some CK, but it still underperforms the best non-CoT prompts.

These findings make MultiWikiAtomic useful not only as a diagnostic benchmark but also as an intervention testbed. Because CK and segment-level recall are measured explicitly, prompt variants can be compared in terms of grounding behavior rather than only end-task correctness.

6. Relation to prior benchmarks, applications, and limitations

MultiWikiAtomic is positioned as a multilingual expansion of WikiAtomic and as a more controlled alternative to counterfactual datasets such as RECALL or FaithEval. Those datasets are useful for conflict scenarios, especially when evaluating whether models override parametric knowledge in contradiction settings, but they do not directly measure CK/PK balance when context and memory are aligned. MultiWikiAtomic instead targets the aligned-case regime and asks how much of an answer is actually sourced from the supplied context (&&&^^^^3search_query3^^^^&&&).

The framework is presented as model- and task-agnostic, applicable to both open and closed models, and more broadly useful than systems that require token-level logits or operate only on specific tasks. A summarization case study extends the same methodology beyond QA and reports that CK-aware prompting improves factual grounding and reduces hallucination without hurting general quality much.

The benchmark’s limitations are explicit. CoPE depends on reliable sentence atomization and semantic entailment, so it is less tested on noisy or informal text such as dialogue and social media. It measures grounding to the provided context, not real-world factual correctness, so stronger CK does not automatically imply higher truthfulness when the context itself is flawed. The multilingual hallucination analysis also relies on an extended FActScore setup, while factual evaluation in Spanish and Danish remains less mature than in English.

Accordingly, MultiWikiAtomic is best understood as an instrument for controlled multilingual studies of grounding, positional bias, and knowledge-source attribution. Its main significance lies in showing that open-ended generation can be decomposed proposition-by-proposition into contextual and parametric components, and that this decomposition exposes systematic failure modes—especially underuse of later context—that are difficult to observe with conventional QA or factuality benchmarks alone.