ECLeKTic: Cross-Lingual QA Benchmark

• ECLeKTic is a multilingual closed-book QA challenge that assesses whether LLMs can retrieve factual knowledge across languages using Wikipedia coverage as a proxy.
• It evaluates models’ ability in zero-shot settings, highlighting the gap between source-language retrieval and target-language transfer.
• The benchmark serves as a diagnostic tool to analyze failure modes such as script mismatch, inference strategies, and response variance in multilingual recall.

Searching arXiv for ECLeKTic and related follow-up work to ground the article in current papers. ECLeKTic is a multilingual closed-book question answering challenge set for evaluating cross-lingual knowledge transfer in LLMs, introduced by Goldman, Shaham, Malkin, Eiger, Hassidim, Matias, Maynez, Gilady, Riesa, Rijhwani, Rimell, Szpektor, Tsarfaty, and Eyal at Google Research and Google DeepMind (Goldman et al., 28 Feb 2025). It is designed to test whether a model can retrieve factual knowledge that was primarily acquired in one language and answer correctly when queried in another language where that knowledge is comparatively absent from the local web footprint, approximated through the presence or absence of Wikipedia articles. The benchmark is explicitly framed as a black-box, zero-shot evaluation of parametric knowledge transfer rather than cross-lingual skill transfer, and it has subsequently become a diagnostic substrate for analyzing script effects, inference-time exploration strategies, and statistical accounts of cross-lingual failure (Goldman et al., 28 Feb 2025, Bandarkar et al., 17 Mar 2026, Diskind et al., 23 Jun 2026, Piratla et al., 17 Oct 2025).

1. Conceptual scope and research objective

ECLeKTic evaluates a specific capability: retrieving factual knowledge across languages when the fact is presumed to have been encountered mainly in one language during pretraining. In the formulation of the original paper, cross-lingual knowledge transfer refers to a model’s ability to retrieve factual knowledge that was primarily acquired in one language and answer questions about it reliably in other languages, which is distinguished from cross-lingual skill transfer, i.e., generalizing a learned capability to new languages (Goldman et al., 28 Feb 2025).

The benchmark is motivated by the asymmetry of web coverage across languages. The central concern is that multilingual LLMs may display strong monolingual factual recall in high-exposure languages while failing to provide comparable service to speakers of other languages, especially lower-resource or non-dominant-script languages. ECLeKTic therefore operationalizes a multilingual fairness problem as a factual recall problem under uneven exposure conditions (Goldman et al., 28 Feb 2025).

The task is deliberately closed-book. Models receive only the question, with no retrieved context, so that evaluation targets parametric memory rather than retrieval-augmented behavior. The original paper justifies this design as a way to avoid the pitfalls of automatic metrics that are less correlated with human judgements and to enable black-box evaluation of proprietary systems without white-box interventions such as editing or neuron inspection (Goldman et al., 28 Feb 2025).

A later line of work uses ECLeKTic to sharpen the diagnosis of transfer failures. One study argues that what appears as a cross-lingual knowledge-transfer problem is, to a substantial extent, a script barrier rather than a language-family effect (Bandarkar et al., 17 Mar 2026). Another recasts multilingual inference over ECLeKTic as an exploration problem over parametric knowledge, where language choice at inference time determines which internal associations are probed (Diskind et al., 23 Jun 2026). A further study treats ECLeKTic as evidence that target-language response variance is a major driver of the source–target accuracy gap (Piratla et al., 17 Oct 2025). Taken together, these developments position ECLeKTic not only as a benchmark, but as an instrument for theorizing failure modes in multilingual parametric recall.

2. Dataset design and construction protocol

ECLeKTic is built around the assumption that the presence or absence of a Wikipedia article is an approximate proxy for the presence or absence of a fact in a language-specific training distribution (Goldman et al., 28 Feb 2025). Using the July 2023 Wikipedia dump, the authors identified articles present in one language’s Wikipedia but lacking equivalent articles in the other 11 languages. Presence is treated as correlated with repeated exposure during pretraining, while absence is treated as indicating sparse exposure in that language (Goldman et al., 28 Feb 2025).

The benchmark uses 12 Wikipedias as both source and target languages: English, French, German, Hebrew, Hindi, Indonesian, Italian, Japanese, Korean, Mandarin Chinese, Portuguese, and Spanish (Goldman et al., 28 Feb 2025). For each language serving as a source, unique articles were sampled under three criteria: at least 200 characters, at least 100 views during 2023, and no equivalent articles in the other 11 languages (Goldman et al., 28 Feb 2025). From each selected article, the first 10 sentences were extracted as context (Goldman et al., 28 Feb 2025).

Question-answer generation was performed by prompting Gemini to produce a single, self-contained, factual question and its answer based strictly on that context, not on general knowledge (Goldman et al., 28 Feb 2025). The generation prompt emphasized specificity, full entity names, clear granularity, and the avoidance of multi-answer or opinion questions. Human validation then filtered candidates to ensure that each question was answerable without explicit access to the context, tied to language-specific knowledge, and not merely generic science or globally well-known material (Goldman et al., 28 Feb 2025). Annotators also disambiguated potentially ambiguous questions, for example by clarifying named entities or specifying an adaptation such as “the Israeli adaptation of Survivor” (Goldman et al., 28 Feb 2025). Items failing these criteria were discarded.

All retained source-language question, answer, and context triples were translated into the other 11 target languages. Verification used English as a pivot to ease non-trivial language pairs, and annotators corrected translation issues or removed untranslatable cases such as questions depending on the meaning of a source-language word (Goldman et al., 28 Feb 2025). This translation design is essential to the benchmark logic: the model is asked in the target language about a fact whose local-language Wikipedia coverage is absent, so a correct answer implies some degree of cross-lingual retrieval rather than mere in-language memorization.

Answer adjudication avoids exact string matching. Instead, ECLeKTic uses an LLM judge with access to the translated context and a prompt asking it to determine whether an answer to the question is supported by a given text, returning YES or NO (Goldman et al., 28 Feb 2025). This judge-based evaluation is intended to be less brittle than string matching and more aligned with semantic correctness.

3. Composition, variants, and benchmark statistics

The main ECLeKTic dataset contains 384 generated question-answer pairs across the 12 source languages, 4224 translated target-language instances, and 4608 evaluated instances in total, counting both source-language and target-language asks (Goldman et al., 28 Feb 2025). No explicit train, development, or test splits are defined; the benchmark is intended for zero-shot evaluation (Goldman et al., 28 Feb 2025).

The per-source-language counts reported in the paper are uneven, reflecting the underlying availability of qualifying articles. The paper lists the following counts: English 39, French 22, German 16, Hebrew 29, Hindi 64, Indonesian 31, Italian 29, Japanese 26, Korean 33, Mandarin Chinese 35, Portuguese 28, and Spanish 32 (Goldman et al., 28 Feb 2025).

A second benchmark variant, ECLeKTic “Popular Pages,” was constructed to test robustness beyond marginal topics (Goldman et al., 28 Feb 2025). For each Wikipedia, the authors sampled the 200 most-viewed April 2023 articles that lack equivalents in at least one of the 12 languages and built QA pairs using the same pipeline (Goldman et al., 28 Feb 2025). This produced 964 unique QA pairs and 6628 evaluated instances, since each pair was evaluated only in target languages lacking Wikipedia coverage for the topic (Goldman et al., 28 Feb 2025).

Component	Main ECLeKTic	“Popular Pages” variant
Unique QA pairs	384	964
Target-language translations	4224	Not stated separately
Total evaluated instances	4608	6628

The benchmark’s examples are intentionally local and often obscure outside the source locale. The original paper notes Tobias Meister as an illustrative case: he is well known in German as Brad Pitt’s dubbing voice, yet this information is not equivalently surfaced across languages (Goldman et al., 28 Feb 2025). Later work describes ECLeKTic questions as local knowledge or factual recall questions about entities and facts that may be obscure outside the source locale or language (Bandarkar et al., 17 Mar 2026). This strongly local character is central to the benchmark’s difficulty profile.

4. Task formulation and evaluation metrics

The ECLeKTic task is multilingual closed-book QA. The input is a question in a target language about a fact that is predominantly present in a source language’s Wikipedia, and the model must answer without any provided context (Goldman et al., 28 Feb 2025). All models in the original evaluation were tested in a zero-shot setting and prompted with the question on its own, without an explicit instruction (Goldman et al., 28 Feb 2025).

The benchmark’s main metric couples source-language recall with target-language transfer. Example-level success for a model $M$ on question-answer pair $(q,a)$ is defined as

$S^{q,a}_M \;=\; \mathds{1}\big(M(q_{l_t}) = a_{l_t} \;\land\; M(q_{l_s}) = a_{l_s}\big),$

where $l_s$ is the source language and $l_t$ is a target language (Goldman et al., 28 Feb 2025). Overall success is the mean of this quantity over the dataset:

$S^{\text{overall}_M \;=\; \frac{\sum_{(q,a) \in D} S^{q,a}_M}{|D|}.$

This metric reflects the combined ability to retrieve the fact in the source language and transfer it to the target language (Goldman et al., 28 Feb 2025). The transfer score further conditions on items the model already answers correctly in the source language. Defining $K=\{(q,a)\mid M(q_{l_s})=a_{l_s}\}$, the score is

$S^{\text{transfer}_M \;=\; \frac{\sum_{(q,a) \in K} S^{q,a}_M}{|K|}.$

This isolates cross-lingual knowledge transfer independently of the model’s source-language factual recall (Goldman et al., 28 Feb 2025).

The original paper does not report EM, F1, or macro versus micro averaging across languages; correctness is determined by the LLM judge’s binary YES/NO decision with access to the question and context (Goldman et al., 28 Feb 2025). Later work often reuses ECLeKTic with different but related metrics. One follow-up uses binary accuracy scored via an auto-rater and fits OLS and logistic regressions to diagnose determinants of success (Bandarkar et al., 17 Mar 2026). Another introduces Knowledge Transfer (KT) as the main ECLeKTic metric over target-language instances where $L_q \neq L_{src}$ and reports macro and micro averages with mean ± SEM using Gemini 2.5 Flash as a deterministic judge (Diskind et al., 23 Jun 2026). These alternate formulations preserve the core idea that ECLeKTic measures whether localized facts can be surfaced outside the language in which they were most prominently encountered.

5. Empirical findings in the original benchmark paper

The original ECLeKTic study evaluates eight LLMs in zero-shot mode: Gemini 2.0 Pro, Gemini 2.0 Flash, GPT-4o, Claude 3.5 Sonnet, Gemma 2 9B, Mistral Nemo Instruct 2407, Qwen 2.5 7B Instruct, and OLMo 2 7B Instruct (Goldman et al., 28 Feb 2025). The reported headline result is that state-of-the-art systems struggle to share knowledge effectively across languages even when they can answer correctly in the source language (Goldman et al., 28 Feb 2025).

The main quantitative results on ECLeKTic are as follows (Goldman et al., 28 Feb 2025):

Model	Overall	Transfer
Gemini 2.0 Pro	41.6±1.5	65.0±1.8
GPT-4o	38.8±1.4	67.0±1.8
Gemini 2.0 Flash	34.6±1.4	62.3±1.9
Claude 3.5 Sonnet	34.4±1.4	60.8±1.9
Gemma 2 9B	8.7±0.8	40.3±3.1
Mistral Nemo	7.1±0.8	38.9±3.4
Qwen 2.5 7B	2.8±0.5	23.5±3.7
OLMo 2 7B	1.6±0.3	17.2±3.7

These numbers imply that even the strongest models combine only moderate source-language retrieval with incomplete transfer. Transfer scores in the low-to-mid 60s coexist with overall success in the mid-30s to low-40s, indicating that the bottleneck lies partly in retrieving the source-language fact and partly in transferring it reliably once retrieved (Goldman et al., 28 Feb 2025).

The “Popular Pages” variant preserves the same ordering and similar performance gaps. Gemini 2.0 Pro reaches 36.7±1.2 overall and 67.8±1.7 transfer; GPT-4o reaches 34.5±1.2 and 67.3±1.7; Gemini 2.0 Flash reaches 31.6±1.2 and 65.8±1.8; Claude 3.5 Sonnet reaches 23.8±1.1 and 56.8±2.0 (Goldman et al., 28 Feb 2025). Smaller open models remain substantially weaker. The similarity of these results is presented as evidence that the core ECLeKTic phenomenon is not limited to only marginal or obscure topics (Goldman et al., 28 Feb 2025).

The paper also reports asymmetries across language directions. Shared script strongly eases transfer: for Gemini 2.0 Pro, transfer is much higher among Latin-script languages and between Chinese and Japanese than for many cross-script pairs, with cited extremes ranging from 23.5 for Portuguese $\rightarrow$ Japanese to 100.0 for German $(q,a)$0 Indonesian (Goldman et al., 28 Feb 2025). The paper further notes source–target asymmetry by language role, observing that Hindi as a source averages 78.6 transfer to others, while transfer into Hindi averages only 59.6 (Goldman et al., 28 Feb 2025). A plausible implication is that source-language prominence and target-language accessibility are not interchangeable properties.

Prompting ablations reinforce the claim that the primary failure is retrieval rather than reasoning over explicit context. For Gemini 2.0 Flash, a general hint suggesting the use of other languages yields only a small change relative to closed-book prompting, but revealing the source language or the source-language Wikipedia title substantially improves performance (Goldman et al., 28 Feb 2025). Providing the correct source-language context in an open-book setup produces near-perfect results: 94.3±0.7 overall and 96.0±0.6 transfer (Goldman et al., 28 Feb 2025). This suggests that cross-lingual reasoning over supplied evidence is relatively strong, whereas cross-lingual retrieval from parametric memory is the more serious bottleneck.

6. Subsequent interpretations: script barriers, exploration, and variance

A 2026 analysis of modern reasoning models uses ECLeKTic to argue that script mismatch is the primary predictor of transfer failure once model capability and question difficulty are controlled (Bandarkar et al., 17 Mar 2026). In this study, models include Qwen3-30B-A3B-Thinking-2507, Qwen3-235B-A22B-Thinking-2507, GLM-4.5-Air, GPT-OSS-120B, and Olmo-3.1-32B-Think (Bandarkar et al., 17 Mar 2026). Using OLS regression on binary accuracy, the paper reports that script match has a coefficient of 0.121 on ECLeKTic and is statistically significant, while language family and in-language indicators are not significant (Bandarkar et al., 17 Mar 2026). Logistic regression average marginal effects corroborate this finding, with script match AME 0.124 (Bandarkar et al., 17 Mar 2026).

Observationally, the same paper reports that when scripts mismatch, accuracy is 13% lower, and that once script matches, cross-language gaps become statistically insignificant after controlling for Belebele target-language reading comprehension and subset difficulty (Bandarkar et al., 17 Mar 2026). This reframes ECLeKTic as evidence for a cross-script transfer barrier rather than a generic cross-lingual barrier. Controlled interventions support that interpretation. Prefixing the model’s thinking trace with the key entity in the source language improves cross-script items more than same-script items, with average gains of +2.1 versus +0.5 across three models (Bandarkar et al., 17 Mar 2026). Synthetic LoRA SFT designed to teach reasoning about transliteration and entity-origin ambiguity yields larger improvements in non-Latin, low-resource, and cross-script settings than in same-script settings (Bandarkar et al., 17 Mar 2026). The study concludes that post-training can reduce the cross-script gap without necessarily changing the underlying knowledge representations (Bandarkar et al., 17 Mar 2026).

Another 2026 paper treats ECLeKTic as a benchmark for inference-time cross-lingual exploration of parametric knowledge (Diskind et al., 23 Jun 2026). It systematizes four design dimensions: language selection, exploration routing, answer selection and aggregation, and inference budget (Diskind et al., 23 Jun 2026). On ECLeKTic, the native query-language baseline is 38.4 macro and 38.4 micro. A single-path English pivot raises macro KT to 47.6, oracle routing through the source language reaches 51.0, and a single-path autonomous origin-aware strategy, denoted $(q,a)$1, reaches 53.1, surpassing the oracle (Diskind et al., 23 Jun 2026). Multi-path exploration with $(q,a)$2 and Minority-Aware aggregation yields 59.3 macro, a gain of 20.9 points over the native baseline (Diskind et al., 23 Jun 2026). The potential upper bound for this configuration is 81.7 macro, indicating that exploration often finds the correct fact but aggregation fails to select it (Diskind et al., 23 Jun 2026). This suggests that ECLeKTic performance is constrained not only by knowledge accessibility but also by answer-selection mechanisms over explored multilingual trajectories.

A separate 2025 study offers a statistical account of ECLeKTic’s cross-lingual gap in terms of bias–variance decomposition (Piratla et al., 17 Oct 2025). It formalizes the gap as

$(q,a)$3

and argues that the dominant term is increased target-language response variance rather than systematic bias (Piratla et al., 17 Oct 2025). On ECLeKTic, sampling 10 generations per prompt and aggregating them reduces divergence between source and target responses, and the paper estimates $(q,a)$4, meaning that roughly 90% of examples are explained by a variance-only component (Piratla et al., 17 Oct 2025). Translation-based input ensembling further improves transfer. For example, the paper reports ECLeKTic transfer-score gains from baseline to TTA-1 of 30.7 to 37.8 for Gemini 2.5-Flash, 37.2 to 49.3 for Gemini 2.5-Pro, and 35.4 to 49.1 for GPT-5 (Piratla et al., 17 Oct 2025). The authors interpret these gains as evidence that target-language generations are often centered near the correct source-language answer but are less stable. This suggests that some ECLeKTic failures arise from stochastic accessibility rather than complete absence of the underlying knowledge.

7. Limitations, assumptions, and significance

ECLeKTic rests on explicit assumptions. The most important is that Wikipedia presence approximates prominence in training data, while absence implies sparse exposure in a language (Goldman et al., 28 Feb 2025). This is a practical operationalization rather than a verified measurement of pretraining exposure. The benchmark also inherits Wikipedia’s cultural and editorial biases, and later work notes that translation through an English pivot may introduce artifacts, especially for rare entities or cross-script transliterations (Goldman et al., 28 Feb 2025, Bandarkar et al., 17 Mar 2026).

Coverage is limited to 12 languages in the original construction (Goldman et al., 28 Feb 2025). Although these languages span Latin and non-Latin scripts, many major languages remain outside the benchmark. Expanding to more languages would increase translation and evaluation cost because each source item must be translated across all other target languages (Goldman et al., 28 Feb 2025). Time sensitivity is another issue: ECLeKTic is built from July 2023 Wikipedia distributions, and localized prominence can change over time (Goldman et al., 28 Feb 2025).

The use of an LLM judge is both a strength and a source of uncertainty. The original benchmark adopts semantic adjudication to avoid brittle string matching (Goldman et al., 28 Feb 2025), and follow-up work continues to rely on automatic judges, sometimes with manual spot checks (Diskind et al., 23 Jun 2026, Piratla et al., 17 Oct 2025). However, judge error on aliases, partial names, or culturally specific renderings remains a validity threat.

Despite these limitations, ECLeKTic has become significant because it isolates a failure mode that broad multilingual benchmarks often obscure. Its source-language labels enable direction-specific analysis rather than only English-to-other-language transfer (Bandarkar et al., 17 Mar 2026). Its local-knowledge construction makes it particularly sensitive to exposure asymmetries, which has allowed later research to identify script mismatch as a major explanatory factor, to demonstrate that inference-time language exploration can outperform source-language oracle routing, and to show that variance-control interventions can materially narrow the gap without parameter updates (Bandarkar et al., 17 Mar 2026, Diskind et al., 23 Jun 2026, Piratla et al., 17 Oct 2025).

In this sense, ECLeKTic is both a benchmark and a methodological lens. It tests whether multilingual LLMs truly abstract knowledge beyond the language in which it was acquired, and subsequent studies suggest that the answer is conditional: current systems often possess the relevant knowledge, but access depends strongly on script alignment, prompting strategy, exploration policy, and generation variance (Goldman et al., 28 Feb 2025, Bandarkar et al., 17 Mar 2026, Diskind et al., 23 Jun 2026, Piratla et al., 17 Oct 2025).