GAIA-v2-LILT: Multilingual Agent Benchmark
- The paper demonstrates that naive machine translation can corrupt benchmark validity by altering task semantics, answer formats, and difficulty.
- GAIA-v2-LILT is a multilingual extension of the GAIA benchmark that utilizes explicit functional, cultural, and difficulty alignment to faithfully adapt tasks.
- Empirical results show significant performance improvements post-audit, suggesting that translation-induced errors skew multilingual evaluation.
GAIA-v2-LILT is a multilingual, re-audited version of the GAIA agent benchmark designed to evaluate whether language-model-based agents can solve the same underlying tasks across languages without the evaluation itself being corrupted by poor translation (Kim et al., 27 Apr 2026). It extends the English GAIA benchmark for “General AI Assistants” into five non-English languages through a workflow that goes beyond machine translation by enforcing explicit functional alignment, cultural alignment, and difficulty calibration. The benchmark’s central claim is methodological as much as empirical: for agentic tasks, multilingual adaptation is a task-level validity problem, not merely a text-level translation problem, because translation errors can alter answer correctness, output format requirements, realism, and difficulty (Kim et al., 27 Apr 2026).
1. Definition, scope, and benchmark lineage
The original GAIA benchmark is an English benchmark for “General AI Assistants,” namely agents that must perform multi-step problem solving with external tools rather than only answer from internal language modeling (Kim et al., 27 Apr 2026). In the task definition used for GAIA-v2-LILT, an agentic task is a single-turn query that requires at least one external action such as web search, page parsing, file inspection, numerical computation, or scripting. GAIA tasks also commonly require strict output formatting and are graded with deterministic reference answers (Kim et al., 27 Apr 2026).
GAIA-v2-LILT is built from the validation split of GAIA via the MAPS multilingual version, but it is not a minimally revised translated copy. It is a re-audited multilingual extension containing 165 query-answer pairs per language in five non-English languages: Arabic, German, Hindi, Korean, and Portuguese (Brazil) (Kim et al., 27 Apr 2026). Its stated purpose is to make multilingual agent evaluation more faithful to the original English task intent, answer conditions, and difficulty.
The benchmark therefore occupies a specific lineage: original English GAIA, then MAPS machine-translated multilingual GAIA, then GAIA-v2-LILT as a human-audited, functionally and culturally aligned multilingual extension (Kim et al., 27 Apr 2026). This lineage is central to the paper’s contribution, because the empirical comparison is explicitly between raw machine-translated tasks and audited multilingual tasks rather than between unrelated benchmark families.
2. Benchmark invalidation under naive multilingual adaptation
The paper argues that standard machine translation plus light post-editing can break benchmark validity in ways that are specific to agentic evaluation (Kim et al., 27 Apr 2026). The core issue is not only reduced fluency; it is that the executable logic of a task can change. A translated query may no longer request the same thing, the translated answer may no longer satisfy the query, the prompt may become culturally off-target, or the task may become easier or harder than the English original.
The most important failure mode is query-answer misalignment. The canonical example is an Arabic task in which an answer that should remain the IOC country code “CUB” was translated into the Arabic word for “lion cub.” That translation is both semantically wrong and functionally incompatible with a query that requires a three-letter country code (Kim et al., 27 Apr 2026). The paper distinguishes under-translation and over-translation as separate subcases. Under-translation occurs when an answer key remains in English even though the task requires localization; the German chess example is that a locally valid move notation such as Td5 would be unfairly rejected if the answer key still expected English Rd5. Over-translation occurs when a string should remain unchanged because the task requires copying or extracting it exactly, but the system translates it anyway (Kim et al., 27 Apr 2026).
A second failure mode is culturally inappropriate or off-target context. The paper lists geography, policy logic, units of measurement, platform conventions, and social norms as recurrent sources of distortion (Kim et al., 27 Apr 2026). Its Korean bottle-return example shows a machine-translated prompt that still referred to U.S. highways, imperial units, and an unsuitable recycling context. The audited version replaced these with Korean routes, converted measurements, updated the date context, and adapted the incentive system to Korean reality where possible. The paper adds a methodological concern: if a translated prompt remains visibly U.S.-centric, an agent may internally translate it back to English and solve the original English problem, so the benchmark ceases to measure multilingual reasoning in a realistic setting (Kim et al., 27 Apr 2026).
A third failure mode is altered task difficulty. Even when broad semantics are preserved, retrieval conditions, arithmetic, clue structure, or local web availability may shift the solving process (Kim et al., 27 Apr 2026). The Korean word-riddle example illustrates this: the draft kept the broad function of the original task but made it easier by embedding an English clue inside otherwise normal Korean text. The corrected version restored the intended challenge by reversing the Korean sentence and localizing the clue word.
These cases motivate the paper’s broader interpretive claim that multilingual performance deficits on translated agent benchmarks may partly be benchmark-induced measurement error rather than direct evidence of weaker model capability in the target language (Kim et al., 27 Apr 2026).
3. Alignment framework and audit workflow
GAIA-v2-LILT organizes multilingual adaptation around three alignment goals: functional alignment, cultural alignment, and difficulty calibration (Kim et al., 27 Apr 2026). Functional alignment requires that the translated task preserve core task mechanics and evaluation logic. The prompt must still request the same output in the same way, and the answer key must remain compatible with the query. This includes deciding when strings should remain unchanged because they are part of an extraction target or answer format, and when they should be localized because the task expects conventional local notation. The paper later reports that this category has the highest association with output flips after correction (Kim et al., 27 Apr 2026).
Cultural alignment requires that the prompt make sense as a realistic task for the target-language audience. The benchmark should not appear to be an English problem with translated surface forms. Localizing roads or place names, units, dates, policy contexts, consumer conventions, or public systems may be necessary, but the paper explicitly states that not everything should be localized blindly; literal source context can be retained when no local equivalent exists (Kim et al., 27 Apr 2026).
Difficulty calibration requires that the target-language task remain approximately as hard as the English original. This is benchmark-specific rather than purely linguistic. A translation can be fluent, culturally appropriate, and still no longer be comparable because evidence is less accessible on the local web or because a clue becomes easier after translation. The paper therefore requires manual verification of the solving process itself, including searches in the target-language environment (Kim et al., 27 Apr 2026).
The workflow uses three review stages. First, deterministic filtering applies rule-based scripts for language identification, string matching for answer leakage, and placeholder recall for fixed terms such as numbers and URLs (Kim et al., 27 Apr 2026). This stage is intentionally non-generative and exact-match based in order to avoid LLM self-preference. Second, LLM judges perform single-axis binary judgments over short contexts, separately evaluating fluency, adequacy, query-answer compatibility, and cultural appropriateness. Third, bilingual human linguists conduct the substantive audit through 1-on-1 training calls, explicit issue checkboxes, visibility into deterministic and LLM flags, one linguist reviewing each task, and a second linguist together with a machine learning researcher conducting meta-review (Kim et al., 27 Apr 2026). The workflow is designed specifically to counter LLM self-preference and human fluency bias.
4. Dataset construction, audit intensity, and corrected error types
GAIA-v2-LILT is based on the machine-translated GAIA version released through MAPS, after which the new workflow is applied to produce the final audited set (Kim et al., 27 Apr 2026). For each language, the benchmark contains 165 validation tasks, each a query-answer pair.
The reported edit rates show that the audit was substantial rather than cosmetic.
| Language | Tasks edited | Word edit rate |
|---|---|---|
| Arabic | 84.8% | 25.4% |
| German | 81.2% | 30.0% |
| Hindi | 100.0% | 55.4% |
| Korean | 92.1% | 36.9% |
| Portuguese | 87.9% | 25.0% |
The corresponding character edit rates are 19.4% for Arabic, 22.2% for German, 46.3% for Hindi, 27.9% for Korean, and 19.5% for Portuguese (Kim et al., 27 Apr 2026). The most striking figure is Hindi, where every task was edited. This supports the paper’s claim that minimal translation is inadequate for this kind of benchmark.
The corrected errors span multiple defect classes. In Arabic, the over-translation of “CUB” corrupted the answer key. In German, the audit corrected translationese, inappropriate use of the formal pronoun Sie, and one case in which the output was in Italian instead of German and also leaked the actual answer into the prompt (“Honolulu, Quincy”) (Kim et al., 27 Apr 2026). In Korean, the audit corrected U.S.-centric road-trip and recycling assumptions and separately repaired a riddle whose translated form had become easier than intended. These examples show that the audit was not limited to fluency editing; it addressed semantic corruption, answer leakage, evaluation mismatch, locale mismatch, and difficulty drift.
5. Experimental protocol and empirical findings
The paper evaluates three frontier models—GPT-5.4, Gemini 3.1 Pro, and Claude Opus 4.6—inside the Open Deep Research agent built with smolagents (Kim et al., 27 Apr 2026). The agents were equipped with web search using Exa, speech recognition, image captioning, and file read tools. For each task, the manager agent was limited to 12 steps and the search subagent to 20 steps. The comparison is between MT, the raw machine-translated benchmark drafts from MAPS; Audit, the final human-corrected GAIA-v2-LILT tasks; and the English original benchmark as baseline (Kim et al., 27 Apr 2026).
Evaluation is reported as pass@1 accuracy [%], with final-answer correctness determined under normalized exact match by removing spaces, lowercasing, and stripping punctuation (Kim et al., 27 Apr 2026). On this metric, auditing improves agent success rates across all languages and all tested models.
For GPT-5.4, Arabic improves from 32.1 to 47.3, German from 47.3 to 63.6, Hindi from 34.6 to 60.0, Korean from 33.3 to 62.4, and Portuguese from 47.3 to 58.2, against an English score of 66.7 (Kim et al., 27 Apr 2026). For Gemini 3.1 Pro, Arabic improves from 34.6 to 52.1, German from 49.7 to 66.7, Hindi from 38.2 to 63.6, Korean from 34.6 to 64.8, and Portuguese from 48.5 to 65.5, against an English score of 73.9. For Claude Opus 4.6, Arabic improves from 32.1 to 49.1, German from 49.7 to 66.7, Hindi from 29.7 to 62.4, Korean from 33.3 to 58.8, and Portuguese from 49.1 to 63.0, against an English score of 79.4 (Kim et al., 27 Apr 2026).
The headline “up to 32.7% improvement” refers to Claude Opus 4.6 on Hindi, which rises from 29.7 to 62.4, a gain of 32.7 percentage points (Kim et al., 27 Apr 2026). Other large gains include GPT-5.4 on Korean at +29.1, GPT-5.4 on Hindi at +25.4, Gemini 3.1 Pro on Korean at +30.2, Gemini 3.1 Pro on Hindi at +25.4, and Claude Opus 4.6 on Korean at +25.5. The closest audited setting to English is GPT-5.4 in German, where audited German reaches 63.6 versus English 66.7, a gap of 3.1 (Kim et al., 27 Apr 2026).
Substantial residual gaps remain. The paper explicitly notes Arabic as the clearest example: after audit, the Arabic gap reaches 19.4 for GPT-5.4, 21.8 for Gemini 3.1 Pro, and 30.3 for Claude Opus 4.6 (Kim et al., 27 Apr 2026). The error-type analysis, using Gemini 3.1 Pro and aggregating across languages, reports that functional alignment has the highest flip rate at 67.9%, while fluency has the lowest at 40.6%; cultural alignment, adequacy, and difficulty calibration flip outcomes roughly about half the time. The paper states that these flip rates are correlational rather than causal because multiple issue types can co-occur on one task (Kim et al., 27 Apr 2026).
6. Interpretation and methodological implications
The paper’s main interpretive claim is that a substantial share of the multilingual performance gap observed on machine-translated benchmarks is benchmark-induced measurement error (Kim et al., 27 Apr 2026). The argument is empirical: raw machine-translated tasks make models appear much worse in non-English languages, auditing raises performance dramatically, and in at least one audited setting the multilingual score comes within 3.1 points of English. The conclusion is not that multilingual model weaknesses disappear, but that existing measurement practices confound model limitations with translation-induced benchmark corruption.
This reframes multilingual agent evaluation as a task adaptation problem rather than a text translation problem (Kim et al., 27 Apr 2026). The practical implications stated in the paper follow directly from that reframing. Researchers should not rely on machine translation plus light post-editing for agent benchmarks. Review should audit benchmark function rather than benchmark language alone, explicitly checking whether the task still asks for the right thing, whether the answer key still matches, whether output-format constraints still hold, whether local context is realistic, and whether solving difficulty is preserved. If resources are limited, the paper suggests prioritizing functional alignment, cultural alignment, and difficulty calibration rather than polishing fluency alone (Kim et al., 27 Apr 2026).
The paper also recommends structured review pipelines that combine deterministic objective checks, narrowly scoped model judgments, trained bilingual human review, and secondary meta-review (Kim et al., 27 Apr 2026). A plausible implication is that multilingual evaluation results should be interpreted together with adaptation quality, because the benchmark itself may be part of the measured error.
7. Limitations, release status, and disambiguation
The paper states several explicit limitations (Kim et al., 27 Apr 2026). The review process did not include baseline agent testing, so hidden technical issues such as search API failures, geo-blocking, poor retrieval quality, or tool bypass through memorization were not directly incorporated into the audit loop. File attachments such as images, audio, PDFs, and other assets remained in English, which means the benchmark is multilingual at the task level but not fully localized across modalities. The flip-rate analysis is correlational rather than causal. The work addresses adaptation of an English benchmark rather than native benchmark creation from scratch. The residual multilingual gap after audit remains only partially explained, leaving open the extent to which it reflects genuine language capability deficits, retrieval environment differences, cultural-data availability differences, or remaining benchmark design residues (Kim et al., 27 Apr 2026).
GAIA-v2-LILT is released as part of the MAPS package, with the dataset available at https://huggingface.co/datasets/Fujitsu-FRE/MAPS/viewer/GAIA-v2-LILT and the experimental code at https://github.com/lilt/gaia-v2-lilt (Kim et al., 27 Apr 2026). This makes the benchmark a released evaluation resource rather than only a methodological proposal.
The name can be confused with other “GAIA” or “Gaia” projects, but these are unrelated. The remote-sensing vision-language dataset “GAIA” is a multi-scale, multi-sensor, multi-modal dataset for remote sensing image analysis and does not discuss “LILT” or any version explicitly named “GAIA-v2” (Zavras et al., 13 Feb 2025). The astronomy papers “GaiaNIR” (Hobbs et al., 2016) and “The Gaia astrophysical parameters inference system (Apsis). Pre-launch description” (Bailer-Jones et al., 2013) concern astrometry, detector concepts, and astrophysical parameter inference for the ESA Gaia mission rather than multilingual agent benchmarking. In this context, GAIA-v2-LILT refers specifically to multilingual adaptation of the GAIA agent benchmark beyond translation.