---
title: 'GAIA-v2-LILT: Multilingual Agent Benchmark'
url: https://www.emergentmind.com/topics/gaia-v2-lilt
type: topic
---

# GAIA-v2-LILT: Multilingual Agent Benchmark

GAIA-v2-LILT is a multilingual, re-audited version of the GAIA agent benchmark designed to evaluate whether language-model-based agents can solve the same underlying tasks across languages without the evaluation itself being corrupted by poor translation [2604.24929]. It extends the English GAIA benchmark for “General AI Assistants” into five non-English languages through a workflow that goes beyond machine translation by enforcing explicit functional alignment, cultural alignment, and difficulty calibration. The benchmark’s central claim is methodological as much as empirical: for agentic tasks, multilingual adaptation is a task-level validity problem, not merely a text-level translation problem, because translation errors can alter answer correctness, output format requirements, realism, and difficulty [2604.24929].

## 1. Definition, scope, and benchmark lineage

The original GAIA benchmark is an English benchmark for “General AI Assistants,” namely agents that must perform multi-step problem solving with external tools rather than only answer from internal language modeling [2604.24929]. In the task definition used for GAIA-v2-LILT, an agentic task is a single-turn query that requires at least one external action such as web search, page parsing, file inspection, numerical computation, or scripting. GAIA tasks also commonly require strict output formatting and are graded with deterministic reference answers [2604.24929].

GAIA-v2-LILT is built from the validation split of GAIA via the MAPS multilingual version, but it is not a minimally revised translated copy. It is a re-audited multilingual extension containing 165 query-answer pairs per language in five non-English languages: Arabic, German, Hindi, Korean, and Portuguese (Brazil) [2604.24929]. Its stated purpose is to make multilingual agent evaluation more faithful to the original English task intent, answer conditions, and difficulty.

The benchmark therefore occupies a specific lineage: original English GAIA, then MAPS machine-translated multilingual GAIA, then GAIA-v2-LILT as a human-audited, functionally and culturally aligned multilingual extension [2604.24929]. This lineage is central to the paper’s contribution, because the empirical comparison is explicitly between raw machine-translated tasks and audited multilingual tasks rather than between unrelated benchmark families.

## 2. Benchmark invalidation under naive multilingual adaptation

The paper argues that standard machine translation plus light post-editing can break benchmark validity in ways that are specific to agentic evaluation [2604.24929]. The core issue is not only reduced fluency; it is that the executable logic of a task can change. A translated query may no longer request the same thing, the translated answer may no longer satisfy the query, the prompt may become culturally off-target, or the task may become easier or harder than the English original.

The most important failure mode is query-answer misalignment. The canonical example is an Arabic task in which an answer that should remain the IOC country code “CUB” was translated into the Arabic word for “lion cub.” That translation is both semantically wrong and functionally incompatible with a query that requires a three-letter country code [2604.24929]. The paper distinguishes under-translation and over-translation as separate subcases. Under-translation occurs when an answer key remains in English even though the task requires localization; the German chess example is that a locally valid move notation such as **Td5** would be unfairly rejected if the answer key still expected English **Rd5**. Over-translation occurs when a string should remain unchanged because the task requires copying or extracting it exactly, but the system translates it anyway [2604.24929].

A second failure mode is culturally inappropriate or off-target context. The paper lists geography, policy logic, units of measurement, platform conventions, and social norms as recurrent sources of distortion [2604.24929]. Its Korean bottle-return example shows a machine-translated prompt that still referred to U.S. highways, imperial units, and an unsuitable recycling context. The audited version replaced these with Korean routes, converted measurements, updated the date context, and adapted the incentive system to Korean reality where possible. The paper adds a methodological concern: if a translated prompt remains visibly U.S.-centric, an agent may internally translate it back to English and solve the original English problem, so the benchmark ceases to measure multilingual reasoning in a realistic setting [2604.24929].

A third failure mode is altered task difficulty. Even when broad semantics are preserved, retrieval conditions, arithmetic, clue structure, or local web availability may shift the solving process [2604.24929]. The Korean word-riddle example illustrates this: the draft kept the broad function of the original task but made it easier by embedding an English clue inside otherwise normal Korean text. The corrected version restored the intended challenge by reversing the Korean sentence and localizing the clue word.

These cases motivate the paper’s broader interpretive claim that multilingual performance deficits on translated agent benchmarks may partly be benchmark-induced measurement error rather than direct evidence of weaker model capability in the target language [2604.24929].

## 3. Alignment framework and audit workflow

GAIA-v2-LILT organizes multilingual adaptation around three alignment goals: functional alignment, cultural alignment, and difficulty calibration [2604.24929]. Functional alignment requires that the translated task preserve core task mechanics and evaluation logic. The prompt must still request the same output in the same way, and the answer key must remain compatible with the query. This includes deciding when strings should remain unchanged because they are part of an extraction target or answer format, and when they should be localized because the task expects conventional local notation. The paper later reports that this category has the highest association with output flips after correction [2604.24929].

Cultural alignment requires that the prompt make sense as a realistic task for the target-language audience. The benchmark should not appear to be an English problem with translated surface forms. Localizing roads or place names, units, dates, policy contexts, consumer conventions, or public systems may be necessary, but the paper explicitly states that not everything should be localized blindly; literal source context can be retained when no local equivalent exists [2604.24929].

Difficulty calibration requires that the target-language task remain approximately as hard as the English original. This is benchmark-specific rather than purely linguistic. A translation can be fluent, culturally appropriate, and still no longer be comparable because evidence is less accessible on the local web or because a clue becomes easier after translation. The paper therefore requires manual verification of the solving process itself, including searches in the target-language environment [2604.24929].

The workflow uses three review stages. First, deterministic filtering applies rule-based scripts for language identification, string matching for answer leakage, and placeholder recall for fixed terms such as numbers and URLs [2604.24929]. This stage is intentionally non-generative and exact-match based in order to avoid LLM self-preference. Second, LLM judges perform single-axis binary judgments over short contexts, separately evaluating fluency, adequacy, query-answer compatibility, and cultural appropriateness. Third, bilingual human linguists conduct the substantive audit through 1-on-1 training calls, explicit issue checkboxes, visibility into deterministic and LLM flags, one linguist reviewing each task, and a second linguist together with a machine learning researcher conducting meta-review [2604.24929]. The workflow is designed specifically to counter LLM self-preference and human fluency bias.

## 4. Dataset construction, audit intensity, and corrected error types

GAIA-v2-LILT is based on the machine-translated GAIA version released through MAPS, after which the new workflow is applied to produce the final audited set [2604.24929]. For each language, the benchmark contains 165 validation tasks, each a query-answer pair.

The reported edit rates show that the audit was substantial rather than cosmetic.

| Language | Tasks edited | Word edit rate |
|---|---:|---:|
| Arabic | 84.8% | 25.4% |
| German | 81.2% | 30.0% |
| Hindi | 100.0% | 55.4% |
| Korean | 92.1% | 36.9% |
| Portuguese | 87.9% | 25.0% |

The corresponding character edit rates are 19.4% for Arabic, 22.2% for German, 46.3% for Hindi, 27.9% for Korean, and 19.5% for Portuguese [2604.24929]. The most striking figure is Hindi, where every task was edited. This supports the paper’s claim that minimal translation is inadequate for this kind of benchmark.

The corrected errors span multiple defect classes. In Arabic, the over-translation of “CUB” corrupted the answer key. In German, the audit corrected translationese, inappropriate use of the formal pronoun **Sie**, and one case in which the output was in Italian instead of German and also leaked the actual answer into the prompt (“Honolulu, Quincy”) [2604.24929]. In Korean, the audit corrected U.S.-centric road-trip and recycling assumptions and separately repaired a riddle whose translated form had become easier than intended. These examples show that the audit was not limited to fluency editing; it addressed semantic corruption, answer leakage, evaluation mismatch, locale mismatch, and difficulty drift.

## 5. Experimental protocol and empirical findings

The paper evaluates three frontier models—GPT-5.4, Gemini 3.1 Pro, and Claude Opus 4.6—inside the Open Deep Research agent built with smolagents [2604.24929]. The agents were equipped with web search using Exa, speech recognition, image captioning, and file read tools. For each task, the manager agent was limited to 12 steps and the search subagent to 20 steps. The comparison is between MT, the raw machine-translated benchmark drafts from MAPS; Audit, the final human-corrected GAIA-v2-LILT tasks; and the English original benchmark as baseline [2604.24929].

Evaluation is reported as pass@1 accuracy [%], with final-answer correctness determined under normalized exact match by removing spaces, lowercasing, and stripping punctuation [2604.24929]. On this metric, auditing improves agent success rates across all languages and all tested models.

For GPT-5.4, Arabic improves from 32.1 to 47.3, German from 47.3 to 63.6, Hindi from 34.6 to 60.0, Korean from 33.3 to 62.4, and Portuguese from 47.3 to 58.2, against an English score of 66.7 [2604.24929]. For Gemini 3.1 Pro, Arabic improves from 34.6 to 52.1, German from 49.7 to 66.7, Hindi from 38.2 to 63.6, Korean from 34.6 to 64.8, and Portuguese from 48.5 to 65.5, against an English score of 73.9. For Claude Opus 4.6, Arabic improves from 32.1 to 49.1, German from 49.7 to 66.7, Hindi from 29.7 to 62.4, Korean from 33.3 to 58.8, and Portuguese from 49.1 to 63.0, against an English score of 79.4 [2604.24929].

The headline “up to 32.7% improvement” refers to Claude Opus 4.6 on Hindi, which rises from 29.7 to 62.4, a gain of 32.7 percentage points [2604.24929]. Other large gains include GPT-5.4 on Korean at +29.1, GPT-5.4 on Hindi at +25.4, Gemini 3.1 Pro on Korean at +30.2, Gemini 3.1 Pro on Hindi at +25.4, and Claude Opus 4.6 on Korean at +25.5. The closest audited setting to English is GPT-5.4 in German, where audited German reaches 63.6 versus English 66.7, a gap of 3.1 [2604.24929].

Substantial residual gaps remain. The paper explicitly notes Arabic as the clearest example: after audit, the Arabic gap reaches 19.4 for GPT-5.4, 21.8 for Gemini 3.1 Pro, and 30.3 for Claude Opus 4.6 [2604.24929]. The error-type analysis, using Gemini 3.1 Pro and aggregating across languages, reports that functional alignment has the highest flip rate at 67.9%, while fluency has the lowest at 40.6%; cultural alignment, adequacy, and difficulty calibration flip outcomes roughly about half the time. The paper states that these flip rates are correlational rather than causal because multiple issue types can co-occur on one task [2604.24929].

## 6. Interpretation and methodological implications

The paper’s main interpretive claim is that a substantial share of the multilingual performance gap observed on machine-translated benchmarks is benchmark-induced measurement error [2604.24929]. The argument is empirical: raw machine-translated tasks make models appear much worse in non-English languages, auditing raises performance dramatically, and in at least one audited setting the multilingual score comes within 3.1 points of English. The conclusion is not that multilingual model weaknesses disappear, but that existing measurement practices confound model limitations with translation-induced benchmark corruption.

This reframes multilingual agent evaluation as a task adaptation problem rather than a text translation problem [2604.24929]. The practical implications stated in the paper follow directly from that reframing. Researchers should not rely on machine translation plus light post-editing for agent benchmarks. Review should audit benchmark function rather than benchmark language alone, explicitly checking whether the task still asks for the right thing, whether the answer key still matches, whether output-format constraints still hold, whether local context is realistic, and whether solving difficulty is preserved. If resources are limited, the paper suggests prioritizing functional alignment, cultural alignment, and difficulty calibration rather than polishing fluency alone [2604.24929].

The paper also recommends structured review pipelines that combine deterministic objective checks, narrowly scoped model judgments, trained bilingual human review, and secondary meta-review [2604.24929]. A plausible implication is that multilingual evaluation results should be interpreted together with adaptation quality, because the benchmark itself may be part of the measured error.

## 7. Limitations, release status, and disambiguation

The paper states several explicit limitations [2604.24929]. The review process did not include baseline agent testing, so hidden technical issues such as search API failures, geo-blocking, poor retrieval quality, or tool bypass through memorization were not directly incorporated into the audit loop. File attachments such as images, audio, PDFs, and other assets remained in English, which means the benchmark is multilingual at the task level but not fully localized across modalities. The flip-rate analysis is correlational rather than causal. The work addresses adaptation of an English benchmark rather than native benchmark creation from scratch. The residual multilingual gap after audit remains only partially explained, leaving open the extent to which it reflects genuine language capability deficits, retrieval environment differences, cultural-data availability differences, or remaining benchmark design residues [2604.24929].

GAIA-v2-LILT is released as part of the MAPS package, with the dataset available at `https://huggingface.co/datasets/Fujitsu-FRE/MAPS/viewer/GAIA-v2-LILT` and the experimental code at `https://github.com/lilt/gaia-v2-lilt` [2604.24929]. This makes the benchmark a released evaluation resource rather than only a methodological proposal.

The name can be confused with other “GAIA” or “Gaia” projects, but these are unrelated. The remote-sensing vision-language dataset “GAIA” is a multi-scale, multi-sensor, multi-modal dataset for remote sensing image analysis and does not discuss “LILT” or any version explicitly named “GAIA-v2” [2502.09598]. The astronomy papers “GaiaNIR” [1609.07325] and “The Gaia astrophysical parameters inference system (Apsis). Pre-launch description” [1309.2157] concern astrometry, detector concepts, and astrophysical parameter inference for the ESA Gaia mission rather than multilingual agent benchmarking. In this context, GAIA-v2-LILT refers specifically to multilingual adaptation of the GAIA agent benchmark beyond translation.

Source: https://www.emergentmind.com/topics/gaia-v2-lilt